Data processing using version-controlled processing elements

Abstract

To provide technologies and solutions for managing subprocess versions and their associated components in computing systems. [Solution] These technologies enable dynamic adaptation and traceability through version control. Subprocess definitions are received and updated to reflect modifications to their components, configuration, or execution order. Changes are identified and propagated using unique version identifiers and events. Iterative refinement is supported through actions such as validating updated subprocesses, reprocessing data, or maintaining the origin chain. The disclosed solutions provide efficient and structured management of subprocesses.

Description

Technical Field

[0001] This disclosure generally relates to version management in data processing workflows.

Background Art

[0002] In modern data management and analysis, knowledge graphs serve as a fundamental framework for compiling, representing, and integrating structured knowledge from diverse sources. A knowledge graph encodes information in a graph-based format where nodes represent entities and edges represent relationships. This interconnected structure facilitates advanced analytics, natural language processing (NLP), and artificial intelligence (AI) applications.

[0003] The processes used to ingest and transform data before integration into a knowledge graph or schema can have a significant impact on subsequent applications such as training neural language models. However, data ingestion pipelines are often ad hoc, making it difficult to track the specific operations performed, especially when pipeline components or their configurations change over time. This lack of traceability complicates the determination of whether a dataset has been consistently processed or whether changes in the pipeline version require reprocessing of previously processed data. Therefore, there is room for improvement.

Summary of the Invention

Means for Solving the Problems

[0004] This "Summary of the Invention" is provided to introduce, in a simplified form, a selection of concepts that are further described below in the "Detailed Description of the Invention". This "Summary of the Invention" is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0005] Techniques and solutions for managing subprocess versions and their associated components in computing systems are disclosed. These techniques enable dynamic adaptation and traceability through version control. Subprocess definitions are received and updated to reflect modifications to their components, configuration, or execution order. Changes are identified and propagated using unique version identifiers and events. Iterative refinement is supported through actions such as validating updated subprocesses, reprocessing data, or maintaining a proofance chain. The disclosed solutions provide efficient and structured management of subprocesses.

[0006] In one embodiment, the disclosure provides a process for managing versions of subprocesses and associated updates within a computing system. A definition of a first version of a subprocess is received. The first version of a subprocess comprises one or more components, each of which is associated with an executable implementation. Processing data using the first version of the subprocess occurs to produce a first result.

[0007] To provide a second version of the subprocess, changes are received to the definition of the first version of the subprocess. The changes include either the version of the components of the first version of the subprocess, or the components themselves or the order in which the components are called. An identifier for the second version of the subprocess is generated.

[0008] An event is generated indicating that the first version of a subprocess has been updated. In response to the event, one or more actions are performed. These actions include triggering a request to validate the second version or second result of the subprocess, reprocessing the data using the second version of the subprocess, or adding the identifier of the second version of the subprocess to the origin chain. The origin chain contains the version identifiers of multiple subprocesses of a process, and multiple subprocesses contain the process.

[0009] This disclosure also includes computing systems and tangible, non-temporary, computer-readable storage media configured to perform or containing instructions for performing the methods described above. Various other features and benefits, as described herein, may be incorporated into the technology as desired. [Brief explanation of the drawing]

[0010] [Figure 1] This diagram shows the architecture and end-to-end process for ingesting data from a source into a graph, and then deploying that information from the graph to one or more channels. [Figure 2] This diagram shows the subprocesses involved in the process of importing data from the source into a graph. [Figure 3] Figure 2 illustrates examples of input and output components that may be used in the subprocess. [Figure 4] This figure shows an example of a processing environment for converting input to output using a configuration for a subprocess, and the steps that can be performed using the processing environment. [Figure 5] This figure shows examples of computing environments in which the disclosed technologies may be implemented. [Figure 6]This diagram shows the relationships between various data artifacts within a schema, such as a core schema, which includes models, processes executed using those models, and components of those processes. [Figure 7] This diagram shows the relationships between various data artifacts in a schema, such as a core schema, which includes types related to model types, process types, and process elements. [Figure 8] This figure shows the implementation of the schema in Figure 6. [Figure 9A] This figure shows a schema representing a combination of the schema in Figure 6 and the schema in Figure 7. [Figure 9B] This figure shows a schema representing a combination of the schema in Figure 6 and the schema in Figure 7. [Figure 10] This diagram shows a graphical representation of various related models, including the core model, taxonomy model, and multiple domain models. [Figure 11] This is a schema diagram showing the implementation of the core model, domain model, and taxonomy model corresponding to the elements in Figure 10. [Figure 12A] This diagram provides a schema showing how core, taxonomy, and domain model elements can be represented and related. [Figure 12B] This diagram provides a schema showing how core, taxonomy, and domain model elements can be represented and related. [Figure 12C] This diagram provides a schema showing how core, taxonomy, and domain model elements can be represented and related. [Figure 13] This figure shows the operation when processing source data using the disclosed technology, and the related events that may be generated as a result of that operation. [Figure 14] This is an example diagram of a user interface where users can define models. [Figure 15]FIG. Example of a user interface that allows a user to define sources used in a model. [Figure 16] FIG. Example of a user interface that allows a user to define a new component. [Figure 17] FIG. Example of a user interface that allows a user to view version information of a subprocess component. [Figure 18] FIG. Example of a user interface for creating a subprocess. [Figure 19] FIG. Example of a user interface for managing versions of a subprocess. [Figure 20] FIG. Example of a user interface for managing the assignment of component versions to subprocess versions. [Figure 21] FIG. Example of a user interface for reviewing the results of the execution of a process, including the results of subprocesses of the process. [Figure 22] FIG. Flow diagram of a process for managing subprocess versions and related updates within a computing system. [Figure 23] FIG. Example of a computing environment in which the disclosed technology may be implemented. [Figure 24] FIG. Example of a cloud computing environment that may be used in conjunction with the technology described herein.

BEST MODE FOR CARRYING OUT THE INVENTION

[0011] Example 1 - Overview Continuing from the background technology, this disclosure provides techniques and solutions for identifying process elements using a specific example of an ingestion pipeline for data that may be used for training purposes, such as training a neural language model. Identifying process elements offers several advantages, including enabling modification or re-execution of the process when the elements change. For example, modifying an algorithm used to transform data can be used to generate an updated process that uses the modified algorithm instead of the original algorithm, or it can be used to notify a user, such as a developer, of a new version of the algorithm, allowing the user to determine whether the process should be updated to use the new algorithm.

[0012] Regarding reruns, it is sometimes desirable to process data used for a common purpose consistently in order to make training as accurate as possible. Therefore, if new data is processed by an updated algorithm, it may be useful to reprocess existing data using the updated algorithm.

[0013] To make processes more directly comparable and to automate actions that respond to changes in process elements, define standard types for process elements. For example, an event may be triggered when a specific action occurs. These actions may include the actions mentioned above, such as triggering an event when a process element changes, and the event triggers processing using the updated process element. Events can also trigger actions that draw the user's attention to processing actions. For example, if data is processed using an updated version of an algorithm, an event may be triggered that triggers an action for the user to examine the results of the processing and determine whether the results are suitable for further processing.

[0014] Information about the processes used to process the data may be added to, or otherwise associated with, the resulting dataset. This can facilitate comparing datasets, for example, to understand where differences might arise. Such annotations can also be used to trigger data reprocessing, as mentioned earlier.

[0015] Example 2 - Example of data ingestion from source to graph and deployment of graph data to channels Figure 1 illustrates a computing environment 100 that may be used in a particular implementation of the disclosed technology. In particular, the computing environment 100 facilitates the process of ingesting data from source 108 into channel 112. Source 108 refers to a data source or provider that may include a database, file system, API, streaming service, or any other system or repository that generates or stores data for processing. Each source 108 may be associated with a specific schema, format, or semantic model and may provide structured, semi-structured, or unstructured data. The sources 108 in Figure 1 represent these sources of data that are ingested into the system for subsequent processing and use in one or more channels 112.

[0016] Channel 112 refers to a target or endpoint to which processed data is delivered or used. Channel 112 may represent a system, application, service, or workflow that consumes the ingested data to perform a specific task such as analytics, visualization, machine learning, or decision-making. Each channel 112 may require data in a specific format or schema and may integrate with multiple data sources. In Figure 1, Channel 112 represents these destinations for processed data, and each destination may leverage one or more sources 108 to meet their operational needs.

[0017] Source 108 and channel 112 can have an N...M relationship. In other words, a given channel 112 can use data from one or more data sources 108. Data from a given data source 108 can be used by multiple channels.

[0018] A processing framework 116 is shown that processes data from one or more sources 108 and provides the data to one or more channels. Generally, the processing framework 116 includes an ingestion process 120 that takes in data from the sources 108 and stores the data in a specific representation. In a particular example, the representation is a graph 124, such as a knowledge graph. The graph 124 may be associated with a schema, such as a semantic schema, as will be further described.

[0019] The schema of graph 124 is called the local schema. Data from source 108 is typically associated with the source schema, and such associations can be part of the ingestion process 120. In some cases, instead of converting data from the source schema to the local schema, the data is instead mapped from the source schema to the local schema, for example, by annotations on the data. The data in the source schema may be called a subgraph. Elements of the subgraph schema that are mapped to the local schema, such as semantic elements of the subgraph schema and instances of those elements, may be called "derivatives" of the elements in the local schema. In certain examples, a derivation relationship can be a predicate type in the knowledge graph, and the linked data corresponds to the subject and object associated by the predicate.

[0020] The ingestion process 120 may include operations such as data formatting and data cleansing. The ingestion process 120 may also include operations such as associating data with elements of a graph. The ingested data may be linked to specific components of the knowledge graph, including, for example, using ontologs defined for the knowledge graph. That is, a particular set of data may be annotated as an instance of a particular class in the knowledge graph, and values ​​within the set of data may be assigned to various properties defined for that class.

[0021] The deployment process 128 may be defined to process data provided to one or more channels 112. The deployment process 128 may include operations such as extracting data from the format of graph 124 or converting it to a format used by the channels. The deployment process 128 may also include operations to send the data to channels 112, such as data stores used by the channels, in any format or transformation.

[0022] The modeling component 132 may be used to perform actions such as associating data from source 108 with a specific schema, such as the schema of graph 124. The lifecycle management component 136 may be used to maintain version information of processing components and data generated during processing. For example, processed data may be tagged with version information of the process or process element used to generate that data. The lifecycle management component 136 may, in some cases, perform actions related to version changes, such as triggering an event or triggering an action in response to a triggered event.

[0023] The computing environment 100 can be called a semantic data layer. Semantic data refers to data that is not merely raw values, but is associated with information (such as metadata) that describes what the data represents. For example, graph 124 can store data values ​​from source 108 and information that links that data to elements in a knowledge graph. Semantic information can facilitate downstream use cases of the data where training a neural language model becomes more effective when the training data includes not only the data but also the semantic context of the data.

[0024] Figure 1 outlines an operation 150 for defining the process of ingesting data from source 108 and deploying that data to channel 112. For example, 154 defines a process that may be performed to retrieve data from source 108 and stage the data for further processing. Defining the process for ingesting data may include defining software functions for extracting information from a repository, database, file, or through an application programming interface (API). Operation 154 may include identifying where the data from source 108 will be stored before further processing, and processing for cleaning or organizing the source data.

[0025] The modeling and ontology generation processes are defined in 158. The operations of 158 may include operations for defining how ontologs or knowledge graphs should be constructed. The operations of 158 may also include defining the processes for how incoming data from source 108 is linked to specific schemas, such as specific knowledge graphs, which may have associated ontologs.

[0026] The operations of 162 include defining a pipeline for processing data, including implementing various functions that are performed as part of the pipeline. Pipeline operations may include actions to clean, transform, or integrate data. For example, a pipeline may be responsible for transforming source data into a standardized format. A pipeline defines operations at a broad level, and specific operations, including data transformation, may be performed in 166.

[0027] The operation for generating graph 124 is defined in 170. The operation for generating graph 124 may include programmatic logic for incorporating the transformed data into the graph, such as processes for creating nodes and edges that represent entities and the relationships between those entities. In a particular implementation, the graph generation operation may define how RDF (Resource Description Framework) triples are generated to represent the data incorporated into graph 124 or a schema linked to the graph.

[0028] Procedures for scrutinizing and validating Graph 124 to ensure that it accurately represents the data and relationships are defined in 174. These procedures may include defining automated validation checks and manual scrutiny processes, such as those performed by domain experts. Processes for correcting any errors or inconsistencies encountered during the validation process may also be defined.

[0029] In section 178, actions are defined for versioning data and managing the release of processed data. These actions may be used to provide users and applications (including as channel 112) with the correct version of graph 124. Processes may be defined for documenting and managing updates or changes to graph 124.

[0030] In section 182, not only can the behavior that defines how data from the graph is provided to channel 112 be defined, but the platform and application on which the graph 124 is deployed can also be defined. For example, the behavior for deploying the knowledge graph to a web application, API, data analytics platform, and other channel 112 where users or computing processes can interact with the data can be defined.

[0031] It should be noted that the computing environment 100, in particular the processing framework 116, can be composed of reusable components. For example, standard processes, subprocesses, and their components can be defined at a broader level. With respect to a particular data acquisition process, elements of these standard processes can be linked, and a standard process can be associated with a particular implementation of the process. A particular implementation can also be reusable. For example, the same data transformation operation may be performed when processing data from two different sources. That is, a particular implementation of a process element can be used, for example, as long as the input is equivalent to the implementation and the output is suitable for downstream processing.

[0032] Example 3 -- Source -- Example of a subprocess in a graph process An overall process can be broken down into different elements, and these elements can be reused between different processes. Sources and subprocesses are two mechanisms for dividing process elements into logical units. For example, an overall process that generates a graph from source data proceeds in various phases called subprocesses. Subprocesses can act as synchronization points, or points where events can be triggered and actions can be taken. A synchronization point itself can be a type of event / action. For example, synchronization may include determining that a subprocess is complete and notifying the user of its completion. The user can then determine whether the result of the subprocess execution indicates that further processing can be performed. Validation actions can be events that themselves can trigger further actions, such as proceeding to the next subprocess of the overall process.

[0033] In a source-graph process, the overall source-graph process can be defined at the source granularity. That is, assuming it is desirable to take data from multiple sources, a separate source-graph process is defined for each source. Although the processes are defined separately, they can have the same general elements, or even specific implementations of such elements. In particular, having different processes for different sources allows the processes to run asynchronously. For example, data sources may be updated at different frequencies, and having separate processes allows updated data from one source to be processed even if another source does not have the updated data.

[0034] The term "source" encompasses many different types of sources, but specific examples of sources that may be used in a source-graph process include the SAP Enterprise Architecture Framework (SEAF), the SAP Enterprise Architecture Reference Library (SEARL), and the American Productivity and Quality Center (APQC). These sources define local schemas from both a technical and semantic perspective. That is, for example, the technical format might be that data is stored in a relational format, while the semantic perspective might include linking data to semantic descriptions such as ontologs, or storing data in a knowledge graph. These data sources typically require at least some differences in the implementation of subprocesses and subprocess components, such as extracting, transforming, and storing content in a graph format. In some cases, multiple sources can extract their data into a common graph, or at least map different graphs into a common format such as a local schema. However, the release cycles of the sources can differ, and the asynchronous nature of the processes for the sources allows data to be processed separately for each source, with the results synchronized with a common graph.

[0035] Figure 2 shows the overall source-graph process 200. Process 200 includes several subprocesses. The external data-source data subprocess 208 is responsible for acquiring source data and staging it for further processing. The source data-source schema process 212 is responsible for mapping source data to a specific schema defined for the source data. The source schema-pipeline subprocess 216 takes the source data, now integrated with the source schema, into a processing pipeline that may include actions to clean, format, or transform the source data. The pipeline-subgraph subprocess 220 acquires data from the pipeline and adds it to a subgraph graph, which may be a subgraph in graph 124 in Figure 1. The subgraph-derivative data subprocess 224 analyzes the data in the subgraph and relates that data to the local schema in graph 124 in Figure 1.

[0036] Subprocesses 208-224 may represent general operations performed between source-graph process 200. These subprocesses 208-224 may be associated with implementations associated with a particular source. In a sense, subprocesses 208-220 can be thought of as base classes in a computing language such as C++, where implementations specific to a given source correspond to derived classes of the base classes.

[0037] As considered in Example 1, process elements such as subprocesses 208-224 can change over time. Subprocesses 208-224 can be associated with version information, which gives time dependency to the data resulting from the subprocesses. For example, data generated by a subprocess can be associated with the subprocess's version identifier. Thus, data can be associated with identifiers that can be used to precisely determine how the data was processed during the subprocess.

[0038] Version control of a subprocess can be associated with version control of the components used within that subprocess. A subprocess may have input components, processing components, and output components, or it may use multiple components of these types. A change to any one of these components can result in a new version of the subprocess. Thus, it may be possible to precisely determine how specific data within a dataset, such as a subgraph, was generated. This information can be used in various ways, including when determining whether the data should be reprocessed to account for changes in the subprocesses of the process used to initially capture the data.

[0039] Figure 3 illustrates how a subprocess implementation can be defined from elements called components. Similar to the subprocess itself, a subprocess component can represent a general data artifact that can be used when defining a subprocess, and moreover, has an implementation for a specific subprocess of a particular process. A data artifact refers to all representations of data within a computing system, including both abstract definitions and concrete instances of the data. An abstract definition may include schema elements, models, classes, or templates that define the structure, relationships, semantics, or associated behavior of the data, such as methods or functions that can be performed on instances of the artifact. Concrete instances may include individual data points, records, objects, or entities that conform to or derive from these definitions. Data artifacts may represent static or dynamic data and can exist in various forms, including structured, semi-structured, or unstructured data. Furthermore, data artifacts may include metadata or annotations related to the data, such as information describing the data's origin, relationships, or intended use, and the behavior or actions associated with the artifact's purpose or role within the system.

[0040] In particular, Figure 3 provides Table 300, which includes columns 310b-310f relating to a specific subprocess of the source-graph process for the source shown in column 310a. In row 320, where the SAP Enterprise Architecture Framework is used as the source, the external data-source data subprocess 310b may have input components: external data and a source data extractor. The output of subprocess 310b is the source data. Note that, in addition to having input and output components, components can have different properties in the sense that they are data (including those represented as data artifacts), inputs or outputs, or processes (also called algorithms). Subprocesses for different sources may have the same general components, but the implementation of the components may differ as needed, taking into account the nature of the source data.

[0041] The subprocesses in columns 310c to 310f are generally similar to the subprocess in column 310b in that they have input and output components, which can be data artifacts or algorithms. The source data-source schema subprocess in column 310c includes input components: source data and a source schema generator algorithm. The output is the source schema.

[0042] The Source Schema-Pipeline subprocess in column 310d has an input component called Source Schema and input algorithm components called SubOntologyGenerator and PipelineGenerator. The output components are SubOntology and Pipeline.

[0043] The pipeline-subgraph subprocess in column 310e has input components of source data (such as that generated by the external data-source data subprocess in column 310a) and a pipeline generated by the subprocess in column 310d. The input components of the subprocess in column 310e further include a graph writer that writes data to the graph and a subontology used by the graph writer. The pipeline-subgraph subprocess outputs a subgraph.

[0044] Column 310f represents a subgraph-derivative subprocess that links data in one graph to data in another graph, such as a local graph. A subgraph-derivative subprocess has input components: a source subgraph and a target subgraph, and attempts to match the source data to the target data (or semantic elements), for example, by using the respective schemas of the source and target. An input component called a derivative writer operates on the subgraph and produces an output component called derivative data, in which annotations link the data between the processed datasets of the source schema and the local schema.

[0045] Example 4 - Example of input-to-output conversion using subprocess configuration Figure 4 shows 400 computing environments and processes collectively involved in the lifecycle management of processes and process elements, including subprocesses and components.

[0046] The computing environment and process 400 includes the environment and runtime 410 on which the processes and their subprocesses and components run. An exemplary subprocess 414 has a configuration 418, which includes components, indicated as components 422a and 422b. Component 422a is a data artifact corresponding to a specific data type or data structure that stores data. Examples of data artifacts include relational database tables, RDF triples, and JSON objects. Component 422b is an algorithm, such as an algorithm that processes data from one or more components 422a and produces one or more outputs, which may also be components 422.

[0047] As shown, components 422a and 422b are arranged sequentially, and a data artifact component 422a is provided as input to an algorithm component 422b, producing an output data artifact component 422a, which in turn can be input to a further algorithm component 422b. A subprocess may have one or more final output data artifact components 422a, which can act as the final output of the entire process or as input to a subsequent subprocess.

[0048] One or more inputs 430, such as data artifacts which are the output of a preceding subprocess, may be provided to subprocess 414. Thus, these inputs 430 can function as input data artifact components 422a. Similarly, the final output data artifact component 422a can become the overall output 434 of subprocess 414, which can then be provided as input to a subsequent subprocess.

[0049] Input 430 and output 434 can be associated with specific events, and specific actions can be triggered based on those events. For example, the availability of a new input 430 may trigger the execution of a subsequent subprocess that operates on the input. Output 434 can be associated with events such as informing the user that new data is available. The user may choose to activate output 434, for example, if it passes a quality check, at which point output 434 will act as input to a downstream subprocess. Manual validation of output 434 may be used, but validation may be skipped or performed in an automated manner. When automated, successfully passing validation may make output 434 available as input to a downstream process.

[0050] Figure 4 also shows process 450 which may be executed when a subprocess is modified or when new or modified input data becomes available. In 454, changes to the input are captured. Changes to the input may include new input data becoming available, which may include modified previously processed data, such as being processed by an updated subprocess that was previously used to provide the input.

[0051] In step 458, changes to the subprocess configuration are received. Changes to the subprocess may include adding, removing, or reorganizing components 422a and 422b. Changes to the subprocess may also include changing the definition or format of data artifact component 422a, or changing the algorithm used in algorithm component 422b. If a configuration update is received, the update can be applied, and then in step 462, the updated configuration can be activated.

[0052] When the input is to be processed by subprocess 414, the subprocess may be executed at 466. At 470, the output of the subprocess may be validated, and if the output is validated, at 474 the output may be activated, making it available for use by downstream subprocesses. At 478, an output change notification may be issued. The output change notification can inform subprocesses that use the output of subprocess 414 that new data is available for processing.

[0053] It should be noted that all or part of the operations of process 450 may be performed. That is, in some cases, an update to the configuration 418 of subprocess 414 may be received in a state where the new data is not available for processing. In this case, operations 458 and 462 will be performed, but the other operations will not. Similarly, without any changes to the configuration 418 of subprocess 414, new input may be made available for processing. In this case, operations 454 and 466-478 will be performed, but operations 458 and 462 will not.

[0054] As described above, the inputs and outputs of a subprocess may be associated with version information that specifies which version of the subprocess and therefore its components were used to produce a particular output. When data is processed using subprocess 414, an incremented version of a previously produced output data artifact component 422a may be generated. This makes it possible to identify the output of different subprocess versions. Version information can be passed between subprocesses so that an incremented version of an output acting as input to a subsequent process further generates an incremented version of the output of the subsequent process.

[0055] Various mechanisms can be used to track version information. In certain cases, semantic versioning may be used when a subprocess or subprocess component is updated. Version numbers can be specified as MAJOR.MINOR.PATCH, where the MAJOR version is associated with incompatible API changes, the MINOR version is associated with backward-compatible added features, and the PATCH version contains backward-compatible bug fixes. This notation can be extended, for example, by having extensions regarding pre-release and build metadata. For instance, an unactivated version may be designated as an alpha version.

[0056] In some cases, multiple components of a subprocess may be modified as part of a single update. In this case, the subprocess's version information may be determined by aggregating the component-level changes. For example, if one component is updated to a new minor version and another component is updated to a new patch version, both the subprocess's minor and patch versions will be updated. In some cases, changes to multiple components may result in multiple instances of the same type of version update being performed, such as when two components are subject to a minor version update. In this case, the subprocess's version identifier is incremented twice, rather than just by one minor version.

[0057] Example 5 - An example of a computing environment for defining and running a process. Figure 5 is a diagram of a computing environment 500 in which the disclosed technology may be implemented. A data store 508, such as a relational database or object store, can store source data 512 and graph data 514. Source data 512 can be data retrieved from an external source, such as source 108 in Figure 1. Graph data 514 can correspond to data in graph 124, or, if the data is not stored directly in the local graph, the graph data can be stored in a separate graph, and at least after processing, the graph data can be mapped to the local graph. Graph data 514 can be stored in a way that directly reflects the structure of the graph, or in a way that does not directly reflect the structure of the graph, but can be used to build the graph and obtain details of its structure. For example, data can be stored as nodes and edges, or, in the case of a knowledge graph, graph data 514 can correspond to RDF triples.

[0058] The process engine 518 can read information from and write information to the data store 508. For example, a subprocess can read source data 512 or graph data 514, or write updated source data or graph data after performing an action defined by a component of the subprocess. The process engine 518 includes one or more subprocess runtimes 522 and one or more algorithm runtimes 524, the algorithm runtimes of which may be invoked by the execution of subprocesses in the subprocess runtimes.

[0059] As part of executing algorithms in the algorithm runtime 524, the algorithm runtime can access algorithms in the algorithm repository 528. Accessing an algorithm may include calling the algorithm to execute it using a specific set of input data.

[0060] User 532 can interact with the algorithm development component 536. In some cases, the algorithm development component 536 can be an integrated development environment (IDE). User 532 can deploy a new algorithm to the algorithm repository 528 or update the version of an algorithm. Deployment of a new algorithm version is broadcast through event management 540. Typically, when a new algorithm is deployed and a corresponding event is created, User 532 may further create a subprocess or modify an existing subprocess to use the new algorithm.

[0061] As already described, updates to algorithms, data artifact components, or subprocesses can be associated with version control information, including cases where a subgraph generated through a source-graph process can include identifiers of the subprocesses used to process the data, or where intermediate data can include identifiers of subprocesses previously executed when generating the intermediate data. Therefore, the algorithm repository 528 can notify the event management component 540 when a component or process is updated. Furthermore, the event management component 540 can trigger an event in the version control component 544.

[0062] The version control component 544 can update the version information of a process, subprocess, or component, including storing the information in version data 546. The event control component 540 can generate additional events in response to version changes, either through an initial notification from the algorithm repository 528 or in response to communication from the version control component 544. The event control component 540 can also trigger other actions, such as triggering actions by the process engine 518. For example, the availability of an updated version of a subprocess might lead the event control component 540 to generate a command to the process engine 518 to reprocess previously processed data using the updated subprocess.

[0063] The semantic modeling component 550 maintains a schema that gives semantic meaning to the source data, such as a knowledge graph or ontology. The metadata repository 554 stores metadata 556, which can be used to maintain and provision parts of the overall process model, including provisioning the process engine 518 with definitions of subprocesses or subprocess components. In other words, the metadata repository 554 stores process definitions and allows the process engine 518 to execute code that implements those process definitions.

[0064] The user interface 560 may allow user 562 to access various components of the computing environment, including the semantic modeling component 550, the metadata repository 554, the version control component 544, and the event management component 540. The user interface 560 may also allow user access to the validation component 566. The validation component 566 can perform various actions. For example, the validation component 566 may provide the user interface 560 with information about the results of executing a subprocess, and accordingly, the user may choose to validate or not validate the results. If the results are validated, the validation component 566 may communicate with the event management component 540, for example, when the event management component notifies the process engine 518 that the output of one subprocess is authorized for use in a downstream subprocess.

[0065] Various operations can be performed in the computing environment 500. User 562 can access the semantic modeling component 550 and define a semantic model through the user interface 560. Through the user interface 560, user 562 can define sources, processes, subprocesses, and components that can be stored in metadata 556. If metadata 556 is modified with respect to an existing source, process, subprocess, or component, a message can be sent to the event manager 540, which can then take action as described above.

[0066] As described, user 532 can access the algorithm development component 536 to define or modify algorithms for use in a subprocess. When an algorithm is activated for use, a communication can be sent to the event manager 540, which can trigger actions such as determining whether an update to the algorithm should lead to data reprocessing. The algorithm's metadata 556 can also be modified, which may trigger events in the event management component 540. For example, a change in the order in which algorithms are called may affect the metadata that defines how the algorithm relates to other components, while the algorithm itself remains unchanged.

[0067] When modifying a source or a process defined for a source using the modeling tool 570, user 562 may choose to activate the new or modified source, process, or process element. In some cases, these new or modified elements may be handled automatically depending on other processes. For example, based on changes to components of a subprocess, such as changing the definition of a data artifact component or the processing performed by an algorithm component, a new version of the expected subprocess may be automatically defined. In at least some cases, the new version of the subprocess will be required to be activated by user 562 before it can be executed.

[0068] A subprocess may be executed in the process engine 518 using a subprocess runtime 522 and an algorithm runtime 524 when triggered for execution. The execution of a subprocess generates one or more outputs, such as output data artifacts, which may be stored in the data store 508. The completion of a subprocess may trigger an event in the event manager 540, in which the event manager notifies user 562 that the output data is available so that the output data can be approved by the user before it is provided as input to a downstream process. Information about the output may be stored in version data 546, such as associating the result with an identifier of the subprocess used to generate the output.

[0069] Example 6 - Example schema for models, processes, and process elements Figures 6–9 provide further details on how processes and their components can be represented and related. In this context, the term “model” refers to a description of the overall configuration, data, and process structure used to process source data and store it in a graph. Model types provide process templates and express dependencies between processes, subprocesses, and components. Model types can be used to generate executable processes that use specific implementations of process elements specific to a particular source. Thus, the disclosed techniques provide a structured way of representing processes that facilitates the reuse of subprocesses and components and establish origin chains that identify how particular data was generated. In this context, an origin chain is a record or genealogy that tracks the sequence of processes, subprocesses, components, and their respective versions involved in generating particular data. This allows the system to relate data outputs to specific inputs, configurations, and processing steps, including versions of those elements, enabling traceability, reproducibility, and accountability.

[0070] Figure 6 provides a schema 600 that describes how a model can be associated with model components. A definition 610 of a model data artifact is associated with a definition 614 of a process data artifact. In practice, a given model can be associated with multiple processes, while each process is associated with a single model. Processes can also be nested within other processes, enabling hierarchical relationships between processes.

[0071] The definition of a model artifact 610 is associated with the definition of a source data artifact 618. A model can have multiple sources, but each source is associated with a single model. A source defines a specific process for retrieving specific data, such as identifying the location of the repository from which the data is retrieved, and methods, such as APIs, used to retrieve the data.

[0072] The definition of process data artifacts 614 is also associated with the definition of source data artifacts 618. In particular, a given process data artifact is associated with exactly one source, while each source may be associated with multiple processes.

[0073] The definition 614 of process artifacts is associated with the definition 622 of subprocess data artifacts. A given subprocess may be associated with one process, while a given process may be associated with one or more subprocesses. The definition 622 of subprocess data artifacts is also associated with the definition 618 of source artifacts. In particular, each subprocess is associated with a single source, while a given source may be associated with multiple subprocesses.

[0074] Definition 622 of the subprocess data artifact is related to Definition 626 of the subprocess component data artifact. In particular, a subprocess contains one or more subprocess components, while a given subprocess component is associated with a single subprocess. Note that Definition 626 of the subprocess component data artifact is labeled "abstract". In this case, the subprocess component acts as a template, and when used, a class that implements the abstract subprocess component is defined, and thus, for example, a common type of subprocess component can be associated with different implementations, such as an implementation suitable for use in a particular subprocess or a particular source.

[0075] Definition 626 of a subprocess component data artifact is associated with definition 630 of a component data artifact. A subprocess component can refer to one component of a given role (input, processing, output), and a given component can be referred to by multiple subprocess components. While a component can refer to, for example, a data artifact or a type of algorithm, a subprocess component refers to a component in a particular context of a particular subprocess, including its interactions with other subprocess components.

[0076] The definition 630 for component data artifacts is also associated with the definition 618 for source data artifacts and the definition 610 for model data artifacts. In particular, a component is associated with a single model, while a given model can have one or more components. Each component is associated with a single source, while a given source can be associated with one or more components.

[0077] In implementation, the definitions of data artifacts shown in schema 600 may be extended to include attributes other than those shown, such as attributes that allow related instances of the data artifact to be tracked. For example, the definition 626 of a subprocess component data artifact may include attributes that act as foreign keys to process identifiers within instances of the process data artifact definition 614.

[0078] Figure 7 provides schema 700, which shows the relationships between different data artifacts that define types, such as the types of data artifacts represented in schema 600. For example, schema 700 shows that the definition 710 of a model type data artifact is associated with the definition 714 of a process type data artifact, the definition 718 of a subprocess type data artifact, the definition 722 of a component type data artifact, the definition 726 of a component category data artifact, the definition 730 of a subprocess component type data artifact, and the definition 734 of a subprocess component category data artifact.

[0079] Figure 8 provides a specific implementation 800 of schema 700. It can be seen that the model type data artifact 810 is linked to the process type data artifact 814, which contains three different process types. The model type data artifact 810 is also linked to the subprocess type data artifact 818. The subprocess type data artifact 818 provides identifiers for several subprocess types included in the model type. These subprocess types can be related to a specific process type for the model type, such as a subprocess of the SourceToGraph process type represented in the process type data artifact 814.

[0080] The process type hierarchy data artifact 822 defines the relationships between process types in the process type data artifact 814. For example, both source-graph processes and graph-channel processes may be defined as child processes of a source-channel process.

[0081] The subprocess type hierarchy data artifact 826 associates a specific subprocess type in the subprocess type data artifact 818 with a specific process in the process type data artifact 814. In the example shown, all subprocesses in this hierarchy are subprocesses of the source-graph process.

[0082] A given subprocess type may be associated with one or more subprocess component types of the subprocess component type data artifact 830. The subprocess component type data artifact 830 may be used to assign components of component type data artifact 834 to specific roles in the subprocess. For example, the subprocess component type data artifact 830 may associate an external data-source data subprocess type with input components, processing components, and output components, and specific components of component type data artifact 834 may be assigned to each role.

[0083] Figures 9A and 9B show schema 900, illustrating how schema 600 from Figure 6 and schema 700 from Figure 7 can be combined with data artifacts that provide version information. For clarity, the data artifacts from Figures 6 and 7 retain their respective reference numbers.

[0084] In general, Figures 9A and 9B illustrate how models, processes, subprocesses, and components can be associated with types and with data artifacts that provide version information. For example, model 610 is associated with model type 710 and definition 908 for the model version data artifact. Definition 908 is associated with definition 912 for the process version data artifact, and further, definition 912 is associated with definition 916 for the subprocess version data artifact. Note that definition 916 for the subprocess version data artifact contains methods for creating subprocess version components and activating subprocesses.

[0085] The subprocess data artifact 622 is associated with the subprocess type data artifact 718 and the subprocess version data artifact definition 916. The subprocess component data artifact 626 is associated with the subprocess data artifact 622 and the subprocess component type data artifact 730. Both the subprocess version data artifact definition 916 and the subprocess component data artifact 626 are associated with the subprocess version component data artifact definition 920 shown in Figure 9B.

[0086] Continuing to refer to Figure 9B, the subprocess version component data artifact definition 920 is associated with the component version data artifact definition 924, and furthermore, definition 924 is associated with the component version data artifact definition 928 and the component data artifact 630 in Figure 9A.

[0087] Returning to Figure 9A, in addition to being associated with the subprocess component type data artifact 730, the subprocess component data artifact 626 is shown as being associated with the input component data artifact definition 940, the processing component data artifact definition 944, and the output component data artifact definition 948. Data artifacts 940, 944, and 948 can function as subclasses of the subprocess component data artifact 626. Data artifacts 940, 944, and 948 are also associated with the subprocess category data artifact 734, and the subprocess category data artifact 734 is also associated with the subprocess component type data artifact 730.

[0088] Model 610 is associated with component data artifact 630, which in turn is associated with the definition 924 of the component version data artifact in Figure 9B. Model 610 is associated with source data artifact 618, which in turn is associated with component data artifact 630, subprocess data artifact 622, and process data artifact 614. Component data artifact 630 is also associated with component type data artifact 722, which may be associated with algorithm data artifact 970, configuration data artifact 974, or component version data artifact 924. Data artifacts 970, 974, and 978 may act as subclasses of component data artifact 630. Data artifacts 970, 974, and 978 are associated with component category data artifact 726, which in turn is associated with component type data artifact 722.

[0089] Example 7 - Examples of core, taxonomy, and domain models, and the relationships between them, as well as an example of a domain model instance. The techniques described may include relating ingested data to a semantic context, such as by representing the data in a knowledge graph or by otherwise associating the data with a contextual schema.

[0090] Figure 10 shows a modeling environment 1000 illustrating how the core data model can relate to the taxonomy model, where the core model and taxonomy model can be associated with one or more domain models. Furthermore, instances such as instances of classes in the knowledge graph and their associated properties can be associated with elements of the core data model and one or more domain models.

[0091] The computing environment provides a core data model 1010 with a core node 1014 (an empty circle). Core node 1014 of the core data model 1010 represents a specific organizing concept for a schema, such as a knowledge graph. In this case, core node 1014 includes node 1014a, which represents a stereotype. A stereotype refers to a generalizable and reusable template or archetype within the core data model that defines a conceptual structure or behavior that can be realized or instantiated in other models. For example, a stereotype in core data model 1010 could represent a high-level organizational concept such as an "entity," "attribute," or "relationship type," which can be specialized or instantiated as specific nodes and relationships in a taxonomy model or domain model. These realizations enable the consistent application of semantic concepts across different layers of the modeling framework.

[0092] At least some of the nodes 1022 of the taxonomy model 1018, such as nodes 1022a and 1022b, can be realizations of a fixed form. The core node 1014 includes a relation type node 1014b that defines a specific type of relationship between the nodes 1022 of the taxonomy model 1018, such as the relationship 1024 between nodes 1022c and 1022d. An example of a relation type could be a "property of," such as when one node in the relation corresponds to a class and another node is a property of the class.

[0093] Node 1014 of the core data model 1010 can also provide organizational classification for the nodes of the domain model, and Figure 10 includes domain models 1026a, 1026b, and 1026c. The core data model 1010, taxonomy model 1018, and domain models 1026a-1026c can be implemented in several ways, but in a specific example, they are implemented as a knowledge graph.

[0094] To aid in understanding Figure 10, it may be useful to consider the nodes of a given model in relation to the elements of a relational database data model. The core data model 1010 can provide basic organizational components, such as defining the concepts of tables, columns, and the relationships between tables and columns. The taxonomy model 1018 can represent standardized semantic concepts and relationships, such as specific table names and specific attributes available for use in tables, or provide additional details about the structural components of the core data model 1010, such as specific table or column types.

[0095] Domain model 1026 is a specific realization of at least some part of taxonomy model 1018, where node and relation names may differ from those used in the taxonomy model, but links between nodes in the taxonomy model and nodes in the domain model allow the domain model to be mapped to the standardized taxonomy of the taxonomy model.

[0096] Some nodes in the taxonomy model 1018 or domain model 1026 can represent data objects such as tables or views. Other nodes can represent attributes (columns / fields) of tables or views and are modeled as classes. A foreign key relationship between two database tables can be an example of the type of relationship between nodes.

[0097] Core nodes 1014 are associated by core edges 1016. Core edges 1016 define a specific type of association between pairs of connected core nodes 1014. While the core data model 1010 is shown as having a single core edge 1016 between any pair of connected core nodes, in at least some implementations, multiple core edges may exist between pairs of core nodes.

[0098] Core edge 1016 helps define how core nodes 1014 and their associated semantic concepts can be used to generate data models that can be implemented in a computing system. Core edge 1016 can also be used to describe hierarchical relationships between core nodes 1014, which can then be used to define more complex modeling concepts from the core nodes.

[0099] The taxonomy model 1018 can be structured in a similar manner to the core model 1010, in that node 1022 of the taxonomy model can be connected by taxonomy edge 1028. For simplicity, not all relationships between taxonomy nodes 1022 are shown in Figure 10. Continuing with the example of how taxonomy nodes 1022 and taxonomy edge 1028, or nodes and edges of the domain model, can be used to represent the data model of a relational database, one node may represent a table and other nodes may represent attributes. A complete table can be defined by associating the node representing the table with nodes representing attributes using edges of the appropriate relationship type. For example, a node representing a table may be linked to a taxonomy node representing its attribute using an edge of type "has attribute" or "has component".

[0100] It should be noted that relationships between nodes can be expressed from the "perspective" of either node. Using the previous example, a node representing an attribute of a table can be associated with a node representing the table using an edge of type "attribute of" or "component of". A one-way relationship between nodes can be called a "relationship" (which may also be called a "relation" or "predicate"), while a relationship that can be thought of in the opposite direction can be called an "inverse relationship" (or "inverse relation" or "inverse predicate"). A given relationship or inverse relationship can represent an instance of a particular relationship type.

[0101] As described, domain model 1026 represents at least some specific implementations of the taxonomy nodes 1022 and their associated relationships (including those indicated by taxonomy edges 1028) of taxonomy model 1018. Domain model 1026 is shown as including domain node 1032. In at least some implementations, the relationships between domain nodes 1032 in at least one given domain model 1026 are not represented using edges between domain nodes. Rather, edges 1036 link domain nodes 1032 to their corresponding taxonomy nodes 1022. The relationships between domain nodes 1032 can be determined by analyzing the taxonomy edges 1028 that exist between pairs of taxonomy nodes 1022 that are linked to the domain nodes by their corresponding edges 1036. In other cases, edge 1040 can be used to directly link domain node 1032 of domain model 1026, and the edge can optionally be linked (via edge 1036) to the corresponding taxonomy model edge 1028.

[0102] Figure 10 illustrates how domain nodes 1032 from multiple domain models 1026 can be linked to a common node 1022 in taxonomy model 1018. For example, edges 1036a and 1036b link domain nodes 1032a and 1032b to taxonomy node 1022e. In practice, many domain nodes 1032 from multiple domain models 1026 are linked to a common taxonomy node 1022. For simplicity of presentation, a single common taxonomy node 1022a is shown.

[0103] As an example of the relationship between domain nodes 1032 in different domain models 1026, consider a node in the first domain that represents a "business process" and has a relationship with the corresponding taxonomy node 1022. The second domain may include a domain node 1032 that contains a "process element" node, which is linked to the same taxonomy node 1022 as the domain node in the first domain. Thus, domain nodes 1032 in the first and second domain models 1026 can represent the same semantic concept in the form of a taxonomy node 1022.

[0104] In addition to relating domain nodes 1032 of different domain models 1026 through taxonomy node 1022, or alternatively, domain nodes of different domains can be directly related, for example, by using edge 1044. Since taxonomy model 1018 represents general semantic concepts represented in different domain models 1026, the taxonomy data model may not "recognize" that different domains exist or that at least two domain models have a more direct relationship.

[0105] The concept of "derivative" has been discussed previously. The term "derivative" can be used to express that domain node 1032 is an actualization of taxonomy node 1022. The term "derivative" can also be used to indicate that related domain notes 1032 in different domains refer to the same semantic concept or an instance thereof.

[0106] One or more instances 1050 of domain node 1032 are created and associated with the domain node via edge 1054. Instance 1050 is specific to a particular domain node 1032 and therefore specific to a particular domain model 1026. Note that associations can also be established between domain instance nodes 1050. For example, node 1050a in a first domain may represent a business process called "accrual management," while node 1050b in a second domain may have a process element with a similar meaning, "manage accruals." Associations between domain instance nodes 1050 can be represented as edges, similar to edges 1030, 1036, and 1040, as indicated by edge 1056.

[0107] Figure 10 provides additional examples of how core nodes 1014 can be linked to nodes of other models or instances of domain models. For example, node 1014c represents a domain concept and can be linked to domain model 1026. Core node 1014d represents a type realized by domain node 1032, and core node 1014e represents an instance of a type realized by instance 1050. Core node 1014f represents an association between domain nodes 1032, while core node 1014g represents an association between instances.

[0108] Figure 11 provides an exemplary schema 1100 illustrating the elements of the modeling environment 1000 in Figure 10 and their interactions. Schema 1100 includes the model data artifacts 610 and source data artifacts 618 described with respect to Figure 6. These data artifacts are each associated with domain data artifacts 1110, which represent specific realizations of at least some part of the taxonomy model. Source data artifacts 618 are further linked to source schema version data artifacts 1114, capturing the evolution of the schema over time. This versioning supports the ability to map changing schema elements to a consistent semantic standard of the taxonomy model.

[0109] Domain data artifact 1110 is associated with type data artifact 1118 and domain category data artifact 1122. Type data artifact 1118 acts as an abstract organizing construct derived from the core data model and is further associated with derivative data artifact 1126, reflecting its dynamic adaptation to support schema mapping or transformation. Type data artifact 1118 also defines structural relationships with additional constructs derived from the core model: property data artifact 1130 and association data artifact 1134. For example, properties can represent attributes or characteristics of a domain concept, while associations define specific interactions or dependencies between domain elements. The domain model is linked to the taxonomy model through the relationship between type data artifact 1118 (taxonomy model artifact) and standard data artifact 1160 (also a taxonomy model artifact), and the standard data artifact is further associated with taxonomy version data artifact 1164.

[0110] Schema 1100 also shows how the taxonomy model interacts with the core data model. The taxonomy model extends the core model by introducing taxonomy-specific artifacts such as attribute data artifact 1150 and related type data artifact 1154, which are linked to property data artifact 1130 and related data artifact 1134, respectively. These data artifacts refine structural elements defined in the core model to support standardized, reusable schema components. For example, attribute data artifacts may represent specific attribute structures used across multiple domains, while related type artifacts define standardized relationships such as "belongs to" or "is part of" that can be universally applied.

[0111] Taxonomy models further incorporate semantic concepts that standardize domain-independent representations of schema elements. For example, a taxonomy model might define a concept like "order" that acts as a semantic anchor for mapping domain-specific elements from different source schemas. For instance, a domain-specific "purchase order" or "sales order" might map to a standardized "order" element in the taxonomy model. This mapping provides domain-wide consistency while enabling semantic alignment.

[0112] The source schema is dynamically mapped to the taxonomy model so that elements specific to its domain are linked to corresponding taxonomy artifacts through derivatives and other mappings. For example, the source schema version might define specific data structures such as "customer_id" or "order_date," which are linked to standardized concepts in the taxonomy model, such as "Customer" or "Order Date." The taxonomy model is supported by schema versioning mechanisms to ensure that these mappings remain consistent even as the source schema evolves over time.

[0113] The core data model continues to provide a regular structure that underlies both the taxonomy model and the domain model. The core model includes constructs such as types, properties, and associations, which are refined in the taxonomy model and further instantiated in the domain model. For example, a domain model 1110 might define a specific reality of a taxonomy element, such as a company-specific schema for "Customer Data," and that specific reality then maps its element back into the taxonomy model for consistency and interoperability.

[0114] Figure 11 illustrates the interaction between the core model, the taxonomy model, and the domain model, showing how elements of each layer support schema mapping, alignment, and standardization. The dual role of the taxonomy model—as a structural extension of the core model and as a semantic framework for domain alignment—enables the integration of diverse schemas and supports evolving source schemas through version-controlled mapping.

[0115] Figures 12A–12C show an exemplary data model 1200 organized by a modeling environment 1000 that associates the captured data with semantic information. For example, the source data artifact 618 and model artifact 610 in Figure 6 can be associated with domains, which are linked to semantic models such as knowledge graphs, which may contain associated ontologaries.

[0116] In data model 1200, artifact 1208 corresponds to a node in core data model 1010 in Figure 10. Data artifacts 1212, 1216, 1220, 1224, 1228, 1232, 1268, and 1284 correspond to nodes in taxonomy model 1018 in Figure 10, representing standardized semantic concepts and structural extensions of the core model. Data artifacts 1240, 1244, 1252, 1264, 1272, 1276, 1280, and 1292 correspond to nodes in domain model 1026 in Figure 10, representing domain-specific implementations of taxonomy elements. Data artifacts 1248 and 1256 also correspond to domain model 1026 in Figure 10, but are part of a different domain than the domains associated with data artifacts 1240, 1252, and 1264. Linking data artifacts from different domains facilitates the alignment of common semantic concepts and enables actions such as relating data from one domain to data from another.

[0117] The term “derivative” is used to describe relationships in two complementary contexts. Firstly, a derivative represents a relationship between a source schema and a local schema (taxonomy model), capturing the semantic mapping of domain-specific constructs in the source schema to standardized elements in the taxonomy model. For example, a source schema element such as “sales order” may map to the standardized semantic concept “order” in the taxonomy model through a derivative relationship. Secondly, a derivative indicates a relationship between artifacts from different domains that share a common semantic concept. For example, domain-specific representations of “customer” in two different domain models may be linked as derivatives, reflecting their shared mapping to the standardized concept of “customer” in the taxonomy model.

[0118] Furthermore, the concept of derivatives offers significant advantages when relating data from two different sources. By reusing data from one source within the context of another, the second source can add new properties or relationships to the original data, enriching its context and usability. This approach supports the decoupling of data from different sources, enabling mapping of corresponding entities while allowing each source to maintain its integrity. For example, in Figure 12B, the entity represented by 1244 with ShortCode ACMP41 is reused in another source and represented by 1248 with ShortCode SACH-ACMP42. The second source adds properties or relationships to the instance from the original source, facilitating comprehensive data integration and enhancing the ability to perform cross-source analysis and reporting.

[0119] The concept of derivatives is applicable even within the same source, particularly when different subgraphs are created from the same source data at different times or under different conditions. This is especially useful for tracking relationships between data ingested using different processes. For example, data from the same source may be processed using different ingestion pipelines, resulting in subgraphs that reflect various transformations or enhancements. Derived relationships allow instances to be linked across these subgraphs, maintaining a clear lineage of how the data has progressed through different processing stages. For instance, a subgraph created using one process might contain an instance of a product with a specific property, while another subgraph created using a different process might contain an updated instance of the same product with additional properties or relationships. Derived relationships enable these instances to be linked, reflecting their progression and supporting consistent data usage. This technique enhances traceability, consistency, and integration within the same source, making it easier to manage and analyze data over time.

[0120] Example 8 - Examples of events that occur in response to changes in a process during execution or in the process definition. Figure 13 provides a diagram illustrating how various actions relating to a model and its elements, such as subprocesses or components, can trigger various events, including those that affect the versioning of model components.

[0121] Through the user interface (UI) 1308, the user can perform actions such as validating the alpha version of component 1312, which may contain a new version of an algorithm or input or output data artifact. Depending on the validation, the user can choose to activate or discard the alpha version of component 1312. Activating component 1312 triggers an event, which can be handled with respect to the subprocess component 1316. Therefore, changes to component 1312 result in events that affect the subprocesses that use that component. Since multiple subprocesses can use the same component 1312, changes to the component may trigger multiple events corresponding to the subprocesses that use the component.

[0122] As part of handling events related to component 1312 activated in connection with a subprocess, events indicating changes to the subprocess component may be triggered. Accordingly, an alpha version of subprocess 1320 may be generated. Again, since the same component 1312 can be used by multiple subprocesses, changes to the component, and by extension the subprocess component, may result in multiple events being triggered to create an alpha subprocess version.

[0123] Through a user interface 1324, which may be the same as or different from user interface 1308, the user can verify the validity of a new version of subprocess 1320 and optionally choose to activate it. If the user activates the alpha version of subprocess 1320, an event may be triggered on process engine 1328. In response to the event, process engine 1328 may trigger the execution of the subprocess. The execution produces a physical representation of the alpha version of the output component. Note that as part of the event indicating that the subprocess component has changed, a new version of output component 1312 may be created, which can store the results of the execution of process engine 1328. This means that changing component 1312 may indirectly change other components as well as the component that is directly changed. For example, a change to component 1312 such as an algorithm will result in a new version of that component, but a new output component will also be generated because the output is now different from the output produced using the old version of the component.

[0124] Example 9 - Example of a user interface for defining and managing processes Figures 14 to 21 illustrate various user interfaces that may be provided when implementing the technology of this disclosure. User interface 1400 in Figure 14 allows a user to create a new model, which includes a model name, model type, and description. The model type provides a template for the model being created, including what subprocesses and components are available for selection and customization.

[0125] The user interface 1500 in Figure 15 allows the user to select or define a source, and the user can provide the source name, code, and description of the source. In practice, the user interface 1500 may provide additional user interface elements, such as user interface elements that allow the user to configure connection to the source and retrieval of data from the source. For example, an address or link to the physical repository of the source may be provided along with administrative information such as a password or login information.

[0126] A component can be created using the user interface 1600 in Figure 16. The user interface 1600 allows the user to select a specific source to which the component should be associated and displays the name of the model to which the component is defined. The user can enter a name for the component and select the type of component. In this particular example, the component is of type ExternalData (ED). A description of the component may be provided, along with a link to the code that implements the component, such as a repository hosted in a Git-hosted repository.

[0127] The user interface 1700 in Figure 17 allows the user to perform actions related to component versions. The user interface 1700 provides a component version history, with each component version associated with a version identifier, a version description, the date and time the component version was created (or activated, depending on the implementation), and a status such as whether the component version is active or inactive. As shown, one of the components is an alpha version, and the user can change the status of the alpha version, for example, by using user interface element 1720 to activate that component as an active version. Alternatively, the date and time may refer to the time when a particular status of the component was set. Further information about the component is provided, including its name, source, category (such as data artifact, configuration, or algorithm), type, and the name of the model to which the component is associated.

[0128] A user can define a new subprocess using the user interface 1800 in Figure 18. The name of the model in which the subprocess is defined is provided, and the user can select the source of the subprocess and provide the subprocess name, subprocess type, and subprocess description. Subprocess types may include those shown in Figure 3, such as the ExternalDataToSourceData (ED2SD) subprocess. A link to the code implementing the subprocess, such as a Git repository, may be provided. A subprocess may, in some cases, define the order in which components should be called and specify the particular inputs for those components.

[0129] Figure 19 provides a user interface 1900 that allows the user to view subprocess version information and take version-related actions, such as activating a new version of a subprocess. User interface 1900 provides information about the subprocess, including the name of the subprocess, the source from which the subprocess is used, and the type of the subprocess. The subprocess version information includes a version identifier for each version of the subprocess, as well as a description, the date and time the subprocess was defined (or activated), and its status. Alternatively, the date and time may refer to the time when a particular status of the subprocess was set. The alpha version of the subprocess may be activated by selecting user interface element 1910. User interface 1900 also provides user interface element 1920 that can be used to view and edit components assigned to a particular subprocess.

[0130] Figure 20 provides a user interface 2000 that can be used to view components associated with a subprocess, including versions of such components. In a particular embodiment, the user interface 2000 may be displayed in response to selecting user interface element 1920 of user interface 1900 in Figure 19. The user interface 2000 provides information about the subprocess itself, including the name of the subprocess, the source from which the subprocess is used, the type of the subprocess, and the version of the subprocess. The user interface 2000 also provides the status of the subprocess version, which can be changed using user interface element 2010, for example, to activate the alpha version of the subprocess.

[0131] User Interface 2000 also provides information about the components used in subprocesses. User Interface 2000 provides each component's role (such as input, output, or processing), component identifier, version of the component used in a specific version of the subprocess displayed in the user interface, component description, date and time associated with the component (such as when the component was created or activated), and status. Alternatively, the date and time may indicate when a particular status of the component was set.

[0132] Figure 21 provides a user interface 2100 that displays process details, including the execution status of subprocesses. The user can select the desired date for process details. The user can also select a specific source from which the process is defined.

[0133] The user interface 2100 displays a sequence of subprocesses of a process and provides details of the subprocess execution, such as when a subprocess was last executed, which version of the subprocess was executed, and the execution status, such as whether the subprocess has completed, failed, is still in progress, or is pending because approval is required to run a new version of the subprocess.

[0134] The user interface 2100 provides user interface elements 2110 and 2114 that can be used to navigate to a user interface that displays metadata and version control information about a process, or a validation user interface that validates the output of one subprocess for use in a subsequent subprocess, or validates a new version of a subprocess or subprocess component before activation.

[0135] Metrics of the execution may also be provided, such as the number of domains represented in the results, the number of types (or classes) in the subgraphs generated by the process, the number of associations in the subgraphs, and the total number of triples used to define the subgraphs.

[0136] Example 10 -- Example of operation Figure 22 shows process 2200 for managing subprocess versions and associated updates within a computing system. In 2210, the definition of the first version of the subprocess is received. The first version of the subprocess includes one or more components, each of which is associated with an executable implementation. In 2214, processing data using the first version of the subprocess to produce a first result occurs.

[0137] At 2218, changes are received to the definition of the first version of the subprocess in order to provide a second version of the subprocess. The changes include either the version of the components of the first version of the subprocess, or the components themselves or the order in which the components are called. An identifier for the second version of the subprocess is generated at 2222.

[0138] At 2226, an event is generated indicating that the first version of the subprocess has been updated. At 2230, one or more actions are performed in response to the event. These actions include triggering a request to validate the second version or second result of the subprocess, reprocessing the data using the second version of the subprocess, or adding the identifier of the second version of the subprocess to the origin chain. The origin chain contains version identifiers for multiple subprocesses of a process, and the multiple subprocesses contain the process.

[0139] Example 11 -- Additional Example Example 1 is a computing system comprising at least one hardware processor, at least one memory coupled to the at least one hardware processor, and one or more computer-readable storage media. The computer-readable storage media contains computer-executable instructions that cause the computing system to perform an action when executed. The action comprises receiving a definition of a first version of a subprocess. The first version of the subprocess comprises one or more components, each of which is associated with an executable implementation. Data is processed using the first version of the subprocess to produce a first result.

[0140] Changes to the definition of the first version of the subprocess are received in order to provide a second version of the subprocess. The changes are either changes to the version of the components of the first version of the subprocess, or changes to the components or the order in which the components are called in the first version of the subprocess. An identifier for the second version of the subprocess is generated. An event is generated indicating that the first version of the subprocess has been updated.

[0141] In response to the event generation, one of the following occurs: a request is triggered to validate the second version of the subprocess or the second result; the data is reprocessed using the second version of the subprocess; or the identifier of the second version of the subprocess is added to the origin chain containing the version identifiers of multiple subprocesses of the process, where multiple subprocesses include the process.

[0142] Example 2 is the computing system from Example 1, where one or more components are associated with a version identifier. The operation further includes receiving changes to the component's implementation. In response to receiving changes to the component's implementation, the component's version identifier is updated.

[0143] When a component's version identifier is updated, an event is generated indicating that the component's version identifier has been updated. This event results in a subprocess receiving a change to its definition.

[0144] Example 3 is a computing system like that in Example 1 or Example 2, where a component is used in multiple subprocesses and events are sent to each of the subprocesses within those multiple subprocesses.

[0145] Example 4 is a computing system of Example 3 in which at least one of the multiple subprocesses is used in multiple different processes.

[0146] Example 5 is a computing system like any of Examples 1-4, where the components are data artifacts and the changes are to the definition of the data artifacts.

[0147] Example 6 is a computing system like those in Examples 1-4, where the component is an algorithm that operates on data, and the changes are to processes defined in the algorithm.

[0148] Example 7 is a computing system from any of Examples 1-6, where a subprocess is part of a process for taking data from a source into a first graph data structure.

[0149] Example 8 is a computing system like any of Examples 1-7, in which a subprocess associates data from a source with a source schema defined for that source.

[0150] Example 9 is a computing system of any of Examples 1-8. Its operation further includes storing data from a source, which has associations with a source schema, in a second graph data structure.

[0151] Example 10 is the computing system of Example 9. The operation further includes generating relationships between data in a second graph data structure and corresponding elements in a first graph data structure.

[0152] Example 11 is a computing system of any of Examples 1-10. The operation further includes receiving user input through a user interface to validate a second version of a subprocess or a second result, in response to a triggered request to validate a second version of a subprocess or a second result.

[0153] Example 12 is a computing system from any of Examples 1-11 in which data reprocessing using a second version of the subprocess is performed in response to user input.

[0154] Example 13 is a computing system of any of Examples 1 through 12. The operation involves creating a process, allocating multiple subprocesses to the process such that each subprocess contains other subprocesses, and further defining the order of the subprocesses among the multiple subprocesses. Defining the order involves allocating the output of the first subprocess among the multiple subprocesses as the input to the second subprocess among the multiple subprocesses.

[0155] Example 14 is the computing system of Example 13. The operation further includes allocating one or more components to each of the subprocesses of a plurality of subprocesses.

[0156] Example 15 is a computing system of Example 13 or Example 14 in which a subprocess among multiple subprocesses and a component of one or more components are used in multiple sources, and the implementation of a component among one or more components differs among at least two of the multiple sources.

[0157] Example 16 is a computing system of any of Examples 1-15. The operation further includes using subprocesses to retrieve data from a first source having a first source schema, and matching the data from the first source having the first source schema to elements of a second source schema in a second source, wherein the second source schema is different from the first source schema.

[0158] Example 17 is a computing system of any of Examples 1-15. The operation involves using subprocesses to ingest data from a first source having a first source schema and from a second source having a second source schema, wherein the second source schema is either the same as or different from the first source schema, and the second source is either the same as or different from the first source, and further includes matching data from the first source of the first source schema with data from the second source of the second source schema.

[0159] Example 18 is a computing system of Example 17 in which the second source differs from the first source, and the second source schema differs from the first source schema.

[0160] Example 19 is a method implemented in a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor. The method includes the step of receiving a definition of a first version of a subprocess, wherein the first version of the subprocess comprises one or more components, and the one or more components are associated with their respective executable implementations. Data is processed using the first version of the subprocess to produce a first result.

[0161] Changes to the definition of the first version of the subprocess are received in order to provide a second version of the subprocess. The changes are changes to the version of the components of the first version of the subprocess, or changes to the components of the first version of the subprocess or the order in which the components are called. An identifier for the second version of the subprocess is generated. An event is generated indicating that the first version of the subprocess has been updated.

[0162] In response to the generation of an event, one or more of the following will occur: a request is triggered to validate a second version of the subprocess or a second result; data is reprocessed using the second version of the subprocess; or the identifier of the second version of the subprocess is added to the origin chain containing multiple subprocess version identifiers for the process, where multiple subprocesses include the process.

[0163] Example 20 is one or more non-temporary computer-readable storage media. The storage media includes computer-executable instructions that cause a computing system to perform an action when executed by a computing system which includes at least one hardware processor and at least one memory coupled to the at least one hardware processor.

[0164] The operation involves receiving a definition of a first version of a subprocess, the first version of the subprocess containing one or more components, each of which is associated with an executable implementation. Data is processed using the first version of the subprocess to produce a first result.

[0165] Changes to the definition of the first version of the subprocess are received in order to provide a second version of the subprocess. The changes are changes to the version of the components of the first version of the subprocess, or changes to the components of the first version of the subprocess or the order in which the components are called. An identifier for the second version of the subprocess is generated. An event is generated indicating that the first version of the subprocess has been updated.

[0166] In response to the generation of an event, one or more of the following will occur: a request is triggered to validate a second version of the subprocess or a second result; data is reprocessed using the second version of the subprocess; or the identifier of the second version of the subprocess is added to the origin chain containing multiple subprocess version identifiers for the process, where multiple subprocesses include the process.

[0167] Example 12 - Computing System Figure 23 shows a generalized example of a suitable computing system 2300 in which the described innovation may be implemented. Since the innovation may be implemented in a variety of general-purpose or dedicated computing systems, computing system 2300 is not intended to imply any limitation on the scope of use or functionality of this disclosure.

[0168] Referring to Figure 23, the computing system 2300 includes one or more processing units 2310, 2315 and memories 2320, 2325. In Figure 23, this basic configuration 2330 is enclosed within the dashed lines. The processing units 2310, 2315 execute computer executable instructions, such as to implement the technologies described in Examples 1-10. The processing units can be a general-purpose central processing unit (CPU), a processor in an application-specific integrated circuit (ASIC), or any other type of processor. In a multiprocessing system, multiple processing units execute computer executable instructions to increase processing power. For example, Figure 23 shows a central processing unit 2310 and a graphics processing unit or auxiliary processing unit 2315. The tangible memories 2320, 2325 may be volatile memory (e.g., registers, cache, RAM), non-volatile memory (e.g., ROM, EEPROM, flash memory, etc.), or any combination of the two, accessible by the processing units 2310, 2315. The memories 2320 and 2325 store software 2380 that implements one or more of the innovations described herein in a form of computer executable instructions suitable for execution by the processing units 2310 and 2315.

[0169] The computing system 2300 may have additional features. For example, the computing system 2300 may include storage 2340, one or more input devices 2350, one or more output devices 2360, and one or more communication connections 2370. Interconnection mechanisms (not shown), such as buses, controllers, or networks, connect the components of the computing system 2300 to each other. Generally, operating system software (not shown) provides an operating environment for other software running in the computing system 2300 and coordinates the activities of the components of the computing system 2300.

[0170] The tangible storage 2340 may be removable or non-removable and includes magnetic disks, magnetic tapes or cassettes, CD-ROMs, DVDs, or any other media that can be used to store information in a non-temporary manner and can be accessed within the computing system 2300. The storage 2340 stores instructions for software 2380 that implement one or more innovations described herein.

[0171] The input device 2350 may be a keyboard, mouse, pen, or other touch input device such as a trackball, an audio input device, a scanning device, or another device that provides input to the computing system 2300. The output device 2360 may be a display, printer, speaker, CD writer, or another device that provides output from the computing system 2300.

[0172] Communication connection 2370 enables communication with another computing entity via a communication medium. The communication medium carries information, such as computer executable instructions, audio or video inputs or outputs, or other data, within a modulated data signal. A modulated data signal is a signal that sets or modulates one or more of its characteristics in a way that encodes information within the signal. The communication medium may, but not exclusively, be electrical, optical, RF, or other carriers.

[0173] Innovation can be described in a broader context where computer executable instructions, such as those contained within a program module, are executed on a target real or virtual processor within a computing system. Generally, a program module or component includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The functionality of a program module may be combined or divided among program modules as desired in various embodiments. The computer executable instructions of a program module may be executed within a local or distributed computing system.

[0174] The terms “system” and “device” are used interchangeably herein. Neither term implies any limitation to any type of computing system or computing device unless the context explicitly indicates otherwise. Generally, a computing system or computing device can be local or distributed and may include any combination of dedicated and / or general-purpose hardware and software that implements the functions described herein.

[0175] In the various examples described herein, a module (e.g., a component or engine) can be “coded” to perform a particular action or provide a particular function, and computer executable instructions for a module can be executed to perform such an action, cause such an action to be performed, or otherwise provide such a function. The functions described with respect to a software component, module, or engine can be executed as separate software units (e.g., programs, functions, class methods), but do not have to be implemented as separate units. That is, the functions can be incorporated into a larger or more general-purpose program, such as one or more lines of code in a larger or more general-purpose program.

[0176] For the sake of presentation, detailed explanations will use terms such as “decide” or “use” to describe the operation of a computer in a computing system. These terms are high-level abstractions of actions performed by computers and should not be confused with actions performed by humans. The actual computer operations corresponding to these terms will vary depending on the implementation.

[0177] Example 13 -- Cloud Computing Environment Figure 24 shows an exemplary cloud computing environment 2400 in which the described technologies may be implemented. The cloud computing environment 2400 includes a cloud computing service 2410. The cloud computing service 2410 may include various types of cloud computing resources, such as computer servers, data storage repositories, and networking resources. The cloud computing service 2410 can be centrally located (for example, provided by a company or organization's data center) or distributed (for example, provided by various computing resources located in different locations, such as different data centers, and / or in different cities or countries).

[0178] The cloud computing service 2410 is utilized by various types of computing devices (e.g., client computing devices) such as computing devices 2420, 2422, and 2424. For example, computing devices (e.g., 2420, 2422, and 2424) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablet computers or smartphones), or other types of computing devices. For example, computing devices (e.g., 2420, 2422, and 2424) can utilize the cloud computing service 2410 to perform computing operations (e.g., data processing, data storage, etc.).

[0179] Example 14 -- Implementation While some of the operations of the disclosed methods are described in a specific sequential order for the sake of brevity, please understand that this method of description is reorderful unless a specific order is required by certain wording listed below. For example, operations described in order may be reordered or performed simultaneously. Furthermore, for simplicity, the accompanying diagrams may not illustrate the various ways in which the disclosed methods may be used in conjunction with other methods.

[0180] Any of the disclosed methods may be implemented as computer executable instructions or computer program products stored on one or more computer-readable storage media, such as tangible non-transient computer-readable storage media, and executed on a computing device (e.g., any available computing device, including a smartphone or other mobile device with computing hardware). Tangible computer-readable storage media are any available tangible media that can be accessed within a computing environment (e.g., one or more optical media discs such as DVDs or CDs, volatile memory components (such as DRAM or SRAM), or non-volatile memory components (such as flash memory or hard drives)). Referring to Figure 23 as an example, the computer-readable storage media includes memory 2320 and 2325 and storage 2340. The term computer-readable storage media does not include signals and carriers. Furthermore, the term computer-readable storage media does not include communication connections (e.g., 2370).

[0181] Any computer-executable instructions for implementing the disclosed technology, and any data created and used during the implementation of the disclosed embodiments, may be stored on one or more computer-readable storage media. Computer-executable instructions may, for example, be part of a dedicated software application or part of a software application accessed or downloaded via a web browser or other software application (such as a remote computing application). Such software may run, for example, on a single local computer (e.g., any suitable commercial computer) or in a network environment using one or more network computers (e.g., via the Internet, a wide area network, a local area network, a client-server network (such as a cloud computing network), or any other such network).

[0182] For clarity, only specific selected aspects of software-based implementations are described. Other details well known in the art are omitted. For example, it should be understood that the disclosed technology is not limited to any particular computer language or program. For example, the disclosed technology may be implemented by software written in C, C++, C#, Java, Perl, JavaScript, Python, R, Ruby, ABAP, SQL, Xcode, GO, Adobe Flash, or any other suitable programming language, or in some examples, a markup language such as HTML or XML, or a combination of a suitable programming language and a markup language. Similarly, the disclosed technology is not limited to any particular type of computer or hardware. Specific details of suitable computers and hardware are well known and do not need to be detailed in this disclosure.

[0183] Furthermore, any software-based embodiment (including, for example, computer executable instructions for causing a computer to perform any of the disclosed methods) may be uploaded, downloaded, or remotely accessed via preferred means of communication. Such preferred means of communication include, for example, the Internet, the World Wide Web, intranets, software applications, cables (including fiber optic cables), magnetic communications, electromagnetic communications (including RF, microwave, and infrared communications), electronic communications, or other such means of communication.

[0184] The methods, apparatus, and systems disclosed should not be construed as limiting in any way. Rather, this disclosure covers all novel, non-obvious features and aspects of the various disclosed embodiments, both individually and in various combinations and partial combinations thereof. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, and the disclosed embodiments do not require any particular advantage or solution to any particular problem.

[0185] Any example of technology may be combined with any one or more of the other examples described. Given the many possible embodiments to which the principles of the disclosed technology may be applied, it should be recognized that the shown embodiments are examples of the disclosed technology and should not be considered limitations on the scope of the disclosed technology. Rather, the scope of the disclosed technology is encompassed by the scope and spirit of the following claims. [Explanation of Symbols]

[0186] 100 Computing Environments 108 Sources 112 channels 116 Processing Frameworks 120 Ingestion Process 124 Graphs 128 Deployment Process 132 Modeling Components 136 Lifecycle Management Components 150 operations 200 Overall Source-Graph Process 208 External Data - Source Data Subprocess 212 Source Data - Source Schema Process 216 Source Schema - Pipeline Subprocess 220 Pipeline-Subgraph Subprocesses 224 Subgraphs - Derivative Data Subprocesses 300 tables Column 310a Column 310b, External Data - Source Data Subprocess 310c row 310d column 310e row 310f row 320 lines 400 Computing Environments and Processes 410 Environments and Runtimes 414 subprocesses 418 composition 422a, 422b components 430 inputs 434 output 450 processes 454~478 operation 500 Computing Environments 508 Datastores 512 Source Data 514 Graph Data 518 Process Engine 522 Subprocess Runtime 524 Algorithm Runtime 528 Algorithm Repositories 532 users 536 Algorithm Development Components 540 Event Management Components 544 Version Control Components 546 Version Data 550 Semantic Modeling Components 554 Metadata Repository 556 Metadata 562 users 560 User Interface 566 Validation Components 570 Modeling Tools 600 schemas 610 Definition of Model Data Artifacts 614 Definition of Process Data Artifacts 618 Definition of Source Data Artifacts 622 Definition of Subprocess Data Artifacts 626 Definition of Subprocess Component Data Artifacts 630 Definition of Component Data Artifacts 700 schemas 710 Definition of Model Type Data Artifacts 714 Definition of Process Type Data Artifacts 718 Definition of Subprocess Type Data Artifacts 722 Definition of Component Type Data Artifacts 726 Definition of Component Category Data Artifacts 730 Definition of Subprocess Component Type Data Artifacts 734 Definition of Subprocess Component Category Data Artifacts 800 implementation 810 Model Type Data Artifacts 814 Process Type Data Artifacts 818 Subprocess Type Data Artifact 822 Process Type Hierarchy Data Artifacts 826 Subprocess Type Hierarchical Data Artifacts 830 Subprocess Component Type Data Artifact 834 Component Type Data Artifacts Sharp00 schema 908 Definition of Model Version Data Artifacts 912 Definition of Process Version Data Artifacts 916 Definition of Subprocess Version Data Artifacts 920 Definition of Subprocess Version Component Data Artifacts 924 Definition of Component Version Data Artifacts 928 Definition of Component Version Data Artifacts 940 Definition of Input Component Data Artifacts 944 Definition of Processing Component Data Artifacts 948 Definition of Output Component Data Artifacts 970 Algorithmic Data Artifacts 974 Configuration Data Artifacts 978 Data Artifacts 1000 Modeling Environments 1010 Core Data Model 1014 core nodes 1014a node 1014b Relation type node 1014c node 1014d Core Node 1014e Core Node 1014f core node 1014g core node 1016 Core Edge 1018 Taxonomy Model 1022 nodes 1022a node, taxonomy node 1022b node 1022c node 1022d node 1022e Taxonomy Node 1024 relationships 1026 Domain Model 1026a, 1026b, 1026c Domain Models 1028 Taxonomy Edge 1030 Edge 1032 Domain Nodes 1032a, 1032b domain nodes 1036 Edge 1036a, 1036b edge 1040 Edge 1044 Edge 1050 instances, domain instance nodes 1050a node 1050b node 1054 Edge 1056 Edge 1100 schema 1110 Domain Data Artifacts 1114 Source schema version data artifact 1118 Type Data Artifact 1122 Domain Category Data Artifacts 1126 Derivative Data Artifacts 1130 Property Data Artifacts 1134 Related Data Artifacts 1150 Attribute Data Artifacts 1154 Related Type Data Artifacts 1160 Standard Data Artifacts 1164 Taxonomy Version Data Artifact 1200 data models 1208 Artifacts 1212, 1216, 1220, 1224, 1228, 1232, 1240, 1244, 1248, 1252, 1256, 1264, 1268, 1272, 1276, 1280, 1284, 1292 Data Artifacts 1308 User Interface (UI) 1312 components 1316 Subprocess Components 1320 subprocesses 1324 User Interface 1328 Process Engine 1400 User Interfaces 1500 User Interfaces 1600 User Interfaces 1700 User Interface 1720 User Interface Elements 1800 User Interface 1900 User Interface 1910 User Interface Elements 1920 User Interface Elements 2000 User Interface 2010 User Interface Elements 2100 User Interface 2110, 2114 User Interface Elements 2200 processes 2300 Computing Systems 2310 Processing Unit, Central Processing Unit 2315 Processing unit, graphics processing unit, auxiliary processing unit 2320, 2325 memory 2330 Basic configuration 2340 storage 2350 Input Devices 2360 Output Devices 2370 Communication connection 2380 Software 2400 Cloud Computing Environments 2410 Cloud Computing Services 2420, 2422, 2424 Computing Devices

Claims

1. At least one hardware processor, At least one memory connected to the at least one hardware processor, When executed, the computing system Receiving a definition of a first version of a subprocess, wherein the first version of the subprocess includes one or more components, and each of those components is associated with an executable implementation. Processing data using the first version of the subprocess to generate a first result, To provide a second version of the subprocess, to receive a change to the definition of the first version of the subprocess, wherein the change is a change to the version of a component of the first version of the subprocess, or a change to the component or the order in which the component is called of the first version of the subprocess. To generate an identifier for the second version of the subprocess, To generate an event indicating that the first version of the subprocess has been updated, and A computing system comprising one or more computer-readable storage media including computer-executable instructions that cause the plurality of subprocesses to perform an operation including adding, in response to the generation of the event, (1) triggering a request to verify the validity of the second version of the subprocess or the validity of the results generated by the second version of the subprocess, or (2) reprocessing the data using the second version of the subprocess, or (3) adding the identifier of the second version of the subprocess to an origin chain including version identifiers of a plurality of subprocesses of the process, wherein the plurality of subprocesses include the process.

2. One or more of the aforementioned components are associated with a version identifier, and the operation is Receiving changes to the implementation of the said component, In response to receiving the changes to the implementation of the component, update the version identifier of the component. The computing system according to claim 1, further comprising generating an event indicating that the version of the identifier of the component has been updated in response to an update of the version identifier of the component, wherein the event results in the subprocess receiving a change to the definition of the subprocess.

3. The computing system according to claim 2, wherein the component is used in a plurality of subprocesses, and the event is sent to each of the subprocesses of the plurality of subprocesses.

4. The computing system according to claim 3, wherein at least one of the plurality of subprocesses is used in a plurality of different processes.

5. The computing system according to claim 2, wherein the component is a data artifact, and the modification is to the definition of the data artifact.

6. The computing system according to claim 2, wherein the component is an algorithm that operates on the data, and the modification is for an operation defined in the algorithm.

7. The computing system according to claim 1, wherein the subprocess is part of a process for importing data from a source into a first graph data structure.

8. The computing system according to claim 7, wherein the subprocess associates data from the source with a source schema defined with respect to the source.

9. The aforementioned operation, The computing system according to claim 8, further comprising storing data from the source having associations with the source schema in a second graph data structure.

10. The aforementioned operation, The computing system according to claim 9, further comprising generating relationships between data in the second graph data structure and corresponding elements of the first graph data structure.

11. The aforementioned operation, The computing system according to claim 1, further comprising receiving user input through a user interface to validate the second version or second result of the subprocess in response to a triggered request to validate the second version or second result of the subprocess.

12. The computing system according to claim 11, wherein the reprocessing of the data using the second version of the subprocess is performed in response to the user input.

13. The aforementioned operation, Creating a process, Assigning multiple subprocesses to the process, wherein the multiple subprocesses include the subprocesses; The computing system according to claim 1, further comprising defining the order of the subprocesses among the plurality of subprocesses, which includes allocating the output of a first subprocess among the plurality of subprocesses as input to a second subprocess among the plurality of subprocesses.

14. The aforementioned operation, The computing system according to claim 13, further comprising allocating one or more components to each of the processes of the plurality of subprocesses.

15. The computing system according to claim 13, wherein a subprocess among the one or more subprocesses and a component among the one or more components are used in multiple sources, and the implementation of a component among the one or more components differs among at least two of the multiple sources.

16. The aforementioned operation, Using the aforementioned subprocess, data is retrieved from a first source having a first source schema, The computing system according to claim 1, further comprising matching data from the first source of the first source schema to elements of the second source schema of the second source, wherein the second source schema is different from the first source schema.

17. The aforementioned operation, Using the aforementioned subprocess, data is retrieved from a first source having a first source schema, Importing data from a second source having a second source schema, wherein the second source schema is either the same as or different from the first source schema, and the second source is either the same as or different from the first source. The computing system according to claim 1, further comprising matching data from the first source of the first source schema with data from the second source of the second source schema.

18. The computing system according to claim 17, wherein the second source differs from the first source, and the second source schema differs from the first source schema.

19. A method implemented in a computing system comprising at least one hardware processor and at least one memory coupled to the at least one hardware processor, A step of receiving a definition of a first version of a subprocess, wherein the first version of the subprocess includes one or more components, and the one or more components are associated with their respective executable implementations. A step of processing data using the first version of the subprocess to generate a first result, A step of receiving a change to the definition of the first version of the subprocess in order to provide a second version of the subprocess, wherein the change is a change to the version of a component of the first version of the subprocess, or a change to the component of the first version of the subprocess or the order in which the components are called. A step of generating an identifier for the second version of the subprocess, A step of generating an event indicating that the first version of the subprocess has been updated, A method comprising, in response to the generation of the event, (1) triggering a request to verify the validity of the second version of the subprocess or the validity of the results generated by the second version of the subprocess, or (2) reprocessing the data using the second version of the subprocess, or (3) adding the identifier of the second version of the subprocess to an origin chain which includes version identifiers of a plurality of subprocesses of the process, wherein the plurality of subprocesses include the process.

20. A computer executable instruction that causes a computing system to receive a definition of a first version of a subprocess when executed by a computing system including at least one hardware processor and at least one memory coupled to the at least one hardware processor, wherein the first version of the subprocess includes one or more components, and the one or more components are associated with their respective executable implementations, A computer executable instruction, when executed by the computing system, causes the computing system to process data using the first version of the subprocess to produce a first result, A computer executable instruction that causes the computing system to receive a change to the definition of the first version of the subprocess in order to provide the computing system with a second version of the subprocess when executed by the computing system, wherein the change is a change to the version of a component of the first version of the subprocess, or a change to the component of the first version of the subprocess or the order in which the components are called, A computer executable instruction that causes the computing system to generate an identifier for the second version of the subprocess when executed by the computing system, A computer executable instruction that, when executed by the computing system, causes the computing system to generate an event indicating that the first version of the subprocess has been updated, One or more non-temporary computer-readable storage media comprising a computer-executable instruction that causes the computing system to perform the following actions when executed by the computing system: (1) trigger a request to verify the validity of the second version of the subprocess or the validity of the results generated by the second version of the subprocess; (2) reprocess the data using the second version of the subprocess; or (3) add the identifier of the second version of the subprocess to an origin chain including multiple subprocess version identifiers of the process, wherein the multiple subprocesses include the process.

Images