Dataset multiplexer for data processing systems

Abstract

A data processing system is provided with a dataset multiplexer that allows applications to be written to specify access to a dataset as an operation on a logical dataset. During execution of an application by the data processing system, an operation to access a dataset is performed by accessing an entry in a dataset catalog for the logical dataset. The entry contains information for accessing a physical data source that stores the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. The catalog entry may be created upon registration of a data source with the dataset multiplexer and may be automatically updated upon changes in the storage of the dataset. Maintenance of this catalog may be partially or fully automated such that the system automatically adapts to changes in the storage of the dataset without the need to modify applications.

Description

Technical Field

[0001] Cross - reference to Related Applications This application claims the benefit of priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 163,709, filed Mar. 19, 2021, entitled "DATASET MULTIPLEXER FOR DATA PROCESSING SYSTEM", and U.S. Provisional Patent Application No. 63 / 143,898, filed Jan. 31, 2021, entitled "DATASET MULTIPLEXER FOR DATA PROCESSING SYSTEM", which are hereby incorporated by reference in their entirety.

[0002] Aspects of the present disclosure relate to techniques for efficiently operating a data processing system having multiple datasets that may be stored in any of a number of data stores.

Background Art

[0003] Modern data processing systems manage an enormous amount of data within an enterprise. For example, in a large organization, there can be millions of datasets. Having such a large number of datasets can be extremely valuable to an enterprise because this data can support multiple aspects of the enterprise's business. Some datasets can support routine processes, such as tracking a customer's account balance or sending account statements to a customer. In other examples, processing data from one or more datasets can generate business insights, such as the conclusion that a requested transaction is fraudulent or that as a result of overall transactions in a particular geographic region, the enterprise is exposed to a certain level of financial risk. In still other examples, processing data from one or more datasets can generate technical insights, such as the conclusion that the enterprise is exposed to the risk of a technical failure as a result of an incorrect technical process.

[0004] Physical storage for these datasets can be provided in one of several ways. For example, datasets may be stored in a structured manner and managed by a database system within the enterprise. In this case, datasets may be stored as one or more tables managed by the database. Alternatively, simple datasets may be stored in files accessible to the data processing system, such as CSV or XML files, or flat files. The computer storage where the datasets reside, whether as files, database tables, or other formats, may be physically implemented in one of several forms, such as locally at the data processing system, distributed across the enterprise, or distributed across a network cloud managed by a third party.

[0005] Enterprise designers can select physical storage for a dataset based on its expected characteristics, such as dataset size, required access time, dataset retention period, or the impact of dataset loss or corruption on the enterprise. Commercial considerations, such as storage pricing or concerns about being tied to third-party storage vendors, can also influence choices made regarding the implementation of physical storage for enterprises. As a result, data stores that hold datasets used within an enterprise can take any of several forms.

[0006] To support a wide range of functions, data processing systems can run applications, whether to implement routine processes or extract insights from datasets. These applications can be programmed to access data stores and read and write data. [Overview of the project] [Means for solving the problem]

[0007] In some embodiments, a method performed by a data processing system enables efficient data analysis in a dynamic environment with multiple datasets by generating and / or using entries in a dataset catalog to enable access to physical datasets in a data store. The data processing system may be configured to run a data processing application programmed to access logical datasets. Each logical dataset contains a schema for data independent of the format of the corresponding data in the physical dataset. The data processing system includes a dataset multiplexer that can be configured to provide the application with access to physical datasets in the data store. The method includes creating a plurality of entries in a dataset catalog, each of which is associated with a logical dataset and a physical dataset, and associated with a computer executable instruction for accessing the physical dataset; receiving an input that at least partially identifies a first logical dataset to be accessed in order to perform an operation in a data processing application that specifies access to the dataset; calling a computer executable instruction for accessing the physical dataset associated with the entry in the dataset catalog associated with the first logical dataset when performing an operation in the data processing application; and dynamically updating the entries in the dataset catalog in response to an event indicating a change in the physical dataset associated with the logical dataset.

[0008] In one embodiment, creating multiple entries in a dataset catalog includes receiving information relating to a first physical dataset among physical datasets stored in a first data store among data stores, wherein the first physical dataset corresponds to a first logical dataset; generating a first program based on the information relating to the first physical dataset, which includes computer executable instructions for accessing the first physical dataset from the first data store; and storing a link in the first entry in the dataset catalog to the first program, which enables a data processing application to access the first physical dataset using the first program.

[0009] In one embodiment, generating a first program for accessing a first physical dataset from a first data store includes identifying the type of the first data store from received information, selecting a first program template for the type of the first data store, and generating the first program by inputting one or more values ​​for one or more parameters of the first program template into the first program template.

[0010] In one embodiment, receiving an input that at least partially identifies a first logical dataset includes providing a user interface that allows the user to at least partially identify the first logical dataset.

[0011] In one embodiment, calling a computer executable instruction includes enabling access to an entry in a dataset catalog associated with a first logical dataset, and, based on the information in the entry, enabling access to a data store that stores a physical dataset corresponding to the first logical dataset.

[0012] In one embodiment, dynamically updating entries in a dataset catalog includes detecting an event indicating a change associated with a physical dataset corresponding to a first logical dataset, and correcting the entry in the dataset catalog associated with the first logical dataset based on the detection of the event.

[0013] In one embodiment, modifying an entry in a dataset catalog includes modifying a computer executable instruction for accessing a physical dataset corresponding to a first logical dataset.

[0014] In some embodiments, a method is provided by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by registering datasets in a dataset catalog to facilitate access to multiple physical datasets in a data store. The data processing system is capable of operating with multiple physical datasets stored in a data store. The data processing system includes a dataset multiplexer configurable to provide an application with access to a physical dataset among the multiple physical datasets, the physical datasets being stored in a data store among the data stores. The physical datasets correspond to logical datasets containing schemas for data independent of the format of the corresponding data within the physical datasets. The method includes receiving information relating to a first physical dataset among multiple physical datasets stored in a first data store among the multiple data stores, the first physical dataset corresponding to a first logical dataset; generating a first program containing computer executable instructions for accessing the first physical dataset from the first data store based on the information relating to the first physical dataset; and storing a link to the first program in a first object in a library of objects to enable an application to access the first physical dataset using the first program.

[0015] In one embodiment, the method includes determining whether or not to modify a first program for accessing a first physical dataset based on the detection of an event indicating a change associated with the first physical dataset.

[0016] In one embodiment, the method includes generating a modified first program based on a decision to modify the first program, and replacing the first program with the modified first program as a link target.

[0017] In one embodiment, generating a modified first program includes generating a modified first program without modifying the application or the first logical dataset.

[0018] In one embodiment, the information relating to the first physical dataset includes information about the type of the first data store.

[0019] In one embodiment, the dataset multiplexer includes a library of objects that store information for accessing multiple physical datasets, wherein a first object in the library of objects includes an identifier for a first physical dataset.

[0020] In one embodiment, the dataset multiplexer further includes an API, and the method further includes providing the application with access to a first object via the API.

[0021] In one embodiment, the method further includes assigning identifiers to objects in a library based on the schema and logical name of each logical dataset in which information is stored for the object.

[0022] According to one aspect, the method further includes receiving a command to register a first physical data set in a data set catalog, and based on the received command, generating a first object and storing the first object in a library.

[0023] According to one aspect, the identifier of the first physical data set is a physical identifier.

[0024] According to one aspect, the first object further includes a second identifier, and the second identifier is a logical identifier of a logical data set associated with the first object.

[0025] According to one aspect, the method further includes, in response to detecting an event indicating that the first physical data set has been changed from being stored in a first data store to being stored in a second data store, modifying the physical identifier in the first object without modifying the logical identifier.

[0026] According to one aspect, the first object includes the values of parameters accessed during the execution of a first program, and the method further includes modifying the values of the parameters stored in the first object based on detecting an event indicating a change in the values of the parameters accessed in the first program.

[0027] According to one aspect, the first program includes access logic and conversion logic, and when an application is executed, the access logic and conversion logic of the first program are executed to provide access to the first physical data set and to perform a conversion between the format used in the first physical data set and the format used in the first logical data set.

[0028] According to one aspect, the first program includes one or more parameters such that the values of the one or more parameters that affect the operation of the first program affect the access to the first physical data set via the first program.

[0029] According to one aspect, the application is configured to supply the values of one or more parameters used when calling the first program.

[0030] According to one aspect, the method further includes generating the first program by detecting the type of the first data store and selecting a template from a plurality of templates based on the detected type.

[0031] According to one aspect, the first program includes a first part configured for read access to the first data store and a second part for write access to the first data store.

[0032] According to one aspect, the first program is configured as an executable data flow graph including logic for accessing the first physical data set.

[0033] In some embodiments, a method is provided by a data processing system for enabling efficient data analysis in a dynamic environment with multiple datasets by using entries in a dataset catalog to allow an application to access multiple physical datasets in multiple data stores. The data processing system can operate with an application and multiple physical datasets stored in multiple data stores. The application is programmed to access a logical dataset that contains a schema for data independent of the format of the corresponding data in the physical dataset. The method includes providing a user interface that at least partially identifies the logical datasets for the application to access, running the application, enabling access to objects in a library of objects associated with the logical dataset when an operation involving access to the identified logical dataset is performed, and enabling access to the data store that stores the physical datasets corresponding to the identified logical dataset, based on the information in the objects.

[0034] In one embodiment, the method further includes updating information within an object based on events associated with the storage of data corresponding to an identified logical dataset.

[0035] In one embodiment, the information within an object includes an executable program for accessing a physical dataset.

[0036] In one embodiment, an executable program for accessing a physical dataset encodes logic for converting data between the format used within the physical dataset and the format used within the logical dataset.

[0037] In one embodiment, the object is an executable program for accessing a physical dataset.

[0038] In one embodiment, the information within an object includes the type of data store.

[0039] According to one embodiment, the information within an object includes a record format or schema associated with a physical dataset.

[0040] In one embodiment, the information within an object includes one or more parameters that specify how to access the physical dataset, and one or more of these parameters include at least one parameter that indicates whether the data in the physical dataset is compressed.

[0041] In one embodiment, the information within an object includes one or more parameters that specify how to access a physical dataset, and one or more of these parameters include at least one parameter that indicates the type of access.

[0042] In one embodiment, the type of access includes indication of read access or write access.

[0043] In one embodiment, the type of access includes indicating access via high-speed or low-speed connection.

[0044] In one embodiment, the data processing system includes a repository of metadata related to a logical dataset, and providing a user interface includes presenting a menu of logical datasets based on the metadata in the repository.

[0045] According to several embodiments, the method performed by the data processing system enables efficient data analysis in a dynamic environment with multiple datasets by generating entries in a dataset catalog to allow access to physical datasets in a data store. The data processing system is configured to run a data processing application programmed to access logical datasets. Each logical dataset contains a schema for the data that is independent of the format of the corresponding data in the physical dataset, and the data processing system includes a dataset multiplexer that can be configured to provide the application with access to the physical datasets in the data store. The method includes receiving information relating to a first physical dataset stored in a first data store of a data store, wherein the application is programmed for accessing a first logical dataset, and the first physical dataset corresponds to the first logical dataset; generating a first program for accessing the first physical dataset from the first data store based on the received information, wherein generating the first program includes identifying the type of the first data store from the received information, selecting a first program template for the type of the first data store, and generating the first program by populating the first program template with one or more values ​​for one or more parameters of the first program template; and storing information in an object for calling the execution of the first program from within the application programmed for accessing the first logical dataset.

[0046] In one embodiment, inputting data into a first program template includes automatically discovering one or more values ​​of one or more first parameters of the first program template based on information related to a first physical dataset.

[0047] In one embodiment, one or more first parameters include information about a record format or schema associated with a first physical dataset.

[0048] In one embodiment, storing information in an object for calling the execution of a first program from within an application programmed to access a first logical dataset includes storing an identifier for the first data store.

[0049] In one embodiment, storing information for calling the execution of a first program from within an application programmed to access a first logical dataset in an object includes storing a logical identifier of the first logical dataset.

[0050] In one embodiment, generating a first program further includes obtaining information about one or more second parameters of a first program template, wherein one or more second parameters are different from one or more first parameters.

[0051] In one embodiment, one or more second parameters specify how to access the first physical dataset.

[0052] In one embodiment, generating a first program further includes determining whether a program template is available for a first data store type, and, based on the determination that a first program template is available for a first data store type, selecting an available template as the first program template.

[0053] In one embodiment, the method includes creating a program structure based on user input, on the basis of determining that a program template is not available for a first data store type, and generating a first program for accessing the first data store based on the created program structure.

[0054] In one embodiment, the method includes receiving information relating to a second physical dataset stored in a second data store of a data store, and generating a second program for accessing the second physical dataset from the second data store based on the information relating to the second physical dataset.

[0055] In one embodiment, the data processing system is configured to run in multiple environments, each environment containing an instance of the data processing system, and objects are assigned unique identifiers within the scope of each of the multiple environments, and include at least a portion that are common across the multiple environments.

[0056] In some embodiments, a method is provided by a data processing system to enable efficient analysis in a dynamic environment having multiple datasets by updating entries in a dataset catalog to facilitate access to physical datasets in a data store. The data processing system is configured to run a data processing application programmed to access data represented as logical datasets. Each logical dataset contains a schema for the data independent of the format of the corresponding data in the physical dataset, and the data processing system includes a dataset multiplexer configurable to provide the application with access to the physical datasets in the data store. The method includes receiving information relating to a first physical dataset stored in a first data store corresponding to a first logical dataset; generating a first program for accessing the first physical dataset from the first data store based on the received information; detecting events indicating changes associated with the physical dataset corresponding to the first logical dataset; and modifying the first program for accessing the physical dataset corresponding to the first logical dataset based on the detection of events.

[0057] In one embodiment, the physical dataset is a first physical dataset, and events indicating a change associated with the physical dataset include events indicating a change from a first datastore storing the first physical dataset to a second datastore, and the method further includes modifying the first program to access the first physical dataset from the second datastore in response to detecting an event indicating a change from the first datastore to the second datastore.

[0058] In one embodiment, the physical dataset is a first physical dataset, and events indicating changes associated with the physical dataset include events indicating changes in the values ​​of parameters used to generate a first program for accessing the first physical dataset.

[0059] In one embodiment, detecting an event indicating a change associated with a physical dataset includes detecting an event indicating that a first physical dataset is replaced with a second physical dataset corresponding to a first logical dataset, and modifying a first program for accessing a physical dataset includes replacing the first program with a second program for accessing a second physical dataset.

[0060] In one embodiment, a data processing system is configured to invoke a first program to perform an operation within an application that specifies access to a first logical dataset, the data processing system is configured to run in multiple environments, the first environment including a first instance of the data processing system, the second environment including a second instance of the data processing system, the first data store and the first program associated with the first instance of the data processing system, and the method further includes generating a second program for performing an operation within an application that specifies access to a first logical dataset within the second instance of the data processing system.

[0061] In one embodiment, this involves running an application in a second environment that specifies access to a first logical dataset, and accessing a second program to access a second physical dataset in response to the application's operations on the first logical dataset.

[0062] In one embodiment, a method is provided by a data processing system for enabling efficient data analysis in a dynamic environment with multiple datasets by using entries in a dataset catalog to allow an application to access multiple physical datasets in multiple data stores. The data processing system is configured to run a data processing application programmed to access logical datasets. Each logical dataset contains a schema for data independent of the format of the corresponding data in the physical dataset, and the data processing system includes a dataset multiplexer configurable to provide the application with access to multiple physical datasets in multiple data stores. The method includes performing operations within the application that specify access to the logical dataset by accessing the dataset catalog and selecting an object associated with the logical dataset, and based on the selected object, calling a program configured for accessing the data source that stores the physical dataset corresponding to the logical dataset.

[0063] In one embodiment, the method further includes dynamically updating objects in a dataset catalog in response to an event indicating a change in the physical storage of a logical dataset represented by objects in the data catalog.

[0064] The various embodiments described above can be used alternatively or in addition to any embodiments of the systems, methods, and / or processes described herein. Furthermore, a data processing system may be configured to operate in accordance with one or more of the embodiments described herein. Such a data processing system may include at least one computer hardware processor and at least one non-temporary computer-readable medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform such a method. Furthermore, the non-temporary computer-readable medium may, when executed by the at least one computer hardware processor of the data processing system, contain processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform one or more of the embodiments described herein. Thus, the above is a non-limiting summary of the invention as defined by the appended claims.

[0065] The following diagrams illustrate various aspects. Please note that the diagrams are not necessarily drawn to scale. Items appearing in multiple diagrams are indicated by the same or similar reference numbers in all of them. [Brief explanation of the drawing]

[0066] [Figure 1A] This is a block diagram of an exemplary enterprise IT system comprising a data processing system having a dataset multiplexer, according to one aspect of the technology described herein. [Figure 1B] Figure 1A is a block diagram of an exemplary enterprise IT system in which a dataset multiplexer is operational in a first time period, facilitating access between an application configured to access a logical dataset and a first data store that stores the physical dataset corresponding to the logical dataset. [Figure 1C]Figure 1B is a block diagram of an exemplary enterprise IT system in which a dataset multiplexer is in operation in a second time, facilitating access between an application configured to access a logical dataset and a second data store that stores the physical dataset corresponding to the logical dataset. [Figure 2A] Figure 1A is a block diagram of an exemplary enterprise IT system in which the data processing system is instantiated into multiple instances to provide multiple environments, and the application runs on the first instance, where a dataset multiplexer facilitates access between the application and the first physical dataset. [Figure 2B] Figure 2A is a block diagram of an exemplary enterprise IT system where the application runs on a third instance, which facilitates access between the application and a second physical dataset via a dataset multiplexer. [Figure 3A] This is a schematic diagram illustrating the graphical development environment for applications written as data flow graphs. [Figure 3B] Figure 3A is a schematic diagram of the dataflow graph, where the input nodes of the dataflow graph are configured or programmed in terms of logical datasets. [Figure 3C] Figure 3A is a schematic diagram of the dataflow graph, modified to access information in the dataset catalog to enable access to the physical dataset for performing operations within the application that specify access to the logical dataset. [Figure 4] This is a summary of information that may be reflected in objects in a dataset catalog, which provides information about the physical dataset corresponding to a logical dataset. [Figure 5A] Figure 1A is a block diagram of an exemplary enterprise IT system, illustrating the details of adding a dataset multiplexer. [Figure 5B]Figure 1A is an exemplary block diagram of an IT system showing data multiplexer components that can be optionally used when interfacing with a running application. [Figure 6A] This is a block diagram of an exemplary enterprise IT system in a first operating state at a first time point, as depicted in Figure 1A or Figure 5A. [Figure 6B] Figure 6A is a block diagram of an example enterprise IT system in the second state during the second time period. [Figure 7] This block diagram shows information used in a data processing system comprising a dataset multiplexer according to some aspects of the technology described herein. [Figure 8] This is a flowchart illustrating an exemplary method for operating a data processing system having a dataset multiplexer according to one aspect of the technology described herein. [Figure 9] This is a block diagram of an exemplary computing system environment that may be used when implementing some aspects of the technology described herein. [Modes for carrying out the invention]

[0067] The inventors recognize and understand that a dataset multiplexer can enable the efficient operation of a data processing system. In a company with numerous datasets that may be stored in various data stores, a dataset multiplexer enables the use of applications written in terms of one or more logical datasets, rather than in terms of physical datasets. These applications written in terms of logical datasets do not need to be modified for proper operation even if the data store that stores the one or more physical datasets represented by the logical datasets changes. To support this dynamic updating of data stores, a dataset multiplexer can maintain a catalog of datasets such that each entry in the catalog provides information for accessing the data store where the one or more physical datasets represented by the logical datasets are stored. A dataset multiplexer can enable efficient analysis, for example, in a dynamic environment where the physical storage of datasets may evolve or change.

[0068] By using a dataset multiplexer, applications can be written and executed without the application having knowledge of the formats supported by the data stores it accesses (e.g., record formats or schemas), or the physical locations of these data stores. For example, a business user who understands how to extract business insights from data but lacks knowledge of physical datasets and data stores can write applications from the perspective of logical datasets rather than physical datasets. A dataset multiplexer can automatically provide connections between the application and the appropriate data stores that store the physical datasets represented by the logical datasets, thereby avoiding the need for applications and users to have knowledge of data store implementations.

[0069] The dataset catalog may be updated in response to events indicating changes in the storage of datasets, such as physical datasets represented by logical datasets. Applications and / or logical datasets may not need to be modified in response to these events. By retrieving information from the catalog to access the datastore that stores the physical dataset corresponding to the application's logical dataset at the time of access, the appropriate datastore can be accessed without having to maintain the application to adapt to changes in the datastore. For enterprises, this capability can facilitate the migration of datasets from one storage location to another, enabling efficient use of computer storage while maintaining proper application execution. For example, a dataset can be migrated from one storage location to another, or from one type of storage to another, throughout its entire lifecycle. Such migrations can occur without modifying the application and while maintaining proper application execution. Avoiding the need to modify the application even when such changes occur results in reliable and efficient application execution, and can lead to significant cost savings for enterprises by avoiding the costs and downtime associated with modifying and retesting applications.

[0070] As a concrete example, a physical dataset may initially be stored as a file. Storing it as a file can allow for the use of low-cost computer storage. As the volume of data in the physical dataset increases, or as the value of the data increases, the physical dataset may be migrated to a database system to enable faster processing of large datasets or higher fault tolerance. By updating the catalog entry for the logical dataset corresponding to the physical dataset, applications written to access the logical dataset via a dataset multiplexer will continue to operate without modification when the physical dataset is migrated from a file to a database system.

[0071] Catalog entries may contain information for accessing physical datasets that can adapt to other types of changes to the storage of data associated with logical datasets. This information may include programs that access data from the datastore at runtime and translate it into a representation of the logical dataset. As a specific example, the format of fields in the physical dataset used to store logical entities can be changed without affecting applications that reference the logical entities, because modifications to the dataset catalog entries may include modifications to programs that translate the datastore data into the format used by the logical dataset.

[0072] Dataset multiplexers can also facilitate application development by simplifying transitions between programming environments. For example, applications are traditionally developed in a development environment, tested in a test environment, and then moved to a production environment. In the production environment, the application may read from and write to one or more datastores containing "live" data used across the enterprise. In the test and development environments, the application may operate using offline datastores that are less likely to impact the enterprise if corrupted by improper application behavior. In the development environment, the datastore may be relatively small, but in the test environment, the datastore may be structured to provide robust test cases, including extreme test cases that may not appear with current live data.

[0073] Regardless of whether different environments require different datasets, each environment may have its own dataset catalog information. An instance of the data processing system providing a development environment can access data catalog information scoped for the development environment. Similarly, an instance of the data processing system providing a test or production environment can access data catalog information scoped for their respective environments to access the appropriate data store. In this way, an application written to access a logical dataset can run in any environment and automatically access the appropriate data store for each environment without the need to adapt the application to a specific environment. When the execution of an application involves operations on a logical dataset, the data processing system automatically uses the appropriate data catalog information for the appropriate environment to access the data store containing the physical dataset for that environment that stores the data corresponding to the logical dataset.

[0074] The value of such a dataset multiplexer can be enhanced by one that can automatically construct entries in a dataset catalog for datastores. A dataset multiplexer can, for example, maintain a set of program templates applicable to different types of datastores. When a datastore is registered using a dataset multiplexer, the dataset multiplexer can detect the type of datastore and select the appropriate template. A program for accessing that datastore can be constructed by populating the selected template with the parameter values ​​discovered through the analysis of the datastore. Some or all of the parameter values ​​may, alternatively or additionally, be obtained from a metadata repository that maintains the datastore's metadata, supplied by user input, or obtained in other ways.

[0075] The data processing system may be implemented to achieve any or more of the aforementioned objectives and benefits. These objectives and benefits may be used individually or in any appropriate combination.

[0076] Typical data processing systems equipped with a dataset multiplexer Figure 1A is a block diagram of an IT system 100, which includes an exemplary data processing system 104 and a dataset multiplexer 105 integrated with the data processing system 104, according to some aspects of the technology described herein. The IT system 100 may be the IT system of a company, such as a financial company. For simplicity, elements of the company's IT system, such as networks, cloud storage, and user devices, are not explicitly shown.

[0077] The data processing system 104 is configured to access data stores 102-1, 102-3, 102-3, ..., and 102-n (for example, to read data from these data stores and / or to write data to these data stores). Each of the data stores 102-1, 102-3, 102-3, ..., and 102-n can store one or more physical datasets. A data store can store any suitable type of data or set of data in any suitable method or format. A data store may store data as flat text files, spreadsheets, etc., using a database system (e.g., a relational database system). These data stores may be internal or external to the enterprise. For example, an external data store may reside "in the cloud" or in storage hardware managed by a third party. Thus, the data stores can provide a federated environment in which different data stores used by the enterprise may be in different locations and / or managed by different entities, internal or external to the enterprise.

[0078] In some cases, a data store can store transaction data. For example, a data store can store credit card transactions, telephone record data, or bank transaction data. It should be understood that the embodiments of the technology described herein are not limited in this respect, and the data processing system 104 may be configured to access any appropriate number of data stores of any appropriate type. A data store that the data processing system 104 may configure to read data from is sometimes called a data source. A data store that the data processing system 104 may configure to write data to is sometimes called a data sink. However, technologies such as those described herein can be applied to data stores that hold other types of data used by enterprises.

[0079] Each datastore may be implemented with one or more storage devices and may include data management software or other control mechanisms to support the storage of physical datasets in any suitable type of one or more formats. The one or more storage devices may be of any suitable type and may include, for example, one or more servers, one or more disk arrays, a cluster of one or more disk arrays, one or more portable storage devices, one or more non-volatile storage devices, one or more volatile storage devices, and / or any other one or more devices configured to electronically store data. In embodiments in which the datastore includes multiple storage devices, the storage devices may be located in one physical location (e.g., within a building) or distributed across multiple physical locations (e.g., in multiple buildings, different cities, states, or countries). The embodiments of the technology described herein are not limited in this respect, and the storage devices may be configured to communicate with each other using any suitable type of one or more networks.

[0080] Data management software can organize data in physical storage and provide a mechanism for accessing data so that data can be written to or read from physical storage. Data management software may be, for example, a database system or a file management system. Depending on the type of data management software, one or more storage devices may store physical datasets using one or more formats, such as database tables, spreadsheet files, flat text files, and / or files in any other suitable format (e.g., the mainframe's native format). Data stores 102-1, 102-2, 102-3, ..., and 102-n may be of the same type (e.g., all may be relational databases) or of different types (e.g., one may be a relational database, while another may be a data store that stores data in flat files). When multiple data stores are of different types, the storage environment may be referred to as a heterogeneous or federated data environment 102. The data store may be, for example, a SQL Server database, an Oracle database, a Teradata database, a flat file, a multi-file data store, a HADOOP distributed database, a DB2 data store, a Microsoft SQL Server data store, an INFORMIX data store, a table, a group of tables or other subparts of a database, and / or any other suitable type of data store, as the embodiments of the technology described herein are not limited in this respect.

[0081] The data processing system 104 supports various applications 106 to perform functions to access (e.g., read access and / or write access) physical datasets stored in data stores 102-1, 102-3, 102-3, ..., and 102-n. The applications 106 can then perform operations based on the data in the data stores. The data processing system 104 can support applications 106-1, 106-2, 162-3, ..., and 106-n, which may be of the same or different types. In some examples, when executed, an application can read or write transaction data to one or more physical datasets in a data store. In other examples, when executed, an application can read or write data to physical datasets stored across multiple different data stores and analyze the data to extract business insights from the datasets.

[0082] Application 106 can be developed as a dataflow graph, for example, as shown in Figure 3A. A dataflow graph may include components called “nodes” or “vertices” that represent data processing operations performed on data, and links between components that represent the flow of data. Techniques for performing computations encoded by a dataflow graph are described in U.S. Patent No. 5,966,072, entitled “Executing Computations Expressed as Graphs,” which is incorporated herein by reference in its entirety. Environments for developing applications (e.g., computer programs) as dataflow graphs are described in U.S. Patent Application Publication No. 2007 / 0011668, entitled “Managing Parameters for Graph-Based Applications,” which is incorporated herein by reference in its entirety. A dataflow graph may include data sources (such as input data stores 302 or 304 in Figure 3A) and data sinks (such as output data store 314 in Figure 3A). These are represented by terminal nodes in flows indicating access to datastores 102-1, 102-3, 102-3, ..., or 102-n.

[0083] However, the application itself does not need to be programmed with a specific data store included in the application. Rather than being hardcoded to access a single physical dataset, application 106 can be programmed in terms of logical datasets. A logical dataset may refer to a logical representation of one or more datasets. The data processing system 104 may store definitions of multiple logical datasets and other metadata about those logical datasets. This information may be managed, for example, by a metadata management module (e.g., metadata management module 526 in Figure 5A). Tools used with the data processing system 104 can access metadata about logical datasets and perform functions based on that metadata. For example, a program development environment may provide a user interface that allows users to select available logical datasets and use them in programming the application.

[0084] A logical dataset may have a schema that defines the data, regardless of the format of the corresponding data in the physical dataset / datastore. For example, a logical dataset may have a schema that defines the logical entities within the logical dataset. Logical entities may be recognizable and / or understandable to human users. For example, a logical dataset may contain a logical entity such as a customer name. In the physical dataset corresponding to this logical dataset, the customer name may be stored in a single row of a data table as three fields, each holding data corresponding to the customer's first name, the initial of their middle name, and their last name, respectively. However, a logical dataset may simply contain the logical entity Customer_Name, regardless of the format of the data in the physical storage.

[0085] The data processing system 104 may include an interface (not shown) that can define a schema for a logical dataset. The interface may be a user interface that allows a user to introduce a logical dataset into the system by specifying the logical dataset or the schema for the logical dataset. In some embodiments, the data processing system 104 may store a set of logical entities commonly used in a company's business. Examples of commonly used logical entities may include one or more of the following: name, identification number, telephone number, address, nationality, account balance, transaction amount, or date. These business terms may be used to specify, at least partially, the schema for a logical dataset. However, the schema may be defined to include other logical entities in place of, or in addition to, predefined logical entities.

[0086] By enabling application programming from the perspective of logical datasets, programmers creating applications no longer need to understand the format of the datastore that stores the corresponding physical datasets. As a result, data analysts can develop applications using logical datasets without needing to understand the format of the data in the datastore that holds the physical datasets.

[0087] As a more detailed example, a programmer within a company might define a logical dataset to store new customers. The schema of the logical dataset might include logical entities such as customer name, customer address, customer identifier, and customer acquisition date. A data analyst can write applications in terms of the logical dataset and these logical entities, regardless of the storage format of the corresponding physical dataset. As a result, a data analyst can write applications without knowledge of the physical dataset that stores the data accessed by the application.

[0088] When an application is executed, the data of the physical dataset corresponding to the logical dataset may be stored in one or more of the data stores 102-1, 102-3, 102-3, ..., and 102-n. Each operation that specifies access to the logical dataset for the execution of the application may be performed by a data processing system 104 that reads or writes data from the corresponding physical dataset stored in one of the data stores 102-1, 102-3, 102-3, ..., and 102-n. In some embodiments, a dataset multiplexer 105 can enable the automatic execution of such operations by automatically accessing the corresponding physical dataset. This access may include conversion between the format of the data stored in the physical data store and the format specified in the schema of the logical dataset. As another example, the conversion may result in associating data from the physical dataset with metadata associated with the logical dataset. Specifically, the conversion may associate fields in the physical dataset with fields in the logical dataset tagged with indications that they hold personally identifiable information. As a result, metadata can be used in operations on data from physical datasets, such as filtering or masking personally identifiable information.

[0089] As shown in Figure 1A, the data processing system 104 includes a data set multiplexer 105 for automating access to corresponding physical data sets and conversion between the formats of logical and physical data sets. The data set multiplexer 105 can maintain a catalog 107 of data sets, where each entry in the catalog corresponds to a logical data set and provides information for accessing one or more physical data sets. For example, a catalog entry may identify a physical data set in data store 102-1, 102-3, 102-3, ..., or 102-n that corresponds to a logical data set. The catalog entry may, or may not, include information for converting data stored in a physical data set to the format of a logical data set. This information may be an executable program, or may include an executable program. For example, the catalog information may identify a program for converting data from multiple fields in a physical data set to the format of the corresponding logical entity in a logical data set. Other information may, or may not be, stored as catalog information for accessing one or more physical data sets, or may be reflected in such catalog information.

[0090] The dataset multiplexer 105 enables application 106 to seamlessly access one or more physical datasets based on one or more programmed logical datasets using information in the dataset catalog. Figure 1B shows an application (e.g., application 106-3) programmed to access data according to logical datasets. When an operation to access a logical dataset (e.g., read and / or write) is performed, the dataset multiplexer 105 of the data processing system 104 can enable access to the corresponding one or more physical datasets in the data store (e.g., data store 102-1). For example, if the catalog information stored with respect to the logical dataset is an access control program, or includes an access control program, that program may be executed. As a result, even though application 106-3 is programmed in terms of logical datasets, when a data access operation is performed, the physical datasets stored in data store 102-1 are accessed.

[0091] The dataset multiplexer 105 may access a catalog of datasets and select entries associated with logical datasets referenced in application 106-3. Information may then be used for data access to identify physical datasets stored in datastore 102-1 and / or to convert data in the format of datastore 102-1 to the format of the logical dataset.

[0092] In some cases, this access may be dynamic. Catalog information may be used when performing operations of an application that requires data access. Entries in the dataset catalog associated with a logical dataset may be updated in response to events indicating changes to the storage of information associated with the logical dataset. Access to the physical data store via catalog information can ensure that the application continues to run despite changes that may occur at any point in time throughout the entire IT system 100, even if the data analyst who wrote application 106-3 or other users are unaware of those changes.

[0093] For example, a physical dataset may be migrated from datastore 102-1 to datastore 102-n. The logical dataset in which the application is programmed does not need to be modified to account for this change. By updating the catalog entry for the logical dataset, the dataset multiplexer 105 can automatically utilize the updated catalog information to provide application 106-3 with access to the correct physical dataset, regardless of the datastore in which it resides.

[0094] Figure 1C shows application 106-3 accessing datastore 102-n via the dataset multiplexer 105 of the data processing system 104. The access conditions in Figures 1B and 1C may be the result of running application 106-3 at different times. Since catalog information is dynamic and changes to take into account the storage of datasets, no changes to application 106-3 are necessary for correct access to the desired data.

[0095] In Figure 1B, the solid lines show the data flow from datastore 102-1 to application 106-3 when performing an operation to access (e.g., read and / or write) a logical dataset. The dashed lines show interactions between components that can control the data flow in the operation. For example, application 106-3 may interact with dataset multiplexer 105 to obtain information from a catalog entry associated with the logical dataset for accessing the physical dataset corresponding to the logical dataset. Dataset multiplexer 105 may obtain information from one or more corresponding physical datasets in datastore 102-1 to generate the appropriate catalog entry. Similarly, the solid lines in Figure 1C show the data flow from datastore 102-n to application 106-3 when performing an operation to access (e.g., read and / or write) a logical dataset, and the dashed lines show interactions between components (e.g., dataset multiplexer 105, application 106-3, and datastore 102-n) that can control the data flow in the operation.

[0096] Using dynamic data can ensure correct operation despite any of several other types of changes within the IT system 100. In addition to changes in the data store where the physical datasets are stored, the type of data store holding the datasets may change. For example, the type of data store may change. For instance, data store 102-1 may be an Oracle database, while data store 102-n may be an SQL Server data store. As another example, the schema of a physical dataset may change to include an additional field, for example, for name data. Such changes are automatically compensated for by modifying the conversion logic in the catalog.

[0097] Dynamically using dataset catalog information for data access allows for automatic handling of other types of changes. As another example, a user might run different instances of a data processing system for different purposes. When running on different instances, it may be desirable for the same application to access different physical datasets. Such execution can be ensured by providing different catalog information on different instances, or when it is desirable for the application to access different physical datasets corresponding to the same logical dataset in different contexts.

[0098] Figure 2A shows an application (e.g., application 106-2) accessing one or more physical datasets in a data store (e.g., data store 102-2) via a dataset multiplexer of an instance of a data processing system (e.g., instance 104-1 of data processing system 104) according to some aspects of the technology described herein. In the environment created by instance 104-1, access to logical datasets is resolved to datasets in data store 102-2. The same application running in different environments created by different instances 104-n of the data processing system may access different physical datasets. Figure 2B shows application 106-2 accessing data store 102-n (e.g., database data store) in an environment created by instance 104-n of data processing system 104. For the sake of illustration simplicity, separate lines showing the control flow between the illustrated components are not shown in Figures 2A and 2B. However, it should be understood that the components of the data processing system may interact to control the operations described herein. Therefore, control interactions may be omitted for simplification.

[0099] The operations shown in Figures 2A and 2B can be generated by scoped catalog information for each instance so that references to the same logical dataset within each scope can access the physical dataset via the catalog information of that scope. All or part of the identifier for a logical dataset may be persistent across multiple scopes. For example, a logical dataset may be identified by a combination of name and schema that may be the same regardless of the environment. However, the catalog information for the dataset associated with that logical identifier may differ in different instances.

[0100] In the embodiments of Figures 2A and 2B, different instances 104-1, 104-2, ..., 104-n of the data processing system 104 may be provided for different programming environments. As a specific example, a company may run the data processing system in development, test, and production environments. The datasets used by the same application may differ in each of these environments. Live data used in the production environment may not be used in the development or test environment to avoid corruption of the live data and / or to minimize the risk of sensitive information leakage. The data store for the production environment may be large and offer high-speed data access, and therefore may be very expensive. On the other hand, datasets for the development environment may be stored in a small, low-cost data store to reduce the application development cost. The dataset for the test environment may include data that may occur in rare operational scenarios that are not present in the live dataset, during application testing, to ensure robust testing and complete code coverage. Making the application usable in any environment allows for efficient movement between environments such as development, test, and production, improving the efficiency of application development and the overall operation of the IT system.

[0101] Each instance of the data processing system 104 may include a dataset multiplexer that maintains a catalog of datasets for the corresponding environment. Each dataset multiplexer may have access to its respective catalog of datasets for the appropriate environment to provide access to one or more appropriate data stores. For example, Figure 2A shows application 106-2 accessing development environment data store 102-2 (e.g., a flat file data store) via instance 104-1 of the data processing system 104. Figure 2B shows application 106-2 accessing production environment data store 102-n (which may be a database) via instance 104-n of the data processing system 104.

[0102] Typical application development techniques using dataset multiplexers In some embodiments, applications executed by the data processing system may be written in a graphical programming language by a human user of the data processing system. In other embodiments, a procedural language or other type of programming language may be used, either alternatively or additionally.

[0103] Figure 3A shows a graphical user interface in which a data analyst or other human user can write applications in a graphical development environment, and is used herein as an example of application development. In this example, the data processing system includes a library of components that perform operations on data. Although not explicitly shown in Figure 3A for simplicity, the graphical development environment may include a toolbar or other user interface elements from which the user can select components from its library. The user can also specify connections between these components to form a graph. For example, a component may specify an operation to transform data, or a data source or data sink to access. Components may be represented by icons having different shapes depending on the operation performed by the component, or the type of data store that holds the data in the data source or data sink.

[0104] Users can write applications by selecting components corresponding to desired operations and connecting them in an order that specifies the desired data flow through the operations represented by the components. Each component can be configured by user parameter inputs. The values ​​of some configuration parameters can specify the manner in which a component operates. For example, a component representing a dataset may accept a parameter that specifies its operation as a data source or data sink.

[0105] In embodiments where an application is written using logical datasets, the values ​​of some configuration parameters can specify a particular logical dataset and / or logical entities within that logical dataset for use when performing operations on the component. For example, a component representing a dataset can be configured to represent a specified logical dataset by supplying the identifier of the logical dataset as the value of its parameter. The component may also consist of user inputs that specify logical entities to be used as keys in particular operations.

[0106] A data processing system may include a repository of information about logical datasets and / or logical entities that can be used when configuring application components. Entries in this repository may be created by the user writing the application. However, in a corporate setting, many individuals may be involved in data generation and analysis, and the information in the repository may not have been created by the user developing the application. For example, logical dataset information may have been generated by other users or by automated analysis of a specific physical dataset.

[0107] The user interface provided in the development environment may include user interface elements that allow the user to specify logical datasets or logical entities within the repository as parameter values ​​that make up the components of the graph. These user interface elements may include elements for the user to enter search queries. The query may be a facet query, for example, in which the user specifies one or more values ​​of dimensions that describe a logical dataset or logical entity. These dimensions may include, for example, words entered into the repository to describe the logical dataset or the field names contained within the dataset.

[0108] A data processing system can perform a search according to a query and return a list of options selected by the data processing system based on the query. The user can then select the returned values ​​to configure a component, which then operates according to the selection. For example, if a dataset component is configured as a data source configured to output data from a logical dataset, that component will operate by supplying data in the format of the specified logical dataset when the application is run.

[0109] Applications do not necessarily have to be entirely developed by human programmers. All or part of a program may be generated by other means, such as templates, or converted by a machine from another programming language or pseudo-language. Regardless of how an application is developed, specifying the data the application manipulates in terms of one or more logical datasets makes it possible to write applications without knowledge of or dependence on the physical storage of the data. Because human users can specify operations involving access to data in terms of logical datasets and / or logical entities within those datasets, this ability simplifies every part of the development process performed by human users. For example, a data analyst may be able to write an application without understanding the details of a particular physical dataset. Furthermore, avoiding dependence on physical storage in applications can extend the functionality of data processing systems. For instance, an application can be written even if the programmer is unaware of or has not yet established the details of the physical datasets that exist at the time the application is executed.

[0110] For further simplification, the data processing system may be configured to perform operations specified in terms of a logical dataset or logical entities within a logical dataset. These operations may be specified to be performed within an application, in which case they may be performed on data accessed within the physical dataset corresponding to the logical dataset.

[0111] For example, a logical entity may be associated with a company-wide list of valid values, and changes to this list can be made at the company level without requiring each application accessing that logical entity to change. As a concrete example, a logical entity for gender may be defined within a data processing system. At some point, the metadata associated with that logical entity may indicate that the acceptable values ​​are M and F. Later, the acceptable values ​​may change to M, F, and X. Each application written in terms of that logical entity can automatically adapt to the changed list, regardless of which physical dataset stores the gender information. This is advantageous because it automatically affects all applications using the gender logical entity, for example, by indicating the value "X" as a newly acceptable value in the metadata.

[0112] As another example, validation rules may be specified in terms of logical entities and may apply regardless of the physical dataset from which the data is accessed. Specifically, a data processing system may consist of data validation rules for logical elements used in email addresses. These data validation rules may be applied to data from any physical dataset storing emails when one or more fields in that dataset are identified as corresponding to logical elements used in email addresses. Validation rules can be used within an application in one or more ways. For example, a rule may be invoked from within the application against data in a particular physical dataset, or the application may access the results of applying the rule to a particular physical dataset even if the application is triggered from outside the application.

[0113] As yet another example, components that perform mask or filter operations can be specified in terms of logical entities and / or metadata about logical entities, and can operate within an application regardless of the physical data store from which the processed data is retrieved. As a specific example, a logical entity that functions as a person identifier can be assigned a privacy level. A logical entity may be defined for multiple identifiers of a person, such as an email address and a social security number. The metadata associated with these logical entities may assign a medium privacy level to the email address, while a high privacy level may be given to the social security number. A filter or mask component specified in terms of a logical entity can be configured to exclude records with specific field values ​​associated with a privacy level exceeding a threshold from its output, or to hide the values ​​of those fields. When these operations are performed on a physical dataset that has fields corresponding to email addresses or social security numbers, they may be based on the privacy level. By defining logical datasets and associated metadata, such as privacy levels, in a repository that can be used in application development, such functionality can be efficiently implemented and updated across the enterprise. Such definitions can also be used to enforce corporate policies related to data access by ensuring that physical datasets containing sensitive information (i.e., datasets containing fields with sensitive information) are handled appropriately.

[0114] Figure 3A shows an application (e.g., Application 106-3) being developed as a dataflow graph via a user interface in a development environment. Here, components are represented as nodes in the graph. The dataflow graph in this example includes input nodes 302, 304 for each physical dataset from which data is read, and output node 314 for each physical dataset to which data is written. An example of generating such input and output nodes based on the functionality they provide (e.g., data sink or data source functionality) is described in U.S. Patent No. 9,977,659, titled "Managing Data Set Objects," which is incorporated herein by reference in its entirety. The dataflow graph also includes nodes 306, 308, 310, 312 for various data processing operations (e.g., filtering, sorting, or joining operations) performed on the data read from the physical datasets. When the graph is executed by a data processing system, the results of the data processing operations are written to the physical dataset associated with output node 314.

[0115] Each input node may consist of parameter values ​​associated with its respective data source. These values ​​may indicate how to access data from the data source. Similarly, each output node may consist of parameter values ​​associated with its respective data sink. These values ​​may indicate how to write results to the data sink.

[0116] Traditionally, applications, including those written as data flow graphs as shown in Figure 3A, required manual updates to account for changes in how data is stored. For example, if a dataset is migrated from one data store to another, an experienced developer would manually modify the configuration of the input and / or output nodes in the data flow graph affected by the migration. Such manual updates must be performed by an experienced developer with knowledge of the data flow graphs and data stores supported by the data processing system (e.g., programming knowledge). In data processing systems supporting numerous datasets where changes in how data is stored occur frequently, errors during updates, or failure to update applications with each change, can propagate throughout the enterprise. For example, running a data flow graph with input nodes configured with incorrect or outdated parameter values ​​associated with a data source can result in data being read from the wrong data source or in the wrong format. Input data errors can lead to data processing operations being performed on incorrect data, resulting in inaccurate output. Incorrect output may be easily recognizable, such as a job crash or a report of missing intended information. In other scenarios, errors are more difficult to detect, and in this case, incorrect data may be written to a physical dataset and used in subsequent processing without indicating that the data has been corrupted by the error. Finding and correcting errors can be time-consuming and costly, as the erroneous data can propagate within the enterprise, and by the time it is recognized, many datasets may be corrupted. Furthermore, migrating from one data store to another is costly and time-consuming because it requires identifying all physical datasets affected by this change and then manually editing the applications that use and test them.

[0117] The inventors have developed a technique to circumvent these problems by automatically providing access to the appropriate physical dataset without requiring the application / dataflow graph to be maintained to adapt to changes in data storage. By enabling the data processing system to adapt to changes in data storage, the risk of errors that occur when modifying applications is significantly reduced, thereby eliminating the error propagation that is common in conventional systems.

[0118] Such access can be enabled by a dataset multiplexer 105 that automatically provides a connection between the application and the appropriate physical dataset. The application may be programmed in terms of one or more logical datasets. For example, a business user with minimal knowledge of physical datasets (e.g., their location or format) can write an application in terms of one or more logical datasets. The dataset multiplexer 105 can maintain a catalog of datasets, each entry in the catalog being associated with a logical dataset and providing information for accessing the corresponding physical dataset, whatever data store it is stored in at runtime. In response to indication that the execution of the dataflow graph involves operations on a logical dataset, the dataset multiplexer 105 can retrieve information for accessing the physical dataset from the catalog entry associated with the logical dataset and automatically provide a connection between the dataflow graph and the physical dataset based on that information. In some embodiments, the information for accessing the physical dataset may include a program that provides access to the physical dataset. When executed by the application, the program can access the physical dataset from the data store and convert it to the format of the logical dataset.

[0119] Figure 3B schematically illustrates how the input node 302 in Figure 3A is configured or programmed in terms of logical datasets. The input node 302 may be configured to represent a specific logical dataset specified via user input provided through a user interface. For example, user input may be provided via user interface 315. A list 370 of logical datasets available for configuring input and output nodes in a dataflow graph may be provided in user interface 315. Logical datasets available for configuring input and output nodes may be logical datasets for which entries exist in a dataset catalog. The user can browse the list to select a specific logical dataset for configuring the input node 302. The user can enter a search query via user interface element 372, where the user can specify one or more values ​​for dimensions that describe a logical dataset or logical entity. Those dimensions may include words entered into the repository to describe the logical dataset or fields contained within the logical dataset. Figure 3B depicts the user selecting the "loyalty" logical dataset 375, and the input node 302 being configured to represent this selected logical dataset.

[0120] A concurrently pending application, titled “Data Processing System with Manipulation of Logical Dataset Groups,” assigned agent reference number A1041.70070US02, describes various search interfaces that allow a user to search for datasets and / or groups of datasets as targets for operation. The interfaces and techniques described in this concurrently pending application can be used in the data processing system described herein for the purpose of constituting components of an application.

[0121] A dataset catalog may contain an entry for a selected logical dataset that provides information for accessing the corresponding physical dataset. This information may be, or may include, a program for accessing the physical dataset. If the execution of an application involves operations on the selected logical dataset, the dataset multiplexer can utilize appropriate data catalog information to provide access to the physical dataset. For example, an identifier associated with the selected logical dataset can be used to identify an appropriate entry in the dataset catalog that contains a program, which can then be executed to access the physical dataset from the datastore. The dataset multiplexer may also expose a link to a program so that access to the physical dataset is achieved by the execution of the program at that link.

[0122] Figure 3C schematically illustrates how such connections can be made using a dataset catalog. This figure schematically illustrates application 106-3 described above in relation to Figure 3B. As shown in Figure 3C, when the program is executed, the input nodes 302, 304 and output node 314 in Figure 3B are replaced with programs that provide access to the physical datasets corresponding to the logical datasets in which their components are composed. For example, the input nodes 302 and 304 are replaced with programs 330 and 340 that provide access to each physical dataset in the datastore where they are currently stored. The output node 314 may also be replaced with program 350 that indicates a program that provides access to each physical dataset to which data is written in the datastore where the data currently resides. These programs can also perform conversions between the format of the logical dataset in which the application is programmed and the format in which the physical datasets are stored in the datastore.

[0123] Representative dataset catalogs The dataset catalog 107 may contain multiple objects, each object storing information associated with a logical dataset. In this context, an object refers to a set of information stored in a computer-readable medium that captures information related to a logical dataset. This information may be stored in any suitable format. For example, it may be stored in a contiguous block of computer memory, distributed across multiple locations in computer memory, stored in a single file or other data structure, distributed across multiple data structures, or stored in a way that allows the information reflected in the object to be associated with the logical dataset.

[0124] An object may be associated with a logical dataset in any appropriate manner. An object may have a predefined format that includes information, which may be formatted as a header, identifying the logical and / or physical dataset to which the information relates. However, this information may be formatted in ways other than the header. For example, a catalog may store a list of pointers to objects, indexed by logical dataset identifiers, so that a computer accessing the catalog can find objects associated with that logical dataset as targets of pointers, by accessing pointers that have a specific logical dataset identifier as an index. Alternatively or additionally, some or all of the catalog information about a logical dataset may be stored as an addendum to a repository of information that may otherwise exist within the data processing system. For example, a data processing system may include a repository of metadata related to logical and / or physical datasets. The catalog information may be appended to this repository and / or stored in a separate metadata repository.

[0125] Information about a logical dataset can be reflected in an object in any suitable format. For example, the information may be stored as one or more descriptors, each having a value. Alternatively or additionally, the information may be stored as computer executable instructions, or may include computer executable instructions. In some embodiments, a physical dataset can be reflected in an object because a program stored with the object to access that physical dataset is hardcoded to access it. In other embodiments, information identifying the physical dataset corresponding to a logical dataset may be stored as the value of a field in the data structure that stores the object. This value may be passed as a runtime parameter to the program stored with the object to access the physical dataset, or it may be used to access the physical dataset.

[0126] Figure 4 shows an exemplary object 400 in the catalog 107 of datasets maintained by the dataset multiplexer 105. Figure 4 shows various pieces of information captured in object 400, some of which, such as discovery information 406 and / or access information 408, may be optional.

[0127] The information captured in object 400 may include information to identify the physical dataset corresponding to the logical dataset. In this example, the object is identified by the logical dataset identifier 404.

[0128] The information reflected in object 400 may be, or may include, an executable program 402 for accessing a physical dataset. When executed, the program may access a physical dataset corresponding to a logical dataset and convert the data in the physical dataset to the format of the logical dataset, or vice versa. The program may reflect in the catalog object by storing a copy of the program's computer executable instructions in computer memory allocated for the object. In other embodiments, the program may be stored elsewhere using only a pointer to the program stored in computer memory allocated for the object, or other identifiers of the program.

[0129] In some embodiments, the program may be created using discovery information 406 identified during the registration process of the physical dataset, and / or access information 408 used to access the physical dataset.

[0130] The object may reflect information about the physical data source that stores the corresponding physical dataset, enabling access to and transformation of data within the physical dataset. This information may be obtained in one of several ways, such as through user input or through an automated discovery process that reads data or metadata from the data source that stores the physical dataset. In some embodiments, the discovery information 406 may be automatically discovered as part of the registration process of the physical dataset using the dataset multiplexer 105. As part of the registration process, the user may specify that the physical dataset corresponds to a logical dataset, or the correspondence between the logical dataset and the physical dataset may be determined in another appropriate way. The automatically discovered information may include a physical identifier associated with the data store and / or the physical dataset, a reference to the storage location of the data store and / or the physical dataset, the type of data store, the record format or schema of the physical dataset, and / or other information.

[0131] In some embodiments, a copy of this discovery information may be stored in an object. In other embodiments, the discovery information 406 may be reflected in an object, as it is used to create a program for accessing a physical dataset that is stored as part of the object. For example, data store and / or physical dataset type and format information may be used to create a program that has conversion logic for converting data in a physical dataset to the format of a logical dataset.

[0132] Access information may include parameters 408 that specify how to access the physical dataset and / or datastore. In some embodiments, these parameters may be design-time parameters and / or runtime parameters. Design-time parameters may be applied to specify the functionality of program 402. Since the program is generated based on the design-time parameters, the values ​​of these parameters do not need to be stored separately in object 400. In the case of runtime parameters, their values ​​may be stored in the object and supplied as input to the program at runtime.

[0133] Parameter 408 may include one or more parameters that specify the type of access to the physical dataset. In some embodiments, the type of access may indicate read access or write access. In other embodiments, the type of access may indicate the amount of bandwidth allocated for accessing a particular logical dataset. For example, the value of parameter 408 may indicate dedicated access or shared access. The datastore as a whole may support several connections to applications 106 that can use a predetermined amount of bandwidth or less to access the datastore. Allocation techniques may be applied to enable applications performing higher-priority tasks to use more of the total available bandwidth of the data source. As a specific example, the data source may support dedicated access and shared access, with dedicated access for an application resulting in more available bandwidth being allocated to the application than if shared access were provided. By specifying dedicated access for logical datasets used by higher-priority applications and shared access for logical datasets used by lower-priority applications, the bandwidth available in the data source can be allocated as desired.

[0134] As another example, access parameters may, alternatively or additionally, indicate the type of connection used to access the datastore that holds the physical datasets corresponding to the logical datasets (such as a fast or slow connection).

[0135] As another example, parameter 408 may include one or more parameters that specify security-related information. In some embodiments, one or more parameters may indicate whether the data in the physical dataset is encrypted. In embodiments where the data is encrypted, parameter 408 may include information such as a security key for decrypting or otherwise making the information available. For enhanced security, the security key may be provided at runtime by application 106 and may not be stored in the dataset catalog 107. In other embodiments, one or more parameters may indicate whether the data in the physical dataset is compressed. In embodiments where parameter 408 is used to create program 402, the value of parameter 408 indicating that the data in the physical dataset is encrypted may be used to include decryption logic in the program.

[0136] As a further example, parameter 408 may include one or more parameters that specify the criteria for the filtering operation. For example, one or more parameters may specify dates that can be used to filter information when accessing the physical dataset.

[0137] In some embodiments, some or all of the values ​​of parameter 408 may be automatically discovered. This automatic discovery process may occur when the physical dataset is registered using a component of a data processing system that creates a dataset catalog. During the discovery process, for example, the data processing system component may access metadata in the datastore to determine the information reflected in the object. Alternatively or additionally, the data processing system component may analyze the data read from the physical dataset to recognize patterns in the data that indicate the record format, encryption, compression, or other information about the physical datastore.

[0138] However, it should be understood that discovery information 406 may be obtained by means other than direct interaction with the data source, such as reading from a repository of metadata associated with logical and / or physical datasets maintained by the data processing system. For example, security information such as encryption or compression may be applicable to all datasets in a data store. If security information for one physical dataset in a data store is stored anywhere in the system, that security information may be reflected in objects used when accessing other physical datasets in the same data store.

[0139] Some or all of the information reflected in an object may be entered by the user, even though it is shown as being discovered in the example in Figure 4. In other embodiments, some of the discovery information 406 and / or access information 408 may be specified by the user via the user interface as part of the registration process. However, it should be understood that user input may be provided in other ways, such as when defining a logical dataset. As a specific example, the priority of a logical dataset may be specified when the logical dataset is defined, or it may be specified after the logical dataset has been defined by editing the metadata stored for that logical dataset.

[0140] Furthermore, it should be understood that Figure 4 shows an object that, at a given point in time, is associated with a logical dataset and is configured to access a physical dataset. The data processing system can detect events that affect the storage of data associated with a logical dataset. In such cases, the object of that logical dataset may be updated. For example, whenever a change in a parameter is detected, the value of one of the parameters may be updated. Alternatively or additionally, if a new physical dataset is registered and the input indicates that the physical dataset stores data from a logical dataset for which an object already exists in the catalog, the object of that logical dataset may be modified. The modification may be carried out, for example, by overwriting all or part of the object with new information to reflect the new physical dataset, or by replacing it with a new object. However, the object of the logical dataset can still be accessed in the same way through the dataset catalog. Thus, if an application written to perform data access operations based on a logical dataset is configured to access the corresponding physical dataset through the dataset catalog, it will continue to correctly access the correct physical dataset regardless of any changes.

[0141] In some embodiments, program 402 may be configured as an executable dataflow graph containing logic for accessing physical datasets. As described above in relation to Figures 3A-3C, in embodiments where the application is developed as a graph, program 402 may be configured as a subgraph, meaning that it runs as part of a dataflow graph that implements the application. For example, Figure 3C shows a first program 330 configured as a subgraph containing logic for accessing input datasets, a second program 340 configured as a subgraph containing logic for accessing input datasets, and a third program 350 configured as a subgraph containing logic for accessing output datasets.

[0142] Since subgraphs are updated periodically based on events indicating changes in the appropriate mechanism for data access to the storage associated with the logical dataset, these subgraphs can be considered dynamic subgraphs (DSGs). Therefore, using data access operations on subgraphs within an application results in dynamic access to the physical dataset that holds the correct data at that time. Accordingly, a DSG is used as an example in Program 402 herein.

[0143] Representative dataset multiplexers with dataset catalogs Figure 5A is a block diagram highlighting the components of the dataset multiplexer 105 of the data processing system 104. As shown in Figure 5, the dataset multiplexer 105 includes, among its many components, a registration module 520, a dynamic subgraph (DSG) generator 524, a metadata management module 526, an operation metadata module 528, a catalog service interface 522, and a user interface 530.

[0144] In some embodiments, the registration module 520 is configured to register a physical dataset using a dataset multiplexer 105. Registration may be triggered by the addition of a physical dataset to the IT infrastructure or by the use of a physical dataset from an application. Alternatively or additionally, the registration module 520 may receive commands to register a physical dataset via a user interface 530. For example, a user may provide input via the user interface 530 to initiate the process of registering a physical dataset. This input may be in the form of a direct command to register the physical dataset.

[0145] Alternatively or additionally, the input can indirectly indicate that registration is about to begin. For example, registration may be triggered when a user writing an application selects a logical dataset associated with a physical dataset for which there is no information in the dataset catalog, or for which the catalog information is not up-to-date. Other actions that act as indirect commands may include instructions to migrate a physical dataset from one datastore to another, or commands to modify metadata associated with a logical dataset that may affect conversions between physical and logical datasets. Regardless of how the registration process is triggered, user input can specify a logical dataset corresponding to a physical dataset so that objects in the logical dataset catalog can be created or overwritten with the latest information.

[0146] Other information for creating or updating objects in the catalog may be collected from one or more sources. The registration module 520 may discover information about the physical dataset and / or the data store in which it is stored during the registration process. The information thus collected may include the type of data store, the record format or schema of the physical dataset, the physical location of the data store, the compression and / or encryption status, and / or other information.

[0147] The registration module 520 may provide the acquired information to the DSG generator 524. The DSG generator 524 may create a DSG based on the received information. The DSG generator 524 may have access to several program templates, each corresponding to a specific type of datastore. The DSG generator 524 may detect the type of datastore from the received information and select an appropriate program template from among the above several program templates that corresponds to the detected type. For example, the data processing system may be pre-configured with templates for read and / or write access to data tables in an Oracle database or a HADOOP distributed database. By detecting the type of datastore that stores the physical dataset, the DSG generator 524 may be able to select an appropriate template for accessing the physical dataset that corresponds to the logical dataset on which the DSG is being created.

[0148] The DSG generator 524 can generate a program based on a selected program template. The DSG generator 524 can detect the parameter values ​​of the selected program template from the received information and populate the program template with the detected values. Some or all of the parameter values ​​may be obtained, either or additionally, from the metadata management module 526, which in this example may maintain metadata for physical datasets, data stores, and / or logical datasets. The parameters may also be supplied, either or additionally, via user input using the user interface 530, or obtained by other means.

[0149] The DSG generator 524 generates a DSG that includes access logic for accessing the physical dataset and conversion logic for converting between the format of the physical dataset and the format of the corresponding logical dataset. The DSG generator 524 can generate a logical-to-physical layer mapping for the physical dataset and the corresponding logical dataset. The DSG generator 524 may also generate a mapping between one or more fields of the logical dataset and one or more fields of the physical dataset that represent the same information. This mapping can be generated using information from various sources, including information available within the data processing system, user input, and / or information derived by semantic discovery. The DSG generator 524 can use the mapping to generate conversion logic. For example, the customer name in the physical dataset may be stored in a single row of a data table as three fields, each holding data corresponding to the customer's first name, middle initial, and last name, respectively. However, the logical dataset may simply contain the logical entity Customer_Name. The DSG generator 524 can generate a mapping between these three fields of the physical dataset and the logical entity in the logical dataset. The conversion logic may include logic to convert between the "Customer First Name, Middle Initial, and Last Name" format of the physical dataset and the "Customer_Name" format of the logical entity. When the DSG is executed, the access logic is executed to retrieve information from the three fields of the physical dataset, and the conversion logic is executed to convert between the format of the physical dataset and the format of the logical dataset.

[0150] In some embodiments, the DSG generator 524 creates a DSG for each of several physical datasets in the datastore. The created DSGs may be included in the dataset catalog 107. The dataset catalog 107 may include objects associated with logical datasets, each of which may be a DSG for accessing the physical dataset corresponding to the logical dataset, or may include a DSG.

[0151] The registration module 520 may also provide discovery information to the metadata management module 526 so that the metadata management module 526 can receive and maintain metadata for physical datasets and / or data stores. In some embodiments, the metadata management module 526 may also be a source of information for the dynamic subgraph generator 524 when generating a DSG, and may additionally store metadata about datasets that can be used in other operations involving datasets in the data processing system. The metadata management module 526 may maintain, for example, information that functions as metadata about logical datasets, information about logical entities within logical datasets, relationships between logical entities in a dataset, and relationships with other logical datasets and / or entities in other logical datasets.

[0152] The metadata management module 526 may also store mappings between logical datasets and physical datasets, which may be based on user input or, in some embodiments, derived by monitoring operations in which the user directly or indirectly specifies associations between logical datasets and physical datasets as part of a data processing operation. Regardless of how it is obtained, in some embodiments, the metadata management module 526 may maintain tables or other data structures that map identifiers of logical datasets to identifiers of corresponding physical datasets. This information may be used by the dynamic subgraph generator 524 when creating objects representing logical datasets and / or when determining that previously created objects require updating because the storage of data associated with a logical dataset has changed.

[0153] The metadata management module 526 can maintain a list of known logical datasets in the data processing system 104. When programming an application in terms of logical datasets, the list of known logical datasets may be presented to the user through the application's user interface, and the user can select a specific logical dataset from the presented list. This logical information maintained by the metadata management module 526 can be used, for example, to enable the user to find a specific logical dataset to use when writing an application. Information about physical datasets, including their correspondence to logical datasets (which can also be stored by the metadata management module 526), ​​can also be used when searching for the appropriate dataset. For example, this logical and physical information can be used to define the dimensions of a dataset facet search.

[0154] Data processing systems may maintain other types of metadata about datasets, and users searching for datasets for specific scenarios may utilize this metadata. For example, metadata about dataset usage may be captured and stored when the dataset is used. This operational metadata may also be used by dataset search tools to enable users to search for datasets based on how other users are using them.

[0155] The operation metadata module 528 can collect operation metadata about a dataset. Operation metadata may be collected during or after the execution of an application or other program that accesses the dataset. Operation metadata collected during execution may include identification information about the accessed physical dataset, the date and time of access, whether the dataset was updated, the values ​​of parameters associated with the execution of one or more subgraphs that accessed the dataset, and / or other operation data. Operation metadata collected or determined after execution may include information about the frequency of access to the physical or logical dataset, information about the recency of the access, or information about the size of the accessed data (e.g., the number of records read and / or written). Part of the operation metadata may be social information, such as information about the user who created the dataset or the user who accessed the dataset. This social information may include the user's role in the organization, the permissions granted to the user, and / or other information about people in the organization.

[0156] In the example in Figure 5A, the catalog service interface 522 integrates access to various types of metadata about a dataset. It may provide a faceted search tool that allows searching for any of several facets that may exist in any of the logical, physical, and / or operational metadata that may be stored with respect to a physical dataset and / or logical dataset, which a user may want to select when writing an application or specifying operations performed on a dataset. The facets in the search may be based on information about logical datasets, physical datasets, and / or operational metadata stored within the data processing system. For example, a dataset search may be conditional on returning only datasets for which entries / objects exist in the dataset catalog. This facet can be combined with other facets related to logical or physical datasets to provide a powerful search interface. For example, a search query may be conditional on returning only datasets accessed within the last week, and only logical datasets that have an email field and whose corresponding physical datasets are stored in a fast-accessible data store.

[0157] Figure 5A shows separate modules managing different types of metadata, but it should be understood that this depiction is based on functional separation, and that hardware and / or software components that capture and / or provide multiple types of metadata may be divided in other ways, including integrating all such metadata capture and management into a single module, or into more modules than shown.

[0158] The catalog service interface 522 also allows the application 106 to be programmed in terms of logical datasets. When a user selects a logical dataset for programming an application, the catalog service interface 522 can provide information that enables the application, written in terms of that logical dataset, to access the appropriate physical dataset. The catalog service interface 522 may also access the catalog 107 of datasets, where each object corresponds to a logical dataset and provides information for accessing the physical dataset corresponding to the logical dataset. Catalog objects may be programs (shown as DSGs in this example) for accessing the physical dataset corresponding to the logical dataset, or may contain such programs.

[0159] The catalog service interface 522 can enable applications to access physical datasets by providing information about programs within objects of logical datasets selected in the catalog 107 of datasets. When an operation to access a logical dataset is performed from within the application, the application can use that information to access the corresponding physical dataset in the datastore. In this way, the program identified from the catalog object can be executed to access the physical dataset from the datastore. For example, the catalog service interface 522 can expose a link to a DSG, and the development environment in which the application is being developed can use that link to build the application so that access to the physical dataset is achieved by executing the DSG at the time the application is executed. In some embodiments, the catalog service interface 522 provides this link via an application programming interface (API).

[0160] As described above, the catalog object associated with a logical dataset, and therefore the DSG of that object, may be updated in response to events indicating a change in the storage of information associated with the logical dataset. For example, the physical dataset corresponding to a logical dataset may be migrated from one datastore to another. The catalog object of the logical dataset may be updated to take this change into account. In some embodiments, the program for accessing the physical dataset may be modified so that the application accesses the physical dataset from the correct datastore. By updating the catalog object of the logical dataset, the application written to access the logical dataset will continue to operate without modification even if the physical dataset is migrated from one datastore to another. Such dynamic updates will be described in more detail below with reference to Figures 6A-6B.

[0161] Other events that do not need to be associated with the location of a physical dataset may result in changes to objects in the dataset catalog. For example, the appropriate catalog object may be updated in response to an event indicating a change in the format of a physical dataset. For instance, if the format of a physical dataset is changed by adding a field to the dataset, the corresponding catalog object may be updated to take the added field into account. In some embodiments, the conversion logic in a program for accessing the physical dataset may be modified to take this change into account. As another example, in response to an event indicating a change in the value of a parameter used to generate a program or accessed in a program, the values ​​of the parameter stored in the catalog object may be updated and / or the program may be regenerated with the new values. As yet another example, an event indicating a change associated with a physical dataset corresponding to a logical dataset may include an event indicating that the physical dataset is replaced with another physical dataset corresponding to the same logical dataset. In this example, the catalog object corresponding to the first physical dataset may be replaced or substituted with the catalog object corresponding to the other physical dataset. These changes may be carried out by the dynamic subgraph generator 524, which may be triggered to update the catalog object when it detects an event. Updates may be performed, for example, by overwriting all or part of the memory location where the catalog objects are stored, or by associating objects stored in other memory locations with catalog entries in the dataset so that they are updated when the catalog object for a particular catalog entry is replaced by a new object. Such changes may be triggered by user input or automatically detected by the dynamic subgraph generator 524, the catalog service interface 522, or other components of the data processing system.

[0162] When an application written in terms of logical datasets is executed and the dataset catalog 107 is accessed to provide the application with access to the physical datasets corresponding to the logical datasets, it should be understood that one or more components such as the registration module 520, the dynamic subgraph generator 524, the metadata management module 526, the operation metadata module 528, and / or the user interface 530 may be optional, as shown in Figure 5B. When an operation to access a logical dataset is performed from within the application, the application may retrieve information about the DSG associated with the logical dataset from the data catalog 107 via the catalog service interface 522, based on the identifier associated with the logical dataset. In some embodiments, the catalog service interface 522 may provide this information to the application by exposing a link to the DSG. When the DSG is executed, it provides the application with access to the physical datasets corresponding to the logical datasets.

[0163] Typical techniques for updating dataset catalog objects Objects within a data catalog can be used by programmed applications to perform data access operations in terms of logical datasets. These catalog objects can be updated in response to events to ensure appropriate data access is provided by using the current information within the objects during application execution. One such event is a change in the storage location of physical datasets, as shown in Figures 6A and 6B. Figure 6A is a block diagram of an exemplary enterprise IT system, as shown in Figure 1A or 5A, in an operational state at a first time point, facilitating access between applications 106-1 and 106-3 and data stores 102-1 and 102-2.

[0164] Application 106-3 may be developed as a dataflow graph in a development environment that implements references to logical datasets in the application specification using information from a dataset catalog. Components 330 and 340 of Application 106-3, which represent the input nodes of the dataflow graph, may be programmed in terms of logical datasets, and the information stored in computer memory for the execution of the application includes links for those components to catalog objects corresponding to the logical datasets. For example, component 330 may be linked to a catalog object corresponding to a first logical dataset, and component 340 may be linked to a catalog object corresponding to a second logical dataset. The links may be stored in any format that conveys enough information to identify the information in the object necessary to access the physical datasets corresponding to the logical datasets referenced by those components. For example, the links may be stored as object identifiers or as paths through a directory structure to files that store programs for accessing the physical datasets.

[0165] Application 106-1 can also be developed as a dataflow graph. Components 610 and 620 of Application 106-1, which represent the input nodes of the dataflow graph, may be programmed in terms of logical datasets, and the components are linked to catalog objects corresponding to the logical datasets. For example, component 610 may be linked to a catalog object corresponding to a first logical dataset, and component 620 may be linked to a catalog object corresponding to a third logical dataset.

[0166] As shown in Figure 6A, component 330 of application 106-3 and component 610 of application 106-1 may be programmed in terms of the same logical dataset and may be linked to the same catalog object in the dataset catalog 107.

[0167] The data processing system 104 may maintain a catalog of datasets 107, which contains catalog objects corresponding to logical datasets. Each catalog object may be a DSG for accessing the physical dataset corresponding to the logical dataset, or may contain a DSG. As shown in Figure 6A, the catalog of datasets contains a first set of DSGs, each DSG in the first set is programmed to access the physical dataset from data source 102-2. The catalog of datasets 107 also contains a second set of DSGs, each DSG in the second set is programmed to access the physical dataset from data source 102-1.

[0168] The data processing system 104 enables applications 106-3 and 106-1 to access physical datasets from data stores 102-2 and 102-1 based on their respective programmed logical datasets, using information in the dataset catalog 107. When programming application 106-3, the user can select a first logical dataset from a list of known logical datasets, associate that logical dataset with component 330, and a second logical dataset associated with component 340. Similarly, when programming application 106-1, the user can select a first logical dataset to associate with component 610 and a third logical dataset to associate with component 620.

[0169] When an operation is performed to access the logical dataset associated with component 330, the data processing system 104 may select the DSG linked to component 330. When an operation is performed to access the logical dataset associated with component 340, the data processing system 104 may select the DSG linked to component 340. When an operation is performed to access the logical dataset associated with component 610, the data processing system 104 may select the DSG linked to component 610. When an operation is performed to access the logical dataset associated with component 620, the data processing system 104 may select the DSG linked to component 620.

[0170] Figure 6B is a block diagram of an exemplary data processing system as shown in Figure 1A or Figure 5A, in which the data processing system is operational in a second time period, facilitating access between applications 106-1 and 106-3 and data stores 102-1 and 102-1', when the physical dataset from data store 102-1 is migrated to data store 102-1'.

[0171] In this example, the migration of physical datasets from datastore 102-1 to datastore 102-1' is an event that causes the data processing system 104 to update the dataset catalog 107. The objects in the dataset catalog 107 corresponding to the logical datasets mapped to the physical datasets in datastore 102-1 may be updated to take into account the change in datastore. This update may modify the second set of DSGs to access the physical datasets from datastore 102-1' instead of datastore 102-1. As shown in Figure 6B, the link between applications 106-3, 106-1 and the dataset catalog 107 remains unchanged, and applications 106-3, 106-1 continue to operate regardless of the change in the physical storage of the datasets. Nevertheless, the execution of operations within the applications that specify access to logical datasets results in access to the physical datasets in the updated location.

[0172] Typical applications configured for data access via dataset catalog objects Figure 7 is a block diagram showing various pieces of information maintained by the dataset multiplexer 105. This information may enable application 106-2 to be configured to access physical datasets based on programmed logical datasets. This information can also be recorded as a result of application execution once configured. This recorded information can provide operational metadata for other functions performed by the data processing system, including providing a search interface that allows users to later retrieve datasets for use in the application based on previous operations on the datasets.

[0173] In this example, application 106-2 is written to read data from a dataset containing customer information. It then extracts records representing preferred customers from that dataset and writes the results to a second dataset. When executed, application 106-2 reads from and writes to a physical dataset. However, application 106-2 may also be programmed in terms of a first logical dataset associated with the input data store 710 and a second logical dataset associated with the output data store 720.

[0174] When application 106-2 is being written, the user can provide configuration inputs for an input data store 710 that specify a logical dataset from which data is to be read. In this example, the logical dataset is identified as "abbott.customers". This dataset can be selected by user input, such as by selecting from a list of all logical datasets registered using the data processing system, or by selecting from a limited list returned in response to a user query for a dataset with user-specified parameters. Such a selection interface may be provided by the development environment for application 106-2.

[0175] Similarly, the output datastore 720 may consist of a logical dataset. In this example, the logical dataset is identified as "abbott.preferred-cust".

[0176] To enable application execution, the development environment can associate selected logical datasets with information that allows read and write operations to be performed on the physical datasets corresponding to the specified logical datasets at the time the application is executed. This can be done, for example, by obtaining information via the catalog service interface 522 (Figure 5A). The catalog service interface 522 can, for example, in response to a request for catalog information related to a logical dataset, provide information about a program that is maintained to access the physical dataset corresponding to a particular logical dataset at that time when the program is executed. In this example, the program information is provided as a path in a directory structure to a file where the program is stored. In this example, the link to the program for accessing the physical dataset corresponding to the input logical dataset "abbott.customers" is stored at the path "common20 / abbott / customers / DSG". However, the link to the program may be supplied in any appropriate format.

[0177] Similarly, a program for accessing the physical dataset corresponding to the output logical dataset "abbott.preferred-cust" is obtained. In this example, the path is "common10 / abbott / preferred-cust / DSG". These links to the program that can access the physical dataset may be exposed by the catalog service interface 522 during application execution. These links may be stored as part of the computer executable representation of the application so that the program can be executed when an operation within the application that accesses these datasets is performed. Alternatively, sufficient information to execute the program that accesses the physical dataset can be obtained at any point before the operation that accesses the data source is performed, including during application execution.

[0178] Regardless of when information about the program providing access to the physical dataset is identified in connection with the execution of the application, the dataset multiplexer 105 can provide information about that program. Figure 7 shows that the dataset multiplexer 105 maintains enough information to associate a logical dataset with a program for accessing the physical dataset corresponding to that logical dataset. This information may be stored, for example, as a dataset catalog object for the logical dataset. In some embodiments, this information may be fetched or provided by the dataset multiplexer 105 at runtime or design / build time of the application. Doing so at design / build time avoids adding time costs and / or dependencies to runtime operation.

[0179] In the example in Figure 7, the information is stored and illustrated as two relationships. The physical identifier of the physical dataset is used as a key to link the information 702, 704, and 706. First, information 702 provides information that links each logical dataset to the identifier of the physical dataset that currently stores the data corresponding to that logical dataset, by the logical ID used for that logical dataset. Second, information 704 provides the relationship between the physical dataset and the program that may be used to access it.

[0180] In the example in Figure 7, information 702 links the logical dataset "abbott.customers" to the physical dataset identified by identifier "123". The program at path "common20 / abbott / customers / DSG" is associated with the physical dataset having identifier "123" via information 704.

[0181] Similarly, the logical dataset "abbott.preferred-cust" is associated with physical dataset ID "247" by information 702. Also, the program at path "common10 / abbott / preferred-cust / DSG" is associated with physical dataset 247 by information 704.

[0182] Similar information may be maintained by the dataset multiplexer, such as within the dataset catalog object, for each logical dataset in which a corresponding physical dataset is registered. Alternatively or additionally, some or all of this information may be maintained by the metadata management module 526 or other modules in the data processing system. Regardless of how the information is maintained, the dataset multiplexer 105 can provide information about programs that access the physical datasets corresponding to the logical datasets.

[0183] In the example in Figure 7, the identified program at the path "common20 / abbott / customers / DSG" is stored as DSG715 at the specified input datastore 710 location, along with the information used to invoke it. DSG715 is sometimes called a "read DSG" as it reads data from the physical dataset corresponding to the input logical dataset "abbott.customers". Similarly, the program at the path "common10 / abbott / preferred-cust / DSG" is stored as DSG725 at the specified output datastore 720 location, along with the information used to invoke it. DSG725 is sometimes called a "write DSG" as it writes data to the physical dataset corresponding to the output logical dataset "abbott.preferred-cust".

[0184] Information indicating the programs to be executed within an application can be stored together with the program instructions that make up the application. In a scenario where an application is written as a dataflow graph and programs for accessing data sources are written as subgraphs, these subgraphs can be dynamically linked to the dataflow graph at appropriate locations within the dataflow graph for execution. These locations may correspond to input and / or output nodes of the dataflow graph. During or immediately before execution of the dataflow graph, link or path information of subgraphs exposed by or obtained from the catalog service interface 522 may be provided to the input and / or output nodes, and the corresponding subgraphs may be linked and / or stored in place of the input and / or output nodes. Exemplary techniques for dynamically linking subgraphs to a dataflow graph via a subgraph interface may be used, as described in U.S. Patent No. 10,180,821, entitled "Managing Interfaces for Sub-Graphs," which is incorporated entirely herein. However, other methods for storing information for executing programs may be used alternatively or additionally.

[0185] When application 106-2 is executed and an operation occurs to access the logical dataset associated with the input data store 710, the linked DSG 715 may be invoked. Invocation of DSG 715 may result in the execution of its access and conversion logic. When executed, the input data store 710 is accessed, data from the input data store and / or the corresponding physical dataset of the input data store may be read and converted to the format of the logical dataset. Invocation of the DSG may require providing parameters to a controller module (not shown) in the data processing system.

[0186] In the example in Figure 7, the parameters supplied for the execution of DSG 715 are shown as parameter 730. In this example, one of the parameters 730 identifies the DSG, for example, by providing its path. The value of this parameter may be stored when the input data source 710 is configured for a specific logical dataset.

[0187] Other parameters 730 may be provided so that they can be supplied to the DSG 715 by the controller module for execution. These runtime parameters (i.e., supplied at runtime) may affect the execution of the DSG. For example, the values ​​of parameters "Param1" and "Param2" may be supplied to the DSG at runtime. The value of one such parameter may, for example, specify that the DSG 715 should be executed in a particular read mode (single record, batch, quick, shared, etc.). The parameter values ​​may, as another example, reflect the access priority of an application.

[0188] The values ​​of these runtime parameters can be obtained in one or more ways. They may be encoded in application 106-2 based on user-provided input during application development, for example. For example, the parameter values ​​may be derived from information entered as configuration parameters of input data source 710 in the development environment. Alternatively, or additionally, the parameter values ​​may be derived from other user inputs during application development or in response to runtime prompts. As yet another example, the application may identify parameter values ​​at runtime from a variety of inputs, such as an external input indicating the time, the current system load, or other inputs that depend on data provided as inputs to a data flow graph.

[0189] As yet another example, the values ​​of the parameters may be obtained from other modules, either alternatively or additionally. Specifically, at least some values ​​of parameter 730 may be read from information in a metadata repository that stores information about logical datasets associated with the input data store 710, or obtained by processing the information. As yet another example, at least some values ​​of parameter 730 may be read from information in an access control module that maintains information about users, or obtained by processing the information, which may reflect access priorities or mechanisms to the data store that are set based on the role of the user who developed the application or the user running the application.

[0190] The values ​​of other parameters in the input data source parameter 730 may be included so that the controller module or other components of the data processing system can capture operational metadata. For example, the logical identifier of the dataset to which access is encoded may be stored for this reason. Similarly, the identifier of the physical dataset to be accessed may be stored. The value of this parameter may be supplied by the dataset multiplexer from information 702, etc., which is up-to-date at runtime. By capturing such information, for example, the operational metadata module 528 (Figure 5A) may be able to supply information that supports additional facets of data retrieval.

[0191] In the example in Figure 7, the dataset multiplexer 105 is shown to store information 706 collected during the execution of application 106-2. For example, information 706 may include the date the dataset was accessed, the size of the dataset at the time it was accessed, and / or the amount of data read into or written from the dataset, the host ID of the computer hardware involved in the data access, such as the execution of the application or access program or the physical storage of the data. Other parts of information 706 may indicate the logical dataset associated with the output data store 720, the accessed physical dataset, the values ​​of parameters such as "Param1" and "Param2" supplied to the program, when the physical dataset was accessed, and / or other information. Such entries may be stored for each access to the dataset, or for a number of previous accesses to the dataset, or for a predetermined period of time after accessing the dataset. This information can be analyzed after execution to determine other operational parameters, such as the frequency or relevance / freshness of the dataset's use.

[0192] Similar information may be stored for the output data store 720. When an operation is performed to access the logical dataset associated with the output data store 720, the linked DSG 725 may be invoked. Invocation of DSG 725 may result in the execution of its access and conversion logic. Once executed, the output data store 720 may be accessed, and the data may be converted from the format of the logical dataset to the format of the output data store and / or the format of the corresponding physical dataset in the output data store, and then written to the output data store. Parameter 740 represents a parameter whose value is supplied to the controller module and which may be used by DSG 725 during execution. Although not shown in Figure 7, entries in the operation metadata repository may also be created based on access to the physical dataset corresponding to the output data store 720.

[0193] Typical methods for registering datasets in a dataset catalog Figure 8 is a flowchart of an exemplary process 800 for registering a physical dataset in a dataset catalog so that the physical dataset can be accessed by an application configured to access a logical dataset corresponding to the physical dataset. Process 800 may be performed by a data processing system 104, such as in a dataset multiplexer 105 as described with reference to Figures 1A-1C. Process 800 may also include other acts, including acts as described elsewhere in this specification in relation to other embodiments.

[0194] Process 800 may be initiated in response to a detected event (801). The event may be an indication that there is no catalog entry in the dataset catalog that provides a mechanism for accessing physical datasets in the IT system that correspond to logical datasets defined in the data processing system. The detected event may be the automatic detection of physical datasets that exist in the IT system but do not yet have a catalog entry. Such indications may be in the form of user input, such as a user input command to register the physical dataset in the data processing system as corresponding to a logical dataset. Alternatively, the event may be an indication that the catalog entry in the dataset catalog that provides a mechanism for accessing physical datasets in the IT system is outdated. However, other events (including other events described herein) may trigger the execution of process 800. For example, a new physical dataset may be identified in the data store as part of a periodic (weekly, bi-weekly, etc.) import feed execution. This identification may trigger the execution of process 800.

[0195] Process 800 may proceed to Act 802, in which information about a physical dataset stored in a data store is retrieved. The physical dataset may be the physical dataset mentioned in the context of the above-mentioned initiation 801 of Process 800. In some embodiments, some of the information, such as a physical identifier associated with the data store and / or the physical dataset, a reference to the storage location of the data store and / or the physical dataset, the type of data store, the record format or schema of the data store and / or the physical dataset, and / or other information (such as the information described in the context of Figure 4), may be discovered automatically.

[0196] Act 804 can generate logical-to-physical layer mappings for physical datasets and their corresponding logical datasets. In some embodiments, the dataset multiplexer 105 can generate mappings between one or more fields in a logical dataset and one or more fields in a physical dataset that represent the same information. These mappings can be generated using information from various sources, including information available within the data processing system, user input, and / or information derived by semantic discovery. For example, a field in a physical dataset where most entries contain the characters "@" and "." might relate to a field in a logical dataset called "email." This relationship can be derived by semantic discovery and used to generate the mapping. Similar relationships between fields may be specified by user input or by other means. Mappings between logical and physical datasets can be generated by applying these relationships. In some embodiments, information about unique keys and / or foreign keys specifying relationships between datasets can be used to generate the mapping.

[0197] These relationships allow a program accessing a physical dataset to be configured to perform the necessary mappings between fields in the physical dataset and fields in the logical dataset. A template for the program is selected and then configured to implement the mappings so that both access and conversion of data formats are provided. In order to obtain a template, Act 806 may determine the type of datastore based on the information obtained in Act 802. In Act 808, a decision may be made as to whether a program template is available for the type of datastore. Many datastores can have a consistent access paradigm, which can be incorporated into templates. Thus, a data processing system may store a library of templates for widely used types of data processing systems, such as Oracle databases or SQL Server databases.

[0198] In response to the decision that a program template is available, the process proceeds to Act 810, where an available program template is selected, and then to Act 812, where a program is generated based on the selected program template. Program generation can enable both access to the target physical dataset and the application of the mapping generated in Act 804 to perform conversions between the data formats of the logical dataset and the physical dataset.

[0199] In Act 812, a program is generated for accessing a physical dataset from a data store. The program may be generated by populating a program template selected in Act 812a based on one or more first parameters, and by retrieving information about one or more other parameters in Act 812b.

[0200] In Act 812a, the selected program template may be populated by identifying the value of the first parameter of the program template based on information obtained in Act 802 (such as information automatically discovered during the registration process).

[0201] In Act 812b, information regarding one or more other parameters of the program template may be obtained. One or more other parameters may specify how to access the physical dataset. For example, some information may be obtained from a metadata repository that maintains the metadata of the data store. As another example, some information may be obtained through user input. For example, the user may specify information regarding the type of access or security-related information. User input regarding other parameters may be obtained during the registration process.

[0202] In some embodiments, in response to a decision in Act 808 that a program template is not available, the process proceeds to Act 820, where a program structure used to generate the program is created. In some embodiments, the program structure may be created by prompting the user for input. For example, the user may provide a file containing the program structure and / or parameter values. Then, in Act 822, a program for accessing a physical dataset from a datastore may be generated based on the program structure entered by the user.

[0203] Acts 802, 804, 806, 808, 810, 812, 820, and 822 may be performed without departing the scope of this disclosure to generate a program for accessing different physical datasets in a datastore, or to generate a program for accessing physical datasets in different datastores. For example, a first program may be generated for accessing a first physical dataset in a datastore, and a second program may be generated for accessing a second physical dataset in a datastore. As another example, the first program may be generated for accessing a first physical dataset in a first datastore, and the second program may be generated for accessing a second physical dataset in a second datastore different from the first datastore.

[0204] When a program is generated, information for invoking the execution of the program from within a programmed application in terms of logical datasets is stored in an object in the dataset catalog 107. The stored information may include the datastore or the physical identifier of the physical dataset stored in the datastore, the logical identifier of the logical dataset, the values ​​of parameters used when the program is executed, and / or other information. In some embodiments, the object may be a program or may contain a program.

[0205] In this way, the program generated in Act 812 or 822 is available to applications that specify access to the logical dataset corresponding to the physical dataset. Therefore, in Act 814, which may be performed at any time after registration (or may not be performed at all), the program generated in Act 812 or 822 is linked to one or more applications. This link allows applications programmed in terms of the logical dataset to access the physical dataset using the generated program. When an operation to access the logical dataset is performed, the linked program is executed, thereby providing access to the physical dataset corresponding to the logical dataset.

[0206] Regardless of whether the generated program is linked to an application that accesses the logical dataset, Act 816 makes a decision on whether an event indicating a change to the storage of data corresponding to the logical dataset has been detected. For example, such a change may indicate a migration from a first data store to a second data store, a change in the format of the logical dataset, or a change in the format of the physical dataset. In response to the detection of such an event, the process loops back to Act 802, where the process may be repeated. By repeating the process, a new program for accessing the physical dataset corresponding to the logical dataset may be generated, or an existing program for accessing the physical dataset corresponding to the logical dataset may be updated. However, the link to that program may remain the same so that the application configured to use that link to access the data corresponding to the logical dataset continues to operate with the correct data.

[0207] In some embodiments, in response to the decision in act 816 that no change events were detected, process 800 continues to monitor for change events so that the program for accessing the physical dataset corresponding to the logical dataset from which the access information was generated continues to operate as intended.

[0208] Further details on implementation forms Figure 9 shows an example of a suitable computing system environment 900 in which the technologies described herein may be implemented. The computing system environment 900 is merely an example of a suitable computing environment and is not intended to imply any limitations on the scope or functionality of the technologies described herein. The computing environment 900 should not be construed as having any dependencies or requirements related to any one or combination of the components exemplified in the exemplary operating environment 900.

[0209] The technologies described herein can be used in conjunction with a number of other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the technologies described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network PCs, minicomputers, mainframe computers, and distributed computing environments encompassing any of the above systems or devices.

[0210] A computing environment can execute computer executable instructions, such as program modules. Generally, program modules include routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. The techniques described herein may be executed in a distributed computing environment in which tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may reside in both local and remote computer storage media, including memory storage devices.

[0211] Referring to Figure 9, an exemplary system implementing the techniques described herein encompasses a general-purpose computing device in the form of a computer 900. The components of computer 910 may include, but are not limited to, a processing unit 920, system memory 930, and a system bus 921 that connects various system components, including system memory, to the processing unit 920. The system bus 921 may be any of several types of bus structures that include a memory bus or memory controller, peripheral bus, and local bus, using any of various bus architectures. For example, and not limited to, such architectures include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Extended ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, also known as a mezzanine bus.

[0212] Computer 910 generally encompasses a variety of computer-readable media. Computer-readable media may be any available media accessible by computer 910 and include both volatile and non-volatile media, removable and non-removable media. For example, and not limited to, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store desired information and can be accessed by computer 910. Communication media generally encompass all information distribution media that embody computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transport mechanisms. The term “modulated data signal” means a signal that has one or more of its characteristic sets, or that has been modified to encode information in a signal. By example, and not limited to, communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the above shall also be included within the scope of computer-readable media.

[0213] The system memory 930 includes computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 931 and random access memory (RAM) 932. The basic input / output system 933 (BIOS), which contains basic routines that help transfer information between elements within the computer 910, such as during startup, is generally stored in the ROM 931. The RAM 932 generally contains readily available data and / or program modules currently being manipulated by the processing unit 920. As an example, and not limited to, Figure 9 illustrates an operating system 934, an application program 935, other program modules 936, and program data 937.

[0214] Computer 910 may also include other removable / non-removable, volatile / non-volatile computer storage media. As a mere example, Figure 9 illustrates a hard disk drive 941 for reading from or writing to a non-removable, non-volatile magnetic medium, a flash drive 951 for reading from or writing to removable, non-volatile memory 952 such as flash memory, and an optical disk drive 955 for reading from or writing to removable, non-volatile optical disk 956 such as a CD-ROM or other optical medium. Other removable / non-removable, volatile / non-volatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital multipurpose disks, digital videotapes, solid RAM, solid ROM, etc. The hard disk drive 941 is generally connected to the system bus 921 through a non-removable memory interface such as interface 940, and the magnetic disk drive 951 and optical disk drive 955 are generally connected to the system bus 921 through a removable memory interface such as interface 950.

[0215] The drives described above and illustrated in Figure 9, and the computer storage media associated with them, provide storage for computer-readable instructions, data structures, program modules, and other data of the computer 910. In Figure 9, for example, the hard disk drive 941 is illustrated as storing the operating system 944, application programs 945, other program modules 946, and program data 947. Note that these components may be the same as or different from the operating system 934, application programs 935, other program modules 936, and program data 937. The operating system 944, application programs 945, other program modules 946, and program data 947 are given different numbers here to illustrate that they are at least different copies. An actor can input commands and information into the computer 910 using input devices such as the keyboard 962 and a pointing device 961, commonly called a mouse, trackball, or touchpad. Other input devices (not shown) may include microphones, joysticks, gamepads, satellite dishes, scanners, etc. These and other input devices are often connected to the processing unit 920 by a user input interface 960 coupled to the system bus, but may also be connected by other interfaces and bus structures such as a parallel port, game port, or Universal Serial Bus (USB). A monitor 991 or other type of display device is also connected to the system bus 921 via an interface such as a video interface 990. In addition to the monitor, the computer may also include other peripheral output devices such as a speaker 997 and a printer 996, which can be connected through an output peripheral interface 995.

[0216] Computer 910 can operate in a networked environment using logical connections to one or more remote computers, such as remote computers 980. The remote computers 980 may be personal computers, servers, routers, network PCs, peer devices, or other common network nodes, and generally, although only the memory storage device 981 is illustrated in Figure 9, it encompasses many or all of the elements described above in relation to computer 910. The logical connections depicted in Figure 9 include a local area network (LAN) 971 and a wide area network (WAN) 973, but may also include other networks. Such networking environments are common in offices, enterprise-scale computer networks, intranets, and the internet.

[0217] When used in a LAN networking environment, computer 910 is connected to LAN 971 via a network interface or adapter 970. When used in a WAN networking environment, computer 910 generally includes a modem 972 or other means for establishing communication over WAN 973, such as the Internet. The modem 972, which may be internal or external, may be connected to the system bus 921 via an actor input interface 960 or other appropriate mechanism. In a networked environment, program modules, or parts thereof, written in relation to computer 910 may be stored in a remote memory storage device. As an example, and not limited to, Figure 9 illustrates a remote application program 985 residing in memory device 981. The network connections shown are illustrative, and it will be understood that other means may be used to establish communication links between computers.

[0218] The techniques described herein are not limited to any particular implementation and can be implemented in any of many ways. Examples of specific implementations are provided herein for illustrative purposes only. Furthermore, the aspects of the techniques described herein are not limited to the use of any particular technique or combination of techniques; therefore, the techniques disclosed herein may be used individually or in any suitable combination.

[0219] As described herein, several aspects of the technology have been explained, but it should be understood that various changes, modifications, and improvements are possible.

[0220] For example, consider a scenario where a user writes an application that specifies access to logical data. In some embodiments, the user may be a human user. In other embodiments, the user may be a program with artificial intelligence (AI). The AI ​​may, for example, derive a data processing algorithm by processing a dataset, which can then be applied to other datasets.

[0221] As another example, information 702, 704, and 706 are depicted as being maintained in separate tables. However, the information may be maintained in a single table, or it may be combined in any appropriate way within any data structure.

[0222] Such changes, modifications, and improvements are intended to be part of this disclosure and to be within the spirit and scope of this disclosure. Furthermore, while the advantages of the technology described herein are shown, it should be understood that not all embodiments of the technology described herein encompass all described advantages. Some embodiments may not implement any of the features described herein as advantageous, and in some cases, one or more of the described features may be implemented to obtain further embodiments. Accordingly, the above description and drawings are merely examples.

[0223] The above embodiments of the technology described herein may be implemented in any of many ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. If implemented in software, the software code can run on any suitable processor or group of processors, whether provided on a single computer or distributed across multiple computers. Such processors may be implemented as integrated circuits and have one or more processors in an integrated circuit component that includes commercially available integrated circuit components known in the industry by names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processor may be implemented in custom circuitry such as ASICs, or in semi-custom circuitry resulting from the configuration of programmable logic devices. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercial, semi-custom, or custom. As one specific example, some commercial microprocessors have multiple cores such that one or a subset of multiple cores can constitute a processor. However, the processor can be implemented using circuitry of any suitable format.

[0224] Furthermore, it is understood that a computer may be embodied in any of many forms, such as a rack-mount computer, a desktop computer, a laptop computer, or a tablet computer. In addition, a computer may be incorporated into a device that is not generally considered a computer but has appropriate processing power, including a personal digital assistant (PDA), a smartphone, or any other appropriate portable or fixed electronic device.

[0225] Furthermore, a computer may have one or more input and output devices. These devices can be used, in particular, to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for a visual representation of the output, and a speaker or other sound-generating device for an audible representation of the output. Examples of input devices that can be used for the user interface include a keyboard, as well as pointing devices such as a mouse, touchpad, and digitizer tablet. As another example, a computer may receive input information by speech recognition or in other audible formats.

[0226] Such computers can be interconnected by one or more networks of any appropriate form, encompassing a corporate network or a local area network or wide area network such as the Internet. Such networks may be based on any appropriate technology, may operate according to any appropriate protocol, and may encompass wireless networks, wired networks, or fiber optic networks.

[0227] Furthermore, the various methods or processes outlined herein may be encoded as software executable for one or more processors using any one of various operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code to be executed against a framework or virtual machine.

[0228] In this regard, embodiments of the technology described herein may be embodied as computer-readable storage media (or more computer-readable media) (e.g., computer memory, one or more floppy disks, compact discs (CDs), optical discs, digital video discs (DVDs), magnetic tape, flash memory, field-programmable gate arrays or circuit configurations in other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform the methods for carrying out the various embodiments described above. As is evident from the above examples, computer-readable storage media can retain information for a time sufficient to provide computer-executable instructions in a non-temporary form. Such one or more computer-readable storage media may be portable so that one or more programs stored therein can be loaded onto one or more different computers or other processors to carry out the various embodiments of the technology described above. In this specification, the term “computer-readable storage media” encompasses only non-temporary computer-readable media that can be considered as products (i.e., manufactured goods) or machines. Alternatively or additionally, embodiments of the technology described herein may be embodied in computer-readable media other than computer-readable storage media, such as propagating signals.

[0229] The terms “program” or “software” are used herein in a general sense to refer to any type of computer code or set of computer executable instructions or set of processor executable instructions that can be used to program a computer or other processor to implement various aspects of the techniques described herein. In addition, according to certain aspects of these embodiments, it will be understood that, when executed, one or more computer programs that perform the methods of the techniques described herein do not need to reside on a single computer or processor, but may be distributed modularly across a number of different computers or processors to implement various aspects of the techniques described herein.

[0230] Computer executable instructions can take many forms, such as program modules, which are executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Generally, the functionality of program modules may be combined or distributed as desired in various embodiments.

[0231] Furthermore, the data structure may be stored in a computer-readable medium in any appropriate form. For the sake of simplicity in illustration, the data structure may be shown having fields that are related by location within the data structure. Such relationships can also be achieved by assigning locations in the computer-readable medium that convey the relationships between fields to the storage of the fields. However, relationships between information in the fields of the data structure may be established using any appropriate mechanism, including the use of pointers, tags, or other mechanisms for establishing relationships between data elements.

[0232] Various aspects of the technology described herein may be used individually, in combination, or in various arrangements not specifically described in the embodiments described above, and therefore, in their application, are not limited to the details and arrangements of components described above or illustrated in the drawings. For example, an aspect described in one embodiment may be combined in any way with an aspect described in another embodiment.

[0233] Furthermore, the techniques described herein may be embodied as the methods exemplified herein, including by reference to Figure 8. Acts performed as part of any method may be ordered in any appropriate manner. Thus, embodiments may be constructed in which the acts are performed in an order different from that illustrated (this may include performing several acts simultaneously, even if they are shown as sequential acts in the embodiments for explanation).

[0234] Furthermore, certain actions are described as being performed by an "actor" or "user." It is understood that an "actor" or "user" does not necessarily have to be a single individual, and in some embodiments, actions attributed to an "actor" or "user" may be performed by a team of multiple individuals and / or by an individual in combination with computer-aided tools or other mechanisms.

[0235] The use of ordinal terms such as “first,” “second,” and “third” in a claim to modify a claim element does not in itself imply a temporal order in which one claim element takes precedence, precedence, or sequence or method of acting over another claim element, and these claim elements are distinguished by being used merely as labels to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms).

[0236] Furthermore, the expressions and terms used herein are for illustrative purposes only and are not intended to be limiting. The use herein of “including,” “comprising,” “having,” “containing,” “involving,” and their variations means that they encompass the items and their equivalents listed thereafter, as well as any additional items. [Explanation of symbols]

[0237] 100 IT Systems 102 Federated Data Environment 102-1, 102-2, 102-3, 102-n Datastore 104 Data Processing System 104-1, 104-2, 104-n Instances of the data processing system 105 Dataset Multiplexer 106, 106-1, 106-2, 106-3, 106-n applications 107 Dataset Catalog 302,304 Input data source, input data store 306,308 filters 310 Sort 312 Combine 314 Output data source, output data store 315 User Interface 330,340 Programs for accessing input data sources 350 Programs for accessing output data sources List of 370 logical datasets 372 User Interface Elements 375 "loyalty" logical data list 500 IT Systems 520 Registration Modules 522 Catalog Service Interface 524 Dynamic Subgraph Generator 526 Metadata Management Module 528 Operation Metadata Module 530 User Interface 610,620 components Information 702, 704, 706 710 Input Data Store 715 Dynamic Subgraph (DSG) 720 Output Datastore 725 DSG 730 Input data source parameters 740 Output data source parameters 900 Computing System Environment 910 Computer 920 Processing Unit 921 System Bus 930 System Memory 931 Read-only memory (ROM) 932 Random Access Memory (RAM) 933 Basic Input / Output System (BIOS) 934 Operating Systems 935 Application Programs 936 Other program modules 937 Program Data 940 Non-removable non-volatile memory interface 941 Hard Disk Drive 944 Operating Systems 945 Application Programs 946 Other program modules 947 Program Data 950 Removable Non-Volatile Memory Interface 951 Magnetic Disk Drive 952 Removable Non-Volatile Memory 955 Optical Disc Drive 956 Removable Non-Volatile Optical Disc 960 User Input Interface 961 Pointing device 962 keyboard 970 Network Interface or Adapter 971 Local Area Network (LAN) 972 Modem 973 Wide Area Network (WAN) 980 Remote Computer 981 Memory Storage Device 985 Remote Application Program 990 Video Interface 991 Monitor 995 Output Peripheral Interface 996 Printer 997 Speaker

Claims

1. A method performed by a data processing system to enable efficient data analysis in a dynamic environment having multiple datasets by generating and / or using entries in a dataset catalog to enable access to physical datasets in a data store, wherein the data processing system is configured to run a data processing application programmed to access logical datasets, each logical dataset includes a schema for data independent of the format of the corresponding data in the physical datasets, and the data processing system includes a dataset multiplexer that can be configured to provide the application with access to the physical datasets in the data store, and the method is Creating multiple entries in the dataset catalog, each of which is associated with a logical dataset and a physical dataset, and also associated with a computer executable instruction for accessing the physical dataset; Receiving input that at least partially identifies a first logical dataset to be accessed in order to perform operations within a data processing application that specify access to the dataset, When the operation is performed within the data processing application, the computer executable instruction is invoked to access the physical dataset associated with the entry in the dataset catalog associated with the first logical dataset, Dynamically updating entries in the dataset catalog in response to events indicating changes to the physical dataset associated with the logical dataset, A method including, Dynamically updating entries in the aforementioned dataset catalog is possible. To detect events indicating changes associated with the physical dataset corresponding to the first logical dataset, This includes modifying the entry in the dataset catalog associated with the first logical dataset based on the detection of the event, Modifying the entry in the aforementioned dataset catalog A method comprising modifying the computer executable instructions for accessing the physical dataset corresponding to the first logical dataset.

2. Creating multiple entries in the aforementioned dataset catalog Receiving information relating to a first physical dataset among the physical datasets stored in a first data store among the data stores, wherein the first physical dataset corresponds to a first logical dataset. Based on the information relating to the first physical dataset, a first program is generated which includes the computer executable instructions for accessing the first physical dataset from the first data store. In the first entry of the dataset catalog, a link to the first program is stored to enable the data processing application to access the first physical dataset using the first program. The method according to claim 1, including the method described in claim 1.

3. To generate the first program for accessing the first physical dataset from the first data store, Identifying the type of the first data store from the received information, Selecting a first program template for the type of the first data store, The first program is generated by inputting one or more values ​​of one or more parameters of the first program template into the first program template, The method according to claim 2, including the method described in claim 2.

4. Receiving an input that at least partially identifies the first logical dataset, The method according to any one of claims 1 to 3, comprising providing a user interface in which a user identifies the first logical dataset, at least partially.

5. Calling the aforementioned computer executable instruction, In the dataset catalog, access to the entry associated with the first logical dataset is enabled, Based on the information in the entry, access to the data store that stores the physical dataset corresponding to the first logical dataset is enabled, The method according to any one of claims 1 to 4, including

6. A method performed by a data processing system to enable efficient data analysis in a dynamic environment having multiple datasets by registering datasets in a dataset catalog to facilitate access to multiple physical datasets in a data store, wherein the data processing system is operable with the multiple physical datasets stored in the data store, and includes a dataset multiplexer that can be configured to provide an application with access to one of the multiple physical datasets, wherein the physical datasets are stored in one of the data stores, and the physical datasets correspond to logical datasets that include schemas for data independent of the format of the corresponding data in the physical datasets, and the method is Receiving information relating to a first physical dataset among a plurality of physical datasets stored in a first data store among a plurality of data stores, wherein the first physical dataset corresponds to a first logical dataset. Based on the information relating to the first physical dataset, a first program is generated which includes computer executable instructions for accessing the first physical dataset from the first data store. To enable the application to access the first physical dataset using the first program, a link to the first program is stored in the first object within the object library, A method including, The method described above is Based on the detection of an event indicating a change associated with the first physical dataset, a decision is made as to whether or not to modify the first program for accessing the first physical dataset. Based on the decision to modify the first program, To generate the corrected first program, The target of the link is to replace the first program with the modified first program, Methods that further include the above.

7. The method according to claim 6, wherein generating the modified first program includes generating the modified first program without modifying the application or the first logical dataset.

8. The method according to claim 6 or 7, wherein the information relating to the first physical dataset includes information relating to the type of the first data store.

9. The method according to any one of claims 6 to 8, wherein the dataset multiplexer includes a library of objects for storing information for accessing the plurality of physical datasets, and the first object in the library of objects includes an identifier for the first physical dataset.

10. The method according to claim 9, wherein the dataset multiplexer further includes an API, and the method further includes providing the application with access to the first object via the API.

11. The method described above is The method of claim 9, further comprising assigning identifiers to objects in the library based on the schema and logical name of each logical dataset in which information is stored.

12. The method described above is Receiving a command to register the first physical dataset in the dataset catalog, Based on the received command, the first object is created and the first object is saved to the library. The method according to claim 9, further comprising:

13. The method according to claim 9, wherein the identifier of the first physical dataset is a physical identifier.

14. The method according to claim 13, wherein the first object further includes a second identifier, the second identifier being a logical identifier of a logical dataset associated with the first object.

15. The method described above is The method according to claim 14, further comprising, in response to detecting an event indicating that the first physical dataset has changed from being stored in the first data store to being stored in the second data store, modifying the physical identifier in the first object without modifying the logical identifier.

16. The first object includes the values ​​of parameters accessed during the execution of the first program. The method described above is The method according to claim 9, further comprising correcting the value of the parameter stored in the first object based on the detection of an event indicating a change in the value of the parameter accessed in the first program.

17. The method according to any one of claims 6 to 16, wherein the first program includes access logic and conversion logic, and when the application is executed, the access logic and conversion logic of the first program are executed to provide access to the first physical dataset and to perform conversion between the format used in the first physical dataset and the format used in the first logical dataset.

18. The method according to any one of claims 6 to 17, wherein the first program includes one or more parameters such that the values ​​of one or more parameters affecting the operation of the first program affect access to the first physical dataset through the first program.

19. The method according to claim 18, wherein the application is configured to supply values ​​for one or more parameters used when calling the first program.

20. The method described above is To detect the type of the first data store, Based on the detected type, select a template from multiple templates, The method according to any one of claims 6 to 19, further comprising generating the first program by...

21. The method according to claim 20, wherein the first program includes a first portion configured for read access to the first data store and a second portion for write access to the first data store.

22. The method according to any one of claims 6 to 21, wherein the first program is configured as an executable dataflow graph including logic for accessing the first physical dataset.

23. A method performed by a data processing system to enable efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores, wherein the data processing system is operable with the application and the multiple physical datasets stored in the multiple data stores, and the application is programmed to access a logical dataset which includes a schema for data independent of the format of the corresponding data in the physical dataset, and the method is To provide a user interface that at least partially identifies the logical dataset that the user accesses in the application, When the aforementioned application is executed and an operation is performed that involves accessing the identified logical dataset, To enable access to objects in the object library associated with the aforementioned logical dataset, Based on the information in the object, access to the data store that stores the physical dataset corresponding to the identified logical dataset is enabled. A method including, The above method further, The further includes updating the information in the object based on the events associated with the storage of the data corresponding to the identified logical dataset, A method wherein the information in the object includes an executable program for accessing the physical dataset.

24. The method according to claim 23, wherein the executable program for accessing the physical dataset encodes logic for converting data between a format used in the physical dataset and a format used in the logical dataset.

25. The method according to claim 23 or 24, wherein the object is an executable program for accessing the physical dataset.

26. The method according to any one of claims 23 to 25, wherein the information in the object includes the type of the data store.

27. The method according to any one of claims 23 to 26, wherein the information in the object includes a record format or schema associated with the physical dataset.

28. A method performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores, wherein the data processing system is operable with the application and the multiple physical datasets stored in the multiple data stores, and the application is programmed to access a logical dataset which includes a schema for data independent of the format of the corresponding data in the physical datasets, and the method is To provide a user interface that at least partially identifies the logical dataset that the user accesses in the application, When the aforementioned application is executed and an operation is performed that involves accessing the identified logical dataset, To enable access to objects in the object library associated with the aforementioned logical dataset, Based on the information in the object, access to the data store that stores the physical dataset corresponding to the identified logical dataset is enabled. A method including, A method wherein the information in the object includes one or more parameters that specify the method for accessing the physical dataset, and the one or more parameters include at least one parameter indicating whether the data in the physical dataset is compressed.

29. The method according to any one of claims 23 to 28, wherein the information in the object includes one or more parameters that specify the method for accessing the physical dataset, and the one or more parameters include at least one parameter indicating the type of access.

30. The method according to claim 29, wherein the type of access includes an indication of read access or write access.

31. The method according to claim 29, wherein the type of access includes indicating access via a high-speed connection or a low-speed connection.

32. The data processing system includes a repository of metadata related to a logical dataset, The method according to any one of claims 23 to 31, wherein providing the user interface includes presenting a menu of logical datasets based on metadata in the repository.

33. A method performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores, wherein the data processing system is operable with the application and the multiple physical datasets stored in the multiple data stores, and the application is programmed to access a logical dataset which includes a schema for data independent of the format of the corresponding data in the physical datasets, and the method is To provide a user interface that at least partially identifies the logical dataset that the user accesses in the application, When the aforementioned application is executed and an operation is performed that involves accessing the identified logical dataset, To enable access to objects in the object library associated with the aforementioned logical dataset, Based on the information in the object, access to the data store that stores the physical dataset corresponding to the identified logical dataset is enabled. A method including, A method wherein the information in the object includes one or more parameters that specify the method for accessing the physical dataset, the one or more parameters including at least one parameter that indicates whether the data in the physical dataset is encrypted or specifies a criterion for filtering the data.

34. To enable access to the data store that stores the aforementioned physical dataset, The method according to claim 23, comprising using the executable program to enable access to the data store that stores the physical datasets corresponding to the identified logical datasets.

35. A method performed by a data processing system to enable efficient data analysis in a dynamic environment having multiple datasets by generating entries in a dataset catalog to enable access to physical datasets in a data store, wherein the data processing system is configured to run a data processing application programmed to access logical datasets, each logical dataset includes a schema for data independent of the format of the corresponding data in the physical datasets, and the data processing system includes a dataset multiplexer that can be configured to provide the application with access to the physical datasets in the data store, and the method is Receiving information relating to a first physical dataset stored in the first data store of the data store, wherein the application is programmed to access the first logical dataset, and the first physical dataset corresponds to the first logical dataset. Based on the received information, a first program is generated for accessing the first physical dataset from the first data store, wherein generating the first program is Identifying the type of the first data store from the received information, Selecting a first program template for the type of the first data store, The first program is generated by inputting one or more values ​​of one or more parameters of the first program template into the first program template, Including generating, The information for calling the execution of the first program from within the application programmed to access the first logical dataset is stored in an object, Methods that include...

36. The method according to claim 35, wherein inputting data into the first program template includes automatically discovering one or more values ​​of one or more first parameters of the first program template based on the information relating to the first physical dataset.

37. The method according to claim 36, wherein the one or more first parameters include information about a record format or schema associated with the first physical dataset.

38. The method according to any one of claims 35 to 37, wherein storing information for calling the execution of the first program from within an application programmed for accessing the first logical dataset in the object includes storing an identifier for the first data store.

39. The method according to any one of claims 35 to 38, wherein storing information for calling the execution of the first program from within an application programmed for accessing the first logical dataset in the object includes storing a logical identifier for the first logical dataset.

40. The first program described above is generated by The method according to claim 36, further comprising obtaining information relating to one or more second parameters of the first program template, wherein the one or more second parameters are different from the one or more first parameters.

41. The method according to claim 40, wherein the one or more second parameters specify a method for accessing the first physical dataset.

42. The first program described above is generated by To determine whether a program template is available for the type of the first data store, Based on the determination that the first program template is available for the type of the first data store, select the available template as the first program template, The method according to any one of claims 35 to 41, further comprising:

43. Based on the determination that a program template is not available for the type of the first data store, Creating a program structure based on user input, Based on the program structure created, the first program for accessing the first data store is generated, The method according to claim 42, further comprising:

44. Receiving information related to a second physical dataset stored in the second data store of the aforementioned data store, Based on the information relating to the second physical dataset, a second program is generated for accessing the second physical dataset from the second data store. The method according to any one of claims 35 to 43, further comprising:

45. The data processing system is configured to run in multiple environments, each environment including an instance of the data processing system. The method according to any one of claims 35 to 44, wherein the object is assigned a unique identifier within the scope of each of the plurality of environments, and includes at least a portion that is common across the plurality of environments.

46. A method performed by a data processing system to enable efficient analysis in a dynamic environment having multiple datasets by updating entries in a dataset catalog to facilitate access to physical datasets in a data store, wherein the data processing system is configured to run a data processing application programmed to access data represented as logical datasets, each logical dataset includes a schema for data independent of the format of the corresponding data in the physical datasets, and the data processing system includes a dataset multiplexer that can be configured to provide the application with access to the physical datasets in the data store, and the method is: Receiving information related to the first physical dataset stored in the first data store corresponding to the first logical dataset, Based on the received information, a first program is generated for accessing the first physical dataset from the first data store. To detect events indicating changes associated with the physical dataset corresponding to the first logical dataset, Based on the detection of the aforementioned event, the first program for accessing the physical dataset corresponding to the first logical dataset is modified. Methods that include...

47. The physical dataset is the first physical dataset, and the event indicating a change associated with the physical dataset includes an event indicating a change from the first data store storing the first physical dataset to the second data store, and the method is The method according to claim 46, further comprising modifying the first program to access the first physical dataset from the second datastore in response to detecting the event indicating the change from the first datastore to the second datastore.

48. The method according to claim 46 or 47, wherein the physical dataset is the first physical dataset, and the event indicating a change associated with the physical dataset includes an event indicating a change in the value of a parameter used to generate the first program for accessing the first physical dataset.

49. The method according to any one of claims 46 to 48, wherein the physical dataset is the first physical dataset, and the event indicating a change associated with the physical dataset includes an event indicating a change to the format of the first physical dataset.

50. Detecting an event indicating a change associated with a physical dataset includes detecting an event indicating that the first physical dataset is replaced with a second physical dataset corresponding to the first logical dataset, The method according to any one of claims 46 to 49, wherein modifying the first program for accessing the physical dataset includes replacing the first program with a second program for accessing the second physical dataset.

51. The data processing system is configured to call the first program to perform an operation within an application that specifies access to the first logical dataset, The data processing system is configured to run in multiple environments, the first environment including a first instance of the data processing system, and the second environment including a second instance of the data processing system. The first data store and the first program are associated with the first instance of the data processing system, The method described above is The method according to any one of claims 46 to 50, further comprising generating a second program for performing an operation within an application that specifies access to the first logical dataset within the second instance of the data processing system.

52. The method according to claim 51, further comprising: running the application that specifies access to the first logical dataset in the second environment; and accessing the second program to access the second physical dataset in response to the application performing operations on the first logical dataset.

53. A method performed by a data processing system to enable efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores, wherein the data processing system is configured to run a data processing application programmed to access logical datasets, each logical dataset includes a schema for data independent of the format of the corresponding data in the physical datasets, and the data processing system includes a dataset multiplexer that can be configured to provide the application with access to multiple physical datasets in the multiple data stores, and the method is Accessing the dataset catalog and selecting objects associated with a logical dataset, Based on the selected object, a program configured for accessing a data source that stores the physical dataset corresponding to the logical dataset is invoked. A method comprising performing an operation within the application that specifies access to the logical dataset, The method described above is The method further includes dynamically updating the objects in the dataset catalog in response to an event indicating a change in the physical storage of a logical dataset represented by an object in the dataset catalog, A method for dynamically updating the objects in the dataset catalog, which includes modifying the program configured for accessing the data source that stores the physical datasets corresponding to the logical datasets.

54. At least one computer hardware processor, A non-temporary computer-readable medium storing processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform the method according to any one of claims 1 to 53, A data processing system that includes this.

55. A non-temporary computer-readable medium comprising, when executed by at least one computer hardware processor, a processor-executable instruction causing the at least one computer hardware processor to perform the method according to any one of claims 1 to 53.

Images