Data set multiplexer for a data processing system - Patents.com

Abstract

A data processing system is provided with a dataset multiplexer that allows applications to be written to specify access to a dataset as an operation on a logical dataset. During execution of an application by the data processing system, an operation to access a dataset is performed by accessing an entry in a dataset catalog for the logical dataset. The entry contains information for accessing a physical data source that stores the logical dataset, including conversion of data from the format of the physical data source to the format of the logical dataset. The catalog entry may be created upon registration of a data source with the dataset multiplexer and may be automatically updated upon changes in the storage of the dataset. Maintenance of this catalog may be partially or fully automated such that the system automatically adapts to changes in the storage of the dataset without the need to modify applications.

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims the benefit of priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63 / 163,709, entitled "DATASET MULTIPLEXER FOR DATA PROCESSING SYSTEM," filed March 19, 2021, and to U.S. Provisional Patent Application No. 63 / 143,898, entitled "DATASET MULTIPLEXER FOR DATA PROCESSING SYSTEM," filed January 31, 2021, each of which is incorporated by reference in its entirety herein.

[0002] Aspects of the present disclosure relate to techniques for efficiently operating a data processing system having multiple data sets that may be stored in any of multiple data stores. [Background technology]

[0003] Modern data processing systems manage vast amounts of data within an enterprise. For example, large institutions may have millions of datasets. Having such a large number of datasets can be extremely valuable to an enterprise, as this data can support multiple aspects of the enterprise's business. Some datasets can support routine processes, such as tracking customer account balances or sending account statements to customers. In other examples, processing data from one or more datasets can produce business insights, such as a conclusion that a requested transaction is fraudulent, or that an enterprise is exposed to a certain level of financial risk as a result of overall transactions in a particular geographic region. In yet other examples, processing data from one or more datasets can produce technical insights, such as a conclusion that an enterprise is at risk of technical failure as a result of an erroneous technical process.

[0004] The physical storage for these datasets may be provided in any of several ways. For example, the datasets may be stored in a structured manner and managed by a database system within the enterprise. In this case, the datasets may be stored as one or more tables managed by the database. Alternatively, simple datasets may be stored in files accessible to the data processing system, such as csv or xml files, or flat files. The computer storage on which the datasets reside, whether as files, database tables, or other formats, may be physically implemented in any of several forms, such as local to the data processing system, distributed across the enterprise, or distributed across a network cloud managed by a third party.

[0005] An enterprise's architects may select physical storage for a dataset based on the expected characteristics of the dataset, such as the size of the dataset, the required access time, the retention period of the dataset, or the impact to the enterprise of loss or corruption of the dataset. Commercial considerations, such as the price of storage or concerns about being locked into a third-party storage vendor, may also influence the choices made regarding the implementation of physical storage for the enterprise. As a result, data stores that hold datasets used within an enterprise may take any of several forms.

[0006] To support a wide range of functions, data processing systems may execute applications, whether to implement routine processes or to extract insights from data sets. Applications may be programmed to access data stores to read and write data. Summary of the Invention [Means for solving the problem]

[0007] According to some aspects, a method performed by a data processing system enables efficient data analysis in a dynamic environment having multiple datasets by generating and / or using entries in a dataset catalog to enable access to physical datasets in a data store. The data processing system may be configured to execute a data processing application programmed to access logical datasets. Each logical dataset includes a schema for the data independent of a format of the corresponding data in the physical dataset. The data processing system includes a dataset multiplexer configurable to provide the application with access to the physical datasets in the data store. The method includes creating a plurality of entries in the dataset catalog, each of the plurality of entries associated with a logical dataset and a physical dataset and associated with computer-executable instructions for accessing the physical dataset; receiving an input at least partially identifying a first logical dataset to access to perform an operation in the data processing application specifying access to the dataset; invoking the computer-executable instructions to access a physical dataset associated with the entry in the dataset catalog associated with the first logical dataset upon execution of the operation in the data processing application; and dynamically updating the entry in the dataset catalog in response to an event indicating a change in the physical dataset associated with the logical dataset.

[0008] According to an aspect, creating a plurality of entries in the dataset catalog includes receiving information related to a first physical dataset among the physical datasets stored in a first data store among the data stores, the first physical dataset corresponding to a first logical dataset; generating a first program including computer-executable instructions for accessing the first physical dataset from the first data store based on the information related to the first physical dataset; and storing, in the first entry in the dataset catalog, a link to the first program to enable a data processing application to access the first physical dataset using the first program.

[0009] According to one aspect, generating a first program for accessing a first physical data set from a first data store includes identifying a type of the first data store from the received information, selecting a first program template for the type of the first data store, and generating the first program by populating the first program template with one or more values ​​of one or more parameters of the first program template.

[0010] According to an aspect, receiving an input at least partially identifying the first logical data set includes providing a user interface through which a user at least partially identifies the first logical data set.

[0011] According to an aspect, invoking the computer-executable instructions includes providing access to an entry in a dataset catalog associated with the first logical dataset and providing access to a data store that stores a physical dataset corresponding to the first logical dataset based on information in the entry.

[0012] According to an aspect, dynamically updating an entry in the dataset catalog includes detecting an event indicating a change associated with a physical dataset corresponding to the first logical dataset, and modifying an entry in the dataset catalog associated with the first logical dataset based on the detection of the event.

[0013] According to an aspect, modifying the entry in the dataset catalog includes modifying computer-executable instructions for accessing a physical dataset that corresponds to the first logical dataset.

[0014] According to some aspects, a method is provided performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by registering a dataset in a dataset catalog to facilitate access to multiple physical datasets in a data store. The data processing system is operable with multiple physical datasets stored in the data store. The data processing system includes a dataset multiplexer configurable to provide an application with access to a physical dataset of the multiple physical datasets, the physical dataset being stored in a data store of the data stores. The physical dataset corresponds to a logical dataset that includes a schema for data independent of a format of corresponding data in the physical dataset. The method includes receiving information related to a first physical dataset of the multiple physical datasets stored in a first data store of the multiple data stores, the first physical dataset corresponding to the first logical dataset; generating a first program including computer executable instructions for accessing the first physical dataset from the first data store based on the information related to the first physical dataset; and storing a link to the first program in a first object in a library of objects to enable an application to access the first physical dataset using the first program.

[0015] According to an aspect, the method includes determining whether to modify a first program for accessing the first physical data set based on detection of an event indicative of a change associated with the first physical data set.

[0016] According to an aspect, the method includes generating a modified first program based on a determination to modify the first program, and replacing the first program with the modified first program as a target of the link.

[0017] According to an aspect, generating the modified first program includes generating the modified first program without modifying the application or the first logical data set.

[0018] According to an aspect, the information associated with the first physical data set includes information regarding a type of the first data store.

[0019] According to an aspect, the dataset multiplexer includes a library of objects that stores information for accessing multiple physical datasets, and a first object in the library of objects includes an identifier of a first physical dataset.

[0020] According to an aspect, the dataset multiplexer further includes an API, and the method further includes providing access to the first object to the application via the API.

[0021] According to one aspect, the method further includes assigning identifiers to objects in the library based on a schema and logical name of the respective logical data set in which information is stored in the object.

[0022] According to an aspect, the method further includes receiving a command to register the first physical dataset in the dataset catalog, and generating a first object based on the received command, and storing the first object in a library.

[0023] According to an aspect, the identifier of the first physical data set is a physics identifier.

[0024] According to an aspect, the first object further includes a second identifier, the second identifier being a logical identifier of a logical data set associated with the first object.

[0025] According to an aspect, the method further includes modifying the physical identifiers in the first object without modifying the logical identifiers in response to detecting an event indicating that the first physical data set has changed from a state stored in the first data store to a state stored in the second data store.

[0026] According to one aspect, the first object includes a value of a parameter accessed during execution of the first program, and the method further includes modifying the value of the parameter stored in the first object based on detection of an event indicating a change in the value of the parameter accessed in the first program.

[0027] According to one aspect, the first program includes access logic and conversion logic, and when the application is executed, the access logic and conversion logic of the first program are executed to provide access to the first physical data set and to convert between a format used in the first physical data set and a format used in the first logical data set.

[0028] According to one aspect, the first program includes one or more parameters such that values ​​of the one or more parameters affecting operation of the first program affect access of the first physical data set via the first program.

[0029] According to one aspect, the application is configured to provide values ​​for one or more parameters for use in invoking the first program.

[0030] According to an aspect, the method further includes generating the first program by detecting a type of the first data store and selecting a template from a plurality of templates based on the detected type.

[0031] According to one aspect, the first program includes a first portion configured for read access to the first data store and a second portion configured for write access to the first data store.

[0032] According to one aspect, the first program is configured as an executable data flow graph that includes logic for accessing a first physical data set.

[0033] According to some aspects, a method is provided performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores. The data processing system is operable with an application and multiple physical datasets stored in multiple data stores. The application is programmed to access a logical dataset that includes a schema for the data independent of a format of the corresponding data in the physical dataset. The method includes providing a user interface that at least partially identifies a logical dataset for a user to access in the application, executing the application, and upon performing an operation involving accessing the identified logical dataset, enabling access to an object in a library of objects associated with the logical dataset, and enabling access to a data store that stores a physical dataset corresponding to the identified logical dataset based on information in the object.

[0034] According to one aspect, the method further includes updating information in the object based on an event associated with storing data corresponding to the identified logical data set.

[0035] According to one aspect, the information in the object includes an executable program for accessing the physical data set.

[0036] According to one aspect, an executable program for accessing the physical dataset encodes logic for converting data between a format used in the physical dataset and a format used in the logical dataset.

[0037] According to one aspect, an object is an executable program for accessing a physical data set.

[0038] According to one aspect, the information in the object includes the type of data store.

[0039] According to one aspect, the information in the object includes a record format or schema associated with the physical data set.

[0040] According to one aspect, the information in the object includes one or more parameters that specify how to access the physical dataset, the one or more parameters including at least one parameter that indicates whether the data in the physical dataset is compressed or not.

[0041] According to one aspect, the information in the object includes one or more parameters that specify how the physical data set is to be accessed, the one or more parameters including at least one parameter that indicates a type of access.

[0042] According to one aspect, the type of access includes an indication of read access or write access.

[0043] According to one aspect, the type of access includes an indication of access via a high speed connection or a low speed connection.

[0044] According to one aspect, the data processing system includes a repository of metadata associated with logical data sets, and providing the user interface includes presenting a menu of logical data sets based on the metadata in the repository.

[0045] According to some aspects, a method performed by a data processing system enables efficient data analysis in a dynamic environment having multiple datasets by generating entries in a dataset catalog to enable access to physical datasets in a data store. The data processing system is configured to execute a data processing application programmed to access logical datasets. Each logical dataset includes a schema for the data independent of a format of the corresponding data in the physical dataset, and the data processing system includes a dataset multiplexer configurable to provide the application with access to the physical datasets in the data store. The method includes receiving information related to a first physical dataset stored in a first data store of a data store, where an application is programmed for access of the first logical dataset, the first physical dataset corresponding to the first logical dataset; generating a first program for accessing the first physical dataset from the first data store based on the received information, where generating the first program includes identifying a type of the first data store from the received information, selecting a first program template for the type of the first data store, and generating the first program by populating the first program template with one or more values ​​of one or more parameters of the first program template; and storing information in an object for invoking execution of the first program from within the application programmed for access of the first logical dataset.

[0046] According to an aspect, populating the first program template includes automatically discovering one or more values ​​for one or more first parameters of the first program template based on information associated with the first physical dataset.

[0047] According to one aspect, the one or more first parameters include information regarding a record format or schema associated with the first physical data set.

[0048] According to an aspect, storing in the object information for invoking execution of the first program from within the programmed application for access of the first logical data set includes storing an identifier of the first data store.

[0049] According to an aspect, storing in the object information for invoking execution of the first program from within the programmed application for access of the first logical data set includes storing a logical identifier of the first logical data set.

[0050] According to one aspect, generating the first program further includes obtaining information regarding one or more second parameters of the first program template, where the one or more second parameters are different from the one or more first parameters.

[0051] According to one aspect, the one or more second parameters specify how the first physical data set is accessed.

[0052] According to one aspect, generating the first program further includes determining whether a program template is available for the first data store type, and selecting an available template as the first program template based on a determination that the first program template is available for the first data store type.

[0053] According to one aspect, the method includes creating a program structure based on user input based on a determination that a program template is not available for a type of the first data store, and generating a first program for accessing the first data store based on the created program structure.

[0054] According to an aspect, the method includes receiving information related to a second physical data set stored in a second data store of the data stores, and generating a second program for accessing the second physical data set from the second data store based on the information related to the second physical data set.

[0055] According to one aspect, a data processing system is configured to execute in multiple environments, each environment including an instance of the data processing system, and an object is assigned a unique identifier within each of the multiple environments and includes at least a portion that is common across the multiple environments.

[0056] According to some aspects, a method performed by a data processing system is provided for enabling efficient analysis in a dynamic environment having multiple datasets by updating entries in a dataset catalog to facilitate access to physical datasets in a data store. The data processing system is configured to execute a data processing application programmed to access data represented as logical datasets. Each logical dataset includes a schema for the data independent of a format of the corresponding data in the physical dataset, and the data processing system includes a dataset multiplexer configurable to provide the application with access to the physical datasets in the data store. The method includes receiving information related to a first physical dataset stored in a first data store corresponding to the first logical dataset, generating a first program for accessing the first physical dataset from the first data store based on the received information, detecting an event indicative of a change associated with the physical dataset corresponding to the first logical dataset, and modifying the first program for accessing the physical dataset corresponding to the first logical dataset based on the detection of the event.

[0057] According to an aspect, the physical dataset is a first physical dataset, and the event indicative of a change associated with the physical dataset includes an event indicative of a change from a first data store storing the first physical dataset to a second data store, and the method further includes modifying the first program to access the first physical dataset from the second data store in response to detecting the event indicative of a change from the first data store to the second data store.

[0058] According to an aspect, the physical dataset is a first physical dataset, and the event indicative of a change associated with the physical dataset includes an event indicative of a change in a value of a parameter used to generate a first program for accessing the first physical dataset.

[0059] According to an aspect, detecting an event indicative of a change associated with the physical dataset includes detecting an event indicative of replacing a first physical dataset with a second physical dataset corresponding to the first logical dataset, and modifying the first program for accessing the physical dataset includes replacing the first program with a second program for accessing the second physical dataset.

[0060] According to one aspect, a data processing system is configured to invoke a first program to perform an operation within an application that specifies access to a first logical data set, the data processing system is configured to run in multiple environments, the first environment includes a first instance of the data processing system, and the second environment includes a second instance of the data processing system, the first data store and the first program are associated with the first instance of the data processing system, and the method further includes generating a second program for performing an operation within the application that specifies access to the first logical data set within the second instance of the data processing system.

[0061] According to one aspect, executing an application in a second environment that specifies access to a first logical dataset, and in response to execution of an operation by the application on the first logical dataset, accessing a second program to access a second physical dataset.

[0062] According to an aspect, a method is provided performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores. The data processing system is configured to execute a data processing application programmed to access logical datasets. Each logical dataset includes a schema for the data independent of a format of the corresponding data in the physical dataset, and the data processing system includes a dataset multiplexer configurable to provide the application with access to the multiple physical datasets in the multiple data stores. The method includes performing an operation in the application specifying access to the logical dataset by accessing the dataset catalog to select an object associated with the logical dataset and invoking a program configured for access of a data source that stores a physical dataset corresponding to the logical dataset based on the selected object.

[0063] According to an aspect, the method further includes dynamically updating the object in the data set catalog in response to an event indicating a change in physical storage of a logical data set represented by an object in the data catalog.

[0064] The various aspects described above may be used alternatively or additionally with aspects in any of the systems, methods, and / or processes described herein. Furthermore, a data processing system may be configured to operate according to a method having one or more of the aforementioned aspects. Such a data processing system may include at least one computer hardware processor and at least one non-transitory computer readable medium having stored thereon processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform such a method. Furthermore, the non-transitory computer readable medium may include processor-executable instructions that, when executed by the at least one computer hardware processor of the data processing system, cause the at least one computer hardware processor to perform a method having one or more of the aforementioned aspects. Thus, the above is a non-limiting summary of the invention as defined by the appended claims.

[0065] Various aspects are described with reference to the following figures. It should be understood that the figures are not necessarily drawn to scale. Items that appear in multiple figures are designated by the same or similar reference numerals in all figures in which they appear. [Brief description of the drawings]

[0066] [Figure 1A] FIG. 1 is a block diagram of an exemplary enterprise IT system including a data processing system having a data set multiplexer, in accordance with one aspect of the technology described herein. [Figure 1B] 1B is a block diagram of the example enterprise IT system of FIG. 1A in which the dataset multiplexer is in an operational state at a first time facilitating access between applications configured to access the logical datasets and a first data store that stores a physical dataset corresponding to the logical dataset. [Figure 1C]FIG. 1C is a block diagram of the example enterprise IT system of FIG. 1B, in which the dataset multiplexer is in an operational state at a second time to facilitate access between applications configured to access the logical dataset and a second data store that stores a physical dataset corresponding to the logical dataset. [Figure 2A] FIG. 1B is a block diagram of an exemplary enterprise IT system in which the data processing system of FIG. 1A is instantiated into multiple instances to provide multiple environments, and an application is executed by a first instance in which a dataset multiplexer facilitates access between the application and a first physical dataset. [Figure 2B] 2B is a block diagram of the example enterprise IT system of FIG. 2A in which an application is executed by a third instance in which a dataset multiplexer facilitates access between the application and a second physical dataset. [Figure 3A] FIG. 1 is a schematic diagram showing a graphical development environment for applications written as dataflow graphs. [Figure 3B] FIG. 3B is a schematic diagram of the dataflow graph of FIG. 3A in which the input nodes of the dataflow graph are configured or programmed in terms of logical data sets. [Figure 3C] FIG. 3B is a schematic diagram of the data flow graph of FIG. 3A modified to access information in a dataset catalog to enable access to a physical dataset for execution of an operation in an application that specifies access to a logical dataset. [Figure 4] This is summary information that may be reflected in an object of a dataset catalog that provides information about a physical dataset that corresponds to a logical dataset. [Figure 5A] FIG. 1B is a block diagram of the example enterprise IT system of FIG. 1A showing additional details of the data set multiplexer. [Figure 5B]1B is a block diagram of the example IT system of FIG. 1A showing components of a data multiplexer that may optionally be used in interfacing with running applications. [Figure 6A] 1A or 5A is a block diagram of an example enterprise IT system in a first operational state at a first time. [Figure 6B] 6B is a block diagram of the example enterprise IT system of FIG. 6A in a second state at a second time. [Figure 7] FIG. 1 is a block diagram illustrating information used in a data processing system configured with a data set multiplexer in accordance with certain aspects of the technology described herein. [Figure 8] 1 is a flowchart of an exemplary method of operating a data processing system having a data set multiplexer, in accordance with one aspect of the technology described herein. [Figure 9] FIG. 1 is a block diagram of an exemplary computing system environment that can be used in implementing certain aspects of the technology described herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0067] The inventors have recognized and understood that a dataset multiplexer can enable efficient operation of a data processing system. In an enterprise with a large number of datasets that may be stored in various data stores, the dataset multiplexer enables the use of applications that are written in terms of one or more logical datasets, rather than written in terms of physical datasets. These applications written in terms of logical datasets do not need to be modified for proper operation when the data store that stores one or more physical datasets represented by the logical datasets is changed. To support this dynamic update of the data store, the dataset multiplexer can maintain a catalog of datasets, with each entry in the catalog providing information for accessing the data store in which one or more physical datasets represented by the logical datasets are stored. The dataset multiplexer can enable efficient analysis in a dynamic environment, for example, where the physical storage of the datasets may evolve or change.

[0068] By using a dataset multiplexer, applications can be written and executed without the application having knowledge of the formats (e.g., record formats or schemas) supported by the data stores that the application accesses, or the physical locations of those data stores. Also, for example, a business user who has no knowledge of the physical datasets and data stores, but who understands how to extract business insights from the data, can write an application in terms of logical datasets rather than in terms of physical datasets. The dataset multiplexer can automatically supply the connection between the application and the appropriate data store that stores the physical dataset represented by the logical dataset, thereby avoiding the need for the application and user to have knowledge of the data store's implementation.

[0069] The catalog of datasets may be updated in response to an event indicating a change in the storage of the dataset, such as the physical dataset represented by the logical dataset. The application and / or the logical dataset may not need to be changed in response to the event. By retrieving information from the catalog to access a data store that stores a physical dataset that corresponds to the logical dataset of the application when accessed, the appropriate data store may be accessed without having to maintain the application to adapt to changes in the data store. In an enterprise, this capability may facilitate the migration of datasets from one storage location to another to enable efficient use of computer storage while maintaining proper execution of the application. For example, a dataset may be migrated from one storage location to another or from one type of storage to another throughout its lifecycle. Such migration may occur without modifying the application and while maintaining proper execution of the application. Avoiding the need to modify the application even when such changes occur may result in reliable and efficient execution of the application and may result in significant cost savings for the enterprise, as the costs and downtime of modifying and retesting the modified application are avoided.

[0070] As a specific example, a physical dataset may initially be stored as a file. Storage as a file may enable the use of low-cost computer storage. As the amount of data in the physical dataset increases, or as the value of the data increases, the physical dataset may be migrated to a database system to enable faster processing of large datasets or higher fault tolerance. By updating the catalog entries of the logical datasets corresponding to the physical datasets, applications written to access the logical datasets through a dataset multiplexer continue to operate without modification when the physical datasets are migrated from files to a database system.

[0071] The catalog entry may include information for accessing a physical dataset that can accommodate other types of changes to the storage of data associated with the logical dataset. This information may include programs that, at run time, access data from a data store and convert it into a representation of the logical dataset. As a specific example, the format of a field of a physical dataset used to store a logical entity may be changed without affecting applications that reference the logical entity, because modification of an entry in the dataset catalog may include modification of a program that converts data in the data store into the format used by the logical dataset.

[0072] Dataset multiplexers can also facilitate the development of applications by simplifying the transition between programming environments. For example, applications are traditionally developed in a development environment, tested in a test environment, and then advanced to a production environment. In the production environment, the application may read and write to one or more data stores that have "live" data used across the enterprise. In the test and development environments, the application may operate with offline data stores that are less likely to impact the enterprise if corrupted by improper operation of the application. In the development environment, the data stores may be relatively small, while in the test environment, the data stores may be structured to provide robust test cases, including extreme test cases that may not appear with the current live data.

[0073] Regardless of the reason that different datasets are desired in different environments, each environment may have its own dataset catalog information. An instance of a data processing system providing a development environment may access data catalog information scoped for the development environment. Similarly, an instance of a data processing system providing a test or production environment may access data catalog information scoped for the respective environment to access the appropriate data store. In this manner, an application written to access a logical dataset may run in either environment and automatically access the appropriate data store for each environment without the need to adapt the application to the specific environment. When execution of the application involves operations on a logical dataset, the data processing system automatically utilizes the appropriate data catalog information for the appropriate environment to access a data store that contains the physical dataset for that environment that stores data corresponding to the logical dataset.

[0074] The value of such a dataset multiplexer can be enhanced by the dataset multiplexer being able to automatically construct a dataset catalog entry for a data store. The dataset multiplexer can, for example, maintain a set of program templates applicable to different types of data stores. When a data store is registered with the dataset multiplexer, the dataset multiplexer can detect the type of data store and select an appropriate template. A program for accessing that data store can be constructed by populating the selected template with values ​​of parameters detected by analysis of the data store. Some or all of the parameter values ​​may alternatively or additionally be obtained from a metadata repository that maintains metadata for the data store, provided by user input, or obtained in other ways.

[0075] Aspects of the data processing system may be implemented to achieve any one or more of the above-mentioned objects and advantages, which may be used alone or in any suitable combination.

[0076] A typical data processing system with a data set multiplexer 1A is a block diagram of an IT system 100 including an exemplary data processing system 104 and a data set multiplexer 105 integrated with the data processing system 104, in accordance with some aspects of the techniques described herein. The IT system 100 may be an IT system of an enterprise, such as, for example, a financial company. For simplicity, elements of the enterprise IT system, such as a network, cloud storage, and user devices, are not explicitly illustrated.

[0077] The data processing system 104 is configured to access (e.g., read data from and / or write data to) the data stores 102-1, 102-3, 102-3, ..., and 102-n. Each of the data stores 102-1, 102-3, 102-3, ..., and 102-n may store one or more physical data sets. The data stores may store any suitable type of data or collections of data in any suitable manner or format. The data stores may store data, for example, using a database system (e.g., a relational database system), as flat text files, spreadsheets, and the data stores may be internal or external to the enterprise. For example, the external data stores may be "in the cloud" or in storage hardware managed by a third party. Thus, the data stores may provide a federated environment in which different data stores used by the enterprise may be in different locations and / or managed by different entities inside or outside the enterprise.

[0078] In some cases, the data store may store transaction data. For example, the data store may store credit card transactions, telephone record data, or bank transaction data. It should be understood that the data processing system 104 may be configured to access any suitable number of data stores of any suitable type, as aspects of the technology described herein are not limited in this respect. A data store from which the data processing system 104 may be configured to read data may be referred to as a data source. A data store to which the data processing system 104 may be configured to write data may be referred to as a data sink. However, the technology as described herein may be applied to data stores that hold other types of data used by an enterprise.

[0079] Each data store may be implemented with one or more storage devices and may include data management software or other control mechanisms to support the storage of physical data sets in any suitable type or formats. The one or more storage devices may be of any suitable type, including, for example, one or more servers, one or more disk arrays, one or more clusters of disk arrays, one or more portable storage devices, one or more non-volatile storage devices, one or more volatile storage devices, and / or any other device or devices configured to electronically store data. In embodiments in which a data store includes multiple storage devices, the storage devices may be collocated in one physical location (e.g., in a building) or distributed across multiple physical locations (e.g., in multiple buildings, different cities, states, or countries). The storage devices may be configured to communicate with each other using one or more networks of any suitable type, as aspects of the technology described herein are not limited in this respect.

[0080] The data management software can organize data in the physical storage and provide mechanisms to access the data so that data can be written to or read from the physical storage. The data management software can be, for example, a database system or a file management system. Depending on the type of data management software, one or more storage devices can store the physical data sets using one or more formats, such as database tables, spreadsheet files, flat text files, and / or files in any other suitable format (e.g., mainframe native format). The data stores 102-1, 102-2, 102-3, ..., and 102-n can be of the same type (e.g., all can be relational databases) or can be of different types (e.g., one can be a relational database while another can be a data store that stores data in flat files). When the data stores are of different types, the storage environment can be referred to as a heterogeneous or federated data environment 102. The data store may be, for example, a SQL Server database, an ORACLE database, a TERADATA database, a flat file, a multi-file data store, a HADOOP distributed database, a DB2 data store, a Microsoft SQL SERVER data store, an INFORMIX data store, a table, a collection of tables or other sub-parts of a database, and / or any other suitable type of data store, as the aspects of the technology described herein are not limited in this respect.

[0081] The data processing system 104 supports various applications 106 to perform functions to access (e.g., read access and / or write access) the physical data sets stored in the data stores 102-1, 102-3, 102-4, ..., and 102-n. The applications 106 can then perform operations based on the data in the data stores. The data processing system 104 can support applications 106-1, 106-2, 102-3, ..., and 106-n, which may be of the same type or different types. In some examples, the applications, when executed, can read or write transactional data to one or more physical data sets in a data store. In other examples, the applications, when executed, can read or write data to physical data sets stored across multiple different data stores and analyze the data to extract business insights from the data sets.

[0082] An application 106 can be developed as a dataflow graph, for example as shown in FIG. 3A. A dataflow graph may include components called "nodes" or "vertices," which represent data processing operations performed on data, and links between components that represent the flow of data. Techniques for executing computations encoded by dataflow graphs are described in U.S. Pat. No. 5,966,072, entitled "Executing Computations Expressed as Graphs," which is incorporated herein by reference in its entirety. An environment for developing applications (e.g., computer programs) as dataflow graphs is described in U.S. Pat. App. Pub. No. 2007 / 0011668, entitled "Managing Parameters for Graph-Based Applications," which is incorporated herein by reference in its entirety. A dataflow graph may include data sources (such as input data stores 302 or 304 of FIG. 3A) and data sinks (such as output data store 314 of FIG. 3A). These are represented by terminal nodes in the flow that indicate access to data stores 102-1, 102-3, 102-3, . . . or 102-n.

[0083] However, the application itself need not be programmed with the particular data store that is included in the application. Rather than being hard-coded to access a single physical data set, the application 106 may be programmed in terms of logical data sets. A logical data set may refer to a logical representation of one or more data sets. The data processing system 104 may store definitions of multiple logical data sets and other metadata about those logical data sets. This information may be managed, for example, by a metadata management module (e.g., metadata management module 526 of FIG. 5A). Tools used with the data processing system 104 may access the metadata about the logical data sets and perform functions based on that metadata. For example, a program development environment may provide a user interface that allows for the selection of available logical data sets and use in programming the application.

[0084] A logical dataset may have a schema that defines the data, regardless of the format of the corresponding data in the physical dataset / data store. For example, a logical dataset may have a schema that defines logical entities within the logical dataset. A logical entity may be recognizable and / or understandable to a human user. For example, a logical dataset may include a logical entity such as a customer name. In a physical dataset corresponding to this logical dataset, the customer name may be stored in one row of a data table as three fields that hold data corresponding to the customer's first name, middle initial, and last name, respectively. However, the logical dataset may simply include the logical entity Customer_Name, regardless of the format of the data in the physical storage.

[0085] The data processing system 104 may include an interface (not shown) through which a schema of a logical data set can be defined. The interface may be, for example, a user interface through which a user can specify a logical data set or introduce a logical data set into the system by specifying a schema of the logical data set. In some embodiments, the data processing system 104 may store a set of logical entities that are commonly used in the business of an enterprise. Examples of commonly used logical entities may include one or more of a name, an identification number, a phone number, an address, a nationality, an account balance, a transaction amount, or a date. These business terms may be used to at least partially specify the schema of the logical data set. However, a schema may be defined to include other logical entities instead of or in addition to the predefined logical entities.

[0086] By enabling programming of applications in terms of logical datasets, the programmer creating the application need not understand the format of the data stores that store the corresponding physical datasets, such that a data analyst may develop an application using logical datasets without understanding the format of the data in the data stores that hold the physical datasets.

[0087] As a more detailed example, a programmer within an enterprise may define a logical dataset that stores new customers. The schema of the logical dataset may include logical entities such as, for example, customer name, customer address, customer identifier, and customer acquisition date. A data analyst can write applications in terms of the logical dataset and these logical entities, regardless of the storage format of the physical dataset that corresponds to the logical dataset. As a result, the data analyst can write applications without knowledge of the physical dataset that stores the data accessed by the application.

[0088] During execution of an application, data of a physical dataset corresponding to a logical dataset may be stored in one or more of the data stores 102-1, 102-3, 102-3, ..., and 102-n. To execute an application, each operation that specifies access to a logical dataset may be executed by the data processing system 104, which reads or writes data from a corresponding physical dataset stored in one of the data stores 102-1, 102-3, 102-3, ..., and 102-n. In accordance with some aspects, the dataset multiplexer 105 may enable automated execution of such operations by automatically accessing the corresponding physical dataset. This access may include translation between the format of the data stored in the physical data store and the format specified in the schema of the logical dataset. As another example, the translation may result in associating data from the physical dataset with metadata associated with the logical dataset. As a specific example, the translation may associate a field of the physical dataset with a field of the logical dataset that is tagged with an indication that it holds personally identifiable information. As a result, the metadata can be used in operations on data from the physical dataset, such as, in that example, to filter or mask personally identifiable information.

[0089] As shown in FIG. 1A, the data processing system 104 includes a dataset multiplexer 105 for automating access to corresponding physical datasets and conversion between logical and physical dataset formats. The dataset multiplexer 105 can maintain a catalog of datasets 107, where each entry in the catalog corresponds to a logical dataset and provides information for accessing one or more physical datasets. For example, a catalog entry may identify a physical dataset in a data store 102-1, 102-3, 102-4, ..., or 102-n that corresponds to a logical dataset. The catalog entry may alternatively or additionally include information for converting data stored in the physical dataset to a format of the logical dataset. The information may be or include an executable program. For example, the catalog information may identify a program for converting data of multiple fields in the physical dataset to a format of a corresponding logical entity in the logical dataset. Other information may alternatively or additionally be stored as or reflected in the catalog information for accessing one or more physical datasets.

[0090] The dataset multiplexer 105 allows the application 106 to seamlessly access one or more physical datasets based on one or more programmed logical datasets using information in the catalog of the datasets. FIG. 1B shows an application (e.g., application 106-3) programmed to access data according to a logical dataset. When an operation is performed to access (e.g., read and / or write) a logical dataset, the dataset multiplexer 105 of the data processing system 104 can enable access to a corresponding one or more physical datasets in a data store (e.g., data store 102-1). For example, if the catalog information stored for the logical dataset is or includes an access control program, that program may be executed. As a result, even though the application 106-3 is programmed in terms of a logical dataset, when a data access operation is performed, a physical dataset stored in the data store 102-1 is accessed.

[0091] The dataset multiplexer 105 may access the catalog of datasets to select an entry associated with a logical dataset referenced in the application 106-3, and the information may then be used to identify the physical dataset stored in the data store 102-1 and / or to convert data in the format of the data store 102-1 to the format of the logical dataset for data access.

[0092] In some cases, this access may be dynamic. The catalog information may be used during execution of an application's operations that require data access. Entries associated with a logical dataset in the catalog of datasets may be updated in response to events that indicate changes to the storage of information associated with the logical dataset. Accessing the physical data stores via the catalog information may ensure that the application continues to run despite changes that may be made at any time throughout IT system 100, even if the data analyst or other user who wrote application 106-3 was unaware of those changes.

[0093] For example, a physical data set may be migrated from data store 102-1 to data store 102-n. The logical data set against which an application is programmed does not need to be modified to account for this change. By updating the catalog entry for the logical data set, the data set multiplexer 105 can automatically utilize the updated catalog information to provide application 106-3 with access to the correct physical data set, regardless of the data store in which it resides.

[0094] Figure 1C shows an application 106-3 accessing a data store 102-n through a dataset multiplexer 105 of a data processing system 104. The access conditions of Figures 1B and 1C may be the result of the execution of the application 106-3 at different times. Because the catalog information is dynamic and changes to account for the storage of datasets, no modification of the application 106-3 is required for proper access to the desired data.

[0095] In FIG. 1B, solid lines indicate the flow of data from data store 102-1 to application 106-3 during the execution of an operation to access (e.g., read and / or write) a logical dataset. Dashed lines indicate interactions between components that may control the flow of data in an operation. For example, application 106-3 may interact with dataset multiplexer 105 to obtain information for accessing a physical dataset corresponding to the logical dataset from a catalog entry associated with the logical dataset. The dataset multiplexer 105 may obtain information from the corresponding one or more physical datasets in data store 102-1 to generate the appropriate catalog entry. Similarly, solid lines in FIG. 1C indicate the flow of data from data store 102-n to application 106-3 during the execution of an operation to access (e.g., read and / or write) a logical dataset, and dashed lines indicate interactions between components that may control the flow of data in an operation (e.g., dataset multiplexer 105, application 106-3, and data store 102-n).

[0096] The use of dynamic data may enable correct operation despite any of several other types of changes in IT system 100. In addition to changes in the data store in which a physical dataset is stored, the type of data store that holds the dataset may change. For example, the type of data store may change. For example, data store 102-1 may be an Oracle database, while data store 102-n may be a SQL Server data store. As another example, the schema of a physical dataset may change, for example to include an additional field for name data. Such changes are automatically compensated for by modifying the transformation logic in the catalog.

[0097] Other types of changes can be handled automatically by dynamically using the dataset catalog information for data access. As another example, a user may run different instances of a data processing system for different purposes. It may be desirable for the same application to access different physical datasets when running on different instances. Such execution may be ensured by providing different catalog information in different instances, or may be ensured when it is desirable for an application to access different physical datasets corresponding to the same logical dataset in different contexts.

[0098] FIG. 2A illustrates an application (e.g., application 106-2) accessing one or more physical data sets in a data store (e.g., data store 102-2) through a data set multiplexer of an instance of a data processing system (e.g., instance 104-1 of data processing system 104) in accordance with some aspects of the techniques described herein. In the environment generated by instance 104-1, accesses to logical data sets are resolved to data sets in data store 102-2. The same application running in different environments generated by different instances 104-n of a data processing system may access different physical data sets. FIG. 2B illustrates application 106-2 accessing a data store 102-n (e.g., a database data store) in an environment generated by instance 104-n of data processing system 104. For simplicity of illustration, separate lines illustrating control flow between the illustrated components are not shown in FIGS. 2A and 2B. However, it should be understood that the components of the data processing system may interact to control the operations described herein. Therefore, control interactions may be omitted for simplicity.

[0099] The operations illustrated by Figures 2A and 2B may be generated by scoping the catalog information of each instance such that references to the same logical dataset in each scope may access the physical dataset via the catalog information of that scope. All or part of the identifier of a logical dataset may be persistent across multiple scopes. As a specific example, a logical dataset may be identified by a combination of name and schema that may be the same regardless of environment. However, the catalog information for the dataset associated with that logical identifier may be different in different instances.

[0100] In the embodiment of FIG. 2A and FIG. 2B, different instances 104-1, 104-2, ..., 104-n of the data processing system 104 may be provided for different programming environments. As a specific example, an enterprise may operate a data processing system in a development environment, a test environment, and a production environment. The data sets used by the same application may be different in each of these environments. The live data used in the production environment may not be used in the development or test environments to avoid corruption of the live data and / or to minimize the risk of leaking confidential information. The data stores for the production environment may be large and provide fast data access, and therefore may be very expensive. On the other hand, the data sets for the development environment may be stored in small, low-cost data stores to reduce the development costs of the application. The data sets for the test environment may include data that may occur in rare operational scenarios that are not present in the live data sets when the application is tested to ensure robust testing and complete code coverage. Making the application available in any environment allows efficient movement between environments such as development, test, and production, which may improve the efficiency of application development and the overall operation of the IT system.

[0101] Each instance of data processing system 104 may include a dataset multiplexer that maintains a catalog of datasets for a corresponding environment. Each dataset multiplexer may access a respective catalog of datasets for an appropriate environment to provide access to one or more appropriate data stores. For example, FIG. 2A illustrates application 106-2 accessing a data store 102-2 (e.g., a flat file data store) of a development environment via instance 104-1 of data processing system 104. FIG. 2B illustrates application 106-2 accessing a data store 102-n (which may be a database) of a production environment via instance 104-n of data processing system 104.

[0102] Typical techniques for developing applications using the Dataset Multiplexer In some embodiments, applications executed by a data processing system may be written in a graphical programming language by a human user of the data processing system, in other embodiments, procedural or other types of programming languages ​​may be used instead or in addition.

[0103] FIG. 3A illustrates a graphical user interface in which a data analyst or other human user can write an application in a graphical development environment, used herein as an example of application development. In this example, a data processing system includes a library of components that perform operations on data. Although not explicitly shown in FIG. 3A for simplicity, the graphical development environment may include a toolbar or other user interface element that allows a user to select a component from the library. A user may also specify connections between these components to form a graph. For example, a component may specify an operation to transform data or specify a data source or data sink to be accessed. A component may be represented by an icon having a different shape depending on the operation performed by the component or the type of data store that holds the data of the data source or data sink.

[0104] A user can write an application by selecting components that correspond to desired operations and connecting them in an order that specifies the desired data flow through the operations represented by the components. Each component can be configured by user parameter input. The values ​​of some configuration parameters can specify aspects of the component's operation. For example, a component that represents a dataset can receive a parameter that specifies its operation as a data source or a data sink.

[0105] In embodiments in which an application is written using logical datasets, the values ​​of some configuration parameters may specify a particular logical dataset and / or logical entities within a logical dataset to use in performing the component's operations. For example, a component that represents a dataset may be configured to represent a specified logical dataset by supplying an identifier for the logical dataset as the value of its parameter. A component may alternatively or additionally be configured with user input specifying a logical entity to be used as a key in a particular operation.

[0106] A data processing system may include a repository of information about logical datasets and / or logical entities that can be used in constructing components of an application. Entries in this repository may have been created by a user writing an application. However, in an enterprise, many individuals may be involved in generating and analyzing data, and the information in the repository may not have been developed by the user developing the application. For example, logical dataset information may have been generated by other users or by automated analysis of a particular physical dataset.

[0107] The user interface provided in the development environment may include user interface elements that allow a user to specify logical data sets or logical entities in the repository as values ​​for parameters that configure components of the graph. The user interface elements may include elements for a user to enter a search query. The query may be, for example, a facet query in which the user specifies one or more values ​​of dimensions that describe the logical data set or logical entity. The dimensions may include, for example, words entered into the repository to describe the logical data set or field names contained within the data set.

[0108] The data processing system can perform a search according to the query and return a list of options selected by the data processing system based on the query. A user can then select a value from the returned to configure a component, which then operates according to the selection. For example, if a dataset component is configured as a data source configured to output data from a logical dataset, the component operates by supplying in the format of the specified logical dataset when the application is executed.

[0109] It is not a requirement that an application be developed entirely by a human programmer. All or parts of a program may be generated in other ways, such as by templates, or may be machine translated from another programming language or pseudo-language. Regardless of how an application is developed, specifying the data that the application operates on in terms of one or more logical data sets allows the application to be written without knowledge of or dependency on the physical storage of the data. This ability can simplify any part of the development process for a human user, since the human user can specify operations that involve accessing data in terms of logical data sets and / or logical entities within the logical data sets. For example, a data analyst may be able to write an application without understanding the details of a particular physical data set. Furthermore, avoiding a dependency on physical storage in an application can extend the functionality of a data processing system. For example, an application can be written even when the details of the physical data sets that exist at the time the application is executed are not known to the programmer or have not yet been established.

[0110] As a further simplification, the data processing system may be configured to perform operations specified in terms of logical data sets or logical entities within the logical data sets. These operations may be specified to be performed within an application and may then be performed on data accessed in a physical data set corresponding to the logical data set.

[0111] For example, a logical entity may be associated with an enterprise-wide list of valid values, and changes can be made to the list at an enterprise level without the need to change each application that accesses the logical entity. As a specific example, within a data processing system, a logical entity for gender may be defined. At one point, metadata associated with that logical entity may indicate that the allowed values ​​are M and F. Later, the allowed values ​​may change to be M, F, and X. Each application written in terms of that logical entity can automatically adapt to the changed list, regardless of which physical dataset stores the gender information. This is advantageous because it can automatically affect all applications that use the gender logical entity, for example by indicating the "X" value as a new allowed value in the metadata.

[0112] As another example, validation rules may be specified in terms of logical entities and applied regardless of the physical dataset from which the data is accessed. As a specific example, a data processing system may be configured with a data validation rule for logical elements used for email addresses. That data validation rule may be applied to data from any physical dataset that stores emails when one or more fields in that physical dataset are identified as corresponding to logical elements used for email addresses. Validation rules may be used within an application in one or more ways. For example, a rule may be invoked on data from within the application on a particular physical dataset, or an application may access the results of the application of a rule to a particular physical dataset even if application of the rule to the dataset was triggered from outside the application.

[0113] As yet another example, components that perform mask or filter operations can be specified in terms of logical entities and / or metadata about the logical entities and can operate within an application regardless of the physical data store from which the data to be processed is drawn. As a specific example, logical entities that serve as identifiers for people can be assigned privacy levels. Logical entities may be defined for multiple identifiers of people, such as email addresses and social security numbers. Metadata associated with these logical entities may assign a medium privacy level to emails, while social security numbers may be given a high privacy level. A filter or mask component specified in terms of logical entities can be configured to exclude from its output records that have specific field values ​​associated with privacy levels above a threshold, or to obscure the values ​​of those fields. When these operations are performed on physical datasets that have fields corresponding to emails or social security numbers, they may be performed based on the privacy levels. Defining logical datasets and associated metadata, such as privacy levels, in a repository that may be used in the development of applications allows functions such as these to be implemented and updated efficiently across an enterprise. Such definitions may also be used to enforce enterprise policies related to data access by ensuring that physical datasets with sensitive information (i.e., datasets that include fields containing sensitive information) are handled appropriately.

[0114] 3A illustrates an application (e.g., application 106-3) being developed as a dataflow graph through a user interface in a development environment, where components are represented as nodes in the graph. The dataflow graph in this example includes input nodes 302, 304 for each of the physical datasets from which data is read, and output nodes 314 for each of the physical datasets to which data is written. An example of generating such input and output nodes based on the functionality they provide (e.g., data sink or data source functionality) is described in U.S. Pat. No. 9,977,659, entitled "Managing Data Set Objects," which is incorporated herein by reference in its entirety. The dataflow graph also includes nodes 306, 308, 310, 312 for various data processing operations (e.g., filter, sort, or join operations) that are performed on the data read from the physical datasets. When the graph is executed by a data processing system, the results of the data processing operations are written to the physical dataset associated with the output node 314.

[0115] Each input node may be configured with parameter values ​​associated with a respective data source, which may indicate how to access data from the data source. Similarly, each output node may be configured with parameter values ​​associated with a respective data sink, which may indicate how to write results to the data sink.

[0116] Traditionally, applications, including applications written as data flow graphs as shown in FIG. 3A, have had to be manually updated to account for changes to how data is stored. For example, when a data set is migrated from one data store to another, an experienced developer manually changes the configuration of the input and / or output nodes of the data flow graph that are affected by the migration. Such manual updates must be performed by an experienced developer with knowledge (e.g., programming knowledge) of the data flow graphs and data stores supported by the data processing system. In a data processing system that supports a large number of data sets with frequent changes to how data is stored, errors during updates or failure to update applications with each change can propagate errors throughout the enterprise. For example, executing a data flow graph with input nodes configured with incorrect or outdated parameter values ​​associated with the data sources can result in data being read from an incorrect data source or in an incorrect format. Errors in the input data cause data processing operations to be performed on incorrect data, resulting in inaccurate output. Incorrect output may be easily recognizable, such as a job crashing or reporting that intended information is missing. In other scenarios, errors are more difficult to detect, where incorrect data may be written to a physical dataset and used in subsequent processing without any indication that the data was corrupted by the error. Finding and correcting an error can be time-consuming and costly, as the incorrect data propagates through the enterprise and by the time it is recognized, many datasets may be corrupted. Furthermore, migration from one data store to another is costly and time-consuming, as it requires identifying all physical datasets affected by the change and then manually editing the applications that use and test them.

[0117] The inventors have developed a technique that avoids these problems by automatically providing access to the appropriate physical data sets without the need to maintain the application / dataflow graph to adapt to changes in data storage. By enabling a data processing system to adapt to changes in data storage, the risk of errors introduced when modifying an application is greatly reduced, thereby eliminating the error propagation common in conventional systems.

[0118] Such access can be made possible by the dataset multiplexer 105, which automatically provides a connection between the application and the appropriate physical dataset. An application can be programmed in terms of one or more logical datasets. For example, a business user with minimal knowledge of the physical datasets (e.g., their location or format) can write an application in terms of one or more logical datasets. The dataset multiplexer 105 can maintain a catalog of datasets, where each entry in the catalog is associated with a logical dataset and provides information for accessing the physical dataset corresponding to the logical dataset, whatever data store it is stored in when the application is executed. In response to an indication that execution of the dataflow graph involves an operation on a logical dataset, the dataset multiplexer 105 can obtain information for accessing the physical dataset from the catalog entry associated with the logical dataset and automatically provide a connection between the dataflow graph and the physical dataset based on the information. In some embodiments, the information for accessing the physical dataset can include a program that provides access to the physical dataset. The program, when executed by the application, can access the physical dataset from the data store and convert it to the format of the logical dataset.

[0119] FIG. 3B illustrates generally how the input node 302 of FIG. 3A is configured or programmed in terms of logical datasets. The input node 302 may be configured to represent a particular logical dataset specified via user input provided via a user interface. For example, the user input may be provided via a user interface 315. A list 370 of logical datasets available for configuration of the input and output nodes of the dataflow graph may be provided in the user interface 315. The logical datasets available for configuration of the input and output nodes may be logical datasets for which there are entries in a catalog of datasets. A user may browse the list to select a particular logical dataset for configuration of the input node 302, and a user may enter a search query via a user interface element 372, where the user may specify one or more values ​​of dimensions that describe the logical dataset or logical entity. Those dimensions may include words entered into the repository to describe the logical dataset or fields contained within the logical dataset. FIG. 3B illustrates that the “loyalty” logical dataset 375 is selected by the user and the input node 302 is configured to represent this selected logical dataset.

[0120] Co-pending application entitled "Data Processing System with Manipulation of Logical Dataset Groups," assigned attorney docket number A1041.70070US02, describes various search interfaces by which a user may search for datasets and / or groups of datasets as targets for manipulation. The interfaces and techniques described in this co-pending application may be used in the data processing systems described herein for purposes of constructing components of applications.

[0121] The catalog of the dataset may include an entry for the selected logical dataset that provides information for accessing a physical dataset corresponding to the selected logical dataset. This information may be or may include a program for accessing the physical dataset. When execution of an application involves an operation on a selected logical dataset, the dataset multiplexer may utilize the appropriate data catalog information to provide access to the physical dataset. For example, an identifier associated with the selected logical dataset may be used to identify an appropriate entry in the catalog of the dataset that includes a program, and the program may be executed to access the physical dataset from the data store. The dataset multiplexer may expose a link to the program such that access to the physical dataset is achieved by execution of the program on the link.

[0122] Fig. 3C shows a schematic of how such a connection can be made using a catalog of data sets. This diagram shows the application 106-3 described above in relation to Fig. 3B. As shown in Fig. 3C, when the programs are executed, the input nodes 302, 304 and output node 314 of Fig. 3B are replaced by programs that provide access to the physical data sets corresponding to the logical data sets for which those components were configured. For example, the input nodes 302, 304 are replaced by programs 330, 340 that provide access to each physical data set in the data store in which they are currently stored. Also, the output node 314 may be replaced by program 350, which represents a program that provides access to each physical data set to which data will be written in the data store in which the data currently resides. These programs may also translate between the format of the logical data sets for which the application is programmed and the format of storage of the physical data sets in the data store.

[0123] Representative Data Set Catalogs The catalog of datasets 107 may include multiple objects, each object storing information associated with a logical dataset. In this context, an object refers to a collection of information stored on a computer-readable medium that captures information related to a logical dataset. The information may be stored in any suitable format. For example, the information may be stored in a contiguous block of computer memory, distributed across multiple locations in computer memory, stored in a single file or other data structure, distributed across multiple data structures, or otherwise stored in a manner that allows the information reflected in the object to be related to the logical dataset.

[0124] Objects may be related to logical datasets in any suitable manner. Objects may have a predefined format, including information that may be formatted as a header that identifies the logical and / or physical datasets to which the information relates. However, the information may be formatted in other than a header. For example, a catalog may store a list of pointers to objects indexed by logical dataset identifiers such that by accessing a pointer that has a particular logical dataset identifier as an index, a computer accessing the catalog can find an object associated with that logical dataset as the target of the pointer. Alternatively or additionally, some or all of the catalog information regarding logical datasets may be stored as an addendum to a repository of information that may otherwise be present in the data processing system. For example, a data processing system may include a repository of metadata related to logical and / or physical datasets. The catalog information may be added to this repository and / or stored in a separate metadata repository.

[0125] Information about the logical dataset may be reflected in the object in any suitable format. For example, the information may be stored as one or more descriptors, each having a value. Alternatively or additionally, the information may be stored as or include computer executable instructions. In some embodiments, a physical dataset may be reflected in an object because a program stored with the object to access the physical dataset is hard-coded to access that physical dataset. In other embodiments, information identifying a physical dataset that corresponds to a logical dataset may be stored as a value of a field in a data structure that stores the object. That value may be passed as a run-time parameter to a program stored with the object to access the physical dataset, or may otherwise be used to access the physical dataset.

[0126] Figure 4 illustrates an example object 400 in the catalogue of datasets 107 maintained by the dataset multiplexer 105. Although Figure 4 illustrates various information captured in the object 400, some of that information, such as discovery information 406 and / or access information 408, may be optional.

[0127] The information captured in the object 400 may include information to identify a physical dataset that corresponds to a logical dataset. In this example, the object is identified by a logical dataset identifier 404.

[0128] The information reflected in the object 400 may be or may include an executable program 402 for accessing a physical dataset. When executed, the program may access a physical dataset that corresponds to a logical dataset and convert data in the physical dataset to the format of the logical dataset, or vice versa. The program may be reflected in the catalog object by storing a copy of the program's computer executable instructions in computer memory allocated for the object. In other embodiments, the program may be stored elsewhere, with only a pointer to the program or other identifier of the program stored in computer memory allocated for the object.

[0129] In some embodiments, the program may be created using discovery information 406 identified during the registration process of the physical dataset and / or access information 408 used to access the physical dataset.

[0130] The objects may reflect information about the physical data sources that store the corresponding physical datasets that allows access to and transformation of the data in the physical datasets. The information may be obtained in any of a number of ways, such as through user input or through an automatic discovery process that occurs by reading data or metadata from the data sources that store the physical datasets. In some embodiments, the discovery information 406 may be automatically discovered as part of the registration process of the physical datasets with the dataset multiplexer 105. As part of the registration process, a user may specify the logical datasets to which the physical datasets correspond, or the correspondence between the logical datasets and the physical datasets may be determined in another suitable manner. The automatically discovered information may include a physical identifier associated with the data store and / or physical dataset, a reference to the storage location of the data store and / or physical dataset, the type of data store, the record format or schema of the physical dataset, and / or other information.

[0131] In some embodiments, a copy of this discovery information may be stored in the object. In other embodiments, the discovery information 406 may be reflected in the object for use in creating programs to access the physical datasets stored as part of the object. For example, data store and / or physical dataset type and format information may be used to create programs having conversion logic to convert data in the physical datasets to the format of the logical datasets.

[0132] The access information may include parameters 408 that may specify how to access the physical datasets and / or data stores. In some embodiments, these parameters may be design-time parameters and / or run-time parameters. Design-time parameters may be applied to specify the functionality of the program 402. Values ​​of these parameters do not need to be stored separately in the object 400 because the program is generated based on the design-time parameters. In the case of run-time parameters, their values ​​may be stored in the object and provided as input to the program at run-time.

[0133] The parameters 408 may include one or more parameters that specify the type of access to the physical data set. In some embodiments, the type of access may indicate read access or write access. In other embodiments, the type of access may indicate the amount of bandwidth allocated for access of a particular logical data set. For example, the value of the parameters 408 may indicate dedicated access or shared access. The data store may support several connections to applications 106 that, as a whole, may use no more than a predetermined amount of bandwidth to access the data store. Allocation techniques may be applied to allow applications performing higher priority tasks to use more of the total available bandwidth of the data source. As a specific example, a data source may support dedicated and shared access, where dedicated access of an application results in the application being allocated more of the available bandwidth than if shared access were provided. By specifying dedicated access for logical data sets used by high priority applications and shared access for logical data sets used by low priority applications, the bandwidth available at the data source may be allocated as desired.

[0134] As another example, the access parameters may alternatively or additionally indicate the type of connection (e.g., a high-speed connection or a low-speed connection) used to access the data store that holds the physical dataset that corresponds to the logical dataset.

[0135] As yet another example, parameters 408 may include one or more parameters that specify security related information. In some embodiments, one or more parameters may indicate whether the data in the physical data set is encrypted. In embodiments where the data is encrypted, parameters 408 may include information such as a security key to decrypt or otherwise make the information usable. For added security, the security key may be provided by application 106 at run time and may not be stored in the catalog of data sets 107. In other embodiments, one or more parameters may indicate whether the data in the physical data set is compressed. In embodiments where parameters 408 are used to create program 402, a value of parameters 408 that indicates that the data in the physical data set is encrypted may be used to include decryption logic in the program.

[0136] As a further example, parameters 408 may include one or more parameters that specify criteria for a filter operation. For example, one or more parameters may specify a date that may be used to filter information when accessing the physical data set.

[0137] In some embodiments, some or all of the values ​​of parameters 408 may be automatically discovered. This automatic discovery process may occur when the physical dataset is registered with a component of a data processing system that creates a dataset catalog. During the discovery process, for example, the component of the data processing system may access metadata in the data store to determine information reflected in the object. Alternatively or additionally, the component of the data processing system may analyze data read from the physical dataset to recognize patterns in the data that are indicative of record format, encryption, compression, or other information about the physical data store.

[0138] However, it should be understood that discovery information 406 may be obtained by other than direct interaction with the data source, such as retrieval from a repository of metadata associated with logical and / or physical datasets maintained by the data processing system. For example, security information, such as encryption or compression, may be applicable to all datasets in a data store. Once security information is stored anywhere in the system for one physical dataset in a data store, the security information may be reflected in objects used to access other physical datasets in the same data store.

[0139] Some or all of the information reflected in the object may be shown as discovered in the example of FIG. 4 or may be entered by a user. In other embodiments, some of the discovery information 406 and / or access information 408 may be specified by a user via a user interface as part of the registration process. However, it should be understood that user input may be provided in other ways, such as when defining a logical dataset. As a specific example, the priority of a logical dataset may be specified when the logical dataset is defined, or may be specified after the logical dataset is defined by editing the metadata stored for that logical dataset.

[0140] It should further be appreciated that FIG. 4 illustrates objects configured to access a physical dataset associated with a logical dataset at a given time. The data processing system may detect events that affect the storage of data associated with a logical dataset. In that case, the objects of that logical dataset may be updated. For example, whenever a change in a parameter is detected, the value of any of the parameters may be updated. Alternatively or additionally, when a new physical dataset is registered and input indicates that the physical dataset stores data of a logical dataset whose object already exists in the catalog, the objects of that logical dataset may be modified. The modification may be implemented, for example, by overwriting all or part of the object with new information or replacing it with a new object to reflect the new physical dataset. However, the objects of the logical dataset may still be accessed via the dataset catalog. Thus, when an application written to perform data access operations based on a logical dataset is configured to access a physical dataset corresponding to the logical dataset via the dataset catalog, it will continue to correctly access the correct physical dataset despite any changes.

[0141] In some embodiments, the program 402 may be configured as an executable dataflow graph that includes logic for accessing a physical data set. As described above in connection with Figures 3A-3C, in embodiments in which an application is developed as a graph, the program 402 may be configured as a subgraph, in the sense that it executes as part of the dataflow graph that implements the application. For example, Figure 3C shows a first program 330 configured as a subgraph that includes logic for accessing an input data set, a second program 340 configured as a subgraph that includes logic for accessing an input data set, and a third program 350 configured as a subgraph that includes logic for accessing an output data set.

[0142] These subgraphs may be considered dynamic subgraphs (DSGs) because the subgraphs are updated from time to time based on events that indicate changes in the appropriate mechanisms for data access to the storage associated with the logical dataset. Thus, use of the subgraph's data access operations within an application dynamically accesses the physical dataset that stores the correct data at that time. Thus, a DSG is used herein as an example of a program 402.

[0143] A typical dataset multiplexer with a dataset catalog Figure 5A is a block diagram highlighting components of the dataset multiplexer 105 of the data processing system 104. As shown in Figure 5, the dataset multiplexer 105 includes, among other components, a registration module 520, a dynamic subgraph (DSG) generator 524, a metadata management module 526, an operational metadata module 528, a catalog service interface 522, and a user interface 530.

[0144] In some embodiments, the registration module 520 is configured to register the physical dataset with the dataset multiplexer 105. The registration may be triggered by the addition of the physical dataset to the IT infrastructure or by the use of the physical dataset from an application. Alternatively or additionally, the registration module 520 may receive a command to register the physical dataset via the user interface 530. For example, a user may provide input via the user interface 530 to initiate the registration process of the physical dataset. The input may be in the form of a direct command to register the physical dataset.

[0145] Alternatively or additionally, the input may indirectly indicate that registration is to be initiated. For example, registration may be triggered when a user writing an application selects a logical dataset associated with a physical dataset for which the dataset catalog does not have information or for which the information in the catalog is not up-to-date. Other actions that serve as indirect commands may include instructions to migrate a physical dataset from one data store to another, or commands to modify metadata associated with a logical dataset that may affect the transformation between the physical dataset and the logical dataset. Regardless of how the registration process is triggered, the user input may specify a logical dataset that corresponds to the physical dataset so that objects in the catalog for the logical dataset may be created or overwritten with the latest information.

[0146] Other information for creating or updating an object in the catalog may be collected from one or more sources. During the registration process, the registration module 520 may discover information about the physical dataset and / or the data store in which it is stored. The information collected in this manner may include the type of data store, the record format or schema of the physical dataset, the physical storage location of the data store, compression and / or encryption status, and / or other information.

[0147] The registration module 520 may provide the acquired information to the DSG generator 524. The DSG generator 524 may create a DSG based on the received information. The DSG generator 524 may have access to several program templates, each program template corresponding to a particular type of data store. The DSG generator 524 may detect the type of data store from the received information and select an appropriate program template corresponding to the detected type from among the several program templates. For example, a data processing system may be pre-configured with templates for read and / or write access to data tables of an ORACLE database or a HADOOP distributed database. Detecting the type of data store that stores the physical dataset may enable the DSG generator 524 to select an appropriate template for access to the physical dataset corresponding to the logical dataset for which the DSG is being created.

[0148] The DSG generator 524 may generate a program based on the selected program template. The DSG generator 524 may detect values ​​of parameters of the selected program template from the received information and populate the program template with the detected values. Some or all of the parameter values ​​may alternatively or additionally be obtained from a metadata management module 526, which in this example may maintain metadata for the physical datasets, data stores, and / or logical datasets. The parameters may alternatively or additionally be provided via user input using a user interface 530 or may be obtained in other ways.

[0149] The DSG generator 524 generates a DSG that includes access logic for accessing the physical dataset and conversion logic for converting between the format of the physical dataset and the format of the corresponding logical dataset. The DSG generator 524 may generate a logical-to-physical layer mapping for the physical dataset and the corresponding logical dataset. The DSG generator 524 may generate a mapping between one or more fields of the logical dataset and one or more fields of the physical dataset that represent the same information. The mapping may be generated using information from various sources, including information available in the data processing system, user input, and / or information derived by semantic discovery. The DSG generator 524 may use the mapping to generate the conversion logic. For example, a customer name in the physical dataset may be stored in one row of a data table as three fields that hold data corresponding to the customer's first name, middle initial, and last name, respectively. However, the logical dataset may simply include a logical entity Customer_Name. The DSG generator 524 may generate a mapping between these three fields of the physical dataset and the logical entities of the logical dataset. The transformation logic may include logic to transform between the "customer first name, middle initial, and last name" format of the physical dataset and the "Customer_Name" format of the logical entity. When the DSG is executed, the access logic is executed to obtain information from the three fields of the physical dataset and the transformation logic is executed to transform between the format of the physical dataset and the format of the logical dataset.

[0150] In some embodiments, DSG generator 524 creates a DSG for each of multiple physical datasets in the data store. The created DSGs may be included in the catalog of datasets 107. The catalog of datasets 107 may include objects associated with logical datasets, and each object may be or include a DSG for accessing the physical dataset corresponding to the logical dataset.

[0151] The registration module 520 may also provide discovery information to the metadata management module 526 so that the metadata management module 526 may receive and maintain metadata for the physical datasets and / or data stores. In some embodiments, the metadata management module 526 may be an information source for the dynamic subgraph generator 524 in generating the DSG and may additionally store metadata about the dataset that may be used in other operations involving the dataset in the data processing system. The metadata management module 526 may maintain, for example, information that serves as metadata about the logical dataset, information about logical entities within the logical dataset, relationships between the logical entities of the dataset, and relationships to other logical datasets and / or entities of other logical datasets.

[0152] The metadata management module 526 may also store mappings between logical and physical datasets, which may be based on user input or, in some embodiments, derived, such as by monitoring operations where a user has directly or indirectly specified an association between a logical dataset and a physical dataset as part of a data processing operation. Regardless of how obtained, in some embodiments, the metadata management module 526 may maintain a table or other data structure that maps identifiers of logical datasets to identifiers of physical datasets that correspond to the logical datasets. This information may be used by the dynamic subgraph generator 524 when creating objects representing the logical datasets and / or when determining that previously created objects require updating because the storage of data associated with the logical dataset has changed.

[0153] The metadata management module 526 can maintain a list of logical datasets known to the data processing system 104. When programming an application in terms of logical datasets, the list of known logical datasets may be presented to a user via the application's user interface, and the user can select a particular logical dataset from the presented list. This logical information maintained by the metadata management module 526 can be used, for example, to allow a user to search for a particular logical dataset to use in writing an application. Information about the physical datasets, including their correspondence to logical datasets (which may also be stored by the metadata management module 526), ​​can also be used in searching for an appropriate dataset. For example, this logical and physical information can be used to define dimensions for faceted searches of datasets.

[0154] The data processing system may maintain other types of metadata about datasets, which may also be available to users searching for datasets for particular scenarios. For example, metadata about the use of a dataset may be captured and stored as the dataset is used. This operational metadata may also be used by dataset discovery tools to allow users to search for datasets based on how they have been used by other users.

[0155] The operational metadata module 528 can collect operational metadata about a dataset. The operational metadata may be collected during or after the execution of an application or other program that accesses the dataset. The operational metadata collected during execution may include identification information about the physical dataset accessed, the date and time of access, whether an update to the dataset was performed, the values ​​of parameters associated with the execution of one or more subgraphs that accessed the dataset, and / or other operational data. The operational metadata collected or determined after execution may include information about how often a physical or logical dataset was accessed, how recently it was accessed, or the size of the data accessed (e.g., the number of records read and / or written). Part of the operational metadata may be social information, such as information about the user who created the dataset or the users who accessed the dataset. This social information may include the user's role in the enterprise, the permissions provided to the user, and / or other information about people in the enterprise.

[0156] In the example of FIG. 5A, the catalog service interface 522 integrates access to various types of metadata about datasets. It may provide a faceted search tool that allows, for example, searching any of several facets that may be present in any of the logical, physical, and / or operational metadata that may be stored for the physical and / or logical datasets that a user may want to select when writing an application or specifying an operation to be performed on the dataset. The facets in the search may be based on information about the logical dataset, the physical dataset, and / or the operational metadata stored in the data processing system. For example, a search for datasets may be conditioned to return only datasets for which an entry / object exists in the dataset catalog. This facet may be combined with other facets related to the logical or physical datasets to provide a powerful search interface. For example, a search query may be conditioned to return only datasets that have been accessed within the last week and only logical datasets with an email field whose corresponding physical dataset is stored in a rapidly accessible data store.

[0157] While FIG. 5A shows separate modules managing different types of metadata, it should be understood that this depiction is separated by functionality, and that hardware and / or software components that capture and / or provide multiple types of metadata may be divided in other manners, including integrating all such metadata capture and management into a single module, or into more modules than are shown.

[0158] The catalog service interface 522 also allows applications 106 to be programmed in terms of logical datasets. Once a user selects a logical dataset for programming an application, the catalog service interface 522 can provide information that allows an application written in terms of that logical dataset to access the appropriate physical datasets. The catalog service interface 522 may access a catalog of datasets 107, where each object in the catalog corresponds to a logical dataset and provides information for accessing the physical datasets that correspond to the logical dataset. A catalog object may be or include a program (shown in this example as DSG) for accessing the physical dataset that corresponds to the logical dataset.

[0159] The catalog service interface 522 can enable an application to access a physical dataset by providing information about a program within a logical dataset object selected in the catalog of datasets 107. When an operation is performed to access a logical dataset from within an application, the application can use the information to access the corresponding physical dataset in the data store. In this manner, a program identified from a catalog object can be executed to access a physical dataset from the data store. For example, the catalog service interface 522 can expose a link to a DSG, which a development environment in which an application is being developed can use to build the application such that access to the physical dataset is achieved by executing the DSG on the link when the application is run. In some embodiments, the catalog service interface 522 provides this link via an application programming interface (API).

[0160] As described above, a catalog object associated with a logical dataset, and therefore the DSG of that object, may be updated in response to an event indicating a change in the storage of information associated with the logical dataset. For example, a physical dataset corresponding to a logical dataset may be migrated from one data store to another. The catalog object of the logical dataset may be updated to account for this change. In some embodiments, programs for accessing the physical dataset may be modified so that the application accesses the physical dataset from the correct data store. By updating the catalog object of the logical dataset, applications written to access the logical dataset continue to operate without modification even if the physical dataset migrates from one data store to another. Such dynamic updates are described in more detail below with reference to FIGS. 6A-6B.

[0161] Other events that do not necessarily have to be associated with the location of the physical dataset may result in changes to objects in the dataset catalog. For example, in response to an event indicating a change in the format of the physical dataset, the appropriate catalog object may be updated. If the format of the physical dataset is changed, for example, by adding a field to the dataset, the corresponding catalog object may be updated to account for the added field. In some embodiments, the translation logic in the program for accessing the physical dataset may be modified to account for this change. As another example, in response to an event indicating a change in the value of a parameter used to generate the program or accessed in the program, the value of the parameter stored in the catalog object may be updated and / or the program may be regenerated with the new value. As yet another example, an event indicating a change associated with a physical dataset corresponding to a logical dataset may include an event indicating the replacement of a physical dataset with another physical dataset corresponding to the same logical dataset. In this example, a catalog object corresponding to a first physical dataset may be replaced or substituted with a catalog object corresponding to the other physical dataset. These changes may be implemented by the dynamic subgraph generator 524, which may be triggered to update the catalog object upon detection of the event. The updates may be performed, for example, by fully or partially overwriting the memory locations storing the catalog objects, or by associating objects stored in other memory locations with catalog entries of the dataset such that the catalog objects of a particular catalog entry are updated when the catalog object is replaced by a new object. The trigger for such a change may be provided by user input or may be automatically detected by the dynamic subgraph generator 524, the catalog services interface 522, or other components of the data processing system.

[0162] When an application written in terms of a logical dataset is executed and the dataset catalog 107 is accessed to provide the application with access to a physical dataset corresponding to the logical dataset, it should be understood that one or more components such as the registration module 520, the dynamic subgraph generator 524, the metadata management module 526, the operation metadata module 528, and / or the user interface 530 may be optional, as shown in FIG. 5B. When an operation is executed to access a logical dataset from within an application, the application may obtain information about the DSG associated with the logical dataset from the data catalog 107 via the catalog service interface 522 based on an identifier associated with the logical dataset. In some embodiments, the catalog service interface 522 may provide this information to the application by exposing a link to the DSG. The DSG, when executed, provides the application with access to the physical dataset corresponding to the logical dataset.

[0163] Typical techniques for updating dataset catalog objects Objects in the data catalog may be used to perform data access operations in applications programmed in terms of logical data sets. The catalog objects may be updated in response to events such that appropriate data access is provided by using the current information in the objects as the application executes. One such event is a change in the storage location of a physical data set, as shown in Figures 6A and 6B. Figure 6A is a block diagram of an exemplary enterprise IT system such as that shown in Figures 1A or 5A, in which the data processing system is in an operational state at a first time facilitating access between applications 106-1, 106-3 and data stores 102-1 and 102-2.

[0164] The application 106-3 may be developed as a dataflow graph in a development environment that uses information from the dataset catalog to implement references to logical datasets in the application's specification. The components 330 and 340 of the application 106-3, which represent input nodes of the dataflow graph, may be programmed in terms of logical datasets, and information stored in computer memory for execution of the application includes, for those components, links to catalog objects corresponding to the logical datasets. For example, component 330 may be linked to a catalog object corresponding to a first logical dataset, and component 340 may be linked to a catalog object corresponding to a second logical dataset. The links may be stored in any format that conveys sufficient information in the objects required to access the physical datasets corresponding to the logical datasets referenced in those components. For example, the links may be stored as identifiers of the objects or as paths through a directory structure to files that store programs for accessing the physical datasets.

[0165] Application 106-1 may also be developed as a dataflow graph. Components 610 and 620 of application 106-1, which represent input nodes of the dataflow graph, may be programmed in terms of logical data sets, with the components linked to catalog objects corresponding to the logical data sets. For example, component 610 may be linked to a catalog object corresponding to a first logical data set, and component 620 may be linked to a catalog object corresponding to a third logical data set.

[0166] As shown in FIG. 6A, component 330 of application 106-3 and component 610 of application 106-1 may be programmed in terms of the same logical data set and may be linked to the same catalog object in catalog 107 of data sets.

[0167] The data processing system 104 may maintain a catalog of datasets 107 that includes catalog objects that correspond to logical datasets. Each catalog object may be or include a DSG for accessing a physical dataset that corresponds to the logical dataset. As shown in FIG. 6A, the catalog of datasets includes a first set of DSGs, where each DSG in the first set is programmed to access a physical dataset from data source 102-2. The catalog of datasets 107 also includes a second set of DSGs, where each DSG in the second set is programmed to access a physical dataset from data source 102-1.

[0168] Data processing system 104 enables applications 106-3 and 106-1 to access physical datasets from data stores 102-2 and 102-1 based on their respective programmed logical datasets using information in catalog of datasets 107. When programming application 106-3, a user may select a first logical dataset, such as from a list of known logical datasets, and associate that logical dataset with component 330, and a second logical dataset associated with component 340. Similarly, when programming application 106-1, a user may select a first logical dataset to associate with component 610 and a third logical dataset to associate with component 620.

[0169] When an operation is performed to access a logical dataset associated with component 330, data processing system 104 may select the DSG linked to component 330. When an operation is performed to access a logical dataset associated with component 340, data processing system 104 may select the DSG linked to component 340. When an operation is performed to access a logical dataset associated with component 610, data processing system 104 may select the DSG linked to component 610. When an operation is performed to access a logical dataset associated with component 620, data processing system 104 may select the DSG linked to component 620.

[0170] FIG. 6B is a block diagram of the example data processing system of FIG. 1A or FIG. 5A in an operational state at a second time when the data processing system facilitates access between applications 106-1, 106-3 and data stores 102-1, 102-1' when the physical data set of data store 102-1 is migrated to data store 102-1'.

[0171] The migration of the physical dataset from data store 102-1 to data store 102-1' in this example is an event that causes data processing system 104 to update catalog of datasets 107. The objects in catalog of datasets 107 that correspond to the logical datasets mapped to the physical datasets in data store 102-1 may be updated to take into account the change in the data store. With this update, the second set of DSGs may be modified to access the physical datasets from data store 102-1' instead of data store 102-1. As shown in FIG. 6B, the links between applications 106-3, 106-1 and catalog of datasets 107 remain unchanged and applications 106-3, 106-1 continue to operate despite the changes to the physical storage of the datasets. Nonetheless, execution of operations within the applications that specify access of the logical datasets results in access to the physical datasets in the updated location.

[0172] A typical application configured for data access through a Dataset Catalog object 7 is a block diagram illustrating various information maintained by the dataset multiplexer 105. This information may allow applications 106-2 to be configured to access physical datasets based on programmed logical datasets. Once configured, this information may also be recorded as a result of the execution of the applications. This recorded information may provide operational metadata for other functions performed by the data processing system, including providing a search interface that allows a user to later search for datasets for use by an application based on previous operations on the datasets.

[0173] In this example, application 106-2 is written to read data from a dataset containing information about customers. It then extracts records from that dataset that represent preferred customers and writes the results to a second dataset. When executed, application 106-2 reads from and writes to a physical dataset. However, application 106-2 may also be programmed in terms of a first logical dataset associated with input data store 710 and a second logical dataset associated with output data store 720.

[0174] When application 106-2 is being written, a user may provide configuration input for input data store 710 that specifies the logical dataset from which data is to be read. In this example, the logical dataset is identified as "abbott.customers." The dataset may be selected by user input, such as by selecting from a list of all logical datasets registered with the data processing system, or by selecting from a limited list returned in response to a user query for a dataset having user-specified parameters. Such a selection interface may be provided by the development environment of application 106-2.

[0175] Similarly, the output data store 720 may be comprised of a logical dataset, in this example, identified as "abbott.preferred-cust."

[0176] To enable execution of the application, the development environment can associate the selected logical dataset with information that enables read and write operations to be performed on the physical dataset corresponding to the specified logical dataset at the time the application is executed. This can be done, for example, by obtaining the information via catalog service interface 522 (FIG. 5A). Catalog service interface 522 can provide information about programs that are maintained to access the physical dataset corresponding to the particular logical dataset at the time the program is executed, for example in response to a request for catalog information related to the logical dataset. In this example, the information about the program is provided as a path in a directory structure to a file that stores the program. In this example, a link to a program for accessing a physical dataset corresponding to the input logical dataset "abbott.customers" is stored in the path "common20 / abbott / customers / DSG". However, the link to the program may be provided in any suitable format.

[0177] Similarly, a program for accessing a physical dataset corresponding to the output logical dataset "abbott.preferred-cust" is obtained. In this example, the path is "common10 / abbott / preferred-cust / DSG". These links to programs that can access the physical datasets may be exposed by the catalog service interface 522 during execution of the application. These links may be stored as part of the computer-executable representation of the application such that the programs can be executed upon execution of operations within the application that access these datasets. Alternatively, sufficient information to execute a program to access a physical dataset may be obtained at any time prior to execution of operations that access the data source, including during execution of the application.

[0178] Regardless of when information about a program that provides access to a physical dataset is identified in connection with the execution of an application, the dataset multiplexer 105 can provide information about that program. FIG. 7 shows that the dataset multiplexer 105 maintains sufficient information to associate a logical dataset with a program for accessing the physical dataset corresponding to that logical dataset. This information may be stored, for example, as a dataset catalog object for the logical dataset. In some embodiments, this information may be fetched or provided by the dataset multiplexer 105 at run-time or design / build time of the application. Doing so at design / build time can avoid adding time costs and / or dependencies to run-time operations.

[0179] In the example of Figure 7, the information is shown stored as two relationships. The physical identifiers of the physical datasets are used as a key to link information 702, 704, and 706. First, information 702 provides information that links each logical dataset, by the logical ID used for that logical dataset, to the identifier of the physical dataset that currently stores the data corresponding to that logical dataset. Second, information 704 provides the relationship between the physical dataset and the program that can be used to access it.

[0180] 7, information 702 links the logical dataset "abbott.customers" to a physical dataset identified by identifier "123." A program at the path "common20 / abbott / customers / DSG" is associated, via information 704, with the physical dataset having identifier "123."

[0181] Similarly, the logical dataset “abbott.preferred-cust” is related to the physical dataset ID “247” by information 702. Also, the program at the path “common10 / abbott / preferred-cust / DSG” is related to the physical dataset 247 by information 704.

[0182] Similar information may be maintained by the dataset multiplexer, such as in a dataset catalog object, for each logical dataset that has a corresponding physical dataset registered. Alternatively, or in addition, some or all of this information may be maintained by the metadata management module 526 or other modules in the data processing system. Regardless of how the information is maintained, the dataset multiplexer 105 may provide information regarding programs that access the physical datasets that correspond to a logical dataset.

[0183] In the example of Figure 7, the identified program in the path "common20 / abbott / customers / DSG" is saved to the specified input data store 710 location as DSG 715 along with information used to invoke it. DSG 715 may be referred to as the "read DSG" that reads data from the physical dataset corresponding to the input logical dataset "abbott.customers". Similarly, the program in the path "common10 / abbott / preferred-cust / DSG" is saved to the specified output data store 720 location as DSG 725 along with information used to invoke it. DSG 725 may be referred to as the "write DSG" that writes data to the physical dataset corresponding to the output logical dataset "abbott.preferred-cust".

[0184] Information indicative of programs executed within an application may be stored together with the program instructions that make up the application. In a scenario where an application is written as a dataflow graph and programs for accessing data sources are written as subgraphs, these subgraphs may be dynamically linked to the dataflow graph at appropriate locations in the dataflow graph for execution. This location may correspond to input and / or output nodes of the dataflow graph. During or immediately prior to execution of the dataflow graph, link or path information of the subgraphs exposed by or obtained from the catalog service interface 522 may be provided to the input and / or output nodes, and the corresponding subgraphs may be linked and / or stored in place of the input and / or output nodes. Exemplary techniques for dynamically linking subgraphs to dataflow graphs via subgraph interfaces may be used, as described in U.S. Pat. No. 10,180,821, entitled "Managing Interfaces for Sub-Graphs," which is incorporated herein in its entirety. However, other methods of storing information for executing programs may alternatively or additionally be used.

[0185] When application 106-2 is executed and an operation occurs that accesses a logical dataset associated with input data store 710, linked DSG 715 may be invoked. Invocation of DSG 715 may result in execution of its access and conversion logic. When executed, input data store 710 may be accessed and data from the input data store and / or the input data store's corresponding physical dataset may be read and converted to the format of the logical dataset. Invocation of the DSG may require providing parameters to a controller module (not shown) in the data processing system.

[0186] 7, the parameters provided for execution of DSG 715 are shown as parameters 730. In this example, one of the parameters 730 identifies the DSG, such as by providing its path. The value of this parameter may be saved at the time the input data source 710 is configured for a particular logical data set.

[0187] Other parameters 730 may be provided such that they may be supplied by the controller module to the DSG 715 for execution. These run-time parameters (i.e., supplied at run-time) may affect the execution of the DSG. For example, values ​​of parameters "Param1" and "Param2" may be supplied to the DSG at run-time. The value of one such parameter may specify, for example, that the DSG 715 should be executed in a particular read mode (single record, batch, quick, shared, etc.). The value of a parameter may reflect, as another example, the access priority of an application.

[0188] The values ​​of these run-time parameters may be obtained in one or more ways. They may be encoded in the application 106-2, for example, based on input provided by a user during development of the application. For example, the values ​​of the parameters may be derived from information entered as configuration parameters of the input data sources 710 in the development environment. As another example, the values ​​of the parameters may alternatively or additionally be derived from other user input during development of the application or in response to prompts at run-time. As yet another example, the application may identify the values ​​of the parameters at run-time from various inputs, such as external inputs indicating the time of day, the current system load, or other inputs that depend on data provided as inputs to the dataflow graph.

[0189] As yet another example, the values ​​of the parameters may alternatively or additionally be obtained from other modules. As a particular example, the values ​​of at least some of the parameters 730 may be read from or obtained by processing information in a metadata repository that stores information about the logical datasets associated with the input data store 710. As yet another example, the values ​​of at least some of the parameters 730 may be read from or obtained by processing information in an access control module that maintains information about users, and may reflect access priorities or mechanisms to the data store that are set based on the role of the user that developed the application or the user that is running the application.

[0190] Values ​​of other parameters of input data source parameters 730 may be included to enable the controller module, or other components of the data processing system, to capture operational metadata. For example, a logical identifier of the data set to which the access is encoded may be stored, for example, for this reason. Similarly, an identifier of the physical data set being accessed may be stored. The value of this parameter may be provided by the data set multiplexer, such as from information 702 that is current at run time. Capturing such information may enable, for example, operational metadata module 528 (FIG. 5A) to provide information that supports additional facets of searching the data.

[0191] In the example of FIG. 7, the dataset multiplexer 105 is shown to store information 706 collected during the execution of the application 106-2. For example, the information 706 may include the date the dataset was accessed, the size of the dataset at the time it was accessed, and / or the amount of data read into or written from the dataset, the host ID of the computer hardware involved in the data access, such as by execution of the application or accessing program or by physical storage of the data. Other portions of the information 706 may indicate the logical dataset associated with the output data store 720, the physical dataset accessed, the values ​​of parameters such as "Param1" and "Param2" supplied to the program, when the physical dataset was accessed, and / or other information. Such entries may be stored for each access to the dataset, or for some number of previous accesses to the dataset, or for a predetermined time after the dataset is accessed. This information can be analyzed after execution to determine other operational parameters, such as frequency of use or recency / freshness of the dataset.

[0192] Similar information may be stored for the output data store 720. When an operation is executed that accesses a logical dataset associated with the output data store 720, the linked DSG 725 may be invoked. The invocation of the DSG 725 may result in the execution of its access and conversion logic. When executed, the output data store 720 may be accessed and data may be written to the output data store after being converted from the format of the logical dataset to the format of the output data store and / or the format of the corresponding physical dataset of the output data store. Parameters 740 represent parameters whose values ​​may be provided to the controller module and utilized by the DSG 725 during execution. Although not shown in FIG. 7, entries in the repository of operational metadata may similarly be created based on the access of the physical dataset corresponding to the output data store 720.

[0193] A typical way to register a dataset in the Dataset Catalog 8 is a flow chart of an example process 800 for registering a physical dataset in a dataset catalog such that the physical dataset may be accessed by applications configured to access a logical dataset corresponding to the physical dataset. Process 800 may be performed by data processing system 104, such as in dataset multiplexer 105 described with reference to FIGS. 1A-1C. Process 800 may alternatively or additionally include other acts, including acts as described elsewhere herein with respect to other embodiments.

[0194] Process 800 may be initiated (801) in response to a detected event. The event may be an indication that there is no catalog entry in a dataset catalog that provides access to a physical dataset in an IT system that corresponds to a logical dataset defined in the data processing system. The detected event may be an automatic detection of a physical dataset that exists in the IT system that does not yet have a catalog entry. Such an indication may be in the form of a user input, such as, for example, a user-input command to cause the data processing system to register the physical dataset as corresponding to a logical dataset. Alternatively, the event may be an indication that a catalog entry in a dataset catalog that provides access to a physical dataset in an IT system is out of date. However, other events (including other events described herein) may trigger execution of process 800. For example, a new physical dataset may be identified in a data store as part of running a periodic (weekly, biweekly, etc.) import feed. This identification may trigger execution of process 800.

[0195] Process 800 may proceed to act 802 where information about the physical datasets stored in the data store is obtained. The physical datasets may be the physical datasets mentioned in the context of start 801 above of process 800. In some embodiments, some of the information may be automatically discovered, such as a physical identifier associated with the data store and / or physical dataset, a reference to the storage location of the data store and / or physical dataset, a type of data store, a record format or schema of the data store and / or physical dataset, and / or other information (such as the information described in the context of FIG. 4 ).

[0196] In act 804, a logical-to-physical layer mapping may be generated for the physical dataset and the corresponding logical dataset. In some embodiments, the dataset multiplexer 105 may generate a mapping between one or more fields of the logical dataset and one or more fields of the physical dataset that represent the same information. This mapping may be generated using information from various sources, including information available within the data processing system, user input, and / or information derived by semantic discovery. For example, a field of the physical dataset where most entries contain the characters "@" and "." may be related to a field of the logical dataset called "email". This relationship may be derived by semantic discovery and used to generate the mapping. Similar relationships between fields may be specified by user input or in other ways. The mapping between the logical dataset and the physical dataset may be generated by applying these relationships. In some embodiments, information about unique keys and / or foreign keys that specify relationships between the datasets may be used to generate the mapping.

[0197] Due to these relationships, a program that accesses the physical dataset may be configured to perform the necessary mapping between the fields of the physical dataset and the fields of the logical dataset. A template for the program may be selected and then configured to implement the mapping so that both access and conversion of the data format are provided. To obtain the template, in act 806, the type of data store may be determined based on the information obtained in act 802. In act 808, a determination may be made as to whether a program template is available for the type of data store. Many data stores may have consistent access paradigms that may be captured in a template. Thus, the data processing system may store a library of templates for commonly used types of data processing systems, such as ORACLE databases or SQL Server databases.

[0198] In response to determining that a program template is available, the process proceeds to act 810 where an available program template is selected and then to act 812 where a program is generated based on the selected program template. The generation of the program may enable both access to the target physical dataset and applying the mapping generated in act 804 to convert between the data format of the logical dataset and the data format of the physical dataset.

[0199] A program for accessing the physical data set from the data store is generated in Act 812. The program may be generated by populating a selected program template based on the one or more first parameters in Act 812a and obtaining information regarding one or more other parameters in Act 812b.

[0200] In act 812a, the selected program template may be populated by identifying a value for a first parameter of the program template based on information obtained in act 802 (such as information automatically discovered during the registration process).

[0201] In Act 812b, information regarding one or more other parameters of the program template may be obtained. The one or more other parameters may specify how to access the physical dataset. For example, some information may be obtained from a metadata repository that maintains metadata for the data store. As another example, some information may be obtained via user input. For example, a user may specify information regarding a type of access or security related information. User input regarding the other parameters may be obtained during a registration process.

[0202] In some embodiments, in response to determining in act 808 that a program template is not available, the process proceeds to act 820 where a program structure used to generate a program is created. In some embodiments, the program structure may be created by prompting a user. For example, the user may provide a file that includes the program structure and / or parameter values. Then, in act 822, a program for accessing a physical data set from a data store may be generated based on the program structure entered by the user.

[0203] It will be appreciated that acts 802, 804, 806, 808, 810, 812, 820, and 822 may be performed to generate a program for accessing different physical data sets in a data store or to generate a program for accessing a physical data set in a different data store without departing from the scope of the present disclosure. For example, a first program may be generated for accessing a first physical data set in a data store and a second program may be generated for accessing a second physical data set in the data store. As another example, a first program may be generated for accessing a first physical data set in a first data store and a second program may be generated for accessing a second physical data set in a second data store different from the first data store.

[0204] Once a program is created, information for invoking execution of the program from within the programmed application in terms of the logical dataset is stored in an object in the catalog of datasets 107. The stored information may include a physical identifier for the data store or physical dataset stored in the data store, a logical identifier for the logical dataset, values ​​of parameters used when executing the program, and / or other information. In some embodiments, the object may be or include a program.

[0205] In this manner, the program generated in act 812 or 822 is available to an application that specifies access to a logical dataset that corresponds to a physical dataset. Thus, in act 814, which may optionally occur any time after registration (or not at all), the program generated in act 812 or 822 is linked to one or more applications. This linking enables an application that is programmed in terms of a logical dataset to access a physical dataset using the generated program. When an operation is performed to access the logical dataset, the linked program is executed, thereby providing access to the physical dataset that corresponds to the logical dataset.

[0206] Regardless of whether the generated program is linked to an application that accesses the logical dataset, in act 816, a determination is made as to whether an event is detected that indicates a change to the storage of data corresponding to the logical dataset. For example, the change may indicate a migration from a first data store to a second data store, a change in format of the logical dataset, or a change in format of the physical dataset. In response to detecting such an event, the process may loop back to act 802 where the process may be repeated. Repeating the process may result in a new program being generated to access the physical dataset corresponding to the logical dataset or an existing program being updated to access the physical dataset corresponding to the logical dataset. However, the link to the program may be the same so that an application configured using the link to access the data corresponding to the logical dataset continues to operate on the correct data.

[0207] In some embodiments, in response to determining that no change event was detected in act 816, process 800 continues to monitor for change events so that the program for accessing the physical dataset corresponding to the logical dataset for which the access information was generated continues to operate as intended.

[0208] Further implementation details 9 illustrates an example of a suitable computing system environment 900 on which the technology described herein may be implemented. The computing system environment 900 is only one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the technology described herein. Neither the computing environment 900 should be interpreted as having any dependency or requirement relating to any one or combination of components illustrated in the exemplary operating environment 900.

[0209] The technology described herein can be used with numerous other general purpose or special purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations that may be suitable for use with the technology described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network PCs, minicomputers, mainframe computers, distributed computing environments that incorporate any of the above systems or devices, and the like.

[0210] A computing environment can execute computer-executable instructions such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The techniques described herein may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including memory storage devices.

[0211] With reference to FIG. 9, an exemplary system for implementing the techniques described herein includes a general purpose computing device in the form of a computer 900. Components of the computer 910 may include, but are not limited to, a processing unit 920, a system memory 930, and a system bus 921 that couples various system components including the system memory to the processing unit 920. The system bus 921 may be any of several types of bus structures including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. By way of example, and without limitation, such architectures include Industry Standard Architecture (ISA) bus, MicroChannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, also known as Mezzanine bus.

[0212] Computer 910 typically includes a variety of computer readable media. Computer readable media may be any available media that can be accessed by computer 910, and includes both volatile and nonvolatile media, removable and non-removable media. By way of example, and not limitation, computer readable media may include computer storage media and communication media. Computer storage media includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store the desired information and that can be accessed by computer 910. Communication media typically embodies computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism and includes any information delivery media. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer readable media.

[0213] The system memory 930 encompasses computer storage media in the form of volatile and / or nonvolatile memory such as read only memory (ROM) 931 and random access memory (RAM) 932. A basic input / output system 933 (BIOS), containing the basic routines that help to transfer information between elements within the computer 910, such as during start-up, is typically stored in ROM 931. RAM 932 typically contains data and / or program modules that are immediately available and / or presently being operated on by the processing unit 920. By way of example, and not limitation, FIG. 9 illustrates operating system 934, application programs 935, other program modules 936, and program data 937.

[0214] The computer 910 may also include other removable / non-removable, volatile / non-volatile computer storage media. By way of example only, FIG. 9 illustrates a hard disk drive 941 that reads from or writes to non-removable, non-volatile magnetic media, a flash drive 951 that reads from or writes to a removable, non-volatile memory 952, such as a flash memory, and an optical disk drive 955 that reads from or writes to a removable, non-volatile optical disk 956, such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that may be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital versatile disks, digital video tape, solid state RAM, solid state ROM, and the like. The hard disk drive 941 is typically connected to the system bus 921 through a non-removable memory interface, such as interface 940, and the magnetic disk drive 951 and optical disk drive 955 are typically connected to the system bus 921 by a removable memory interface, such as interface 950.

[0215] The drives and their associated computer storage media, discussed above and illustrated in FIG. 9, provide storage of computer readable instructions, data structures, program modules, and other data for the computer 910. In FIG. 9, for example, hard disk drive 941 is illustrated as storing operating system 944, application programs 945, other program modules 946, and program data 947. Note that these components may be the same as or different from the operating system 934, application programs 935, other program modules 936, and program data 937. The operating system 944, application programs 945, other program modules 946, and program data 947 are numbered differently here to illustrate that, at a minimum, they are different copies. Actors can enter commands and information into the computer 910 through input devices such as a keyboard 962 and pointing device 961, commonly referred to as a mouse, trackball, or touch pad. Other input devices (not shown) may include a microphone, joystick, game pad, satellite dish, scanner, or the like. These and other input devices are often connected to the processing unit 920 by a user input interface 960 coupled to the system bus, but may also be connected by other interface and bus structures, such as a parallel port, game port, or universal serial bus (USB). A monitor 991 or other type of display device is also connected to the system bus 921 via an interface, such as a video interface 990. In addition to a monitor, computers may also include other peripheral output devices such as speakers 997 and printer 996, which may be connected through an output peripheral interface 995.

[0216] The computer 910 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer 980. The remote computer 980 may be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 910, although only a memory storage device 981 is illustrated in FIG. 9. The logical connections depicted in FIG. 9 include a local area network (LAN) 971 and a wide area network (WAN) 973, but may also include other networks. Such networking environments are commonplace in offices, enterprise-wide computer networks, intranets and the Internet.

[0217] When used in a LAN networking environment, the computer 910 is connected to the LAN 971 through a network interface or adapter 970. When used in a WAN networking environment, the computer 910 typically includes a modem 972 or other means for establishing communications over the WAN 973, such as the Internet. The modem 972, which may be internal or external, may be connected to the system bus 921 via the actor input interface 960 or other appropriate mechanism. In a networked environment, program modules depicted relative to the computer 910, or portions thereof, may be stored in a remote memory storage device. By way of example, and not limitation, FIG. 9 illustrates remote application programs 985 as residing on memory device 981. It will be appreciated that the network connections shown are exemplary and other means of establishing a communications link between the computers may be used.

[0218] The techniques described herein are not limited to any particular implementation aspect and may be implemented in any of a number of ways. Example implementation details are provided herein for illustrative purposes only. Furthermore, aspects of the technology described herein are not limited to the use of any particular technique or combination of techniques and the techniques disclosed herein may be used individually or in any suitable combination.

[0219] Thus, having described several aspects of the technology described herein, it will be appreciated that various alterations, modifications, and improvements may be made.

[0220] For example, describe a user writing an application that specifies access to logical data. In some embodiments, the user may be a human user. In other embodiments, the user may be a program equipped with artificial intelligence (AI). The AI, for example, derives a data processing algorithm by processing a data set, which can then be applied to other data sets.

[0221] As another example, information 702, 704, and 706 are depicted as being maintained in separate tables, however, the information may be maintained in one table or combined in any suitable manner in any data structure.

[0222] Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to be within the spirit and scope of this disclosure. Furthermore, advantages of the technology described herein are shown, but it is to be understood that not all embodiments of the technology described herein include all described advantages. Some embodiments may not implement any of the features described herein as advantageous, and in some cases, one or more of the described features may be implemented to obtain further embodiments. Thus, the above description and drawings are merely examples.

[0223] The above aspects of the technology described herein may be implemented in any of a number of ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such processors may be implemented as integrated circuits, with one or more processors in integrated circuit components, including commercially available integrated circuit components known in the industry by names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processors may be implemented in custom circuits, such as ASICs, or semi-custom circuits resulting from the configuration of programmable logic devices. As yet a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercially available, semi-custom, or custom. As a specific example, some commercially available microprocessors have multiple cores, such that one or a subset of the multiple cores may constitute a processor. However, the processors may be implemented using circuits of any suitable format.

[0224] Further, it should be understood that the computer may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer, etc. Additionally, the computer may be incorporated into devices not generally considered computers, but equipped with suitable processing capabilities, including a personal digital assistant (PDA), a smart phone, or any other suitable portable or fixed electronic device.

[0225] A computer may also have one or more input and output devices. These devices may be used, among other things, to present a user interface. Examples of output devices that may be used to provide a user interface include a printer or display screen for a visual representation of the output, and a speaker or other sound generating device for an audible representation of the output. Examples of input devices that may be used for a user interface include keyboards and pointing devices such as mice, touch pads, and digitizer tablets. As another example, a computer may receive input information by voice recognition or in other audible formats.

[0226] Such computers may be interconnected by one or more networks of any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and operate according to any suitable protocol, and may include wireless networks, wired networks, or fiber optic networks.

[0227] Also, the various methods or processes outlined herein may be encoded as software executable for one or more processors using any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine code or intermediate code that runs on a framework or virtual machine.

[0228] In this regard, aspects of the technology described herein may be embodied as a computer-readable storage medium (or multiple computer-readable media) (e.g., a computer memory, one or more floppy disks, compact disks (CDs), optical disks, digital video disks (DVDs), magnetic tapes, flash memories, circuitry in field programmable gate arrays or other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods for implementing the various embodiments described above. As is evident from the above examples, a computer-readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such one or more computer-readable storage media may be portable such that one or more programs stored thereon may be loaded into one or more different computers or other processors to implement various aspects of the technology described above. As used herein, the term "computer-readable storage medium" covers only non-transitory computer-readable media that may be considered to be an article (i.e., an article of manufacture) or a machine. Alternatively or additionally, aspects of the technology described herein may be embodied as a computer-readable medium other than a computer-readable storage medium, such as a propagated signal.

[0229] The terms "program" or "software" are used herein generically to refer to any type of computer code or set of computer-executable instructions or set of processor-executable instructions that can be used to program a computer or other processor to perform various aspects of the technology described above. Additionally, in accordance with certain aspects of the present embodiments, it is to be understood that one or more computer programs that, when executed, perform the methods of the technology described herein need not reside on a single computer or processor, but may be distributed in a modular manner among a number of different computers or processors to perform various aspects of the technology described herein.

[0230] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0231] Additionally, data structures may be stored in a computer-readable medium in any suitable form. For ease of illustration, data structures may be shown with fields related by location within the data structure. Such relationships may similarly be achieved by assigning locations within the computer-readable medium that convey the relationship between the fields to the storage of the fields. However, relationships between information in fields of a data structure may be established using any suitable mechanism, including through the use of pointers, tags, or other mechanisms that establish relationships between data elements.

[0232] Various aspects of the technology described herein may be used alone, in combination, or in various arrangements not specifically set forth in the embodiments described above, and therefore are not limited in their application to the details and arrangements of components set forth in the above description or illustrated in the drawings. For example, aspects described in one embodiment can be combined in any manner with aspects described in other embodiments.

[0233] Additionally, the techniques described herein may be embodied as methods, examples of which are provided herein, including with reference to Figure 8. Acts performed as part of any method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that shown (which may include performing some acts simultaneously, even though in the illustrative embodiments they are shown as sequential acts).

[0234] Additionally, some actions are described as being performed by an "actor" or a "user." It is to be understood that an "actor" or "user" need not be a single individual, and that in some embodiments, actions attributed to an "actor" or "user" may be performed by a team of individuals and / or an individual in combination with computer-assisted tools or other mechanisms.

[0235] The use of ordinal terms such as "first," "second," "third," etc. in the claims to modify claim elements does not, of itself, imply a priority, precedence, or ordering of one claim element over another, or the chronological order in which acts of a method are performed, but rather is used merely as a label to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms) that distinguishes between the claim elements.

[0236] Also, the phraseology and terminology used herein are for purposes of description and not of limitation. The use of "including," "comprising," "having," "containing," "involving," and variations thereof herein are meant to encompass the items listed thereafter and equivalents thereof, as well as additional items. [Explanation of symbols]

[0237] 100 IT Systems 102 Federated Data Environment 102-1, 102-2, 102-3, 102-n Datastore 104 Data Processing System 104-1, 104-2, 104-n Instances of Data Processing Systems 105 Data Set Multiplexer 106, 106-1, 106-2, 106-3, 106-n Applications 107 Dataset Catalog 302,304 Input data source, Input data store 306,308 Filters 310 Sorting 312 Combine 314 Output Data Source, Output Data Store 315 User Interface 330,340 Programs for accessing input data sources Program for accessing 350 output data sources 370 List of logical datasets 372 User Interface Elements 375 "loyalty" logical data list 500 IT Systems 520 Registration Module 522 Catalog Service Interface 524 Dynamic Subgraph Generator 526 Metadata Management Module 528 Operation Metadata Module 530 User Interface 610,620 Components 702,704,706 Information 710 Input Data Store 715 Dynamic Subgraph (DSG) 720 Output Datastore 725 DSG 730 Input Data Source Parameter 740 Output Data Source Parameter 900 Computing System Environment 910 Computer 920 Processing Equipment 921 System Bus 930 System Memory 931 Read-only memory (ROM) 932 Random Access Memory (RAM) 933 Basic Input / Output System (BIOS) 934 Operating Systems 935 Application Program 936 Other Program Modules 937 Program Data 940 Non-removable non-volatile memory interface 941 Hard Disk Drive 944 Operating Systems 945 Application Program 946 Other Program Modules 947 Program Data 950 Removable Non-Volatile Memory Interface 951 Magnetic Disk Drive 952 Removable Non-Volatile Memory 955 Optical Disk Drive 956 Removable non-volatile optical disk 960 User Input Interface 961 Pointing Device 962 Keyboard 970 Network Interfaces or Adapters 971 Local Area Network (LAN) 972 Modem 973 Wide Area Network (WAN) 980 Remote Computer 981 Memory Storage Devices 985 Remote Application Program 990 Video Interface 991 Monitor 995 Output Peripheral Interface 996 Printer 997 Speaker

Claims

1. 1. A method performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by generating and / or using entries in a dataset catalog to enable access to physical datasets in a data store, the data processing system being configured to execute data processing applications programmed to access logical datasets, each logical dataset including a schema for data independent of a format of corresponding data in a physical dataset, the data processing system including a dataset multiplexer configurable to provide applications with access to the physical datasets in the data store, the method comprising: creating a plurality of entries in the dataset catalog, each of the plurality of entries being associated with a logical dataset and a physical dataset and associated with computer-executable instructions for accessing the physical dataset; receiving an input at least partially identifying a first logical dataset to access to perform an operation within a data processing application specifying access to the dataset; invoking the computer-executable instructions to access a physical dataset associated with an entry in the dataset catalog associated with the first logical dataset during execution of the operation within the data processing application; dynamically updating entries in the dataset catalog in response to events indicating changes to a physical dataset associated with a logical dataset; A method comprising:

2. creating a plurality of entries in the dataset catalog, receiving information relating to a first one of the physical data sets stored in a first one of the data stores, the first physical data set corresponding to a first logical data set; generating a first program including the computer executable instructions for accessing the first physical data set from the first data store based on the information related to the first physical data set; storing, in a first entry in the data set catalog, a link to the first program for enabling the data processing application to access the first physical data set using the first program; The method of claim 1 , comprising:

3. generating the first program for accessing the first physical data set from the first data store; identifying a type of the first data store from the received information; and selecting a first program template for the type of the first data store; generating the first program by populating the first program template with one or more values ​​of one or more parameters of the first program template; The method of claim 2 , comprising:

4. Receiving an input at least partially identifying a first logical data set; 4. The method of claim 1, further comprising providing a user interface through which a user at least partially identifies the first logical data set.

5. Invoking the computer-executable instructions providing access to the entry in the dataset catalog associated with the first logical dataset; enabling access to a data store storing the physical data set corresponding to the first logical data set based on information in the entry; The method of any one of claims 1 to 4, comprising:

6. Dynamically updating entries in the dataset catalog, detecting an event indicative of a change associated with a physical data set corresponding to the first logical data set; modifying the entry in the dataset catalog associated with the first logical dataset based on the detection of the event; The method of any one of claims 1 to 5, comprising:

7. Modifying the entry in the dataset catalog The method of claim 6 , further comprising modifying the computer executable instructions for accessing the physical data set that corresponds to the first logical data set.

8. 1. A method performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by registering a dataset in a dataset catalog to facilitate access to multiple physical datasets in a data store, the data processing system being operable with the multiple physical datasets stored in the data store, the data processing system including a dataset multiplexer configurable to provide an application with access to a physical dataset of the multiple physical datasets, the physical datasets being stored in a data store of the data stores, the physical datasets corresponding to logical datasets including a schema for data independent of a format of the corresponding data in the physical datasets, the method comprising: receiving information relating to a first physical data set of the plurality of physical data sets stored in a first data store of a plurality of data stores, the first physical data set corresponding to a first logical data set; generating a first program comprising computer executable instructions for accessing the first physical data set from the first data store based on the information related to the first physical data set; storing a link to the first program in a first object in a library of objects to enable the application to access the first physical data set using the first program; A method comprising:

9. The method further comprising:

9. The method of claim 8, further comprising determining whether to modify the first program for accessing the first physical data set based on detection of an event indicative of a change associated with the first physical data set.

10. The method further comprises: based on determining to modify the first program, generating a modified first program; replacing the first program with the modified first program as the target of the link; The method of claim 9 further comprising:

11. 11. The method of claim 10, wherein generating the modified first program comprises generating the modified first program without modifying the application or the first logical data set.

12. 12. The method of claim 8, wherein the information relating to the first physical data set includes information about a type of the first data store.

13. 13. The method of claim 8, wherein the dataset multiplexer includes a library of objects that stores information for accessing the plurality of physical datasets, and the first object in the library of objects includes an identifier for the first physical dataset.

14. The method of claim 13 , wherein the dataset multiplexer further comprises an API, the method further comprising providing the application with access to the first object through the API.

15. The method further comprising:

14. The method of claim 13, further comprising assigning identifiers to objects in the library based on a schema and logical name of a respective logical data set in which information is stored in the object.

16. The method further comprising: receiving a command to register the first physical data set in a data set catalog; generating the first object based on the received command and storing the first object in the library; The method of claim 13 further comprising:

17. The method of claim 13 , wherein the identifier of the first physical data set is a physical identifier.

18. 20. The method of claim 17, wherein the first object further comprises a second identifier, the second identifier being a logical identifier for a logical data set associated with the first object.

19. The method further comprising:

20. The method of claim 18, further comprising: in response to detecting an event indicating that the first physical data set has changed from a state stored in the first data store to a state stored in a second data store, modifying the physical identifier in the first object without modifying the logical identifier.

20. the first object includes values ​​of parameters accessed during execution of the first program; The method further comprising:

14. The method of claim 13, further comprising modifying a value of the parameter stored in the first object based on detection of an event indicating a change in a value of the parameter accessed in the first program.

21. 21. A method according to any one of claims 8 to 20, wherein the first program includes access logic and conversion logic, and when the application is executed, the access logic and the conversion logic of the first program are executed to provide access to the first physical data set and to convert between a format used in the first physical data set and a format used in the first logical data set.

22. 22. A method according to any one of claims 8 to 21, wherein the first program includes one or more parameters that affect the operation of the first program, such that values ​​of the one or more parameters affect access of the first physical data set via the first program.

23. 23. The method of claim 22, wherein the application is configured to supply values ​​for the one or more parameters for use in invoking the first program.

24. The method further comprising: Detecting a type of the first data store; selecting a template from a plurality of templates based on the detected type; 24. The method of claim 8, further comprising generating the first program by:

25. 25. The method of claim 24, wherein the first program includes a first portion configured for read access to the first data store and a second portion for write access to the first data store.

26. 26. A method according to any one of claims 8 to 25, wherein the first program is structured as an executable data flow graph including logic for accessing the first physical data set.

27. 1. A method performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores, the data processing system being operable with the application and the multiple physical datasets stored in the multiple data stores, the application being programmed to access a logical dataset that includes a schema for data independent of a format of the corresponding data in the physical datasets, the method comprising: providing a user interface that at least partially identifies a logical data set for a user to access in the application; Upon executing the application and performing operations involving access to the identified logical data set, providing access to objects in a library of objects associated with said logical data set; enabling access to a data store storing the physical data set corresponding to the identified logical data set based on information in the object; A method comprising:

28. The method further comprising:

30. The method of claim 27, further comprising updating the information in the object based on an event associated with the saving of data corresponding to the identified logical data set.

29. 29. A method according to claim 27 or 28, wherein the information in the object comprises an executable program for accessing the physical data set.

30. 30. The method of claim 29, wherein the executable program for accessing the physical data set encodes logic for converting data between a format used in the physical data set and a format used in the logical data set.

31. 31. A method according to any one of claims 27 to 30, wherein the object is an executable program for accessing the physical data set.

32. 32. A method according to any one of claims 27 to 31, wherein the information in the object includes a type of the data store.

33. 33. A method according to any one of claims 27 to 32, wherein the information in the object comprises a record format or schema associated with the physical data set.

34. 34. A method according to any one of claims 27 to 33, wherein the information in the object includes one or more parameters specifying the manner of accessing the physical data set, the one or more parameters including at least one parameter indicating whether or not data in the physical data set is compressed.

35. 35. A method according to any one of claims 27 to 34, wherein the information in the object includes one or more parameters specifying the manner in which the physical data set is to be accessed, the one or more parameters including at least one parameter indicating a type of the access.

36. 36. The method of claim 35, wherein the type of access includes an indication of read access or write access.

37. 36. The method of claim 35, wherein the type of access includes an indication of access over a high speed connection or a low speed connection.

38. the data processing system includes a repository of metadata associated with logical data sets; 38. The method of any one of claims 27 to 37, wherein providing the user interface comprises presenting a menu of logical data sets based on metadata in the repository.

39. A method according to any one of claims 27 to 38, wherein the information in the object includes one or more parameters specifying the manner of accessing the physical dataset, the one or more parameters including at least one parameter indicating whether data in the physical dataset is encrypted or specifying criteria for a filter operation on the data.

40. The method of claim 40, further comprising: providing access to a data store storing said physical data set, said data store comprising:

30. The method of claim 29, comprising using the executable program to enable access to the data store storing the physical data set corresponding to the identified logical data set.

41. 1. A method performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by generating entries in a dataset catalog to enable access to physical datasets in a data store, the data processing system being configured to execute data processing applications programmed to access logical datasets, each logical dataset including a schema for the data independent of a format of the corresponding data in the physical dataset, the data processing system including a dataset multiplexer configurable to provide the application with access to the physical datasets in the data store, the method comprising: receiving information relating to a first physical data set stored in a first data store of the data stores, the application being programmed for accessing a first logical data set, the first physical data set corresponding to the first logical data set; generating a first program for accessing the first physical data set from the first data store based on the received information, wherein generating the first program includes: identifying a type of the first data store from the received information; and selecting a first program template for the type of the first data store; generating the first program by populating the first program template with one or more values ​​of one or more parameters of the first program template; generating storing information in an object for invoking execution of the first program from within the application programmed for access of the first logical data set; A method comprising:

42. 42. The method of claim 41 , wherein populating the first program template comprises automatically discovering one or more values ​​for one or more first parameters of the first program template based on the information related to the first physical data set.

43. 43. The method of claim 42, wherein the one or more first parameters include information regarding a record format or schema associated with the first physical data set.

44. 44. The method of claim 41, wherein storing in the object information for invoking execution of the first program from within an application programmed for access of the first logical data set comprises storing an identifier of the first data store.

45. 45. The method of claim 41, wherein storing in the object information for invoking execution of the first program from within an application programmed for access of the first logical data set comprises storing a logical identifier of the first logical data set.

46. generating the first program, 43. The method of claim 42, further comprising obtaining information regarding one or more second parameters of the first program template, the one or more second parameters being different from the one or more first parameters.

47. 47. The method of claim 46, wherein the one or more second parameters specify how the first physical data set is accessed.

48. generating the first program, determining whether a program template is available for the type of the first data store; selecting an available template as the first program template based on a determination that the first program template is available for the type of the first data store; and 48. The method of any one of claims 41 to 47, further comprising:

49. based on a determination that a program template is not available for the type of the first data store, creating a program structure based on user input; generating the first program for accessing the first data store based on the created program structure; 49. The method of claim 48, further comprising:

50. receiving information relating to a second physical data set stored in a second one of the data stores; generating a second program for accessing the second physical data set from the second data store based on the information related to the second physical data set; 50. The method of any one of claims 41 to 49, further comprising:

51. the data processing system is configured to run in a number of environments, each environment including an instance of the data processing system; 51. A method according to any one of claims 41 to 50, wherein the objects are assigned unique identifiers within each of the multiple environments and include at least some that are common across the multiple environments.

52. 1. A method performed by a data processing system for enabling efficient analysis in a dynamic environment having multiple datasets by updating entries in a dataset catalog to facilitate access to physical datasets in a data store, the data processing system being configured to execute data processing applications programmed to access data represented as logical datasets, each logical dataset including a schema for the data independent of a format of corresponding data in a physical dataset, the data processing system including a dataset multiplexer configurable to provide applications with access to the physical datasets in the data store, the method comprising: receiving information relating to a first physical data set stored in a first data store corresponding to a first logical data set; generating a first program for accessing the first physical data set from the first data store based on the received information; detecting an event indicative of a change associated with a physical data set corresponding to the first logical data set; modifying the first program for accessing a physical data set corresponding to the first logical data set based on the detection of the event; A method comprising:

53. the physical data set is the first physical data set, and the events indicative of changes associated with the physical data set include events indicative of changes from the first data store storing the first physical data set to a second data store, the method comprising:

53. The method of claim 52, further comprising modifying the first program to access the first physical data set from the second data store in response to detecting the event indicating the change from the first data store to the second data store.

54. 54. The method of claim 52 or 53, wherein the physical dataset is the first physical dataset, and the event indicating a change associated with the physical dataset comprises an event indicating a change in a value of a parameter used to generate the first program for accessing the first physical dataset.

55. A method according to any one of claims 52 to 54, wherein the physical dataset is the first physical dataset, and the events indicating changes associated with the physical dataset include events indicating changes to a format of the first physical dataset.

56. detecting an event indicative of a change associated with a physical dataset includes detecting an event indicative of replacing the first physical dataset with a second physical dataset corresponding to the first logical dataset; 56. The method of any one of claims 52 to 55, wherein modifying the first program for accessing the physical data set comprises replacing the first program with a second program for accessing the second physical data set.

57. the data processing system is configured to invoke the first program to perform an operation within an application that specifies access to a first logical data set; the data processing system is configured to run in multiple environments, a first environment including a first instance of the data processing system and a second environment including a second instance of the data processing system; the first data store and the first program are associated with the first instance of the data processing system; The method further comprising:

57. The method of any one of claims 52 to 56, further comprising generating a second program for performing operations within an application that specifies access to the first logical data set within the second instance of the data processing system.

58. 58. The method of claim 57, further comprising: executing the application in the second environment specifying access to the first logical data set; and in response to performance of an operation by the application on the first logical data set, accessing the second program to access a second physical data set.

59. 1. A method performed by a data processing system for enabling efficient data analysis in a dynamic environment having multiple datasets by using entries in a dataset catalog to enable an application to access multiple physical datasets in multiple data stores, the data processing system being configured to execute a data processing application programmed to access logical datasets, each logical dataset including a schema for data independent of a format of corresponding data in a physical dataset, the data processing system including a dataset multiplexer configurable to provide an application with access to multiple physical datasets in the multiple data stores, the method comprising: Accessing the dataset catalog to select an object associated with the logical dataset; invoking a program configured for accessing a data source storing a physical data set corresponding to the logical data set based on the selected object; performing an operation within the application specifying access to the logical data set by

60. The method further comprising:

60. The method of claim 59, further comprising dynamically updating an object in the dataset catalog in response to an event indicating a change in physical storage of a logical dataset represented by the object in the dataset catalog.

61. at least one computer hardware processor; at least one non-transitory computer readable medium having stored thereon processor executable instructions that, when executed by said at least one computer hardware processor, cause said at least one computer hardware processor to perform the method of any one of claims 1 to 60; 23. A data processing system comprising:

62. At least one non-transitory computer readable medium comprising processor-executable instructions that, when executed by at least one computer hardware processor, cause the at least one computer hardware processor to perform the method of any one of claims 1 to 60.

Images