Logical access to preview datasets in the extended view

Abstract

A method implemented by a data processing system, which enables a user to preview the attributes of fields in an extended view of a base dataset and to specify one or more of the fields for use in downstream data processing, and generates a dataset containing one or more of the fields specified for use in downstream data processing, wherein the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all of the fields in the extended view when only one or more of the specified fields are used in downstream data processing.

Description

Technical Field

[0001] (Claim of Priority) This application claims priority to U.S. Patent Application No. 63 / 491,921, filed on March 23, 2023, and to U.S. Patent Application No. 18 / 492,904, filed on October 24, 2023, under 35 U.S.C. § 119(e), the entire contents of both of which are incorporated herein by reference.

[0002] (Field of the Invention) The present disclosure relates to techniques for customizing views to large and complex databases.

Background Art

[0003] Modern data processing systems manage vast amounts of data within an enterprise. For example, large organizations may have millions of data sets. These data sets can support multiple aspects of a business's operations. Complex data processing systems typically process data in multiple stages, with the results generated from one stage being fed into the next. The overall flow of information through such a system can be described in terms of a directed data flow graph, where the nodes or vertices within the graph represent components (either data files or processes), and the links or "edges" within the graph indicate the flow of data between components. A system for performing such graph-based computations is described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," which is incorporated herein by reference.

[0004] Graphs can also be used to directly invoke computations. Graphs created according to this system provide methods for introducing information into individual processes represented by graph components, for retrieving information from individual processes, for moving information between processes, and for defining the execution order for processes. Systems that invoke these graphs include algorithms for selecting inter-process communication methods and algorithms for scheduling process execution and monitoring graph execution.

[0005] To support a wide range of functionalities, data processing systems can run applications, whether they implement routine processes or extract insights from datasets. These applications can be programmed to access data stores to read and write data. [Overview of the project]

[0006] Generally, in the first embodiment, the method implemented by the data processing system includes enabling a user to preview the attributes of fields in an extended view of a base dataset and to specify one or more of the fields for use in downstream data processing, and generating a dataset containing one or more of the fields specified for use in downstream data processing, wherein the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all fields of the extended view when only one or more of the specified fields are used in downstream data processing, and generating, receiving an identification of the base dataset, and based on the identification, generating the dataset of the extended view Receiving a definition of a dataset in an extended view, where the definition of the dataset in the extended view specifies other datasets related to the base dataset or fields of other datasets; outputting a preview of the attributes of the fields in the dataset in the extended view, where the preview is generated from a subset of data in other datasets related to the base dataset or a subset of data in fields of other datasets; receiving an input specifying that one or more of the fields in the preview are available for data processing; and generating an available dataset, based on the input, containing data in the base dataset and data in one or more of the specified fields.

[0007] In a second embodiment, which can be combined with the first embodiment, the method includes providing a dataset of the extended view, which, when generated, includes data from a base dataset and other datasets related to the base dataset, to a preview of the attributes of the extended view dataset.

[0008] In a third embodiment, which can be combined with the first or second embodiment, the definition of an extended view dataset specifies a set of data processing operations performed to generate an extended view dataset containing data from a base dataset and other datasets associated with the base dataset, and the preview is generated by applying the set of data processing operations specified by the definition of an extended view dataset to only a subset of the data in the base dataset and other datasets associated with the base dataset.

[0009] In a fourth embodiment, which can be combined with any of the first to third embodiments, the method includes receiving a specification specifying a data processing operation in response to providing a preview, the data processing operation of the specification being defined at least in part based on user input identifying attributes included in the preview as attributes of that data processing operation.

[0010] In a fifth embodiment, which can be combined with any of the first to fourth embodiments, the method includes updating a defined set of data processing actions by applying one or more optimization rules to a set of data processing actions based on a data processing action that is at least partially defined based on user input identifying attributes included in a preview as attributes of that data processing action; and performing the updated set of data processing actions to generate a dataset that includes only a subset of the data that would have been included in the dataset of the extended view.

[0011] In a sixth embodiment, which can be combined with any of the first to fifth embodiments, the method includes enabling a user to register a definition of a new dataset in a data catalog using a definition that specifies a selected dataset and other datasets related to the selected data, the definition providing logical access to other datasets related to the base dataset without incurring the computational cost of providing the other related datasets.

[0012] In a seventh embodiment, which can be combined with any of the first to sixth embodiments, the method includes accessing a data catalog that specifies one or more datasets, and providing a user interface that indicates that one or more datasets are candidates for generating datasets for an extended view.

[0013] In an eighth embodiment, which can be combined with any of the first to seventh embodiments, the method includes receiving metrics for a specific dataset as a base dataset via a user interface, and automatically generating a dataset definition for an extended view for the specific base dataset in response to the metrics.

[0014] In a ninth embodiment, which can be combined with any of the first to eighth embodiments, the method includes: identifying a particular base dataset as a base dataset and one or more attributes of the particular base dataset; determining a definition of the base dataset from one or more attributes; determining one or more other datasets related to the base dataset based on the definition of the base dataset; and generating a definition of a dataset for an extended view, which specifies the base dataset, one or more other datasets, and one or more relationships between the base dataset and the one or more other datasets, based on the determined one or more other datasets.

[0015] In a tenth embodiment, which can be combined with any of the first to ninth embodiments, the method includes storing the definition of the extended view dataset in a hardware storage device and registering the definition of the extended view dataset in a data catalog.

[0016] In an eleventh embodiment, which can be combined with any of the first to tenth embodiments, the method includes receiving a request for an extended view dataset, and in response to the request, providing an extended view dataset by retrieving a definition of the extended view dataset from a hardware storage device, retrieving a base dataset and one or more other datasets from one or more data sources based on the definition of the extended view dataset, and generating an extended view dataset based on the data in the retrieved datasets.

[0017] In a twelfth embodiment, which can be combined with any of the first to eleventh embodiments, the method includes determining whether to update a data catalog to specify the definition of an extended view dataset as a data source, based on the extended view dataset; storing the definition of an extended view dataset in a hardware storage device; and registering the definition of an extended view dataset in a data catalog.

[0018] In a thirteenth embodiment, which can be combined with any of the first to twelfth embodiments, the method includes determining whether to update the data catalog to specify the definition of the extended view dataset as a data source, based on a provided preview of the attributes of the extended view dataset.

[0019] In a 14th embodiment, which can be combined with any of the 1st to 13th embodiments, generating an available dataset includes using the dataset definition of an extended view to access only datasets having one or more specified fields, and including the data of those accessed datasets in the available dataset.

[0020] In a 15th embodiment, which can be combined with any of the 1st to 14th embodiments, the method includes processing the generated available dataset to obtain results from processing the data of the available dataset.

[0021] In a 16th embodiment, which can be combined with any of the 1st to 15th embodiments, the method includes providing user permissions to access a portion of the base dataset within the extended view dataset, while denying user access to the rest of the base dataset.

[0022] In a 17th embodiment, which can be combined with any of the 1st to 16th embodiments, the definition of an extended view dataset includes a computation graph specifying a set of data processing operations for generating an extended view dataset that includes a base dataset and data from other datasets related to the base dataset, the set of data processing operations including at least one operation for combining data from the base dataset with data from at least one of the other datasets related to the base dataset.

[0023] In an 18th embodiment, which can be combined with any of the 1st to 17th embodiments, the definition of the extended view dataset provides logical access to data from the base dataset and other datasets associated with the base dataset. The aforementioned actions of the method can be combined in any and all combinations.

[0024] In a 19th embodiment, which can be combined with any of the 1st to 18th embodiments, the preview is generated during development and the available dataset is generated at runtime.

[0025] Generally, in the 20th embodiment, a method implemented by a data processing system for generating a dataset that allows a user to preview the attributes of fields in an extended view of a base dataset and specify one or more of the fields for use in downstream data processing, and generates a dataset containing one or more of the fields specified for use in downstream data processing, wherein the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all fields of the extended view when only one or more of the specified fields are used in downstream data processing, the method for generating a dataset that receives an identification of the base dataset and, based on the identification, generates the data of the extended view The process includes receiving a dataset definition, wherein the extended view dataset definition specifies other datasets related to the base dataset or fields of other datasets; outputting a preview of the attributes of the fields in the extended view dataset based on the extended view dataset definition, wherein the preview is generated from a subset of data in other datasets related to the base dataset or a subset of metadata of other datasets; receiving an input specifying that one or more of the fields in the preview are available for data processing; and generating an available dataset based on the input, which includes data in the base dataset and data in one or more of the specified fields.

[0026] In general, in the 21st embodiment, the data processing system includes one or more processing devices and one or more machine-readable hardware storage devices that store instructions that can be executed by one or more processing devices to perform any of the operations of the first to 20 embodiments.

[0027] In general, in the 22nd embodiment, one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform any of the operations of the first to 21st embodiments.

[0028] One or more computer systems can be configured to perform certain operations or actions by installing software, firmware, hardware, or combinations thereof on the system that cause the system to perform actions during operation, including any and all of the aforementioned actions in any combination. One or more computer programs can be configured to perform certain operations or actions by including instructions that cause the device to perform actions when executed by a data processing device, including any and all of the aforementioned actions in any combination.

[0029] One or more of the above aspects can provide one or more of the following advantages.

[0030] An extended view dataset can represent a subset of the data contained within a dataset. Because the definition of an extended view dataset only provides logical access to the dataset and does not include a copy of all the data presented by the original dataset, the definition of an extended view dataset occupies relatively little storage space. In particular, when an extended view dataset is requested, the system generates and stores only the definition of that extended view dataset. This definition provides logical access without the physical cost of materializing the extended view dataset, as described below. This definition allows for a preview of the fields within the extended view dataset without materializing the entire extended view dataset (which can be generated during development or authoring). This preview allows for the selection or specification of which fields are needed for processing. Then, when processing is actually performed (e.g., at runtime), the system uses the definition to access only the datasets containing the required fields, and these accessed datasets become available for processing through the available datasets. The available dataset includes one or more fields from the preview designated for use in downstream data processing, and the generated available dataset is more efficient in terms of speed and data memory compared to generating a dataset containing all fields of the extended view when only one or more of the designated fields are used in downstream data processing. Depending on the embodiment, the extended view dataset can limit the extent to which the underlying dataset is exposed to the outside. A given user may have permission to read the extended view dataset which contains a portion of the underlying base dataset, but is denied access to the rest of the base dataset. The extended view dataset can simplify multiple datasets by combining them into a single virtual dataset. The extended view dataset can function as an aggregated dataset, in which the system aggregates data (sum, mean, etc.) and presents the results as part of the data in the extended view dataset.Extended views of datasets can hide the complexity of the data by transparently partitioning the actual underlying dataset.

[0031] Extended view datasets are the result of executing a set of stored transformation logic, and catalog users can access them in the same way they access persistent data. Extended view datasets are efficient when returning multiple data axes and avoiding data duplication. Extended view datasets use the relationships of the underlying base datasets that maintain relationships to those base datasets by using the relationships of the primary keys and foreign keys of the base datasets to find related datasets.

[0032] Details of one or more embodiments of the present invention are described in the accompanying drawings and the following description. Other features, purposes, and advantages of the present invention will become apparent from the specification and drawings and the claims. [Brief explanation of the drawing]

[0033] [Figure 1] This is a block diagram of the system. [Figure 2] This is a diagram of the system for previewing the attributes of a dataset in an extended view. [Figure 3A] This is a diagram of the system in Figure 2 at the stage of defining the extended view. [Figure 3B] This is a diagram of the system in Figure 2 at the stage of defining the extended view. [Figure 3C] This is a diagram of the system in Figure 2 at the stage of defining the extended view. [Figure 3D] This is a diagram of the system in Figure 2 at the stage of defining the extended view. [Figure 3E] This is a diagram of the system in Figure 2 at the stage of defining the extended view. [Figure 3F] This is a diagram of the system in Figure 2 at the stage of defining the extended view. [Figure 4A] The diagrams illustrate the steps involved in generating the extended view dataset. [Figure 4B-1] The diagrams illustrate the steps involved in generating the extended view dataset. [Figure 4B-2] The diagrams illustrate the steps involved in generating the extended view dataset. [Figure 4C] The diagrams illustrate the steps involved in generating the extended view dataset. [Figure 4D] The diagrams illustrate the steps involved in generating the extended view dataset. [Figure 4E] The diagrams illustrate the steps involved in generating the extended view dataset. [Figure 5] This is a flowchart of the process for previewing the attributes of a dataset in an extended view. [Figure 5A] This is a flowchart of the process for previewing the attributes of a dataset in an extended view. [Figure 5B] This is a flowchart of the process for previewing the attributes of a dataset in an extended view. [Figure 5C] This is a flowchart of the process for previewing the attributes of a dataset in an extended view. [Figure 6] This diagram shows the details of computer systems, such as data processing systems. [Modes for carrying out the invention]

[0034] Referring to Figure 1, a diagram illustrates the complexity of understanding and accessing data when data is distributed across different data sources within an enterprise or system. Furthermore, the diagram shows how computationally inefficient the current system is in aggregating all the data from different data sources that may be needed to perform a particular calculation or operation. In this example, the data scientist desires a deeper understanding of and access to all the data available across the enterprise or system. Therefore, the data scientist submits a request to the data engineer for all the data related to the ActiveLoan dataset in this example.

[0035] The data engineer then needs to identify all the data related to the active loan dataset. They do this by requesting schemas from various systems across the enterprise and generating a program to retrieve all the data related to the active loan. Once the data engineer generates the program, it is sent to a quality engineer who identifies any errors in it. These errors are sent back to the data engineer, and perhaps a month or two later, the data engineer has an updated program to retrieve all the data related to the active loan. However, even this updated program may still have some data missing or still contain some errors. In practice, the data engineer may not be able to identify all the data sources and datasets related to the active data.

[0036] A quality engineer transmits this updated program to a computer, which then runs the program against various data sources. The computer program generates a large dataset, which is already outdated at the very moment it is created. This is because the dataset is generated prior to any program or data operation that actually calls or uses that dataset. In this example, it may be five days or five months later that a data scientist reviews the large dataset to see what data is available in the system. By the time the data scientist reviews this large dataset, it is now five days later, and therefore outdated.

[0037] In this example, a data scientist might request that the average FICO of an active loan be calculated. The data scientist could send this request to a computer, which would then implement the logic to perform the request. This process is highly inefficient because it results in the manifestation of all data related to the active loan when, in reality, only a portion of that data is substantially needed to compute the data scientist's request. In this example, only the loan ID field, status field, and FICO field are needed to complete the data scientist's request. However, when the data scientist generates a dataset to see which data is available, the computer manifests all data related to the active loan, which is computationally inefficient not only because it is costly to remember all the manifested data, but also because the computer needs to scale its resources to combine all this data into a dataset for review by the data scientist.

[0038] Furthermore, as discussed earlier, this dataset is outdated. Therefore, a system is needed that can efficiently generate a dataset containing only the data substantially necessary for computation, and retrieve that data in real time on demand in response to requests. Thus, it is necessary to ensure that the data is not outdated, while simultaneously enabling logical access to that data, allowing for understanding and previewing the available data without actually materializing it.

[0039] Referring to Figure 2, System 10 is shown for generating a preview of the field attributes of an extended view dataset. The extended view dataset includes data in a base dataset and data in one or more datasets related to the base dataset, for example, in a database schema. The base dataset is, for example, a dataset specified or selected by the user. Generally, attributes include the values ​​of fields, and / or information describing the values ​​and / or fields. System 10 also allows the user to specify one or more attributes for use in one or more data processing operations that are optimized in terms of speed and data memory.

[0040] System 10 includes a data processing system 12 and a client device 18. The client device 18 receives data from a data catalog 14, which is rendered by the client device 18 in a browser 19 (specifying which datasets are available for processing and / or available for calculating requested values). The user interface 19a rendered in the browser 19 displays a section 20a (labeled "Data Catalog") that displays datasets from the data catalog 14 and a section 20b (labeled "Field Selector") that displays fields within a dataset. Section 20a lists the datasets from the data catalog 14, and section 20b displays the fields in one of the selected datasets (further consideration below). The user selects a data source they want to preview via the user interface 19a. The user selects a data source from many heterogeneous (e.g., enterprise-wide) data sources and all the fields within those data sources. System 10 can ultimately automatically generate code to access a specific data source across those heterogeneous data sources (e.g., generate a data flow graph).

[0041] The data catalog 14 is a repository of identifiers (e.g., business names or logical names or indexes of logical metadata) for one or more datasets and fields, as well as other data across the entire storage infrastructure that enables users to find and identify data more quickly. Identifiers in the data catalog 14 may be business names that are easily understood by users and provide semantic meaning. The data catalog 14 may also store technical identifiers (also known as technical metadata), such as datasets and fields. For example, this technical metadata may specify technical field names, e.g., field names that appear in the data source itself. For each technical field name, the data catalog may store a logical name or business name to enable users to easily identify the field and dataset. In some examples, system 10 automatically transforms the technical metadata into logical metadata (e.g., business names) by performing semantic discovery on data received from the data source. This is described in U.S. Patent Application Publication 2020 / 0380212 (titled "Discovering a Semantic Meaning of Data Fields from Profile Data of the Data Fields"), the entire contents of which are incorporated herein by reference.

[0042] In this example, client device 18 transmits request 13 to EVD definition generator 22. Request 13 specifies that "active loans.dat" is the base dataset and that request 13 is for an extended view of "active loans.dat". The extended view includes representations, specifications, identifications, or lists, or all datasets related to the base dataset. In response to the request, EVD definition generator 22 identifies the datasets related to the base dataset and generates definitions 13a for these related datasets and the base dataset. This definition, referred to as the EVD definition, specifies the related datasets and the base dataset, and also specifies the logic for generating a dataset (referred to as the extended view dataset) that contains data from the related datasets and the base dataset. At this stage, the EVD definition generator transmits the EVD definition 13a to the metadata repository 25 for storage. At this point, data processing system 12 does not generate the extended view dataset using the EVD definition 13a. This is because the extended view dataset (once generated) would contain many fields (e.g., all fields from the base dataset and related datasets). Therefore, materializing this dataset is costly. Materialization refers to the process of extracting data (of fields) from various sources, combining all of that data into a single dataset, and then storing that combined single dataset. This materialization is costly in terms of processing and memory resources. Therefore, the data processing system 12 only materializes the combined single dataset once the fields required for processing are actually specified, as described below. Furthermore, the data processing system 12 displays visualizations of the fields in the EVD, including previews of the values ​​of those fields. The data processing system 12 generates these visualizations and previews by processing a limited or specified amount of data in the base dataset and related datasets using the EVD definition.In some cases, the data processing system 12 can generate visualizations of fields without previewing their values ​​within the EVD, thereby eliminating the need to process any data in the base dataset and associated datasets. By processing only a specified or limited amount of data, the data processing system 12 conserves processing and memory resources. Thus, the data processing system 12 provides logical access (to physical fields and values) without the cost of materializing the EVD. Logical access includes previews of the fields and / or their values, allowing those fields to be used when specifying logic for the computation process.

[0043] Once these fields are specified, the data processing system can use the EVD definition to materialize a dataset containing only the fields actually needed for processing. This materialized dataset is called the available dataset because it contains the fields that need to be available for processing. In this example, the data processing system 12 does not materialize the EVD dataset. Rather, the EVD definition is used to materialize the available dataset 15.

[0044] User interface 19a includes section 20a which lists data catalog datasets (e.g., Customer, Active Loan, Active Loan EVD) in which the Active Loan EVD dataset is selected (in bold), and section 20b which displays fields (e.g., Loan ID, Status, FICO, Customer SSN (Social Security Number)) corresponding to fields within a dataset, such as the Active Loan EVD dataset. Section 20b allows the user to select which fields (e.g., Loan ID, Status, FICO) to include in the request sent to the expanded view dataset (EVD) definition generator 22. In this example, the fields Customer SSN and Hardship are not included in the request.

[0045] In this example, the metadata manager 24 registers the EVD definition 13a in the data catalog 14 by transmitting information to the data catalog 14 that identifies the EVD that can be generated from the EVD definition 13a, for example. Based on this, section 20a displays a visualization 13b showing that it is a dataset that can be used for processing and defining logic for the active loan EVD.

[0046] User interface 19a includes section 20a which lists data catalog datasets (e.g., Customer, Active Loan, Active Loan EVD) in which the Active Loan EVD dataset is selected (in bold), and section 20b which displays fields (e.g., Loan ID, Status, FICO, Customer SSN (Social Security Number)) corresponding to fields within the dataset, e.g., the Active Loan EVD dataset. Section 20b allows the user to select which fields (e.g., Loan ID, Status, FICO) to include in the request sent to the Extended View Dataset (EVD) Definition Generator 22. In this example, the fields Customer SSN and Hardship are not included in the request.

[0047] In this example, the user browses the data catalog 14 via a browser 19 to identify which fields and datasets are candidate datasets to be used in the computation process. Specifically, the user browses the data catalog 14 on a client device 18 by viewing a user interface 19a that presents the logical metadata of the data catalog 14 and identifiers and visual representations of business names.

[0048] In this example, user interface 19a displays a preview of the fields of the active loan EVD. This preview is generated by the data processing system 12 by using definition 13a to identify the fields and, for example, by accessing the dataset or metadata of the dataset specified in definition 13a and retrieving the fields within those datasets. In this example, only the fields Loan ID, Status, and FICO are selected. User interface 19a also includes an input command field "Input Command" used to select what type of request to send to the EVD definition generator 22. User interface 19a also includes a calculate button 21 to start the calculation of the dataset 59 (Figure 3F) in the extended view. When the calculate button 21 is selected, the client device 18 sends request 13c to the EVD integrator 26. Request 13c is a request to calculate the average FICO of the active loan.

[0049] The EVD integrator 26 sends a request to the metadata manager 24 and retrieves the EVD definition 13a from the metadata repository 25, specifying all datasets related to the active loan dataset. The EVD integrator 26 receives the definition 13a. The EVD integrator 26 integrates (or combines) the request 13c and uses the definition 13a to calculate the average FICO of the active loan.

[0050] The EVD integrator 26 sends the integrated request to the execution engine 28. The execution engine 28 generates code to execute the integrated request. The optimizer 30 optimizes the code to retrieve only the data from the dataset containing the fields selected in the user interface 19a. Execution of this optimized code generates available data 15 containing only the fields necessary to perform the calculations specified in request 13c and / or selected in the user interface 19a. The execution engine 28 executes this optimized code to retrieve data from the data source containing data related to the active loans of the selected fields. The execution engine 28 also executes the optimized code to perform the requested calculations and, as a result, stores the average FICO score of the active loans in the storage system 32. Details of optimizers such as optimizer 30 are disclosed in U.S. Patent Application Publication No. 2019 / 0370407 (titled "Systems and Methods for Dataflow Graph Optimization"), the entire contents of which are incorporated herein by reference.

[0051] Figures 3A to 3F show scaled diagrams of System 10, with specific boxes highlighted in bold. To see the relationship between the bold boxes and other elements in Figure 2 that are not specifically numbered or illustrated in Figures 3A to 3F, please refer to Figure 2.

[0052] Referring here to Figure 3A, the data catalog 14 sends a visualization of the data catalog data to the client device 18. The client device 18 renders the visualization, for example, in the user interface 19b of the browser 19. The user interface 19b includes a section 20a that lists the data catalog datasets (e.g., Customers, Active Loans) and data for Active Loans 20c that displays fields, for example, Loan ID, Customer ID, and Status. The data for Active Loans 20c is rendered with the corresponding fields, Loan ID, Customer ID, and Status. The user interface 19b also includes a control 21a "Request Extended View". When selected, control 21a "Request Extended View", sends "Request Extended View, Dataset - Base Dataset Definition: Base Dataset = Active Loans" to the EVD Definition Generator 22.

[0053] Referring to Figure 3B, the metadata manager 24 and metadata repository 25 send the extracted metadata related to the active loan from the metadata repository 25 to the EVD definition generator 22. The metadata repository 25 sends the extracted metadata (not explicitly readable in Figure 3B) to the EVD definition generator 22 as a metadata model 40. The dataset 59 in the extended view of Figure 3F, when generated, includes the base dataset definition 44 and data from one or more related dataset definitions 46 that correspond to the new dataset and are related to the base dataset definition 44, as shown in Figure 3C.

[0054] Referring to Figure 3C, the EVD definition generator 22 parses the metadata model 40 (not explicitly readable in Figure 3C) to identify the base dataset definition 44 and the related dataset definition 46. The extended view 41 shows the base dataset definition 44. The base dataset definition 44 corresponds to the active_loans.dat base dataset 44a. The active_loans.dat base dataset 44a contains two keys: the primary key loan_id and the foreign key customer_id. The related dataset definition 46 contains the "hardship.dat" dataset 46a, the "loan_details.dat" dataset 46b, and the "settlement.dat" dataset 46c, each of which is related to the active_loans.dat base dataset 44a, which contains the foreign key customer_id, by the "hardship.dat" dataset 46a, the "loan_details.dat" dataset 46b, and the "settlement.dat" dataset 46c, respectively.

[0055] Furthermore, the extended view 41 also shows related datasets, the "customer.dat" dataset 46d and the "FICO.dat" dataset 46e. The "active_loans.dat" base dataset 44a is related to the "customer.dat" dataset 46d by sharing the key "customer_id" with the "customer.dat" dataset 46d (for example, a primary key and foreign key relationship). Similarly, the "FICO.dat" dataset 46e is related to the "customer.dat" dataset 46d by sharing the key "ssn" with the "customer.dat" dataset 46d (for example, another primary key and foreign key relationship).

[0056] An extended view 41 of the base dataset definition 44 and the related dataset definition 46 is also shown in Figure 3C. The extended view 41 includes the related dataset definition 46, which includes "hardship.dat" 46a, "loan_details.dat" 46b, and "settlement.dat" 46c. As described below, the preview is generated by applying the set of data processing actions specified by the definition of dataset 50 (Figure 3F) in the extended view only to a subset of the data in the base dataset 44a and a subset of the data in the datasets related to the base dataset 44a, for example, "hardship.dat" 46a, "loan_details.dat" 46b, and "settlement.dat" 46c, as well as the "customers.dat" dataset 46d and the "FICO.dat" dataset 46e.

[0057] Details regarding the transformation from the metadata model in Figure 3C to the definition graph in Figure 3D are disclosed in U.S. Patent No. 11,423,083 (titled "Transforming a Specification into a Persistent Computer Program") and U.S. Patent No. 11,210,285 (titled "Generation of Optimized Logic from a Schema"), the entire contents of which are incorporated herein by reference.

[0058] One or more related dataset definitions 46 have one or more relationships 43a to 43e (e.g., primary key-foreign key relationships) between the base dataset 44a and one or more related datasets 46a to 46e. The user registers a new dataset definition in the data catalog 14. The base dataset definition 44 specifies the base dataset 44a and the datasets associated with the base dataset 44a. The base dataset definition 44 provides logical access to the related datasets without incurring the computational cost of providing the related datasets 46a to 46e.

[0059] From one or more attributes, system 10 determines a base dataset definition 44, and based on the base dataset definition 44, system 10 determines one or more related datasets 46a to 46e associated with the base dataset 44a. Based on the determined one or more related datasets 46a to 46e, system 10 generates a definition for an extended view dataset 59 (Figure 3F) that specifies the base dataset 44a, one or more related datasets 46a to 46e, and one or more relationships 43a to 43e between the base dataset 44a and the one or more related datasets 46a to 46e.

[0060] In this example, the EVD definition generator 22 includes a graph generator (not shown) as described in U.S. Patent No. 11,210,285 (titled “Generation of Optimized Logic from a Schema”), the entirety of which is incorporated herein by reference. In this example, the base dataset definition 44 and related dataset definition 46, identified in Figure 3C, are the logical data input to the graph generator. In this example, the graph generator is pre-configured with a specification that specifies that a graph should be generated to access all datasets in the base dataset definition 44 and related dataset definition 46 and to join them together, and thus generates an EVD definition (in this example, the active loan EVD definition 50 in Figure 3D). The graph generator uses the specification and logical data to generate a data flow graph (also referred to herein, for convenience, without limitation, as a computation graph or graph). Generally, a data flow graph (or persistent computer program) is generated from a specification such as the following: The specification specifies several modules to be implemented by a computer program to process one or more values ​​of one or more fields in a structured data item (e.g., a data record). A module can be a component of a dataflow graph, or a group of components of a dataflow graph (e.g., a subgraph). In this case, multiple modules are modules for accessing and joining data sets specified by the logical database-base dataset definition 44 and the related dataset definition 46. Thus, the graph generator creates a graph with access data components for each data source represented by the base dataset definition 44 and the related dataset definition 46. Furthermore, the graph generator adds an appropriate number of joining components to join these data sources together and add them to the data sink, as described below. In addition, these multiple modules may include rules, instructions, dataflow graph components, etc.The system described herein transforms the specification into a computer program that implements multiple modules (e.g., logical data) by specifying the processing flow between components or modules, as follows: for each of one or more first modules of the multiple modules, identify one or more second modules of the multiple modules, each of which receives input based at least partially on the output of the first module; format the output data format of the first module such that each (i) is accessible to the first module and (ii) outputs only one or more values ​​of one or more fields of a structured data item designated as input to at least one of the one or more second modules, at least partially on the output of the first module; and store the computer program in persistent memory using a saved computer program that specifies the formatted output data format for each of the one or more first modules, as described in U.S. Patent No. 11,423,083. The system also includes various rules that specify that the contents of each module are included in the computer program and / or converted into instructions in an appropriate format of the computer program. In this example, the graph generator first generates a dataflow graph with data sources represented in the logical data. The graph generator also adds data sinks to the dataflow graph, as the graph flow graph requires them. Next, the graph generator adds various components to the dataflow graph, such as sorting components, which are configured to be automatically added by the graph generator to increase the computational efficiency of the dataflow graph. The graph generator is also configured to add join components to properly combine data from various data sources. Instructions, parameters, or other information for accessing or joining data sources can be included in the logical data. Finally, the graph generator may add transformation components containing computational logic specified in the specification.The transformable component itself may contain various components or subcomponents that represent different data flow graphs when the specification is transformed into a data flow graph as described above.

[0061] Referring here to Figure 3D, System 10 is shown using an actual view of the EVD definition already defined in Figures 3A-3C. The EVD definition generator 22 returns the active loan EVD definition 50 as a computation graph 50'. The active loan EVD definition 50 provides logical access without physical cost by providing the logic of how the active loan EVD is generated without actually generating the dataset (i.e., the active loan EVD) itself, thereby requiring the data necessary for the active loan EVD to be materialized (e.g., read from the data source and stored in memory). The computation graph 50' of the active loan EVD definition 50 includes visualizations 52a-52f of the access data, such as "Access active_loans.dat" 52a, "Access customers.dat" 52b, "Access FICO.dat" 52c, "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f. These components correspond to the relevant datasets 46a to 46e in Figure 3C.

[0062] The EVD definition generator 22 returns a computation graph 50' which includes further join operations. These join operations include join operation 54a, which applies to "Access active_loans.dat" and "Access customers.dat" that are joined based on "customer_id". Join operation 54b, which applies to the result from join operation 54a, joins with "Access FICO.dat" based on "ssn". Join operation 54c, which applies to the result from join operation 54b, joins with "Access hardship.dat" 52d and then with "loan_id". "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f are shown as being merged together into a single input to join operation 54c. As described below, the available dataset is generated from this computation graph 50' and optimized to remove "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f as data sources. This is because "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f" contain data that is not used by downstream components when generating the specified output. In other words, "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f are optimized as described below.

[0063] Referring to Figure 3E, the active loan EVD definition 50 is sent to the metadata manager 24 and the metadata repository 25. The metadata manager 24 sends the EVD identifier (active loan EVD) to the data catalog 14. The active loan EVD definition 50 is sent to the metadata manager 24 and the metadata repository 25. The data catalog 14 sends a visualization of the data catalog data to the client device 18. The client device 18 renders the browser 19 with a user interface 19c that displays a section 20a (labeled "Data Catalog") that displays the dataset from the data catalog 14.

[0064] Browser 19 renders a user interface 19c that displays a visualization of data catalog data, and a data viewing button 21b. Client device 18 sends a request to EVD integrator 26 to view the active loan EVD. Metadata manager 24 and metadata repository 25 receive a specification (active loan EVD definition) that specifies data processing behavior in response to providing a preview. The data processing behavior in the specification is defined at least partially based on user input that identifies attributes included in the preview as attributes of that data processing behavior.

[0065] Referring to Figure 3F, the metadata manager 24 sends the active loan EVD definition 50 from the metadata repository 25 to the EVD integrator 26. The EVD integrator 26 sends the active loan EVD definition 50 to the execution engine 28. The execution engine 28 compiles the computation graph 50' and executes the compiled graph to generate the dataset "active loan EVD" which is generated by executing the data flow graph, which is the active loan EVD definition 50, on the data in the data source 34. That is, when generating the active loan EVD 59, the data from the data source 34 is materialized (e.g., read from the data source 34) and processed to provide the new dataset - active loan EVD 59 - in response to a request. The execution engine 28 also generates a visualization 60 of the active loan EVD 59, allowing the user to view the data in the active loan EVD. For example, a visualization may show only a specified amount of data (e.g., the first five results) to generate the visualization without latency when loading it, even if the visualization contains all the data in the active loan EVD 59 (e.g., one million data items). The generated dataset "active loan EVD" is stored in the storage system 32 and registered in the data catalog 14.

[0066] System 10 receives a request for an extended view dataset 59 and, in response to the request, provides the extended view dataset 59 by retrieving the definition 50 of the extended view dataset 59 from a hardware storage device, retrieving the base dataset and one or more other datasets, i.e., related datasets, from one or more data sources. From the extended view dataset 59, System 10 generates a visualization 60 of the extended view dataset 59.

[0067] Figures 4A to 4E show an example of data catalog editing operation. Figures 4A to 4E show a scaled-down view of system 10, with certain boxes highlighted in bold. Refer to Figure 2 to see the relationship between the bold boxes and other elements in Figure 2 that are not specifically numbered or shown in Figures 4A to 4E.

[0068] Referring here to Figure 4A, the data catalog 14 sends a visualization of the data catalog data to the client device 18. The client device renders the visualization to the browser 19 as a user interface 19d. The user interface 19d includes a data catalog 20a that lists data catalog datasets (e.g., Customers, Active Loans, Active Loan EVDs) as well as operations such as enrich, filter, calculate, and store. The user interface 19d also includes an editor 20d for editing the listed data catalog datasets, e.g., the "Active Loan EVD" dataset. The editor 20d allows the user to define specifications that are compiled into a graph and executed. The editor 20d includes a "Preview" control 21c and an "Execute" control 21d. Based on the selection in the preview control 21c, the client device generates a data preview 62. In Figure 4A, the generated data preview 62 corresponds to the visualization 60 of the Active Loan EVD 59 in Figure 3F.

[0069] In this example, the data preview 62 provides logical access without physical cost, meaning that the preview 62 is generated by running the active-loaned EVD definition 50 on only a fraction of the records in the data source 34, rather than on all the data records in the data source 34. Running the active-loaned EVD definition 50 on all the data records in the data source 34 would consume considerable memory and processing resources to collect data on fields that may never be used (e.g., data on fields that are not used or accessed by the specification defined in editor 20d). However, the user is given logical access to all fields, as they can see the fields defined in definition 50 and, furthermore, see the values ​​of those fields when deciding which fields they want to use in their specification (e.g., which fields they want to filter by). Then, when the actual dataset (active-loaned EVD) is materialized (e.g., generated), it can be optimized to materialize only the fields used by the data processing behavior defined by the specification in editor 20d.

[0070] Referring here to Figure 4B-1, the client device 18 renders the visualization to the browser 19 as a user interface 19d. The client device 18 sends the graph specification to the EVD integrator 26. The user interface 19d includes a data catalog 20a that lists data catalog datasets (e.g., Customer, Active Loan, Active Loan EVD) as well as operations such as enrich, filter, calculate, and store. The user interface 19d also includes an editor 20d for editing the listed data catalog datasets, e.g., the "Active Loan EVD" dataset. The editor 20d includes a "Preview" control 21c and an "Execute" control 21d, as well as an additional preview control 21e for previewing "Status = Active and Filtered" and an additional preview control 21f for "Preview Calculation of Average FICO". Based on the selection of preview controls 21c, 21e, and 21f, the client device 18 sends instructions to the EVD integrator 26 to generate the corresponding data preview (not shown). An example of a graph specification, such as Figure 4B-1, which can be converted into executable code, is given in U.S. Patent Application Publication No. 2021 / 0232579 (titled "Editor for Generating Computational Graphs"), the entire contents of which are incorporated herein by reference.

[0071] In this example, graph 19' is generated by editor 19d. Graph 19' is an example of a declaration graph where each component represents a declared action, which is then transformed into an instruction action as described herein. The client device 18 transmits the graph specification 66 to the EVD integrator 26. The graph specification 66 represents and / or specifies the contents of graph 19'. Based on the graph specification 66, the EVD integrator 26 generates graph 66' as described in U.S. Patent No. 11,593,380.

[0072] In this example, the graph specification 66 includes selection data specifying which icon in the user interface 19d has been selected, along with other information and / or values ​​specified in the user interface 19d. The EVD integrator 26 includes a dataflow graph engine (not shown) that receives the selection data from the client device 18. The selection data indicates the data sources, data sinks, and data processing functions for the desired computation graph. The user of the client device 18 does not need to specify details of data access or other low-level implementation forms, because these details can be derived by the dataflow graph engine. Based on the selection data, the dataflow graph engine generates a dataflow graph 66' or modifies a previously created dataflow graph. In some examples, the dataflow graph engine modifies the dataflow graph by, for example, removing redundancy in the dataflow graph, adding reorders or splits to the dataflow graph, and specifying intermediate metadata (e.g., metadata for transforming or otherwise transforming the dataflow graph), among other optimizations and modifications. The EVD integrator 26 transmits the dataflow graph 66' to an execution engine 28, which includes a compiler that compiles the dataflow graph 66' into a compiled computation graph (e.g., an executable program).

[0073] Referring to Figure 4B-2, an example of generating a dataflow graph 66' from graph specification 66 is described here. In this example, using graph specification 66, the EVD integrator 26 generates a dataflow graph 17 (sometimes referred to as the "initial dataflow graph" or "preliminary dataflow graph") that represents the core components of a compiled graph, such as a modified dataflow graph 50" (Figure 4C) having nodes (or components). The dataflow graph 17 optionally includes parameters (e.g., name, value, location, interpretation). In some implementations, the dataflow graph 17 includes input and output ports in the graph itself, as in a graph intended to be used as a subgraph.

[0074] In some implementations, a node (or component) has, or is, a node "type" that indicates the behavior or function of the node. The node type is used to select a node prototype, to facilitate pattern matching (e.g., to find a reorder node followed by another reorder node), and to determine which components are instantiated in the transformed dataflow graph 50" (or 66, depending on the level of transformation). For example, a garbage node in dataflow graph 23 may be instantiated as a garbage node in the transformed dataflow graph 66. A node (or component) may include input ports, output ports, and parameters, as will be considered below.

[0075] Nodes optionally have labels to identify them. In some implementations, if a node does not have a label, the system assigns one to it. Node labels can contain any combination of alphanumeric characters, whitespace, and punctuation, and do not need to be unique (although they can be unique during conversion to a graph). The system can use node labels to refer to nodes (or their input ports, output ports, or parameters) to define, for example, the inputs and outputs of nodes or the data flow between nodes.

[0076] In some examples, before generating the dataflow graph 17, the EVD integrator 26 includes a template dataflow graph, the dataflow graph 17. The dataflow graph 17 is shown as including nodes 34a to 34n. Each of nodes 34a to 34n includes at least one behavior placeholder field and at least one data placeholder field. For example, the “initial” node 34a has a behavior placeholder field 35a that holds one or more behavior elements 35a', and a data placeholder field 35b that holds one or more data source or data sink elements 35b'. The behavior element 35a' may specify code or the location of code that performs a function in the input and output of data to the initial node 34a. The data source or data sink element 35b' may specify a data source or data sink (for the functionality of the initial node 34a), or the location of a data source or data sink. In some implementations, element 35a' or element 35b', or both, includes a link to a database or a pointer to code contained in the storage system 32, or a link or address to the storage system or storage system 32 included in the EVD integrator 26. In some implementations, element 35a' or element 35b', or both, includes a script.

[0077] During the construction of the dataflow graph 17, each of the nodes 34a to 34n can be modified by taking out the operation element placed in the operation placeholder field and the data source or data sink element placed in the data placeholder field and inputting them into their respective fields. For example, the initial node 34a is modified during construction by taking out the operation element 35a' (e.g., from the storage system based on the operation specified by specification 66) and inputting a link pointing to the specified function or function into the operation placeholder field 35a, and by taking out the data source or data sink element 35b' and inputting a link pointing to the data source or sink into the data placeholder field 35b. Once the modification of a particular node 34a to 34n is complete, the node can be labeled to provide labeled nodes. After each of the nodes 34a to 34n has been modified (and labeled), the completed dataflow graph 17 is stored (e.g., in the storage system 32) and used to generate other dataflow graphs as described below.

[0078] In some implementations, each of the nodes 34a to 34n of the dataflow graph 17 is initially unmodified. For example, each of the nodes 34a to 34n may have an empty operation placeholder field 35a and a data placeholder field 35b, which are later modified to include a specified operation element 35a' and a data source or data sink element 35b', as described above. In some implementations, the dataflow graph 17 is a previously completed dataflow graph, and some or all of the nodes 34a to 34n have a corresponding operation placeholder field 35a that holds the operation element 35a' and a data placeholder field 35b that holds the data source or data sink element 35b'. The completed, such dataflow graph 17 may be further modified (for example, by taking out additional or alternative elements 35a', 35b' to be placed in the respective fields 35a, 35b) and stored as a new or modified dataflow graph.

[0079] In some implementations, certain nodes, such as the initial node 34a, are "reused" to generate new, optionally labeled nodes associated with the previous node 34a. This iterative process of generating new nodes from the initial node 34a continues until the user specifies the desired functionality of the computation graph. Once the iterative process is complete, a completed dataflow graph 17 is provided. The completed dataflow graph 17 includes, for example, multiple nodes 34a~34n instantiated from the initial node 34a. The completed dataflow graph 17 can be stored (for example, in the storage system 32) and used to generate other dataflow graphs, as described below.

[0080] Figure 4B-2 also illustrates one implementation form of the completed (e.g., modified) dataflow graph 17'. The modified dataflow graph 17' is shown as containing four nodes labeled OP-0 to OP-3, each having a corresponding operation placeholder field 35a that holds an operation element and a data placeholder field 35b that holds a data source or data sink element. For example, node 34a labeled OP-0 contains a read operation element 37a' that indicates that the data source element 37b' of "Dataset I" is read. The modified dataflow graph 17' is stored in the storage system 32 as a data structure, for example.

[0081] Generally, the execution engine 28 performs optimizations or other transformations that may be necessary to process the data according to one or more of the operations specified in the dataflow graph 17', or improves processing the data according to one or more of the operations specified in the dataflow graph 17' compared to processing the data without optimization or transformation, or both. For example, the execution engine 28 adds, among other things, one or more sorting operations, data type operations, join operations including join operations based on keys specified in the dataflow graph 17', partition operations, automatic parallel operations, or operations specifying metadata to produce a transformed dataflow graph 50" (Figure 4D) having the desired functionality of the dataflow graph 17'. In some implementations, the transformed dataflow graph 50" is an optimized dataflow graph (or is transformed into an optimized dataflow graph) by applying one or more dataflow graph optimization rules to the transformed dataflow graph to improve the computational efficiency of the transformed dataflow graph compared to the computational efficiency of the transformed dataflow graph before the optimization is applied. Data flow graph optimization rules may include, for example, the removal of dead or redundant components, early filtering, or record reduction, as described below in relation to Figure 4D.

[0082] Referring here to Figure 4C, the metadata manager 24 and metadata repository 25 send the active loan EVD definition 50 to the EVD integrator 26. Based on graph 66' and the active loan EVD definition 50, the EVD integrator 26 generates a data flow graph 50'', for example, as follows. Using the active loan EVD definition 50, the EVD integrator 26 adds the necessary nodes to graph 17' to generate the active loan EVD. In this example, the EVD integrator 26 includes logic to insert the active loan EVD definition into graph 17' (as shown by graph 50' in Figure 3D) by removing the access active loan EVD component from graph 66' and replacing it with the version in graph 50', in which the write component 56 (Figure 3D) is removed from graph 50'. By doing so, the EVD integrator 26 generates a computation graph 50' with the status = active and the operation of calculating filters and average FICO added, and therefore stores to generate computation graph 50''.

[0083] Referring to Figure 4D, the execution engine 28 and optimizer 30 optimize the computation graph 50” to remove components that are substantially unused, for example, among other optimizations. In this example, the components of the average FICO calculation do not require data from the following data sources, namely access hardship.dat, access loan details.dat, or access settlement.dat. Therefore, reading data from these data sources would result in reading data that has never been used by downstream components. This would result in a waste of memory and computational resources. To ensure that the computation graph 50” is computationally efficient and does not waste memory and processing resources when retrieving unused data, optimizer 30 includes optimizer rules and logic configured to analyze the components of the data flow graph and remove components that are not used by downstream components (e.g., read components). Thus, the generation of the available dataset 29 (Figure 4E) is computationally efficient, with reduced use of memory and processing resources compared to the amount of processing and memory resources used when generating the available dataset containing all data sources specified in the active loan EVD definition. In this example, downstream refers to a component whose input is based on the output of another component. Based on this, the execution engine 28 and optimizer 30 perform data processing operations (including, for example, optimization rules and / or data processing operations) to generate a computation graph "active loan EVD definition" that contains only a subset of the data that would have been included in the extended view's dataset definition 50. The "active loan EVD definition computation graph 50" includes "Access active_loans.dat", "Access customers.dat", and "Access FICO.dat".The Active Loan EVD Definition Calculation Graph 50 contains "Access active_loans.dat" and is filtered by status=active, specifically excluding "Access hardship.dat", "Access loan_details.dat", and "Access settlement.dat" in this example. The execution engine 28 and optimizer 30 perform a join operation with "Access active_loans.dat" (filtered by status=active) and "Access customers.dat" and join it to customer_id 54a. The result of joining "Access FICO.dat" with customer_id 54a is joined to ssn 54b. This result is passed to the calculation of the average FICO, and the generated dataset "Active Loan EVD" is stored in the storage system 32.

[0084] The bold "X" in Figure 4D indicates the portion of computation graph 50' (Figure 4C) that is not included in computation graph 50'' in Figure 4D. That is, as described in U.S. Patent Application Publication 2019 / 0370407, the datasets "Access hardship.dat", "Access loan_details.dat", and "Access settlement.dat" are excluded from computation graph 50'' based on optimization.

[0085] In this example, referring back to Figure 4B-1, the specification presented in the editor (also referred to herein as the graph specification) specifies that data from the Active Loan EVD should be sorted by status=active, and then the average FICO score should be calculated. Therefore, according to this specification, not all datasets presented in the Active Loan EVD are substantially required to perform the calculations specified in the editor. Active_loans.dat is required because it is the base dataset. Furthermore, FICO.dat is required because the specification specifies that the average FICO score should be calculated per user. Customer.dat is required because it contains the keys necessary to associate this dataset with FICO.dat and active_loans.dat. In this example, hardship.dat, loan_details.dat, and settlement.dat are not required for the processing described by the specification. Therefore, the optimizer 30 removes the references to those data sources in the Active Loan EVD definition 50. By removing the references to those data sources, the Active Loan EVD does not contain any unnecessary fields and data that are not used by the data processing behavior specified in the specification when it is generated. In this example, the active-loan EVD that is actually generated contains only the data used by the data processing operations specified in the specification (and therefore made available), so the active-loan EVD is referred to as the available dataset. Thus, the available dataset is very efficient in terms of speed and memory (because the system does not need to store data as part of the available dataset that has never been used) (for both generating and processing the available dataset).

[0086] Referring to Figure 4E, the execution engine 28 and optimizer 30 execute a computation graph "Active Loan EVD Definition" which contains only a subset of the data that would have been included in the extended view dataset. The Active Loan EVD Definition computation graph 50" includes "Access active_loans.dat", "Access customers.dat", and "Access FICO.dat". The Active Loan EVD Definition computation graph 50" has "Access active_loans.dat" and filters it by status=active, specifically excluding "Access hardship.dat", "Access loan_details.dat", and "Access settlement.dat" in this example. When executing the compiled version of computation graph 50", the execution engine 28 generates an available dataset 29, for example, upon completion of the "Join to SSN" component and before executing the "Calculate Average FICO" component. The available dataset 29 contains only the data from active_loans.dat, customers.dat, and FICO.dat, as described above. In this example, the available dataset 29 includes all the data from the three data sources, rather than implicitly meaning that only a subset of the data in those data sources will be used to generate the preview. The execution engine 28 and optimizer 30 perform a join operation on "Access active_loans.dat" and "Access customers.dat" (filtered by status=active) and join them to customer_id54a. The result of joining "Access FICO.dat" with customer_id54a is joined to ssn54b. The computation graph 50 calculates the average FICO score and stores the result.

[0087] The computation graph 50" performs calculations directly. The computation graph 50" retrieves the necessary data for each component represented by the graph components, moves the data between components, and defines the execution order of the computation process. It can also provide monitoring of the execution engine 28 or the execution of the computation graph 50". The results of executing the computation graph 50" Active Loan EVD" are stored in the storage system 32 via the storage component of the computation graph 50".

[0088] Referring here to Figure 5, a process 150 is shown performed by the data processing system 12 to enable the user to preview the attributes of fields in an extended view dataset and to grant the user the authority to specify one or more attributes to be used in one or more data processing operations optimized in terms of speed and data memory. Process 150 enables the user to specify one or more fields to be used in downstream data processing to generate a dataset containing one or more of the fields from the preview specified for use in downstream data processing, and the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all fields of the extended view when only one or more of the specified fields are used in the downstream data.

[0089] Process 150 includes receiving an identification of a base dataset (152) and, based on the identification, receiving a definition of an extended view dataset (154). The extended view dataset definition specifies other datasets related to the base dataset or fields of other datasets. Based on the extended view dataset definition, process 150 includes outputting a preview of the attributes of the fields in the extended view dataset (156), the preview being generated from a subset of data in other datasets related to the base dataset or a subset of data in fields of other datasets. Process 150 also includes receiving input specifying that one or more of the fields in the preview are available for data processing (158) and, based on the input, generating an available dataset containing data in the base dataset and data in one or more of the specified fields (159).

[0090] Referring here to Figure 5A, process 160 may include other features such as providing a preview of the attributes of the extended view dataset, which, when generated, includes data from the base dataset and other datasets associated with the base dataset (162). Process 160 includes (163) a definition of the extended view dataset that specifies a set of data processing operations performed to generate the extended view dataset, which includes data from the base dataset and other datasets associated with the base dataset, and the preview is generated by applying the set of data processing operations specified by the definition of the extended view dataset to only a subset of the data in the base dataset and other datasets associated with the base dataset (164). In response to providing the preview, process 160 includes receiving a specification that specifies the data processing operations, which are defined at least in part based on user input that identifies the attributes included in the preview as attributes of those data processing operations (165).

[0091] Referring here to Figure 5B, based on a data processing behavior that is at least partially defined based on user input identifying the attributes included in the preview as attributes of that data processing behavior, process 170 may include updating the defined set of data processing behaviors by applying one or more optimization rules to the set of data processing behaviors (166), and executing the updated set of data processing behaviors to generate a dataset that contains only a subset of the data that would have been included in the extended view dataset (168).

[0092] Process 170 may include enabling a user to register a definition of a new dataset in the data catalog using a definition that specifies a selected dataset and other datasets related to the selected data, thereby providing logical access to other datasets related to the base dataset without incurring the computational cost of providing the other related datasets (169). Process 170 may include accessing a data catalog that specifies one or more datasets and providing a user interface that indicates that one or more datasets are candidates for generating datasets for an extended view (171). Process 170 may include receiving metrics for a particular dataset as a base dataset via the user interface and automatically generating a definition of a dataset for an extended view for a particular base dataset in response to the metrics (172).

[0093] Process 170 may include identifying a specific base dataset as a base dataset and one or more attributes of the specific base dataset; determining the definition of the base dataset from the one or more attributes; determining one or more other datasets related to the base dataset based on the definition of the base dataset; and generating a dataset definition for an extended view that specifies the base dataset, one or more other datasets, and one or more relationships between the base dataset and the one or more other datasets, based on the determined one or more other datasets (174).

[0094] Process 170 may include storing the definition of the extended view dataset in a hardware storage device and registering the definition of the extended view dataset in a data catalog (176).

[0095] Referring here to Figure 5C, process 180 may include receiving a request for an extended view dataset (182), and providing an extended view dataset in response to the request by retrieving a definition of the extended view dataset from a hardware storage device, retrieving a base dataset and one or more other datasets from one or more data sources based on the definition of the extended view dataset, and generating an extended view dataset based on the data in the retrieved datasets (184).

[0096] Process 180 may include deciding whether to update the data catalog to specify the definition of the extended view dataset as a data source based on the extended view dataset (186), storing the definition of the extended view dataset in a hardware storage device, and registering the definition of the extended view dataset in the data catalog (188). Process 180 may also include receiving a request for the extended view dataset, and in response to the request, retrieving the definition of the extended view dataset from a hardware storage device, retrieving a base dataset and one or more other datasets from one or more data sources based on the definition of the extended view dataset, and generating the extended view dataset based on the data in the retrieved datasets.

[0097] A dataflow graph component includes data processing components and / or datasets. A dataflow graph can be represented by a directed graph containing nodes or vertices, where nodes or vertices represent dataflow graph components, connected by directed links or dataflow connections, and represent the flow of work elements (i.e., data) between dataflow graph components. A data processing component includes code for processing data from at least one data input (e.g., a data source) and providing the data to at least one data output of system 10 (e.g., a data sink). Thus, a dataflow graph can perform graph-based computations on data flowing from one or more input datasets through graph components to one or more output datasets.

[0098] System 10 also includes a data processing system 12 for executing one or more computer programs (such as a data flow graph) generated by transforming a specification into a computer program using the transformation generators and techniques described herein. The transformation generator transforms the specification into a computer program that implements multiple modules. In this example, selections made by the user through the user interface described herein form a specification that specifies which fields and datasets are used in the composite aggregation. Based on the specification, the transformations described herein are generated.

[0099] The data processing system 12 may be hosted on one or more general-purpose computers under the control of a suitable operating system, such as a UNIX operating system. For example, the data processing system 12 may include a multi-node parallel computing environment that includes a computer system configuration using multiple central processing units (CPUs) that are local (e.g., a microprocessor system such as an SMP computer), locally distributed (e.g., multiple processors linked as a cluster or MPP), remotely distributed (e.g., multiple processors linked via a LAN or WAN network), or any combination thereof.

[0100] The graph construction approach described above can be implemented using software to run on a computer. For example, the software may form a procedure in one or more computer programs running on one or more systems 10, e.g., computer-programmed systems or computer-programmable systems (which may be of various architectures such as distributed, client / server, or grid), each system including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software may form one or more modules of a larger program that provides other services related to the design and construction of data flow graphs, for example. The nodes and elements of the graph can be implemented as data structures stored in computer-readable media, or as other organized data conforming to data models stored in a data repository.

[0101] The software may be provided on a hardware storage device readable by a general-purpose or dedicated programmable computer, such as a non-temporary storage medium like a CD-ROM, or it may be delivered (encoded into a propagating signal) to a computer on which the software is executed via a network communication medium. All functions may be performed on a dedicated computer or using dedicated hardware such as a coprocessor. The software may be implemented in a distributed manner, with different parts of the data flow specified by the software being performed by different computers. Each such computer program is preferably stored or downloaded onto a non-temporary storage medium or hardware storage device readable by a general-purpose or dedicated programmable computer (e.g., solid-state memory or media, or magnetic or optical media), and the computer is configured and operated to perform the procedures described herein when the non-temporary storage medium or device is read by system 10. System 10 may also be considered to be implemented as a computer-readable storage medium configured with a computer program, and such a storage medium causes system 10 to operate in a specific and predefined manner to perform the functions described herein.

[0102] Exemplary computing environment Referring to Figure 6, an exemplary operating environment for carrying out embodiments of the present invention is shown, and is explicitly referred to as a computing device 120. Essential elements of the computing device 120, or a computer, data processing system, client, or server, are one or more programmable processors 122 for performing actions according to instructions, and one or more memory devices 124 for storing instructions and data. Generally, the computer also includes I / O components 126, such as a display device, a network / communication subsystem (not shown), and one or more mass storage devices 128 for storing data and instructions, etc., and a network communication subsystem 130 powered by a power supply (not shown), or operably connected to them (via a bus 121, fabric, network, etc.). The memory device 124 contains an operating system 124a and an application 124b for application programming.

[0103] Suitable devices for storing computer program instructions and data include, for example, all forms of non-volatile memory, media, and memory devices, including semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory can be complemented by or incorporated into dedicated logic circuits.

[0104] To provide user interaction, embodiments of the subject matter described herein are implemented on a computer having a display device (monitor) for displaying information to the user, a keyboard that allows the user to provide input to the computer, and a pointing device (e.g., a mouse or trackball). Furthermore, the computer can interact with the user by sending documents to and receiving documents from a device used by the user (for example, by sending a web page to a web browser on the user's user device in response to a request received from a web browser).

[0105] Embodiments of the subject matter described herein can be implemented in a computing system that includes a backend component (e.g., as a data server), a middleware component (e.g., an application server), a frontend component (e.g., a user computer having a graphical user interface or a web browser that allows a user to interact with the implementation of the subject matter described herein), or any combination of one or more such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (LANs) and wide area networks (WANs), internetworks (e.g., the Internet), and peer-to-peer networks (e.g., ad-hoc peer-to-peer networks).

[0106] A computing system can include users and servers. Clients and servers are generally geographically separated and typically interact through a communication network. The relationship between a client and a server is established by computer programs running on each computer that have a client-server relationship with each other. In some embodiments, the server transmits data (e.g., an HTML page) to a client device (for example, to display data to a user interacting with a user device and to receive user input from the user interacting with the user device). Data generated on the client device (e.g., the results of user interaction) can be received by the server from the client device.

[0107] This specification includes many details of specific implementations, but these should not be construed as limitations on the scope of any invention or claim, but rather as descriptions of features specific to a particular embodiment of a particular invention.

[0108] Similarly, although the operations are depicted in a specific order in the drawings, it should not be understood that such operations must be performed in that specific or sequential order, or that all illustrated operations must be performed, in order to achieve the desired result. Furthermore, the separation of various system components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged in multiple software products.

[0109] Several embodiments have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the techniques described herein. For example, some of the above steps may be sequence-independent and can therefore be performed in an order different from that described. Furthermore, any of the aforementioned techniques described in relation to data flow graphs can also be implemented and performed in relation to programs. Therefore, other embodiments are also within the scope of the following claims.

Claims

1. A method implemented by a data processing system for enabling a user to preview the attributes of fields in an extended view of a base dataset and to specify one or more of the fields for use in downstream data processing, and generating a dataset from the preview specified for use in the downstream data processing, wherein the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all of the fields in the extended view when only one or more of the fields specified for use in the downstream data processing, Receiving the identification of the base dataset, Based on the aforementioned identification, the definition of the extended view dataset is received, The definition of the dataset in the extended view specifies, receive, other datasets related to the base dataset or fields of other datasets, Outputting a preview of the field attributes of the dataset in the extended view, based on the definition of the dataset in the extended view, wherein the preview is generated from a subset of the data in the other dataset related to the base dataset or a subset of the data in the field of the other dataset. Receiving input that specifies one or more of the fields in the preview can be used for data processing, A method comprising generating a usable dataset based on the input, which includes data in the base dataset and data in one or more of the specified fields.

2. To provide the extended view dataset, which includes the base dataset and data from other datasets related to the base dataset when generated, to the preview of the attributes of the extended view dataset, The method according to claim 1, further comprising:

3. The definition of the dataset of the extended view specifies a set of data processing operations performed to generate the dataset of the extended view, which includes the data from the base dataset and other datasets related to the base dataset. The method of claim 2, wherein the preview is generated by applying the set of data processing operations specified by the definition of the dataset in the extended view only to the base dataset and subsets of the data in the other datasets associated with the base dataset.

4. In response to providing the aforementioned preview, the further includes receiving a specification that specifies a data processing operation, the data processing operation of the specification being defined at least partially based on user input that identifies the attributes included in the preview as attributes of the data processing operation. The method according to claim 2.

5. Updating the defined set of data processing operations by applying one or more optimization rules to the set of data processing operations based on the data processing operations that are at least partially defined based on the user input that identifies the attributes included in the preview as attributes of the data processing operations, To generate a dataset containing only a subset of the data that would have been included in the dataset of the extended view, the updated set of data processing operations is performed, The method according to claim 3, further comprising:

6. The further includes enabling a user to register the definition of a new dataset in a data catalog, using a definition of a new dataset that specifies a selected dataset and other datasets related to the selected dataset, wherein the definition provides logical access to the other datasets related to the base dataset without incurring the computational cost of providing the other related datasets. The method according to claim 1.

7. Accessing a data catalog that specifies one or more datasets, To provide a user interface that indicates that one or more datasets are candidates for generating the dataset for the extended view, The method according to claim 6, further comprising:

8. The user interface receives metrics from a specific dataset as the base dataset, In response to the aforementioned indicator, the definition of the extended view dataset for the specific dataset is automatically generated, The method according to claim 7, further comprising:

9. Identifying the specific dataset as the base dataset, and one or more attributes of the specific dataset, The definition of the base dataset is determined from one or more of the aforementioned attributes, Based on the definition of the base dataset, determine one or more other datasets related to the base dataset, Based on the one or more other datasets determined above, generate the definition of the dataset for the extended view, specifying the base dataset, the one or more other datasets, and one or more relationships between the base dataset and the one or more other datasets. The method according to claim 8, further comprising:

10. The hardware storage device stores the definition of the dataset of the extended view, Registering the definition of the dataset of the extended view in the data catalog, The method according to claim 1, further comprising:

11. Receiving requests for the dataset of the aforementioned extended view, In response to the above request, Retrieving the definition of the extended view dataset from the hardware storage device, Based on the definition of the dataset in the extended view, extract the base dataset and the one or more other datasets from one or more data sources, and To provide the extended view dataset by generating the extended view dataset based on the data in the extracted dataset, The method according to claim 1, further comprising:

12. Based on the dataset of the extended view, determine whether to update the data catalog to specify the definition of the dataset of the extended view as the data source. The hardware storage device stores the definition of the dataset of the extended view, The definition of the dataset of the extended view is registered in the data catalog, The method according to claim 11, further comprising:

13. Based on the provided preview of the attributes of the dataset of the extended view, determine whether to update the data catalog to specify the definition of the dataset of the extended view as a data source. The method according to claim 7, further comprising:

14. Generating the aforementioned usable dataset is The method according to claim 1, comprising using the definition of the datasets of the extended view to access only those datasets having the specified one or more fields, and including the data of the accessed datasets in the available dataset.

15. Processing the generated available dataset and obtaining results from processing the data in the available dataset, The method according to claim 1, further comprising:

16. To provide user permissions to access the portion of the base dataset within the extended view dataset, while denying user access to the rest of the base dataset, The method according to claim 1, further comprising:

17. The method according to claim 1, wherein the definition of the extended view dataset includes a computation graph specifying a set of data processing operations for generating the extended view dataset, which includes the data from the base dataset and the other datasets associated with the base dataset, the set of data processing operations includes at least one operation for combining the data from the base dataset with the data from at least one of the other datasets associated with the base dataset.

18. The method according to claim 1, wherein the definition of the dataset in the extended view provides logical access to data from the base dataset and the other datasets related to the base dataset.

19. A data processing system for enabling a user to preview the attributes of fields in an extended view of a base dataset and to specify one or more of the fields for use in downstream data processing, and for generating a dataset from the preview specified for use in the downstream data processing that the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all of the fields in the extended view when only one or more of the fields specified for use in the downstream data processing, One or more processing devices, The system includes one or more machine-readable hardware storage devices that store instructions that can be executed by one or more processing devices in order to perform an operation, and the operation is Receiving the identification of the base dataset, Based on the aforementioned identification, the definition of the extended view dataset is received, The definition of the dataset in the extended view specifies, receive, other datasets related to the base dataset or fields of other datasets, Outputting a preview of the field attributes of the dataset in the extended view, based on the definition of the dataset in the extended view, wherein the preview is generated from a subset of the data in the other dataset related to the base dataset or a subset of the data in the field of the other dataset. Receiving input that specifies one or more of the fields in the preview can be used for data processing, A data processing system comprising generating a usable dataset based on the input, which includes data in the base dataset and data in one or more of the specified fields.

20. One or more machine-readable hardware storage devices for generating and generating a dataset containing one or more of the fields from the preview specified for use in the downstream data processing, wherein the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all of the fields in the extended view when only one or more of the specified fields are used in the downstream data processing, the one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform the operation, Receiving the identification of the base dataset, Based on the aforementioned identification, the definition of the extended view dataset is received, The definition of the dataset in the extended view specifies, receive, other datasets related to the base dataset or fields of other datasets, Outputting a preview of the field attributes of the dataset in the extended view, based on the definition of the dataset in the extended view, wherein the preview is generated from a subset of the data in the other dataset related to the base dataset or a subset of the data in the field of the other dataset. Receiving input that specifies one or more of the fields in the preview can be used for data processing, One or more machine-readable hardware storage devices, comprising generating a usable dataset based on the input, which includes data in the base dataset and data in one or more of the specified fields.

21. A method implemented by a data processing system for enabling a user to preview the attributes of fields in an extended view of a base dataset and to specify one or more of the fields for use in downstream data processing, and generating a dataset from the preview specified for use in the downstream data processing, wherein the generated dataset has increased efficiency in terms of speed and data memory compared to the efficiency of generating a dataset containing all of the fields in the extended view when only one or more of the fields specified for use in the downstream data processing, Receiving the identification of the base dataset, Based on the aforementioned identification, the definition of the extended view dataset is received, The definition of the dataset in the extended view specifies, receive, other datasets related to the base dataset or fields of other datasets, Outputting a preview of the field attributes of the dataset in the extended view, based on the definition of the dataset in the extended view, wherein the preview is generated from a subset of data in the other dataset related to the base dataset or from metadata of the other dataset. Receiving input that specifies one or more of the fields in the preview can be used for data processing, A method comprising generating a usable dataset based on the input, which includes data in the base dataset and data in one or more of the specified fields.

22. The method according to claim 21, wherein the preview is generated during development and the available dataset is generated at runtime.

Images