Logical access to preview extended view datasets

Abstract

A method implemented by a data processing system for enabling a user to preview attributes of fields of an extended view of a base data set and specify one or more of the fields for downstream data processing, and generating a dataset comprising one or more of the fields from the preview designated for downstream data processing, where when only the designated one or more of the fields are used in downstream data processing, the generated dataset is more efficient than generating a dataset comprising all the fields of the extended view. And the efficiency is improved in terms of speed and data storage.

Description

[0001] Priority Statement

[0002] This patent application claims priority to U.S. Patent Application Serial No. 18 / 492,904, filed October 24, 2023, and U.S. Patent Application Serial No. 63 / 491,921, filed March 23, 2023, pursuant to 35 USC §119(e), the entire contents of which are incorporated herein by reference. Background Technology

[0003] This disclosure relates to techniques for customizing views into large, complex databases.

[0004] Modern data processing systems manage massive amounts of data within an enterprise. For example, large organizations may have millions of datasets. These datasets can support multiple aspects of the enterprise's operations. Complex data processing systems typically process data in multiple stages, with the results produced by one stage being fed to the next. The overall flow of information through such systems can be described according to a directed data flow graph, where nodes or vertices in the graph represent components (data files or processes), and links or "edges" in the graph indicate the data flow between those components. Systems for performing such graph-based computations are described in U.S. Patent No. 5,966,072, entitled "Executing Computations Expressed as Graphs," which is incorporated herein by reference.

[0005] Graphs can also be used to directly invoke calculations. The graphs generated by this system provide methods for obtaining information about the processes represented by the graph components, for moving between processes, and for defining the order in which processes run. Systems that invoke these graphs include algorithms for selecting inter-process communication methods and scheduling process execution, and also provide algorithms for monitoring the execution of the graphs.

[0006] To support a wide range of functionalities, data processing systems can execute applications, whether to implement routine processes or to extract insights from datasets. Applications can be programmed to access data repositories to read and write data. Summary of the Invention

[0007] Generally speaking, in a first aspect, a method implemented by a data processing system includes: enabling a user to preview attributes of fields of an extended view of a base dataset and specify one or more fields of the field for downstream data processing; generating a dataset including the one or more fields of the field specified for the downstream data processing from the preview, wherein the generated dataset has improved efficiency in terms of speed and data storage compared to generating a dataset including all fields of the extended view when only the specified one or more fields of the field are used in the downstream data processing; the method includes: receiving an identifier of a base dataset; receiving, based on the identifier, a definition of an extended view dataset, wherein the definition of the extended view dataset specifies fields of other datasets or other datasets related to the base dataset; outputting, based on the definition of the extended view dataset, a preview of the attributes of the fields of the extended view dataset, wherein the preview is generated from a subset of data in the field of the other dataset or other datasets related to the base dataset; receiving input specifying that one or more fields of the field in the preview can be used for data processing; and generating, based on the input, a usable dataset including data in the base dataset and data in the one or more specified fields of the field.

[0008] In a second aspect that can be combined with the first aspect, the method includes: providing a preview of the property of the extended view dataset, wherein the extended view dataset includes data from the base dataset and other datasets associated with the base dataset when it is generated.

[0009] In a third aspect, which may be combined with the first or second aspect, the definition of the extended view dataset specifies a set of data processing operations to be performed to generate the extended view dataset, which includes the data from the base dataset and the other datasets associated with the base dataset, and wherein the preview is generated by applying the set of data processing operations specified by the definition of the extended view dataset only to a subset of the data in the base dataset and the other datasets associated with the base dataset.

[0010] In a fourth aspect, which may be combined with any of the first to third aspects, the method includes: in response to providing the preview, receiving a specification of a designated data processing operation, wherein the data processing operation of the specification is defined at least in part based on user input that identifies attributes included in the preview as attributes of the data processing operation.

[0011] In a fifth aspect, which may be combined with any of the first to fourth aspects, the method includes: defining a data processing operation based at least in part on the user input that identifies the attribute included in the preview as the attribute of the data processing operation; updating the set of data processing operations by applying one or more optimization rules to the defined set of data processing operations; and performing the updated set of data processing operations to generate a dataset that includes only a subset of the data that should have been included in the extended view dataset.

[0012] In a sixth aspect, which may be combined with any of the first to fifth aspects, the method includes: enabling a user to register a definition of a new dataset with a data catalog, wherein the definition specifies a selected dataset and other datasets associated with the selected dataset, wherein the definition provides logical access to the other datasets associated with the base dataset without incurring computational costs related to providing the other associated datasets.

[0013] In a seventh aspect, which may be combined with any of the first to sixth aspects, the method includes: accessing a data directory of one or more specified datasets; and providing a user interface indicating that the one or more datasets are candidates for generating the extended view dataset.

[0014] In an eighth aspect, which can be combined with any of the first to seventh aspects, the method includes: receiving, via the user interface, an instruction for a specific dataset as the base dataset, and, in response to the instruction, automatically generating a definition of the extended view dataset for the specific base dataset.

[0015] In a ninth aspect, which may be combined with any of the first to eighth aspects, the method includes: identifying the particular base dataset as the base dataset and identifying one or more attributes of the particular base dataset; determining a definition of the base dataset based on the one or more attributes; determining one or more other datasets associated with the base dataset based on the definition of the base dataset; and generating a definition of the extended view dataset based on the determined one or more other datasets, the definition specifying the base dataset, the one or more other datasets, and one or more relationships between the base dataset and the one or more other datasets.

[0016] In a tenth aspect, which can be combined with any of the first to ninth aspects, the method includes: defining the extended view dataset in a hardware storage device; and registering the definition of the extended view dataset with a data directory.

[0017] In an eleventh aspect, which may be combined with any of the first to tenth aspects, the method includes: receiving a request for the extended view dataset; and in response to the request, providing the extended view dataset by: retrieving the definition of the extended view dataset from a hardware storage device; retrieving the base dataset and the one or more other datasets from one or more data sources based on the definition of the extended view dataset; and generating the extended view dataset based on data in the retrieved datasets.

[0018] In a twelfth aspect, which can be combined with any one of the first to eleventh aspects, the method includes: determining, based on the extended view dataset, whether to update the data directory to designate the definition of the extended view dataset as a data source, storing the definition of the extended view dataset in a hardware storage device, and registering the definition of the extended view dataset to the data directory.

[0019] In a thirteenth aspect, which can be combined with any of the first to twelfth aspects, the method includes: determining, based on a provided preview of the property of the extended view dataset, whether to update the data catalog to designate the definition of the extended view dataset as a data source.

[0020] In the fourteenth aspect, which can be combined with any of the first to thirteenth aspects, generating the available dataset includes: using the definition of the extended view dataset to access only those datasets having one or more specified fields, and including the data of those accessed datasets into the available dataset.

[0021] In a fifteenth aspect, which can be combined with any one of the first to fourteenth aspects, the method includes: processing the generated available dataset to obtain a result from processing the available dataset.

[0022] In a sixteenth aspect, which may be combined with any of the first to fifteenth aspects, the method includes: granting user permission to access a portion of the base dataset in the extended view dataset while denying the user access to the remainder of the base dataset.

[0023] In a seventeenth aspect, which can be combined with any of the first to sixteenth aspects, the definition of the extended view dataset includes a computational graph that specifies a set of data processing operations for generating the extended view dataset, the extended view dataset including data from the base dataset and other datasets associated with the base dataset, the set of data processing operations including at least one operation for joining the data from the base dataset and data from at least one of the other datasets associated with the base dataset.

[0024] In the eighteenth aspect, which can be combined with any of the first through seventeenth aspects, this definition of the extended view dataset provides logical access to data from the base dataset and other datasets associated with the base dataset. The foregoing actions of the method can be combined in any and all combinations.

[0025] In the nineteenth aspect, which can be combined with any of the first to eighteenth aspects, the preview is generated during development, and the available dataset is generated at runtime.

[0026] Generally speaking, in the twentieth aspect, a method implemented by a data processing system is provided, the method being used to: enable a user to preview the attributes of a field of an extended view of a base dataset and specify one or more fields of that field for downstream data processing, and generate a dataset including the one or more fields of that field specified for the downstream data processing from the preview, wherein when only the specified one or more fields of that field are used in the downstream data processing, the generated dataset has improved efficiency in terms of speed and data storage relative to the efficiency of generating a dataset including all the fields of the extended view, the method comprising: receiving an identifier of a base dataset; receiving, based on the identifier, a definition of an extended view dataset, wherein the definition of the extended view dataset specifies fields of other datasets or other datasets associated with the base dataset; outputting, based on the definition of the extended view dataset, a preview of the attributes of the fields of the extended view dataset, wherein the preview is generated from a subset of data in the other datasets associated with the base dataset or metadata of the other datasets; receiving input specifying that one or more fields of that field in the preview can be used for data processing; and generating, based on the input, an available dataset including data in the base dataset and data in the one or more specified fields of that field.

[0027] Generally speaking, in the twenty-first aspect, a data processing system includes: one or more processing devices; and one or more machine-readable hardware storage devices storing instructions executable by the one or more processing devices to perform operations of any one of the first to twenty aspects.

[0028] Generally speaking, in the twenty-second aspect, one or more machine-readable hardware storage devices store instructions that can be executed by one or more processing devices to perform the operations of any one of the first to twenty-first aspects.

[0029] A system of one or more computers may be configured to perform specific operations or actions by installing software, firmware, hardware, or combinations thereof on the system, which, in operation, cause the system to perform actions, including any or all of the aforementioned actions in any combination. One or more computer programs may be configured to perform specific operations or actions by including instructions that, when executed by a data processing device, cause the device to perform actions, including any or all of the aforementioned actions in any combination.

[0030] One or more of the above aspects may provide one or more of the following advantages.

[0031] An extended view dataset can represent a subset of the data contained in a dataset. The definition of an extended view dataset occupies relatively little storage space because it only provides logical access to the dataset and does not contain a copy of all the data presented in the original dataset. Specifically, when an extended view dataset is requested, the system only generates and stores its definition. As described below, this definition provides logical access without the physical cost of materializing the extended view dataset. The definition allows previewing fields in the extended view dataset without materializing the entire extended view dataset. This preview (which can be generated during development or authoring) allows selection or specification of which fields are needed for processing. Then, during actual processing (e.g., runtime), the system uses the definition to access only those datasets with the required fields and makes those accessed datasets available for processing through the available dataset. The available dataset includes one or more fields from the preview that are specified for downstream data processing, where the generated available dataset is more efficient in terms of speed and data storage than generating a dataset that includes all such fields of the extended view when only one or more of those fields are used in the downstream data processing. In various ways, extended view datasets limit the exposure of the underlying dataset to the outside world. A given user may have permission to read a portion of the extended view dataset containing the underlying base dataset, while being denied access to the rest of the base dataset. Extended view datasets can join multiple datasets and simplify them into a single virtual dataset. Extended view datasets can act as aggregate datasets, where system-aggregated data (sums, averages, etc.) presents the results as part of the data in the extended view dataset. Extended view datasets can hide the complexity of the data by transparently partitioning the actual underlying dataset.

[0032] Extended view datasets are the result of performing a set of stored transformation logic, accessible to catalog users as if they were accessing a persistent dataset. Extended view datasets are efficient when returning multiple data axes and avoiding data duplication. Extended view datasets leverage the relationships between their underlying base datasets by using the primary key and foreign key relationships of the base datasets to find related datasets, which in turn preserve their relationships with the base datasets.

[0033] Details of one or more embodiments of the present invention are set forth in the following drawings and description. Other features, objects, and advantages of the invention will be apparent from the description and drawings, as well as from the claims. Attached Figure Description

[0034] Figure 1 This is a block diagram of the system.

[0035] Figure 2 This is a diagram of the system used to preview the properties of an extended view dataset.

[0036] Figures 3A to 3F yes Figure 2 The system is illustrated in the phase of defining the extended view.

[0037] Figures 4A to 4E Each example is illustrated with a diagram showing the stages of generating the expanded view dataset.

[0038] Figure 5 , Figures 5A to 5C This is a flowchart process used to preview the properties of an expanded view dataset.

[0039] Figure 6 It is a diagram showing details of a computer system, such as a data processing system. Detailed Implementation

[0040] refer to Figure 1 This diagram illustrates the complexity of understanding and accessing data when it is distributed across different data sources within an enterprise or system. Additionally, the diagram shows how computationally inefficient the current system is when aggregating all data from different data sources that might be required for a specific computation or operation. In this example, the data scientist wants a better understanding and ability to use all the data available across the enterprise or system. Therefore, the data scientist sends a request to the data engineer for all data related to the (in this example) valid loan dataset.

[0041] Next, data engineers must identify all data relevant to the valid loan dataset. Data engineers do this by requesting patterns from various systems across the enterprise and generating programs to retrieve all data related to valid loans. Once the data engineer generates the program, it is passed to quality engineers who identify any errors in the program. These errors are sent back to the data engineer, and perhaps a month or two later, the data engineer has an updated program to retrieve all data related to valid loans. However, even this updated program may miss some data, or it may still contain some errors. The reality is that data engineers may not be able to identify all data sources and datasets relevant to valid data.

[0042] The quality engineer sends the updated program to a computer, which runs it against various data sources. The computer program generates a massive dataset (and, in the case of its creation), that is already outdated. This is because the dataset was generated before any program or data operation actually invoked or used it. In this example, perhaps five days or five months later, a data scientist is reviewing the massive dataset to see what data is available in the system. When the data scientist reviews this massive dataset, it is now outdated because it has now existed for five days.

[0043] In this example, a data scientist might request the calculation of the average FICO for valid loans. The data scientist could send this request to a computer, which would then implement the logic to execute it. This process is highly inefficient because it results in the materialization of all data related to valid loans, when in reality only a portion of that data is needed to calculate the data scientist's request. In this example, only the Loan ID field, Status field, and FICO field are needed to fulfill the data scientist's request. However, when generating a dataset for the data scientist to see which data is even available, the computer materializes all the data related to valid loans. This materialization is not only costly because all the materialized data must be stored, but also computationally inefficient because the computer must scale its resources to join all this data together into a dataset for the data scientist to review.

[0044] Additionally, the dataset is outdated, as discussed previously. Therefore, a system is needed that can efficiently generate the dataset using only the data actually needed for computation, and can pull the data on demand in real time in response to requests, thus ensuring that the data is not outdated, while allowing logical access to the data so that it is possible to understand and preview what data is available without requiring the actual materialization of the data.

[0045] refer to Figure 2The system 10 illustrates a preview of the attributes of fields used to generate an extended view dataset. The extended view dataset includes data from a base dataset and data from one or more datasets related to the base dataset (e.g., related in a database schema). The base dataset is, for example, a dataset specified or selected by the user. Typically, attributes include field values ​​and / or descriptive values ​​and / or information about the field. System 10 also enables the user to specify one or more attributes for use in one or more data processing operations optimized for speed and data storage.

[0046] System 10 includes a data processing system 12 and a client device 18. Client device 18 receives data from a data catalog 14 and presents it in a browser 19 (specifying which datasets are available for processing and / or computation of the requested values). The user interface 19a presented in browser 19 displays a section 20a (labeled "Data Catalog") showing datasets from the data catalog 14 and a section 20b (labeled "Field Selector") showing fields from the datasets. Section 20a lists the datasets from the data catalog 14, and section 20b displays the fields from one of the selected datasets (discussed further below). Through user interface 19a, the user selects which data sources they want to preview. The user selects which data sources from many different (e.g., enterprise-wide) data sources and the fields within those data sources. System 10 can ultimately automatically generate code (e.g., generate data flow charts) to access specific data sources across those different data sources.

[0047] Data catalog 14 is a repository of identifiers (e.g., indexes of business or logical names or logical metadata) for one or more datasets and fields, as well as other data across the storage infrastructure, allowing users to find and identify data more quickly. Identifiers in data catalog 14 can be business names that are easy for users to understand and provide semantic meaning. Data catalog 14 may also store technical identifiers (also known as technical metadata) for datasets and fields. For example, this technical metadata may specify technical field names, such as the field name when it appears in the data source itself. For each technical field name, the data catalog may store logical or business names to enable users to easily identify fields and datasets. In some examples, system 10 automatically transforms technical metadata into logical metadata (e.g., business names) by performing semantic discovery on data received from the data source, as described in U.S. Patent Publication No. 2020 / 0380212 (titled “Discovering a Semantic Meaning of Data Fields from Profile Data of the Data Fields”), the entire contents of which are incorporated herein by reference.

[0048] In this example, client device 18 sends request 13 to EVD definition generator 22. Request 13 specifies that “active loans.dat” is the base dataset, and request 13 is for an extended view of “active loans.dat.”. The extended view includes representations, specifications, identifiers, or lists or all datasets associated with the base dataset. In response to this request, EVD definition generator 22 identifies the datasets associated with the base dataset and generates definitions 13a for these associated datasets and the base dataset. This definition, called an EVD definition, specifies the associated datasets and the base dataset and also specifies the logic for generating a dataset (called the extended view dataset) that includes data from the associated datasets and the base dataset. At this stage, the EVD definition generator sends EVD definition 13a to metadata store 25 for storage. At this point, data processing system 12 does not use EVD definition 13a to generate the extended view dataset. The reason is that the extended view dataset (once generated) will include many fields (e.g., all fields from the base dataset and associated datasets). Therefore, materializing this dataset is expensive. Materialization refers to the process of retrieving data (of fields) from various sources, combining all the data into a single dataset, and then storing that combined single dataset. This materialization is expensive in terms of processing and storage resources. Therefore, once the fields required for processing have actually been specified, the data processing system 12 only materializes the single dataset of that combination, as described below. Additionally, the data processing system 12 displays visualizations of the fields that will be in the EVD and includes previews of the values ​​of those fields. The data processing system 12 generates these visualizations and previews by using the EVD definition to process a limited or specified amount of data in the base dataset and related datasets. In some cases, the data processing system 12 may generate visualizations of the fields that will be in the EVD without previewing the values ​​of those fields, thus avoiding the need to process any data in the base dataset and related datasets. By processing only a specified or limited amount of data, the data processing system 12 saves processing and storage resources. Therefore, the data processing system 12 provides logical access (to physical fields and values) without the cost of EVD materialization. Logical access includes previews of the fields and / or the values ​​of those fields, enabling those fields to be used to specify the logic used in the computation process.

[0049] Once these fields are specified, the data processing system can use the EVD definition to materialize a dataset that includes only the fields actually needed for processing. This materialized dataset is called the available dataset because it includes the fields that need to be used for processing. In this example, data processing system 12 does not materialize the EVD dataset. Instead, the EVD definition is used to materialize the available dataset 15.

[0050] User interface 19a includes a section 20a listing data catalog datasets (e.g., Customers, Active Loans, Active Loan EVD), where the Active Loan EVD dataset is selected (in bold), and a section 20b displaying the fields corresponding to the fields in the dataset (e.g., Active Loan EVD dataset) (e.g., Loan ID, Status, FICO, Customer SSN (Social Security Number)). Section 20b allows the user to select which fields (e.g., Loan ID, Status, FICO) are included in the request sent to the Extended View Dataset (EVD) definition generator 22. In this example, the fields Customer SSN and Hardship are not included in the request.

[0051] In this example, the metadata manager 24 registers the EVD definition 13a with the data directory 14, for example, by sending information identifying EVDs that can be generated from EVD definition 13a to the data directory 14. Based on this, section 20a displays visualization 13b, which indicates that a valid loan EVD is a dataset that can be processed and used to define logic.

[0052] User interface 19a includes a section 20a listing data catalog datasets (e.g., Customers, Active Loans, Active Loan EVD), where the Active Loan EVD dataset is selected (in bold), and a section 20b displaying the fields corresponding to the fields in the dataset (e.g., Active Loan EVD dataset) (e.g., Loan ID, Status, FICO, Customer SSN (Social Security Number)). Section 20b allows the user to select which fields (e.g., Loan ID, Status, FICO) are included in the request sent to the Extended View Dataset (EVD) definition generator 22. In this example, the fields Customer SSN and Hardship are not included in the request.

[0053] In this example, the user browses the data catalog 14 via browser 19 to identify which fields and datasets are candidate datasets to be used in computational processing. Specifically, the user browses the data catalog 14 by viewing user interface 19a on client device 18, which presents identifiers and visual representations of logical metadata and business names in the data catalog 14.

[0054] In this example, user interface 19a displays a preview of the fields in the valid loan EVD. This preview is generated by data processing system 12 using definition 13a to identify the fields, for example, by accessing the metadata of the dataset or dataset specified in definition 13a to obtain the fields in those datasets. In this example, only the fields Loan ID, Status, and FICO are selected. User interface 19a also includes an input instruction field, "Input Instruction," which is used to select which type of request to send to EVD definition generator 22. User interface 19a also includes a calculation button 21 to begin expanding the view of dataset 59. Figure 3FThe calculation of the average FICO of valid loans. When the calculation button 21 is selected, the client device 18 sends a request 13c to the EVD integrator 26. Request 13c is a request to calculate the average FICO of valid loans.

[0055] EVD Integrator 26 sends a request to Metadata Manager 24 to retrieve EVD Definition 13a from Metadata Store 25, which specifies all datasets related to the valid loan dataset. EVD Integrator 26 receives Definition 13a. EVD Integrator 26 integrates (or combines) Request 13c to calculate the average FICO of valid loans with Definition 13a.

[0056] EVD Integrator 26 transmits an integration request to Execution Engine 28. Execution Engine 28 generates code to execute the integration request. Optimizer 30 optimizes the code to retrieve data only from those datasets that include the fields selected in User Interface 19a. Execution of this optimization code produces available data 15, which includes only the fields specified in Execution Request 13c and / or selected in User Interface 19a for the calculations required. Execution Engine 28 executes the optimization code to retrieve data from a data source having data related to valid loans in relation to the selected fields. Execution Engine 28 executes the optimization code, also performs the requested calculations, and stores the results (e.g., the average FICO score of valid loans) in Storage System 32. Details regarding optimizers (such as Optimizer 30) are disclosed in U.S. Patent Publication No. 2019 / 0370407, entitled “Systems and Methods for Dataflow Graph Optimization,” the entire contents of which are incorporated herein by reference.

[0057] Figures 3A to 3F A miniature version of System 10 is shown, with some boxes highlighted in bold. References Figure 2 To show the bolded box and Figure 2 The not in Figures 3A to 3F The specific number or relationship of other elements shown in the text.

[0058] Now for reference Figure 3AThe data catalog 14 transmits a visualization of the data catalog data to the client device 18. The client device 18 presents the visualization in a user interface 19b, such as in a browser 19. The user interface 19b includes a section 20a listing the data catalog datasets (e.g., customers, valid loans) and data for valid loans 20c displaying fields (e.g., loan ID, customer ID, and status). The data for valid loans 20c is presented in the corresponding fields for loan ID, customer ID, and status. The user interface 19b also includes a control 21a, "Request Extended View". When selected, the "Request Extended View" control 21a sends a "Request for Extended View Dataset - Base Dataset Definition: Base Dataset = Valid Loans" to the EVD Definition Generator 22.

[0059] Now for reference Figure 3B The metadata manager 24 and the metadata store 25 transfer the retrieved metadata related to the valid loan from the metadata store 25 to the EVD definition generator 22. The metadata store 25 will then transfer the retrieved metadata (in...) Figure 3B (Not explicitly readable in the text) is transmitted as metadata model 40 to EVD definition generator 22. Figure 3F The extended view dataset 59, when generated, includes data from the base dataset definition 44 and one or more related dataset definitions 46, which correspond to the new dataset and are related to the base dataset definition 44, such as... Figure 3C As shown.

[0060] Now for reference Figure 3C EVD defines the generator 22 to parse the metadata model 40 (in Figure 3C (Not explicitly readable in the text) to identify the base dataset definition 44 and the related dataset definition 46. The base dataset definition 44 is shown in extended view 41. Base dataset definition 44 corresponds to the active_loans.dat base dataset 44a. The active_loans.dat base dataset 44a includes two keys: a primary key `loan_id` and a foreign key `customer_id`. Related dataset definition 46 includes the “hardship.dat” dataset 46a, the “loan_details.dat” dataset 46b, and the “settlement.dat” dataset 46c, each of which is related to the active_loan.dat base dataset 44a by means of its respective “hardship.dat” dataset 46a, “loan_details.dat” dataset 46b, and “settlement.dat” dataset 46c, which include the foreign key `customer_id`.

[0061] Extended view 41 also shows the related datasets “customers.dat” dataset 46d and “FICO.dat” dataset 46e. The base dataset 44a of “active_loan.dat” is related to the dataset 46d of “customers.dat” by sharing the key “customer_id” (e.g., a primary key-foreign key relationship) with the dataset 46d of “active_loan.dat”. The dataset 46e of “FICO.dat” is also related to the dataset 46d of “customers.dat” by sharing the key “ssn” (e.g., another primary key-foreign key relationship) with the dataset 46d of “FICO.dat”.

[0062] Figure 3C The document also shows an extended view 41 of the basic dataset definition 44 and the related dataset definition 46. Extended view 41 includes the related dataset definition 46, which includes “hardship.dat” 46a, “loan_details.dat” 46b, and “settlement.dat” 46c. As described below, this is achieved by using the extended view dataset 50 (… Figure 3F The defined set of data processing operations is applied only to the base dataset 44a and subsets of data in datasets associated with the base dataset 44a (e.g., “hardship.dat” 46a, “loan_details.dat” 46b and “settlement.dat” 46c, and datasets “customers.dat” 46d and “FICO.dat” 46e) to generate a preview.

[0063] Regarding Figure 3C Metadata model conversion in Figure 3D The details of the definition diagram are disclosed in U.S. Patent No. 11,423,083 (titled "Transforming a Specification into a Persistent Computer Program") and U.S. Patent No. 11,210,285 (titled "Generation of Optimized Logic from a Schema"), the entire contents of which are incorporated herein by reference.

[0064] One or more related dataset definitions 46 have one or more relationships 43a to 43e (e.g., primary key-foreign key relationships) between the base dataset 44a and one or more related datasets 46a to 46e. Users register new dataset definitions with data catalog 14. The base dataset definition 44 specifies the base dataset 44a and the datasets associated with the base dataset 44a. The base dataset definition 44 provides logical access to the related datasets without incurring the computational cost of providing the related datasets 46a to 46e.

[0065] Based on one or more attributes, system 10 determines a basic dataset definition 44, and based on the basic dataset definition 44, system 10 determines one or more related datasets 46a to 46e associated with the basic dataset 44a. Based on the determined one or more related datasets 46a to 46e, system 10 generates an extended view dataset 59. Figure 3F The definition of the extended view dataset specifies the base dataset 44a, one or more related datasets 46a to 46e, and one or more relationships 43a to 43e between the base dataset 44a and one or more related datasets 46a to 46e.

[0066] In this example, the EVD definition generator 22 includes a graph generator (not shown), as described in U.S. Patent No. 11,210,285 (titled "Generation of Optimized Logic from a Schema"), the entire contents of which are incorporated herein by reference. In this example, Figure 3C The basic dataset definition 44 and related dataset definition 46, identified in the example, are the logical data input into the chart generator. In this example, the chart generator is pre-configured with a specification that stipulates that a chart should be generated to access and concatenate all datasets in the basic dataset definition 44 and related dataset definition 46, thereby generating the EVD definition (in this example, ). Figure 3D(See EVD definition 50 in the text). The chart generator uses specification and logical data to generate data flow charts (also referred to herein as computational charts or charts for convenience, and not limited to this). Generally, a data flow chart (or persistent computer program) is generated from a specification as follows: The specification specifies multiple modules to be implemented by the computer program for processing one or more values ​​of one or more fields in a structured data item (e.g., a data record). Modules can be components in a data flow chart or groupings of components in a data flow chart (e.g., subcharts). In this case, the multiple modules are modules for accessing and connecting datasets specified by logical data (basic dataset definition 44 and related dataset definition 46). Therefore, the chart generator creates a chart with data access components for each data source in the data sources represented in basic dataset definition 44 and related dataset definition 46. Additionally, the chart generator adds an appropriate number of connection components to connect these data sources and adds data hosts, as described below. Additionally, these multiple modules may include rules, instructions, components of the data flow chart, etc. The system described herein transforms a specification into a computer program implementing multiple modules (e.g., on logical data) by specifying the processing flow between components or modules, specifically as follows: For each of one or more first modules among the multiple modules: identifying one or more second modules among the multiple modules, each second module receiving input at least partially based on the output of the first module; and formatting the output data format of the first module such that the first module outputs only one or more values ​​of one or more fields of a structured data item, each field (i) being accessible by the first module and (ii) being specified as input to at least one of the one or more second modules at least partially based on the output of the first module; and storing the computer program in persistent storage, wherein the stored computer program specifies the formatted output data format of each of the one or more first modules, as described in U.S. Patent No. 11,423,083. The system also includes various rules specifying instructions for the content of each module to be included in the computer program and / or converted to an appropriate format for the computer program. In this example, the chart generator initially generates a data flow chart with a data source represented in logical data. The chart generator also adds a data sink to the data flow chart because the data flow chart requires a data sink. Then, the chart builder adds various components (such as sorting components) to the dataflow chart, configured to automatically add them to improve the computational efficiency of the dataflow chart. The chart builder is also configured to add connection components to appropriately link data from various data sources. Instructions, parameters, or other information for accessing or connecting data sources can be included in the logical data. Finally, the chart builder can add transformation components that include the computational logic specified in the specification.When a specification is transformed into a data flow graph as described above, the transformation component itself may include various components or sub-components representing another data flow graph.

[0067] Now for reference Figure 3D System 10 is shown as having already been Figures 3A to 3C The actual view of the EVD definition defined in the code. EVD definition generator 22 returns the effective loan EVD definition 50 as a computational graph 50'. The effective loan EVD definition 50 provides logical access without physical cost by providing the logic of how to generate the effective loan EVD without actually generating the dataset (i.e., the effective loan EVD) itself and thus having to materialize (e.g., read from the data source and store in memory) the data required for the effective loan EVD. The computational graph 50' of the effective loan EVD definition 50 includes visualizations 52a to 52f of access data, such as “Access active_loan.dat” 52a, “Access customers.dat” 52b, “AccessFICO.dat” 52c, “Access hardship.dat” 52d, “Access loan_details.dat” 52e, and “Accesssettlement.dat” 52f. These correspond to... Figure 3C The relevant datasets are 46a to 46e.

[0068] The EVD definition generator 22 returns a computation graph 50' that further includes join operations. These join operations include join operation 54a applied to "Access active_loan.dat" and "Access customers.dat" joined based on "customer_id". Join operation 54b applied to the result of join operation 54a joined with "AccessFICO.dat." based on "ssn". Join operation 54c applied to the result of join operation 54b joined with "Access hardship.dat" 52d joined with "loan_id". "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f are shown as being combined into a single input to join operation 54c. As described below, a usable dataset is generated from the computation graph 50'. This dataset is optimized to remove "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f as data sources because these datasets include data not used by downstream components when generating the specified output. Specifically, as described below, "Access hardship.dat" 52d, "Access loan_details.dat" 52e, and "Access settlement.dat" 52f are optimized.

[0069] Now for reference Figure 3E The valid loan EVD definition 50 is transmitted to the metadata manager 24 and the metadata store 25. The metadata manager 24 transmits the EVD identifier (valid loan EVD) to the data directory 14. The valid loan EVD definition 50 is transmitted to the metadata manager 24 and the metadata store 25. The data directory 14 transmits a visualization of the data directory data to the client device 18. The client device 18 presents a user interface 19c to a browser 19, which displays a section 20a (labeled "data directory") that displays the dataset from the data directory 14.

[0070] Browser 19 presents a visual user interface 19c displaying data from the data catalog and a view data button 21b. Client device 18 transmits a request to view the valid loan EVD to EVD integrator 26. Metadata manager 24 and metadata store 25 respond by providing a preview to receive a specification (valid loan EVD definition) of a specified data processing operation. The specified data processing operation is defined, at least in part, based on user input that identifies attributes included in the preview as attributes of the data processing operation.

[0071] Now for reference Figure 3F Metadata Manager 24 transfers the Effective Loan EVD Definition 50 from Metadata Store 25 to EVD Integrator 26. EVD Integrator 26 then transfers the Effective Loan EVD Definition 50 to Execution Engine 28. Execution Engine 28 compiles and executes a computational graph 50' to generate a dataset "Effective Loan EVD," which is generated by executing a data flow graph as Effective Loan EVD Definition 50 against data in Data Source 34. That is, in generating Effective Loan EVD 59, in response to the request, data from Data Source 34 is materialized (e.g., read from Data Source 34) and processed to provide a new dataset, Effective Loan EVD 59. Execution Engine 28 also generates a visualization 60 of Effective Loan EVD 59 to allow users to view the data in the Effective Loan EVD. In the example, the visualization may only show a specified amount of data (e.g., the first five results) to generate the visualization without the latency of loading the visualization, which could be the case if the visualization includes all the data in Effective Loan EVD 59 (e.g., 1 million data items). The generated dataset “Effective Loans EVD” is stored in storage system 32 and registered with data directory 14.

[0072] System 10 receives a request for extended view dataset 59 and, in response to the request, provides extended view dataset 59 by retrieving definition 50 of extended view dataset 59 from hardware storage device, and retrieves base dataset and one or more other datasets, i.e., related datasets, from one or more data sources. Based on extended view dataset 59, system 10 generates visualization 60 of extended view dataset 59.

[0073] Figures 4A to 4E The document describes examples of data catalog editing operations. Figures 4A to 4E A miniature version of System 10 is shown, with some boxes highlighted in bold. References Figure 2 To show the bolded box and Figure 2 The not in Figures 4A to 4E The specific number or relationship of other elements shown in the text.

[0074] Now for reference Figure 4AThe data catalog 14 transmits a visualization of the data catalog data to the client device 18. The client device presents the visualization as a user interface 19d in a browser 19. The user interface 19d includes a data catalog 20a, which lists data catalog datasets (e.g., customers, valid loans, valid loan EVD) and operations such as enrichment, filtering, calculation, and storage. The user interface 19d also includes an editor 20d for editing the listed data catalog datasets, such as the "Valid Loan EVD" dataset. The editor 20d allows the user to define the specifications that are compiled into a chart and executed. The editor 20d includes a "preview" control 21c and an "execute" control 21d. Selecting the preview control 21c causes the client device to generate a data preview 62. Figure 4A In the middle, the generated data preview 62 corresponds to Figure 3F Visualization of effective loan EVD 59 60.

[0075] In this example, data preview 62 provides logical access without physical cost. This means that preview 62 is generated by performing the Valid Loan EVD definition 50 on only a few records in data source 34, rather than on all data records in data source 34. This would consume significant memory and processing resources to collect data for fields that might never be used (e.g., data for fields not used or accessed by the specification defined in editor 20d). However, providing logical access to all fields to the user is beneficial because the user can view the fields defined in definition 50 and even see the values ​​of those fields when deciding which fields the user wants to use in the specification (e.g., which fields the user wants to filter). Then, when the actual dataset (Valid Loan EVD) is materialized (e.g., generated), it can be optimized to materialize only those fields used by the data processing operations defined by the specification in editor 20d.

[0076] Now for reference Figure 4B The client device 18 presents the visualization as a user interface 19d in a browser 19. The client device 18 transmits the chart specifications to the EVD integrator 26. The user interface 19d includes a data catalog 20a, which lists data catalog datasets (e.g., customers, valid loans, valid loan EVD) and operations such as enrichment, filtering, calculation, and storage. The user interface 19d also includes an editor 20d for editing the listed data catalog datasets, such as the "Valid Loans EVD" dataset. The editor 20d includes a "preview" control 21c and an "execute" control 21d, and also includes additional preview controls 21e for previewing "Filter by Status = Valid" and 21f for previewing "Calculate Average FICO". Selecting the preview controls 21c, 21e, and 21f causes the client device 18 to transmit instructions to the EVD integrator 26 to generate a corresponding data preview (not shown). Figure 4BAs shown, an example of a graph specification converted into executable code is given in U.S. Patent Publication No. 2021 / 0232579 (titled “Generating Computational Graphs”), the entire contents of which are incorporated herein by reference.

[0077] In this example, diagram 19' is generated in editor 19d. Diagram 19' is an example of a declarative diagram, where each component represents a declarative action, which is then transformed into an imperative action as described herein. Client device 18 sends diagram specification 66 to EVD integrator 26. Diagram specification 66 represents and / or specifies the content of diagram 19'. Based on diagram specification 66, EVD integrator 26 generates graph 66', as described in U.S. Patent 11,593,380.

[0078] In this example, chart specification 66 includes selection data specifying which icons in user interface 19d have been selected, as well as other information and / or values ​​specified in user interface 19d. EVD integrator 26 includes a dataflow chart engine (not shown) that receives the selection data from client device 18. The selection data indicates the data source, data destination, and data processing capabilities for which the chart is expected to be computed. The user of client device 18 does not need to specify data access details or other low-level implementation details, as these details can be derived by the dataflow chart engine. Based on the selection data, the dataflow chart engine generates a dataflow chart 66' or modifies a previously created dataflow chart. In some examples, the dataflow chart engine transforms the dataflow chart by, for example, removing redundancy in the dataflow chart, adding sorting or partitioning to the dataflow chart, specifying intermediate metadata (e.g., metadata used to transform or otherwise modify the dataflow chart), and other optimizations and transformations. EVD integrator 26 sends dataflow chart 66' to execution engine 28, which includes a compiler that compiles dataflow chart 66' into a compiled computation chart (e.g., an executable program).

[0079] refer to Figure 4B' Now, an example of generating a data flow diagram 66' from diagram specification 66 is described. In this example, using diagram specification 66, EVD integrator 26 generates a data flow diagram 17 (sometimes referred to as the "initial data flow diagram" or "preliminary data flow diagram"), which represents a compiled diagram (such as the transformed data flow diagram 50) with nodes (or components). Figure 4C The core structure of the data flow diagram 17 is as follows. The data flow diagram 17 may optionally include parameters (e.g., name, value, location, explanation). In some specific implementations, the data flow diagram 17 includes input and output ports on the diagram itself, as in a diagram intended to be used as a subgraph.

[0080] In some implementations, nodes (or components) possess or belong to a node "type" that indicates the behavior or function of the node. Node types are used to select node prototypes to facilitate pattern matching (e.g., finding sorted nodes followed by sorted nodes) and to determine which components are instantiated in the transformed dataflow graph 50 (or 66' depending on the transformation level). For example, a garbage node in dataflow graph 23 may be instantiated as a garbage node in the transformed dataflow graph 66'. Nodes (or components) may include input ports, output ports, and parameters, as discussed below.

[0081] Nodes may optionally have labels that identify them. In some implementations, if a node does not have a label, the system assigns a label to it. Node labels may include any set of alphanumeric characters, spaces, and punctuation marks, and do not have to be unique (but can be unique during the conversion to a graph). The system may use node labels to refer to nodes (or node input ports, output ports, or parameters) to, for example, qualify a node's input or output, or the data flow between nodes.

[0082] In some examples, the EVD integrator 26 includes a template data flow diagram, namely data flow diagram 17, before generating data flow diagram 17. Data flow diagram 17 is shown as including nodes 34a to 34n. Each node among nodes 34a to 34n includes at least one operation placeholder field and at least one data placeholder field. For example, the "initial" node 34a has an operation placeholder field 35a for storing one or more operation elements 35a' and a data placeholder field 35b for storing one or more data source or data destination elements 35b'. Operation element 35a' can specify the code or code location that will perform a function on data input to or output from the initial node 34a. Data source or data destination element 35b' can specify a data source or data destination for the initial node 34a (for the function of the initial node 34a), or the location of the data source or data destination. In some implementations, element 35a' or element 35b' or both include a link or address to a storage system or storage system 32 included in the EVD integrator 26, such as a link to a database or a pointer to code included in storage system 32. In some implementations, element 35a' or element 35b' or both include a script.

[0083] During the construction of data flow diagram 17, each of nodes 34a to 34n can be modified by retrieving operation elements to be placed in operation placeholder fields and data source or data sink elements to be placed in data placeholder fields to populate the corresponding fields. For example, the initial node 34a is modified during construction by retrieving operation element 35a' to populate operation placeholder field 35a using a specified function or a link to that function, and by retrieving data source or data sink element 35b' to populate data placeholder field 35b using a link to the source or sink of the data. After the modification of a particular node 34a to 34n is completed, the node can be marked to provide the marked node. After each of nodes 34a to 34n has been modified (and marked), the completed data flow diagram 17 is stored (e.g., in storage system 32) and used to generate other data flow diagrams, as described below.

[0084] In some implementations, each of the nodes 34a to 34n of the data flow diagram 17 is initially unmodified. For example, each of the nodes 34a to 34n may have an empty operation placeholder field 35a and a data placeholder field 35b, which are subsequently modified to include a specified operation element 35a' and a data source or data destination element 35b', as described above. In some implementations, the data flow diagram 17 is a previously completed data flow diagram, and some or all of the nodes 34a to 34n have a corresponding operation placeholder field 35a that holds the operation element 35a' and a data placeholder field 35b that holds the data source or data destination element 35b'. Such a completed data flow diagram 17 can be further modified (e.g., by retrieving additional or alternative elements 35a', 35b' to be placed in the corresponding fields 35a, 35b) and stored as a new or modified data flow diagram.

[0085] In some implementations, a specific node (such as the initial node 34a) is "reused" to generate new, optionally labeled nodes associated with the previous node 34a. This iterative process of generating new nodes from the initial node 34a continues until the user has specified functionality for the desired computational graph. Upon completion of the iterative process, a completed data flow graph 17 is provided. The completed data flow graph 17 includes multiple nodes 34a to 34n instantiated from, for example, the initial node 34a. The completed data flow graph 17 can be stored (e.g., in storage system 32) and used to generate other data flow graphs, as described below.

[0086] Figure 4B'A specific implementation of a completed (e.g., modified) data flow diagram 17' is also illustrated. The modified data flow diagram 17' is shown as comprising four nodes labeled OP-0 to OP-3, where corresponding operation placeholder fields 35a store operation elements and data placeholder fields 35b store data source or data destination elements. For example, node 34a labeled OP-0 includes a read operation element 37a' indicating that 'dataset I' data source element 37b' should be read. The modified data flow diagram 17' is stored in storage system 32 as, for example, a data structure.

[0087] Typically, execution engine 28 performs optimizations or other transformations that may be needed to process the data based on one or more operations specified in data flow diagram 17', or improves data processing based on one or more operations specified in data flow diagram 17' compared to processing data without optimization or transformation. For example, execution engine 28 adds one or more sorting operations, data type operations, join operations, including join operations based on keys specified in data flow diagram 17', partitioning operations, automatic parallel operations, or operations specifying metadata, to produce a data flow diagram 50 with the desired functionality of data flow diagram 17'. Figure 4D In some specific implementations, the computational efficiency of a transformed data flow graph is improved relative to the computational efficiency of the transformed data flow graph before optimization is applied by applying one or more data flow graph optimization rules. The transformed data flow graph 50” is (or is transformed into) an optimized data flow graph. Data flow graph optimization rules may include, for example, elimination of failed or redundant components, early filtering, or record reduction, as described below. Figure 4D As described in the context.

[0088] Now for reference Figure 4C Metadata Manager 24 and Metadata Store 25 transmit the Effective Loan EVD Definition 50 to EVD Integrator 26. Based on Graph 66' and Effective Loan EVD Definition 50, EVD Integrator 26 generates Data Flow Graph 50", as shown below. Using Effective Loan EVD Definition 50, EVD Integrator 26 adds the nodes necessary for generating the Effective Loan EVD to Graph 17'. In this example, EVD Integrator 26 includes removing the Effective Loan EVD component from Graph 66' and replacing it with the version from Graph 50', thus retrieving the Effective Loan EVD Definition (as defined by...). Figure 3D The logic of inserting (as shown in Figure 50') into Figure 17', where component 56 ( Figure 3D Remove from chart 50'. By doing so, EVD integrator 26 generates a computation chart 50" with the following new operation: filter by state = valid and calculate the average FICO and store it to thus generate computation chart 50".

[0089] Now for reference Figure 4D The execution engine 28 and optimizer 30 optimize the computation graph 50” to, for example, remove components that are not actually used, as well as other optimizations. In this example, calculating the average FICO component does not require data from the following data sources: accessing hardship.dat, accessing loan details.dat, or accessing settlement.dat. Therefore, reading data from these data sources would result in reading data that downstream components have never used. This would lead to a waste of memory and computational resources. To ensure that the computation graph 50” is computationally efficient and does not waste memory and processing resources when retrieving unused data, optimizer 30 includes optimizer rules and logic configured to analyze the components of the data flow graph and remove components that are not used by downstream components (e.g., read components). Thus, the available dataset 29 ( Figure 4E The generation of the dataset is computationally efficient, with reduced memory and processing resource usage relative to the amount of processing and memory resources required to generate the dataset that includes all data sources specified in the Valid Loan EVD definition. In this example, downstream refers to a component whose input is based on the output of another component. Based on this, execution engine 28 and optimizer 30 perform data processing operations (e.g., including optimization rules and / or data processing operations) to generate a computational graph “Valid Loan EVD Definition” that includes only a subset of the data that would otherwise be included in the extended view dataset definition 50. The “Valid Loan EVD Definition Computational Graph 50” includes “Access active_loan.dat”, “Access customers.dat”, and “Access FICO.dat”. The “Valid Loan EVD Definition Computational Graph 50” has “Access active_loan.dat” filtered by status = valid, and in this example specifically excludes “Access hardship.dat”, “Access loan_details.dat”, and “Access settlement.dat”. Execution engine 28 and optimizer 30 perform join operations on “Access active_loan.dat” (filtered by status = valid) and “Access customers.dat”, which are joined based on customer_id 54a. “Access FICO.dat” and join 54a based on customer_id are joined based on SSN 54b. The result is passed to calculate the average FICO and the generated dataset “Valid Loans EVD” is stored in storage system 32.

[0090] Figure 4D The bold "X" in the figure indicates the calculation chart 50' ( Figure 4C (not included) Figure 4D The portion of the computation graph 50. That is, as disclosed in U.S. Patent Publication No. 2019 / 0370407, the datasets “Access hardship.dat”, “Access loan_details.dat”, and “Accesssettlement.dat” are excluded from the computation graph 50 based on optimization.

[0091] In this example, return a reference. Figure 4B The specification represented in the editor (also referred to herein as the chart specification) specifies that from the Effective Loan EVD, the data is sorted by status = effective, and then the average FICO score is calculated. Therefore, according to this specification, not all datasets represented in the Effective Loan EVD are actually needed to perform the calculations specified in the editor. Active_loan.dat is required because it is the base dataset. Additionally, FICO.dat is required because the specification specifies that the average FICO score is calculated per user. customers.dat is required because this dataset has the keys needed to associate FICO.dat with active_loan.dat. In this example, hardship.dat, loan_details.dat, and settlement.dat are not required for the processing represented by the specification. Therefore, optimizer 30 removes references to those data sources in the Effective Loan EVD definition 50. By removing references to those data sources, the Effective Loan EVD (when generated) does not include unnecessary fields and data that are not used by the data processing operations specified in the specification. In this example, the valid loan EVD is referred to as the available dataset because the actual generated valid loan EVD only includes (and thus makes available) the data used by the data processing operations specified in the specification. Therefore, the available dataset is efficient in terms of both speed (for generating the available dataset and for processing it) and memory (because the system does not have to store data as part of the available dataset that is never used).

[0092] Now for reference Figure 4EThe execution engine 28 and optimizer 30 execute the computation graph "Effective Loan EVD Definition", which includes only a subset of data that should have been included in the extended view dataset. The "Effective Loan EVD Definition Computation Graph 50" includes "Accessactive_loan.dat", "Access customers.dat", and "Access FICO.dat". The "Effective Loan EVD Definition Computation Graph 50" has "Access active_loan.dat" filtered by status = effective, and in this example specifically excludes "Access hardship.dat", "Access loan_details.dat", and "Access settlement.dat". When executing the compiled version of “Calculate Chart 50”, execution engine 28 generates a usable dataset 29, for example, when completing the “Connect based on SSN” component and before executing the “Calculate Average FICO” component. Usable dataset 29 includes only data from active_loan.dat, customers.dat, and FICO.dat, as previously described. In this example, usable dataset 29 includes all data from these three data sources, rather than implying that only a subset of data from these data sources is used to generate the preview. Execution engine 28 and optimizer 30 perform join operations on “Access active_loan.dat” (filtered by status = valid) and “Access customers.dat”, which are connected based on customer_id 54a. “Access FICO.dat” and the customer_id-based join 54a are joined based on SSN 54b. “Calculate Chart 50” calculates the average FICO score and stores the results.

[0093] The calculation graph 50” directly performs calculations. The calculation graph 50” obtains the required data for each component represented by the graph components, moves the data between components, and defines the execution order of the calculation processes. The execution engine 28 can also provide monitoring of the execution of the calculation graph 50”. The results of executing the calculation graph 50” Valid Loan EVD are stored in the storage system 32 via the storage component in the calculation graph 50”.

[0094] Now for reference Figure 5The diagram illustrates a process 150 performed by a data processing system 12, which enables a user to preview the attributes of fields in an extended view dataset and authorizes the user to specify one or more attributes for use in one or more data processing operations optimized for speed and data storage. Process 150 enables the user to specify one or more fields from the preview for use in downstream data processing to generate a dataset comprising the one or more fields specified for downstream data processing from the previewed fields. The generated dataset exhibits improved efficiency in terms of speed and data storage compared to generating a dataset comprising all fields of the extended view when only the specified one or more fields are used in the downstream data.

[0095] Process 150 includes receiving 152 an identifier for a base dataset, and based on that identifier, receiving 154 a definition for an extended view dataset. The definition of the extended view dataset specifies fields from other datasets or other datasets related to the base dataset. Based on the definition of the extended view dataset, process 150 includes outputting 156 a preview of the attributes of the fields of the extended view dataset, wherein the preview is generated from a subset of data from fields in other datasets or other datasets related to the base dataset. Process 150 also includes receiving 158 input specifying that one or more fields in the preview can be used for data processing, and based on that input, generating 159 a usable dataset, which includes data from the base dataset and data from one or more specified fields.

[0096] Now for reference Figure 5A Process 160 may include other features, such as providing a preview of the attributes of an extended view dataset, wherein the extended view dataset includes data from a base dataset and other datasets associated with the base dataset when it is generated. Process 160 may include specifying a definition of the extended view dataset, which includes data from the base dataset and other datasets associated with the base dataset, and specifying a set of data processing operations to be performed to generate the extended view dataset, wherein the preview is generated by applying only a subset of the data in the base dataset and other datasets associated with the base dataset to the set of data processing operations specified by the definition of the extended view dataset. In response to providing the preview, process 160 includes receiving a specification of data processing operations, wherein the specified data processing operations are defined at least in part based on user input that identifies attributes included in the preview as attributes of the data processing operations.

[0097] Now for reference Figure 5BBased on data processing operations defined at least in part based on user input that identifies attributes included in the preview as data processing operations, process 170 may include updating 166 the set of data processing operations by applying one or more optimization rules to the defined set of data processing operations, and performing 168 the updated set of data processing operations to generate a dataset that includes only a subset of the data that should have been included in the extended view dataset.

[0098] Process 170 may include enabling user 169 to register a definition of a new dataset with a data catalog, wherein the definition specifies a selected dataset and other datasets associated with the selected dataset, thereby providing logical access to other datasets associated with the base dataset without incurring computational costs related to providing other associated datasets. Process 170 may include accessing a data catalog 171 specifying one or more datasets, and providing a user interface indicating that one or more datasets are candidates for generating extended view datasets. Process 170 may include receiving, through the user interface 172, an instruction for a specific dataset as a base dataset, and, in response to the instruction, automatically generating a definition of an extended view dataset for the specific base dataset.

[0099] Process 170 may include identifying a particular base dataset 174 as a base dataset and identifying one or more attributes of the particular base dataset; determining the definition of the base dataset based on the one or more attributes; determining one or more other datasets associated with the base dataset based on the definition of the base dataset; and generating the definition of an extended view dataset based on the determined one or more other datasets, the definition specifying the base dataset, one or more other datasets, and one or more relationships between the base dataset and one or more other datasets.

[0100] Process 170 may include storing the definition of 176 extended view dataset in a hardware storage device, and registering the definition of extended view dataset in a data directory.

[0101] Now for reference Figure 5C Process 180 may include receiving 182 requests for extended view datasets; in response to the requests, providing 184 extended view datasets by: retrieving the definition of the extended view dataset from a hardware storage device; retrieving a base dataset and one or more other datasets from one or more data sources based on the definition of the extended view dataset; and generating the extended view dataset based on the data in the retrieved datasets.

[0102] Process 180 may include, based on the extended view dataset, determining 186 whether to update the data catalog to designate the definition of the extended view dataset as a data source, storing 188 the definition of the extended view dataset in a hardware storage device, and registering the definition of the extended view dataset with the data catalog. Process 180 may include receiving a request for an extended view dataset; in response to the request, generating the extended view dataset by: retrieving the definition of the extended view dataset from the hardware storage device; retrieving a base dataset and one or more other datasets from one or more data sources based on the definition of the extended view dataset; and generating the extended view dataset based on data in the retrieved datasets.

[0103] The data flow graph component includes a data processing component and / or a dataset. The data flow graph can be represented by a directed graph comprising nodes or vertices representing the data flow graph components, connected by directed links or data flow connections representing the flow of work elements (i.e., data) between the data flow graph components. The data processing component includes code for processing data from at least one data input (e.g., a data source) and providing data to at least one data output (e.g., a data sink) of system 10. The data flow graph can thus enable graph-based computations performed on data flowing from one or more input datasets through the graph component to one or more output datasets.

[0104] System 10 also includes a data processing system 12 for executing one or more computer programs (such as data flow graphs), which are generated by transforming a specification into the computer program using a transformation generator and the techniques described herein. The transformation generator transforms the specification into a computer program that implements multiple modules. In this example, selections made by the user through the user interface described herein form a specification specifying which fields and datasets to use in a complex aggregation. Based on this specification, the transformation described herein is generated.

[0105] The data processing system 12 may be hosted on one or more general-purpose computers under the control of a suitable operating system, such as UNIX. For example, the data processing system 12 may include a multi-node parallel computing environment, which includes a configuration of computer systems using multiple central processing units (CPUs), which may be local (e.g., a multiprocessor system, such as an SMP computer), or locally distributed (e.g., multiple processors coupled as a cluster or MPP), or remotely distributed (e.g., multiple processors coupled via a LAN or WAN network), or any combination thereof.

[0106] The above-described diagram configuration method can be implemented using software that executes on a computer. For example, the software is formed within one or more computer programs executing on one or more systems 10 (e.g., computer programming or computer-programmable systems, which may have various architectures such as distributed, client / server, or grid) each including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The software may form one or more modules of a larger computer program, for example, providing other services related to the design and configuration of the data flow diagram. The nodes and elements of the diagram may be implemented as data structures stored in a computer-readable medium or as data of other organizations conforming to a data model stored in a data repository.

[0107] The software may be located on a non-volatile storage medium, such as a hardware storage device (e.g., a CD-ROM), which can be read by a general-purpose or special-purpose programmable computer, or delivered (encoded in a propagating signal) to a computer executing the software via a network communication medium. All these functions may be executed on a dedicated computer or using dedicated hardware such as a coprocessor. The software may be implemented in a distributed manner, wherein different portions of the data stream specified by the software are executed by different computers. Each such computer program is preferably stored on or downloaded to a non-volatile storage medium or hardware storage device (e.g., solid-state memory or medium, or magnetic or optical medium) that can be read by a general-purpose or special-purpose programmable computer for configuring and operating the computer to perform the processes described herein when the non-volatile storage medium or device is read by system 10. System 10 may also be considered as a computer-readable storage medium configured with computer programs, wherein the storage medium is configured such that system 10 operates in a specific and predefined manner to perform the functions described herein.

[0108] Example computing environment

[0109] refer to Figure 6An example operating environment for implementing embodiments of the present invention is shown, and this operating environment is generally designated as computing device 120. The basic elements of computing device 120, or computer, or data processing system, or client, or server, are one or more programmable processors 122 for performing actions according to instructions and one or more memory devices 124 for storing instructions and data. Generally, the computer will also include (via bus 121, texture, network, etc.) operatively coupled to I / O components 126, such as display devices, network / communication subsystems, etc. (not shown), and one or more mass storage devices 128 for storing data and instructions, etc., and a network communication subsystem 130 powered by a power supply (not shown). In memory 124 are an operating system 124a and an application program 124b for application programming.

[0110] Devices suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. Processors and memory may be supplemented or incorporated into them by dedicated logic circuitry.

[0111] To provide interaction with the user, the embodiments of the subject matter described in this specification are implemented on a computer having a display device (monitor) for displaying information to the user and a keyboard and pointing device (e.g., a mouse or trackball) that the user can use to provide input to the computer. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the user's device (e.g., by sending a webpage to the web browser in response to a request received from the user's web browser on the user's device).

[0112] Implementations of the subject matter described in this specification can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with a specific implementation of the subject matter described in this specification), or any combination of one or more such back-end components, middleware components, or front-end components. The components of the system can be interconnected via any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include local area networks (“LANs”) and wide area networks (“WANs”), inter-network networks (e.g., the Internet), and peer-to-peer networks (e.g., self-organizing peer-to-peer networks).

[0113] The computing system may include clients and servers. Clients and servers are typically geographically separated and usually interact via a communication network. The relationship between the client and server arises from computer programs running on their respective computers, and they have a client-server relationship. In some implementations, the server sends data (e.g., HTML pages) to the client device (e.g., for the purpose of displaying data to a user interacting with the user device and receiving user input from that user). Data generated at the client device (e.g., the result of user interaction) may be received from the client device at the server.

[0114] Although this specification contains many specific implementation details, these should not be construed as limiting the scope of any invention or potentially claimed content, but rather as a description of the features of specific embodiments specific to a particular invention.

[0115] Similarly, although the accompanying drawings show the operations in a specific order, this should not be construed as meaning that such operations must be performed in the specific or sequential order shown, or that performing all the shown operations must yield the desired result. Furthermore, the separation of the various system components in the embodiments described above should not be construed as requiring such separation in all embodiments, but rather as meaning that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0116] Several embodiments have been described. However, it will be understood that various modifications may be made without departing from the spirit and scope of the techniques described herein. For example, some of the steps described above may be order-independent and therefore may be performed in a different order than that described. Additionally, any of the techniques described with respect to the data flow diagram can also be applied to program implementation and execution. Therefore, other embodiments are also within the scope of the following claims.

Claims

1. A method implemented by a data processing system, the method being used to: enable a user to preview attributes of fields of an extended view of a base dataset and specify one or more fields of the fields for downstream data processing, and generate a dataset including the one or more fields specified for the downstream data processing from the previewed fields, wherein the generated dataset has improved efficiency in terms of speed and data storage compared to generating a dataset including all fields of the extended view when only the specified one or more fields of the fields are used in the downstream data processing, the method comprising: Receive the identifier of the basic dataset. Based on the identifier, receive the definition of the extended view dataset. The definition of the extended view dataset specifies fields from other datasets or other datasets related to the base dataset. Based on the definition of the extended view dataset, a preview of the attributes of the fields of the extended view dataset is output, wherein the preview is generated from a subset of data in the fields of the other dataset associated with the base dataset or the other dataset. Receive input specifying one or more fields in the preview that can be used for data processing, and Based on the input, a usable dataset is generated that includes data from the base dataset and data from one or more specified fields.

2. The method according to claim 1, further comprising: Provides a preview of the properties of the extended view dataset, wherein the extended view dataset includes data from the base dataset and other datasets associated with the base dataset when it is generated.

3. The method of claim 2, wherein the definition of the extended view dataset specifies a set of data processing operations performed to generate the extended view dataset, the extended view dataset comprising data from the base dataset and the other datasets associated with the base dataset, and The preview is generated by applying the set of data processing operations specified by the definition of the extended view dataset only to a subset of the data in the base dataset and the other datasets associated with the base dataset.

4. The method according to claim 2, further comprising: In response to providing the preview, a specification of a data processing operation is received, wherein the specified data processing operation is defined at least in part based on user input that identifies attributes included in the preview as attributes of the data processing operation.

5. The method according to claim 3, further comprising: The data processing operation set is updated by applying one or more optimization rules to the defined set of data processing operations, based at least in part on user input that identifies the attributes included in the preview as attributes of the data processing operation. Perform the updated set of data processing operations to generate a dataset that includes only a subset of the data that should have been included in the extended view dataset.

6. The method according to claim 1, further comprising: This allows users to register the definition of a new dataset with a data catalog, wherein the definition of the new dataset specifies the selected dataset and other datasets associated with the selected dataset, wherein the definition provides logical access to the other datasets associated with the base dataset without incurring computational costs related to providing the other datasets.

7. The method according to claim 6, further comprising: Access the data directory of one or more specified datasets, and Provide a user interface indicating that the one or more datasets are candidates for generating the extended view dataset.

8. The method according to claim 7, further comprising: The user interface receives instructions for a specific dataset that serves as the base dataset. In response to the instruction, the definition of the extended view dataset for the specific dataset is automatically generated.

9. The method according to claim 8, further comprising: The specific dataset is identified as the base dataset, and one or more attributes of the specific dataset are identified. The definition of the basic dataset is determined based on one or more of the aforementioned attributes. Based on the definition of the basic dataset, determine one or more other datasets related to the basic dataset, and Based on one or more other datasets identified, the definition of the extended view dataset is generated, the definition specifying the base dataset, the one or more other datasets, and one or more relationships between the base dataset and the one or more other datasets.

10. The method according to claim 1, further comprising: The definition of the extended view dataset is stored in the hardware storage device, and Register the definition of the extended view dataset with the data catalog.

11. The method according to claim 1, further comprising: Receive a request for the extended view dataset. In response to the request, the extended view dataset is provided by the following operations: Retrieve the definition of the extended view dataset from the hardware storage device. Based on the definition of the extended view dataset, the base dataset and the one or more other datasets are retrieved from one or more data sources. The extended view dataset is generated based on the data retrieved from the dataset.

12. The method according to claim 11, further comprising: Based on the extended view dataset, determine whether to update the data catalog to specify the definition of the extended view dataset as the data source. The definition of the extended view dataset is stored in the hardware storage device, and Register the definition of the extended view dataset with the data directory.

13. The method according to claim 7, further comprising: Based on the preview provided by the properties of the extended view dataset, determine whether to update the data catalog to designate the definition of the extended view dataset as the data source.

14. The method of claim 1, wherein generating the usable dataset comprises: Use the definition of the extended view dataset to access only those datasets that have one or more specified fields, and include the data of those accessed datasets into the available dataset.

15. The method according to claim 1, further comprising: The generated available dataset is processed to obtain the results from the data obtained from processing the available dataset.

16. The method according to claim 1, further comprising: Grant user permissions to access a portion of the base dataset within the extended view dataset, while denying the user access to the remainder of the base dataset.

17. The method of claim 1, wherein the definition of the extended view dataset includes a computational graph specifying a set of data processing operations for generating the extended view dataset, the extended view dataset including data from the base dataset and the other datasets associated with the base dataset, the set of data processing operations including at least one operation for joining the data from the base dataset and data from at least one of the other datasets associated with the base dataset.

18. The method of claim 1, wherein the definition of the extended view dataset provides logical access to data from the base dataset and the other datasets associated with the base dataset.

19. A data processing system, the data processing system being configured to: enable a user to preview attributes of fields of an extended view of a base dataset and specify one or more fields of the fields for downstream data processing, and generate a dataset including the one or more fields of the fields from the preview that are specified for the downstream data processing, wherein the generated dataset has improved efficiency in terms of speed and data storage compared to the efficiency of generating a dataset including all the fields of the extended view when only the specified one or more fields of the fields are used in the downstream data processing, the data processing system comprising: One or more processing devices; and One or more machine-readable hardware storage devices that store instructions executable by the one or more processing devices to perform operations including: Receive the identifier of the basic dataset. Based on the identifier, receive the definition of the extended view dataset. The definition of the extended view dataset specifies fields from other datasets or other datasets related to the base dataset. Based on the definition of the extended view dataset, a preview of the attributes of the fields of the extended view dataset is output, wherein the preview is generated from a subset of data in the fields of the other dataset associated with the base dataset or the other dataset. Receive input specifying one or more fields in the preview that can be used for data processing, and Based on the input, a usable dataset is generated that includes data from the base dataset and data from one or more specified fields.

20. One or more machine-readable hardware storage devices, the one or more machine-readable hardware storage devices being configured to: enable a user to preview attributes of fields of an extended view of a base dataset and specify one or more of the fields for downstream data processing, and generate a dataset including the one or more fields specified for the downstream data processing from the previewed fields, wherein the generated dataset has improved efficiency in terms of speed and data storage compared to the efficiency of generating a dataset including all the fields of the extended view when only the specified one or more fields are used in the downstream data processing, the one or more machine-readable hardware storage devices storing instructions executable by one or more processing devices to perform operations including the following: Receive the identifier of the basic dataset. Based on the identifier, receive the definition of the extended view dataset. The definition of the extended view dataset specifies fields from other datasets or other datasets related to the base dataset. Based on the definition of the extended view dataset, a preview of the attributes of the fields of the extended view dataset is output, wherein the preview is generated from a subset of data in the fields of the other dataset associated with the base dataset or the other dataset. Receive input specifying one or more fields in the preview that can be used for data processing, and Based on the input, a usable dataset is generated that includes data from the base dataset and data from one or more specified fields.

21. A method implemented by a data processing system, the method being configured to: enable a user to preview attributes of fields of an extended view of a base dataset and specify one or more fields of the fields for downstream data processing, and generate a dataset including the one or more fields specified for the downstream data processing from the previewed fields, wherein the generated dataset has improved efficiency in terms of speed and data storage compared to generating a dataset including all fields of the extended view when only the specified one or more fields of the fields are used in the downstream data processing, the method comprising: Receive the identifier of the basic dataset. Based on the identifier, receive the definition of the extended view dataset. The definition of the extended view dataset specifies fields from other datasets or other datasets related to the base dataset. Based on the definition of the extended view dataset, a preview of the attributes of the fields in the extended view dataset is output, wherein the preview is generated from a subset of data in the other datasets associated with the base dataset or from the metadata of the other datasets. Receive input specifying one or more fields in the preview that can be used for data processing, and Based on the input, a usable dataset is generated that includes data from the base dataset and data from one or more specified fields.

22. The method of claim 21, wherein the preview is generated during development, and wherein the available dataset is generated at runtime.

Images