Data processing system involving the manipulation of logical dataset groups

Abstract

This invention provides a data processing system and method that receive user input through a user interface that enables manipulation of hierarchical groups of datasets. [Solution] The method presents a user interface configured to allow the user to select one or more datasets or dataset carts for use in conjunction with data access-related operations in a data processing system; identifies a persona associated with the user of the data processing system; identifies scope information associated with datasets and / or groups of datasets; automatically identifies one or more groups of datasets based at least partially on the correspondence between the user persona and the scope information associated with the automatically identified groups of datasets; and renders an indication of the automatically identified groups of datasets via the user interface.

Description

Technical Field

[0001] Cross - Reference to Related Applications This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 163,699, entitled "DATA PROCESSING SYSTEM WITH MANIPULATION OF LOGICAL DATASET GROUPS," filed on March 19, 2021, and U.S. Provisional Patent Application No. 63 / 143,924, entitled "DATA PROCESSING SYSTEM WITH MANIPULATION OF LOGICAL DATASET GROUPS," filed on January 31, 2021, under 35 U.S.C. § 119(e). The entire disclosures of these patents are incorporated herein by reference in their entireties.

[0002] Aspects of the present disclosure relate to techniques for efficiently operating a data processing system having a number of datasets that can be stored in any of a number of data stores.

Background Art

[0003] Modern data processing systems manage vast amounts of data within enterprises. For example, large institutions may have millions of datasets. Having such a large number of datasets can be extremely valuable to an enterprise because this data can support multiple aspects of its operations. Some datasets can support routine processes, such as tracking customer account balances or sending account statements to customers. In other examples, processing data from one or more datasets can generate business insights, such as concluding that a requested transaction is fraudulent, or that an enterprise is exposed to a certain level of financial risk as a result of overall transactions in a particular geographical area. In yet another example, processing data from one or more datasets can generate technical insights, such as concluding that an enterprise is at risk of technical failure as a result of flawed technical processes.

[0004] Datasets can be accessed by applications executed by the data processing system or through tools invoked by users of the data processing system. Applications can be developed by programmers to perform repetitive processes (such as tracking customer account balances or sending account statements to customers). Programmers can specify datasets that serve as the source of data input to the process or as the destination of results generated by executing the process. Tools can also perform operations using datasets. For example, a data processing system may include tools that allow users to process datasets for purposes such as removing invalid records or generating metrics about the dataset (such as the number of records or fields containing invalid values).

[0005] To assist users, a dataset search capability can be provided to help them find appropriate datasets within an enterprise. For example, an application development environment may include a dataset search interface that allows application programmers to specify the characteristics of the desired dataset. The programmer can then select input or output datasets from the search results. Through a similar search, users can identify datasets as input or output for a tool.

[0006] Searches can be performed based on metadata stored for a dataset. For example, a data processing system may store metadata for a dataset that indicates the values ​​of one or more parameters that characterize the dataset. This metadata may include, for example, the names or descriptions of the dataset's fields or the dataset itself. Another example is that the metadata may indicate the organization within the company that created the dataset, the program that generated the dataset, and the date the dataset was created. These or other types of metadata may be used in a dataset search. [Overview of the project] [Means for solving the problem]

[0007] According to some embodiments, a method is provided for enabling efficient operation of a data processing system in an environment using multiple datasets by forming and presenting dataset groups in order to make selections relating to configuring an operation to access one or more datasets. The method includes receiving input from a first user through one or more first user interfaces to select one or more datasets of a plurality of datasets to associate with one group of a plurality of datasets; storing representations of the plurality of datasets; and presenting a second user interface configured for a second user to select one or more datasets for use in conjunction with an action to access one or more datasets, wherein the second user has a persona and the datasets have a scope at least partially based on the user's persona; and presenting a second user interface that includes automatically identifying one or more groups of datasets based at least partially on the correspondence between the persona associated with the second user of the data processing system and the scope associated with one or more automatically identified groups of datasets; and rendering an indication of one or more automatically identified groups of datasets in the second user interface.

[0008] According to one embodiment, storing representations of multiple groups includes storing information about one or more users who have permission to access the group for each of the multiple groups in the dataset.

[0009] According to one embodiment, one or more first user interfaces include a dataset search interface that includes a faceted search interface, the facets of the faceted search interface being based on metadata values ​​associated with a number of datasets.

[0010] According to one embodiment, one or more first user interfaces include a user interface that displays the lineage of a dataset.

[0011] According to one embodiment, one or more first user interfaces include a user interface that displays metadata related to one of a number of datasets.

[0012] According to one embodiment, the method further includes receiving input from a second user through a second user interface specifying one of one or more automatically identified groups, and performing an action on each of a number of datasets within the selected group based on the input received from the second user.

[0013] According to one embodiment, the operation includes configuring an application for execution by a data processing system.

[0014] According to one embodiment, automatically identifying one or more groups of datasets, at least in part, based on the correspondence between a persona associated with a second user of a data processing system and a scope associated with one or more automatically identified groups of datasets, includes selecting one or more groups of datasets that the second user is authorized to access.

[0015] According to one embodiment, rendering indications for one or more automatically identified groups includes rendering graphical user interface elements that represent groups of datasets for each of the one or more automatically identified groups, the method further includes receiving a selection of rendered graphical user interface elements that represent groups of datasets via a second user interface, and rendering a number of datasets for the groups on the second user interface based on that selection.

[0016] In some embodiments, a method is provided for enabling efficient operation of a data processing system in an environment using multiple datasets by presenting dataset groups for a user of the data processing system to make a selection in relation to configuring an operation to access one or more datasets. The method is to present a user interface configured for a user to select one or more datasets for use in conjunction with an operation to access one or more datasets, the user having a persona, and the datasets having a scope at least partially based on the user's persona, the presenting of a user interface which includes automatically identifying one or more groups of datasets based at least partially on the correspondence between a persona associated with a user of the data processing system and the scope associated with one or more automatically identified groups of datasets, and rendering an indication of one or more automatically identified groups of datasets in the user interface.

[0017] According to one embodiment, the method further includes receiving user input through a user interface that specifies one of one or more groups, and rendering an indication of the datasets within the selected group based on the received input.

[0018] According to one embodiment, the method further includes receiving user input through a user interface that specifies one of one or more groups, and performing an action on each of a number of datasets within the selected group based on the received input.

[0019] According to one embodiment, automatically identifying one or more groups of datasets further includes receiving a search query for the dataset via a user interface and performing a search based on the search query to generate search results.

[0020] According to one embodiment, the operation includes configuring an application for execution by a data processing system.

[0021] According to one embodiment, automatically identifying one or more groups of datasets, at least in part, based on the correspondence between a persona associated with a user of a data processing system and a scope associated with one or more automatically identified groups of datasets, includes selecting one or more groups of datasets that the user is authorized to access.

[0022] According to one embodiment, rendering an indication of one or more automatically identified groups includes rendering a graphical user interface element that represents a group of datasets for each of the one or more automatically identified groups, and the method further includes receiving a selection of the rendered graphical user interface elements that represent a group of datasets, and rendering a number of datasets for the group onto the user interface based on that selection.

[0023] In some embodiments, a method is provided for enabling efficient operation of a data processing system in an environment using multiple datasets by enabling the selection of a group of datasets to perform an operation on each of the multiple datasets in the group. The method includes: receiving a search query via a user interface for searching for datasets to be used in conjunction with operations related to data access in the data processing system; presenting the search results to the user interface based on the search query, by presenting one or more groups of datasets, wherein each of at least some of the groups of datasets presents one or more groups containing one or more of the search datasets; receiving an operation on a first group of datasets of one or more groups of datasets presented to the user interface, wherein the user interface is configured to provide an option to select the first group of datasets via the user interface as a target for operations related to data access; and, once the first group of datasets of one or more groups of datasets presented to the user interface has been selected, performing an operation on each of the one or more datasets contained in the first group of datasets.

[0024] According to one embodiment, performing an operation in each of one or more datasets includes performing data quality rules in each of one or more datasets.

[0025] In one embodiment, the user interface provides an option to expand a first group of datasets to enable the user to select one or more datasets from a first group of datasets as targets for operations related to data access, and as soon as one or more datasets from the first group of datasets are selected, the operation is performed on each of the datasets from the first group of datasets.

[0026] According to one aspect, each of one or more groups of data sets presented in a user interface has a correspondence between a persona associated with a user who entered a search query via the user interface and a scope associated with the one or more groups of data sets.

[0027] According to one aspect, the search results exclude data sets that do not have metadata associated with the user's persona.

[0028] According to some aspects, a method is provided for enabling efficient operation of a data processing system in an environment using multiple data sets by forming groups of data sets. The method includes rendering one or more first user interfaces in which a number of data sets are identified, receiving user input through the one or more first user interfaces to select one or more of the identified data sets for association with one of a number of groups of data sets, and storing representations of the number of groups of data sets.

[0029] According to one aspect, storing representations of a number of groups includes storing information about one or more users having access rights to a group for each group of the number of groups of data sets.

[0030] According to one aspect, the method is rendering a second user interface associated with a user configuration of the data processing system for performing operations related to data access, the second user interface further including rendering a data set selection portion, and rendering the second user interface includes presenting representations of one or more of the number of groups of data sets in the data set selection portion.

[0031] According to one embodiment, the method further includes selecting one or more groups of a number of datasets to be presented to a second user interface, based on the user persona.

[0032] According to one embodiment, the second user interface includes a user interface in a program development environment, and the operations related to data access include configuring a component of the program under development to access a dataset or a group of datasets.

[0033] According to one embodiment, one or more first user interfaces include a dataset search interface.

[0034] According to one embodiment, the dataset search interface includes a faceted search interface, where the facets of the faceted search interface are based on metadata values ​​associated with a large number of datasets.

[0035] According to one embodiment, one or more first user interfaces include a user interface that displays the lineage of a dataset.

[0036] According to one embodiment, one or more first user interfaces include a user interface that displays metadata related to one of a number of datasets.

[0037] In some embodiments, a method is provided for enabling the efficient operation of a data processing system in an environment using multiple datasets. The method includes means for rendering one or more first user interfaces in which datasets are identified; means for receiving user input through one or more first user interfaces for selecting one or more identified datasets to associate with one group of a number of groups of datasets; and means for storing representations of the number of groups of datasets.

[0038] According to one embodiment, the method is a means for rendering a second user interface associated with a user configuration of a data processing system for performing operations related to data access, wherein the second user interface further includes a means for rendering, the second user interface including a dataset selection portion, the means for rendering the second user interface includes presenting a representation of one or more groups of a number of groups of datasets in the dataset selection portion.

[0039] According to one embodiment, the method further includes means for selecting one or more groups of a number of datasets to be presented in a second user interface, based on a user persona.

[0040] In some embodiments, a method is provided for creating a dataset group in a data processing system capable of operating with a large number of datasets. The method includes identifying a set of datasets available for use when the data processing system performs an operation, the operation relating to data access in the data processing system; presenting the identified set of datasets to a first user interface; receiving a user selection of one or more datasets from the presented identified set of datasets via the first user interface; and storing a representation of the group containing the selected one or more datasets.

[0041] According to one embodiment, identifying a set of datasets available for use when performing data access-related operations in a data processing system includes receiving a search query via a user interface that specifies one or more values ​​of facets describing a number of datasets defined in the data processing system, and performing a search based on the search query to generate search results, wherein the search results include a set of datasets available for use when performing operations.

[0042] According to one embodiment, a search query includes a faceted search query, and the faceted search query includes one or more facets for filtering the search results.

[0043] According to one embodiment, one or more facets include a facet indicating whether the dataset is registered in a catalog that associates information for accessing the physical dataset with the logical dataset.

[0044] According to one embodiment, a user interface for receiving search queries includes a number of fields for receiving user input that identifies values ​​for one or more facets, the number of fields including fields for receiving values ​​of logical, physical and / or behavioral metadata associated with a number of datasets.

[0045] According to one embodiment, operations related to data access include configuring components of an application executed by a data processing system.

[0046] In one embodiment, receiving a command to update a group via a second user interface, wherein the command includes a request to add one or more datasets to the group or a request to remove one or more datasets from the group.

[0047] According to one embodiment, metadata relating to a dataset of an identified set of datasets is presented via a first user interface in response to user input requesting metadata related to a dataset.

[0048] In one embodiment, a group is a second group, and receiving a user selection of one or more datasets includes receiving a selection of a first group of datasets previously defined such that the second group includes a hierarchical grouping of datasets.

[0049] According to one embodiment, storing a group representation includes storing scope information for the group.

[0050] According to one embodiment, scope information includes the identification of one or more users who have permission to access the group.

[0051] According to one embodiment, scope information includes the identification of one or more roles that have permission to access the group.

[0052] According to one embodiment, the method is to render a second user interface associated with a user configuration of a data processing system for performing operations related to data access, further comprising rendering the second user interface including a dataset selection portion, wherein rendering the second user interface includes presenting a representation of a group including one or more selected datasets in the dataset selection portion.

[0053] The various embodiments described above can be used alternatively or in addition to any embodiments of the systems, methods, and / or processes described herein. Furthermore, a data processing system may be configured to operate in accordance with one or more of the embodiments described herein. Such a data processing system may include at least one computer hardware processor and at least one non-temporary computer-readable medium that, when executed by the at least one computer hardware processor, stores processor-executable instructions that cause the at least one computer hardware processor to perform such a method. Furthermore, the non-temporary computer-readable medium may, when executed by the at least one computer hardware processor of the data processing system, include processor-executable instructions that cause the at least one computer hardware processor to perform one or more of the embodiments described herein. Thus, the above is a non-limiting summary of the invention as defined by the appended claims.

[0054] The following diagrams illustrate various aspects. Please note that the diagrams are not necessarily drawn to scale. Items appearing in multiple diagrams are indicated by the same or similar reference numbers in all of them. [Brief explanation of the drawing]

[0055] [Figure 1A] This figure illustrates different users of an exemplary enterprise IT system who create and use groups of datasets, such as dataset carts, according to aspects of the technology described herein. [Figure 1B] This figure illustrates a user of an exemplary enterprise IT system who performs various operations related to datasets for the purpose of creating and / or managing groups of datasets, according to aspects of the technology described herein. [Figure 1C] This is a block diagram of an exemplary enterprise IT system comprising a data processing system having a dataset catalog that maintains information about a group of datasets, according to aspects of the technology described herein. [Figure 2A] This is an illustration of a simplified, exemplary graphical user interface, rendered by a data processing system, that allows the user to specify the components of an executable dataflow graph and the interconnections between those components. [Figure 2B] Figure 2A illustrates the operational state of an exemplary graphical user interface, where the user accesses a dataset selection tool to select a dataset as a step in the process of configuring components of an executable dataflow graph to access the dataset. [Figure 2C] Figure 2A is an illustration of an exemplary graphical user interface with additional elements of the user interface being depicted. [Figure 2D] This is an illustration of a simplified, exemplary graphical user interface, rendered by a data processing system, that allows the user to specify the components of an executable dataflow graph and the interconnections between those components. [Figure 2E] This is an illustrative diagram of the operational state of an exemplary graphical user interface, where the user accesses a dataset selection tool to select a dataset cart as a process step that constitutes a component of an executable dataflow graph. [Figure 3] This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows a user to select a logical dataset, with the user providing input requesting to browse data in the physical dataset corresponding to the logical dataset available for selection. [Figure 4A]This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows a user to select a dataset, where the user navigates through a directory of datasets as a first mechanism for restricting the search, and then enters text appearing in the dataset description as a search query as a second mechanism for restricting the search. [Figure 4B] Figure 4A illustrates the operational state after a search query has been executed in an exemplary graphical user interface rendered by a data processing system, allowing the user to select a dataset. A list of datasets matching the search query is presented, allowing the user to select one or more datasets as the target of the operation. [Figure 5] This is an illustration of the operational state after a search query has been executed in an exemplary graphical user interface rendered by a data processing system, allowing the user to select a dataset, with the list of datasets limited to those containing a field for storing email addresses. [Figure 6] This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows users to view or modify information related to a dataset. [Figure 7] This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows users to view or modify information related to a dataset cart. [Figure 8A] This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows users to define a dataset cart. [Figure 8B] Figure 8A illustrates different operating states of an exemplary graphical user interface, where the user can select datasets to include in the dataset cart. [Figure 9]This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows users to specify datasets to include in their dataset cart. [Figure 10A] This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows users to search a dataset. [Figure 10B] Figure 10A illustrates the operational state of an exemplary graphical user interface, where the user specifies additional search criteria to limit search results to datasets registered in the dataset catalog. [Figure 10C] Figure 10A illustrates the operational state of an exemplary graphical user interface, where the user indicates a dataset to be included in a group of datasets, which is shown here as a dataset cart. [Figure 11] This is an illustration of an exemplary graphical user interface, rendered by a data processing system, that allows users to view or modify information related to a group of datasets, which are here identified as a technical group. [Figure 12] This is a block diagram of an exemplary data structure that holds information about a group of datasets, depending on the configuration of the data processing system. [Figure 13] This is a flowchart illustrating an exemplary method for operating a data processing system capable of working with a large number of datasets, according to embodiments of the technology described herein. [Figure 14] This is a flowchart illustrating an exemplary method for operating a data processing system configured to perform operations that access a dataset, according to aspects of the technology described herein. [Figure 15] This is a flowchart illustrating an exemplary method for operating a data processing system configured to run a program for accessing a dataset, according to aspects of the technology described herein. [Figure 16]This is a block diagram of an exemplary computing system environment that can be used when implementing some aspects of the technology described herein. [Modes for carrying out the invention]

[0056] The inventors have recognized and understood that a data processing system can be performed more efficiently and is a more effective tool for data analysis when it supports the manipulation of groups of datasets that can serve as targets for operations performed by the data processing system. Groups of datasets can be presented in a user interface, instead of or in addition to individual datasets, through which the user selects one or more datasets as targets for operations. The user can then manipulate the group, for example, by expanding the group to allow selection of any of its components as targets for operations, or, in some scenarios, by selecting the group as the target for operations so that operations are performed on all of the datasets in the group. Since the datasets processed through the operations can be directly selected by the user through the manipulation of the group presented in the user interface, there is no longer any need to position and configure operations for individual datasets. In other words, the techniques described herein provide a graphical shortcut for initiating the processing of one or more datasets via a user-initiated operation, without the need to iteratively process datasets or configure menus for each of the individual datasets that need to be processed.

[0057] Dataset groups can be scoping so that only specific groups within the scope of that group appear as search results. By scoping dataset groups, data processing systems can automatically present relevant dataset groups at the time a dataset search is performed. In enterprises where there can be literally millions of datasets, search results can exclude datasets irrelevant to the user and / or the tasks being performed by that user. Thus, in addition to delivering more relevant search results, appropriate dataset searches can be performed quickly, with less consumption of processing resources. In other words, dataset groups as described herein assist in performing technical tasks of data storage and retrieval for efficient data management, such as in database management systems. Put another way, dataset groups facilitate access to data in an efficient manner.

[0058] Manipulating groups of datasets can be advantageous in data processing systems where a rich set of metadata about the datasets is maintained. Metadata can be used to search for or specify datasets for use as targets for data access-related actions within the data processing system. While a rich metadata set offers great flexibility in specifying search queries to identify datasets for specific data access actions, this flexibility can lead to complex user interfaces, long search times, or heavy use of computing resources, any or all of which can reduce the effectiveness of the data processing system. Scoping searches of groups of datasets for the user allows for a simpler search interface, enabling the return of equally or more relevant search results with less time and / or fewer computing resources. Metadata can relate to multiple aspects of a dataset, including its logical, physical, and / or operational modes.

[0059] A logical mode can refer to the significance of data in a dataset or fields within a dataset for an enterprise or people within an enterprise. A logical mode can be applied to a dataset regardless of its physical storage. For example, a dataset may be defined to hold customer data. That dataset may have a schema specifying fields that hold certain types of data meaningful within the enterprise, such as customer name, customer identifier, email, address, and telephone number. Fields can be specified as relating to such logical entities, regardless of the underlying physical storage of the data representing those entities.

[0060] In contrast, the physical configuration may relate to how the data in a dataset is stored. A dataset can be stored in a specific data store implemented with specific storage hardware and software. The software can organize the stored dataset into a table with rows of cells. Data corresponding to logical entities can be stored in one or more specific cells in each row. For example, data constituting an email address can be stored in three fields (one identified as the username, another as the domain name, and another as the TLD). Metadata about the physical configuration of a dataset may relate to the configuration of the physical data store, such as the storage schema of the physical storage, the software used to organize the data in the dataset, and / or the hardware that holds the data in the dataset. Alternatively or in addition to this, physical metadata may indicate data characteristics, such as the quantity or quality of the data. Metadata related to the quantity of data may indicate the total amount of data in the dataset, such as the number of records in the dataset. Other metadata related to quantity may indicate the number of records that have a specific value in a particular field. Metadata related to data quality may indicate the number of records that are missing a particular field or that contain an invalid value in a particular field.

[0061] The operational mode may relate to the operations performed using the dataset. For example, operational metadata can be recorded for each job performed by the data processing system. This metadata may indicate the datasets accessed between jobs and other information about the jobs (such as the values ​​of parameters entered into the job, the date and time the job is executed, or the user requesting the job execution).

[0062] A metadata repository for a data processing system can store other items of metadata about a dataset. Such metadata may include items that define the province of the dataset, such as the user who defined the dataset's schema or the system from which the data for the physical dataset was imported. Another example is the recording of text descriptions of the dataset or its fields.

[0063] Regardless of the specific metadata items that can be maintained in a data processing system, metadata can be used to group and / or search for one or more datasets within a large enterprise dataset for use as a target for operations in the data processing system. Metadata of various aspects can be stored by the data processing system in relation to one another. As a result, searches can find datasets that satisfy combinations of metadata aspects. The data processing system can provide a dataset selection tool in a user interface that allows users to search for datasets that satisfy multiple criteria regarding dataset metadata. The user can then select a dataset as a target from the datasets identified by the search. In embodiments where groups of datasets are scoping, the dataset selection tool can restrict searches to return only dataset groups containing datasets within the scope and / or dataset groups that are within the scope.

[0064] For example, a user developing an application in a development environment may select a dataset as input to the application. A dataset selection tool can present a user interface that allows the user to select a dataset, which is then identified in the development environment as a target for actions within the development environment that link the application to the identified dataset. To make a selection, the user can enter a search query that specifies a combination of values ​​for several logical, physical, and / or behavioral metadata aspects. As a specific example, a search query might specify a dataset containing emails, a dataset with email field data quality exceeding a specified threshold amount, and a dataset used in jobs within the last week. For this purpose, a faceted search interface can be used, which has different aspects of dataset metadata that provide facets for searching. The user can then select from a result set returned by the data processing system as a result of executing this query against the system's dataset metadata repository. If the result set contains one or more dataset groups, the user can provide input that acts as a command to expand the dataset group and indicate the datasets contained within that dataset group. The user can then select a dataset from the expanded dataset group. The dataset selected by the user can be returned to the development environment for use as input datasets for the application under development.

[0065] As another example, a dataset selection tool can be used to select datasets on which maintenance may be performed. A user might want to select a dataset for, for example, performing data quality rules. In this example, the dataset selection tool can be used to identify datasets that are offered as targets for a tool to perform a set of data quality rules on the dataset. Through the selection tool, the user can search for frequently used datasets in a job that meet other logical, physical, and / or operational requirements, and then select one or more of those datasets from this result set for a data quality analysis. If the result set includes one or more dataset groups, the user can provide inputs that act as commands to expand the dataset groups and indicate the datasets they contain. A dataset can then be selected from the expanded dataset groups. In some embodiments, the user may select a dataset group rather than a single dataset. In this context, a dataset group may be selected and offered as a target, rather than the user presenting the contents of a dataset group to make a single dataset selection. Once a group is offered as a target for a tool to perform an action on a dataset, that action may be performed on each dataset within the group.

[0066] To assist in the selection process, the dataset selection tool allows the user to access additional information about the datasets returned in response to the search query. This additional information may include, for example, some or all of the metadata stored for the datasets included in the search set. Alternatively, or in addition to this, the additional information may include information about the data in the selected dataset. For example, the additional information may include a view of a few rows of the selected dataset. This additional information may be presented, for example, in response to user interaction with user interface elements.

[0067] For companies with many datasets, enabling the manipulation of datasets in groups improves dataset search functionality. In the exemplary embodiments herein, a group, represented as a dataset cart, can be predefined and, like datasets, may have associated metadata that can define which datasets are members of the group. Associated dataset cart metadata may include logical, physical, and / or behavioral metadata. Dataset search capabilities can return groups of datasets, such as dataset carts, instead of, or in addition to, returning individual datasets. A dataset cart can be represented by a visually distinctive icon that looks different from the representation of individual datasets. The icon may appear, for example, as a shopping cart. In this specification, the description of features in the context of a dataset cart is not limited to dataset carts and applies to any representation of a group of datasets.

[0068] A dataset search may be restricted to returning dataset carts that satisfy some or all of the datasets in a dataset cart, based on specified search criteria. Alternatively, the search interface may include, for example, an option as a search facet, allowing the user to specify that only dataset carts, rather than individual datasets, should be returned in response to a search query.

[0069] A dataset cart allows users to limit the number of datasets considered when selecting datasets as targets for actions in a data processing system. In companies with millions of datasets, even with strictly defined search criteria, the system may return so many datasets that it becomes difficult, or even difficult, for users to identify the most relevant dataset without significant additional effort, such as further processing. For example, a dataset cart can be predefined to hold datasets appropriate for a particular task, thereby reducing the time required to select the right dataset from the cart. It can also generate more search results that are actually relevant to the user.

[0070] A dataset cart can be predefined by the same user performing the dataset search. The user can then consider selecting datasets from only one of their own dataset carts. Alternatively, or in addition to this, dataset carts can be curated by other users of the data processing system. For example, a user responsible for maintaining data about customers participating in a customer loyalty program can curate a dataset cart to include datasets representing the most reliable source of information about the loyalty program. Other users can then limit their selection of datasets for data analysis related to the customer loyalty program to the datasets in the cart. The data processing system can limit the results of a dataset search to any dataset cart accessible to the user requesting the search, or to the datasets within a dataset cart.

[0071] A data processing system that supports dataset carts can provide several benefits within an enterprise. For example, a data processing system can automate process flows that lead to greater efficiency. Figure 1A illustrates how different users of IT system 100 can create and use dataset carts within an enterprise. As shown in Figure 1A, a first user of the data processing system of IT system 100 (e.g., user 111a) or someone with knowledge of datasets, their lineage, and their individual advantages and disadvantages can, for example, define or create dataset carts (e.g., dataset carts 1, 2, 3, 4) suitable for a particular type of data analysis from many datasets (e.g., datasets 1-N). A second user of the data processing system (e.g., users 112a, 113a) or someone with knowledge of data analysis can quickly select one or more of those dataset carts or datasets relevant to a particular analytical task. Another benefit is that the human and computer work of searching for datasets across the enterprise's vast datasets can be done when datasets are assigned to dataset carts. Subsequently, searching for datasets to be used in data access-related operations can be simplified, both in terms of human and computer work. For example, a search interface for selecting datasets to be used in data access-related operations may only include a subset of the search facets or other options of the search interface for selecting datasets to include in a dataset cart, because fewer search facets are required to find relevant datasets when the search results are limited to dataset carts with relevant scope.

[0072] Figure 1B shows various actions (e.g., actions 115a, 115b, 115c, 115d, 115e) that a first user (such as user 111a) can perform to define, create, and / or manage a dataset cart. For example, user 111a can view or modify information about datasets and / or dataset groups / carts via the interfaces described in relation to Figures 6, 7, and 11. As another example, user 111a can define or create a dataset cart via the interfaces described in relation to Figures 8A and 8B. As yet another example, user 111a can select or specify datasets to include in one or more dataset carts via the interfaces described in relation to Figures 9 and 10C. As yet another example, user 111a can search for datasets via the interfaces described in relation to Figures 10A and 10B.

[0073] To perform these or other operations, user 111a may need expertise in some or all of the datasets (dataset 1…dataset N), or user 111a may need to undertake time-consuming searches through many such datasets. However, as shown in Figure 1B, the burden of these operations on users 112a and 113a and the enterprise IT system can be avoided by creating a smaller number of dataset carts. For example, the processing power and network bandwidth required for users 112a and 113a to make such selections can be reduced. Moreover, this reduction may be compounded in terms of computing resources because users such as 112a and 113a frequently perform searches for relevant datasets.

[0074] Dataset grouping can be hierarchical. A group of datasets may include a dataset, as well as subgroups of datasets. The hierarchy can extend to multiple levels, with subgroups containing further subgroups. In an example where a group is represented as a dataset cart, the dataset cart may contain subgroups of datasets, either in place of or in addition to a dataset. These subgroups can be identified as dataset carts within a cart, or the dataset cart can be identified as a top-level grouping with subgroups represented in a different way.

[0075] A dataset selection tool can conditionally perform actions on a group of datasets returned in a search, at least in part, depending on the action that invoked it. For example, if the action requires a single dataset as its target, the dataset selection tool will expand the group to allow the user to select a single dataset, regardless of whether the group is a dataset cart or a subgroup, as a result of the user's group selection following the execution of the search query. Conversely, if the action can be applied to multiple datasets, the user may be prompted to select all datasets in the group as a target, or a mechanism for doing so may be provided, or the system may be made to present multiple datasets in the group that the user can select. Such a selection tool can be implemented, for example, by providing separate navigation and selection controls. Through the navigation control, the user can traverse the hierarchy of dataset groupings. Through the selection control, the user can select a single dataset or a group of datasets as desired. In some examples, the selection control may be context-dependent. For example, the selection control may be configured to exclude the selection of a dataset group in a scenario where only a single dataset is the appropriate target.

[0076] Groups can be scoped so that the groups returned in response to a search query are restricted based on scope. For example, a dataset cart can be scoped based on user personas. A persona can represent, for example, a specific individual or a group of individuals. Individuals can be specified based on their identity, which can be established, for example, by credentials, or based on membership in one or more groups, such as a department within an enterprise or a specific project team. Alternatively or in addition to this, personas can be established based on roles within an enterprise, such as a data analyst, application developer, test engineer, or database programmer. Alternatively or in addition to this, other criteria can be used to identify users authorized to use the dataset cart, or other criteria can be used when specifying personas.

[0077] By scoping dataset carts, the amount of data returned to any particular user in response to a dataset search through a dataset selection tool can be limited. The tool can, for example, check the personal characteristics of the user requesting the dataset search and then limit the result set to only dataset carts and / or datasets that have a scope encompassing those user characteristics. This method allows for the return of fewer and more relevant results from a dataset search.

[0078] Such a selection method can be used, for example, by a data analyst creating a dataset cart containing datasets relevant to a project. A dataset selection tool can be used to select target datasets for multiple operations within a data processing system. In this method, available datasets are followed by the data analyst throughout their work, thereby ensuring that the appropriate datasets are selected quickly and consistently.

[0079] It is not necessary to execute the exact same computer executable instructions to implement a dataset selection tool for each operation in which one or more datasets are selected as targets. In some embodiments, a general-purpose tool can be implemented to support this operation. However, in other embodiments, the dataset selection methodology can be implemented by different computer executable instructions that perform the selection function described above. When different computer executable instructions are used to support dataset selection for different operations performed by the data processing system, each copy of the computer executable instructions may render a similar interface for consistency or ease of use. However, identical interfaces for dataset selection for different operations are not required.

[0080] The data processing system can be implemented to achieve one or more of the aforementioned objectives and benefits. These objectives and benefits can be used individually or in any appropriate combination.

[0081] Typical data processing systems that support dataset carts Dataset groups, such as dataset carts, as described herein, can be used in data processing systems that provide search interfaces to enable users to search for datasets as targets for their actions. These search interfaces can perform searches that return dataset groups / carts, either in addition to or instead of datasets. Other interfaces may enable users to create or modify dataset groups / carts. Such data processing systems may include one or more components that maintain a repository of information about dataset carts (including their scopes).

[0082] The exemplary data processing system can operate not only on physical datasets but also on logical datasets. A logical dataset may be defined based on a schema that includes elements relevant to a company's business but is independent of the physical representation of the data it stores. A logical dataset can correspond to a physical dataset.

[0083] A concurrently pending application, “Dataset Multiplexer for Data Processing System,” assigned agent reference number A1041.70066US02, which is incorporated herein by reference in its entirety, describes a data processing system that enables the specification of operations in logical datasets while ensuring that those operations are applied to the appropriate physical datasets. The application describes a dataset catalog that is updated in response to events affecting the storage of data associated with logical datasets. The techniques for dataset selection described herein can be applied in a data processing system as described in the concurrently pending application.

[0084] The actions related to dataset selection can be applied to logical datasets and / or physical datasets. For example, a logical dataset may be selected. Nevertheless, the selection may involve or be based on the corresponding physical dataset. Such results can be achieved by accessing the dataset catalog and identifying the physical dataset corresponding to the logical dataset, so that when searching for the dataset to select, the dataset selection tool can retrieve physical information for the logical dataset and use it in the dataset selection process.

[0085] Figure 1C is a block diagram of an IT system 100 that includes an exemplary data processing system 104 and a dataset multiplexer 105 integrated with the data processing system 104. The IT system 100 may be the IT system of a company, such as a financial company. For simplicity, elements of the company's IT system, such as networks, cloud storage, and user devices, are not explicitly shown.

[0086] The data processing system 104 is configured to access data stores 102-1, 102-3, 102-3, ..., and 102-n (for example, to read data from these data stores and / or to write data to these data stores). Each of the data stores 102-1, 102-3, 102-3, ..., and 102-n can store one or more physical datasets. The data stores can store any appropriate type of data in any appropriate manner. The data stores may store data as flat text files, spreadsheets, etc., using, for example, a database system (e.g., a relational database system). These data stores may be internal or external to the enterprise. For example, an external data store may reside "in the cloud" or in storage hardware managed by a third party. Thus, the data stores can provide a federated environment in which different data stores used by the enterprise may be in different locations and / or managed by different entities, internal or external to the enterprise.

[0087] In some cases, a data store can store transaction data. For example, a data store can store credit card transactions, telephone record data, or bank transaction data. It should be understood that the embodiments of the technology described herein are not limited in this respect, and the data processing system 104 may be configured to access any appropriate number of data stores of any appropriate type. A data store that the data processing system 104 may configure to read data from is sometimes called a data source. A data store that the data processing system 104 may configure to write data to is sometimes called a data sink. However, technologies such as those described herein can be applied to data stores that hold other types of data used by enterprises.

[0088] Each datastore may be implemented with one or more storage devices and may include data management software or other control mechanisms to support the storage of physical datasets in any suitable type of one or more formats. The one or more storage devices may be of any suitable type and may include, for example, one or more servers, one or more disk arrays, a cluster of one or more disk arrays, one or more portable storage devices, one or more non-volatile storage devices, one or more volatile storage devices, and / or any other one or more devices configured to electronically store data. In embodiments in which the datastore includes multiple storage devices, the storage devices may be located in one physical location (e.g., within a building) or distributed across multiple physical locations (e.g., in multiple buildings, different cities, states, or countries). The embodiments of the technology described herein are not limited in this respect, and the storage devices may be configured to communicate with each other using any suitable type of one or more networks.

[0089] Data management software can organize data in physical storage and provide a mechanism for accessing data so that data can be written to or read from physical storage. The data management software may be, for example, a database system or a file management system. Depending on the type of data management software, one or more storage devices may store physical datasets using one or more formats, such as database tables, spreadsheet files, flat text files, and / or files in any other suitable format (e.g., the mainframe's native format). In one embodiment, data stores 102-1, 102-2, 102-3, ..., and 102-n may be of the same type (e.g., all may be relational databases) or of different types (e.g., one may be a relational database, while another may be a data store that stores data in flat files). When multiple data stores are of different types, the storage environment may be referred to as heterogeneous or federated data environment 102. The data store may be, for example, a SQL Server database, an Oracle database, a Teradata database, a flat file, a multi-file data store, a HADOOP distributed database, a DB2 data store, a Microsoft SQL Server data store, an INFORMIX data store, a table, a group of tables or other subparts of a database, and / or any other suitable type of data store, as the embodiments of the technology described herein are not limited in this respect.

[0090] The data processing system 104 supports various applications 106 to perform functions to access (e.g., read access and / or write access) physical datasets stored in data stores 102-1, 102-3, 102-3, ..., and 102-n. The applications 106 can then perform operations based on the data in the data stores. The data processing system 104 can support applications 106-1, 106-2, 162-3, ..., and 106-n, which may be of the same or different types. In some examples, when executed, an application can read or write transaction data to one or more physical datasets in a data store. In other examples, when executed, an application can read or write data to physical datasets stored across multiple different data stores and analyze the data to extract business insights from the datasets.

[0091] Application 106 can be developed as a dataflow graph. A dataflow graph may include components called “nodes” or “vertices” that represent data processing operations performed on data, and links between components that represent the flow of data. Techniques for performing computations encoded by a dataflow graph are described in U.S. Patent No. 5,966,072, entitled “Executing Computations Expressed as Graphs,” which is incorporated herein by reference in its entirety. Environments for developing applications (e.g., computer programs) as dataflow graphs are described in U.S. Patent Application Publication No. 2007 / 0011668, entitled “Managing Parameters for Graph-Based Applications,” which is incorporated herein by reference in its entirety. A dataflow graph may include data sources and data sinks. These are represented by terminal nodes in the flow that indicate access to data stores 102-1, 102-3, 102-3, ..., or 102-n.

[0092] However, the application itself does not need to be programmed on a specific data store included in the application. Rather than being hardcoded to access a single physical dataset, application 106 can be programmed in terms of logical datasets. A logical dataset may refer to a logical representation of one or more datasets. The data processing system 104 may store definitions of multiple logical datasets and other metadata about those logical datasets. This information may be managed by a dataset multiplexer 105. Tools used with the data processing system 104 can access metadata about logical datasets and perform functions based on that metadata. For example, a program development environment may provide a user interface that allows users to select available logical datasets and use them in programming the application.

[0093] A logical dataset may have a schema that defines the data, regardless of the format of the corresponding data in the physical data store. For example, a logical dataset may have a schema that defines the logical entities within the logical dataset. These logical entities may be recognizable and / or understandable to a human user. For example, a logical dataset may contain a logical entity such as a customer name. In the physical dataset corresponding to this logical dataset, the customer name may be stored in a single row of a data table as three fields, each holding data corresponding to the customer's first name, the initial of their middle name, and their last name, respectively. However, a logical dataset may simply contain the logical entity Customer_Name, regardless of the format of the data in the physical storage.

[0094] The data processing system 104 may include an interface (not shown) that can define a schema for a logical dataset. The interface may be a user interface that allows a user to introduce a logical dataset into the system by specifying the logical dataset or the schema for the logical dataset. The data processing system 104 may store a set of logical entities commonly used in a company's business. Examples of commonly used logical entities may include one or more of the following: name, identification number, telephone number, address, nationality, account balance, transaction amount, or date. These business terms may be used to specify, at least partially, the schema for a logical dataset. However, the schema may be defined to include other logical entities in place of, or in addition to, predefined logical entities.

[0095] By enabling application programming from the perspective of logical datasets, programmers creating applications no longer need to understand the format of the datastore that stores the corresponding physical datasets. As a result, data analysts can develop applications using logical datasets without needing to understand the format of the data in the datastore that holds the physical datasets.

[0096] As a more detailed example, a programmer within a company might define a logical dataset to store new customers. The schema of the logical dataset might include logical entities such as customer name, customer address, customer identifier, and customer acquisition date. A data analyst can write applications in terms of the logical dataset and these logical entities, regardless of the storage format of the corresponding physical dataset. As a result, a data analyst can write applications without knowledge of the physical dataset that stores the data accessed by the application.

[0097] When the application is executed, the data of the physical dataset corresponding to the logical dataset may be stored in one or more of the data stores 102-1, 102-3, 102-3, ..., and 102-n. Each operation that specifies access to the logical dataset in order to execute the application may be performed by a data processing system 104 that reads or writes data from the corresponding physical dataset stored in one of the data stores 102-1, 102-3, 102-3, ..., and 102-n. The dataset multiplexer 105 can enable the automatic execution of such operations by automatically accessing the corresponding physical dataset and can enable conversion between the format of the data stored in the physical data store and the format specified in the schema of the logical dataset.

[0098] As shown in Figure 1C, the data processing system 104 includes a dataset multiplexer 105 for automating access to corresponding physical datasets and conversions between the formats of logical datasets and physical datasets. The dataset multiplexer 105 can maintain a dataset catalog 107, where each entry in the catalog corresponds to a logical dataset and provides information for accessing one or more physical datasets. For example, a catalog entry may identify a dataset in datastore 102-1, 102-3, 102-3, ..., or 102-n that corresponds to a logical dataset. The catalog entry may, or may not, include information for converting data stored in a physical dataset to the format of a logical dataset. This information may be an executable program, or may include an executable program. For example, the catalog information may identify a program for converting data from multiple fields in a physical dataset to the format of the corresponding logical entity in a logical dataset. Other information may, or may not be, stored as catalog information for accessing one or more physical datasets, or may be reflected in such catalog information.

[0099] The dataset multiplexer 105 enables application 106 to seamlessly access physical datasets based on programmed logical datasets using information in the dataset catalog. As soon as an operation to access a logical dataset (e.g., read and / or write) is performed in the application (e.g., application 106-3), the dataset multiplexer 105 of the data processing system 104 can enable access to the corresponding physical dataset in the data store (e.g., data store 102-1). For example, if the catalog information stored with respect to the logical dataset is an access control program, or includes an access control program, that program may be executed. As a result, even though application 106-3 is programmed in terms of logical datasets, when a data access operation is performed, the physical dataset stored in data store 102-1 is accessed.

[0100] The dataset multiplexer 105 may access a catalog of datasets and select entries associated with logical datasets referenced in application 106-3. Information may then be used for data access to identify the corresponding physical datasets stored in data store 102-1 and / or to convert data in the format of data store 102-1 to the format of the logical dataset.

[0101] This access may be dynamic. Catalog information may be used when performing operations of an application that requires data access. Entries in the dataset catalog associated with a logical dataset may be updated in response to events indicating changes to the storage of information associated with the logical dataset. Access to the physical data store via catalog information can ensure that the application continues to run despite any changes that may occur at any point in time throughout the entire IT system 100, even if the data analyst who wrote application 106-3 or other users are unaware of those changes.

[0102] For example, a physical dataset may be migrated from datastore 102-1 to datastore 102-n. The logical dataset in which the application is programmed does not need to be modified to account for this change. By updating the catalog entry for the logical dataset, the dataset multiplexer 105 can automatically utilize the updated catalog information to provide application 106-3 with access to the correct physical dataset, regardless of the datastore in which it resides.

[0103] Regardless of how a particular data store is accessed as part of an operation related to accessing a dataset, a user can provide input specifying which dataset is the target of a particular operation. In enterprise data processing systems with many datasets, one or more search interfaces may be provided to enable the selection of the appropriate dataset. A dataset selection tool may provide a user interface that provides interface elements configured to receive input specifying dataset search and selection commands, for example.

[0104] Information that enables the retrieval of datasets and operations within dataset groups can be stored within the IT system 100. In this example, this information can be stored within the dataset multiplexer 105, which may include one or more metadata repositories. The metadata repositories may store information about logical and / or physical datasets having different types of metadata that provide facets for performing dataset retrieval. This metadata can be collected using manual or automated techniques, including techniques known in the art.

[0105] In addition, one or more repositories can store information about a dataset group. For example, a dataset group repository 120 that holds such information is shown in Figure 1C. This information can be stored in a non-volatile computer-readable medium in a manner that associates multiple types of information. For example, related information can be stored in the same data structure or associated through links.

[0106] This information can be shared among multiple users of the data processing system. As a result, different users can create, modify, and / or access information about dataset groups. The information can be scoping to ensure that information about each dataset group is only leaked to users who have personas within the scope of the dataset group. Alternatively, or in addition to this, the repository storing information about dataset groups can implement access restrictions to limit which users can create, modify, and / or access some or all of the dataset groups.

[0107] Restrictions on access to information within a repository may be in parallel with scope restrictions on access to dataset groups. Access may be granted to a user to create or modify dataset groups with a scope specific to that user. Alternatively, or in addition to this, access may be granted to users in groups with specific roles and / or other characteristics as part of their persona within the scope of a dataset group. However, in some embodiments, the privilege to create and modify dataset groups may be set separately from the scope for use of those dataset groups. Different access controls for the management and use of dataset groups may make it possible to capture the expertise of certain workers within an enterprise and automatically disseminate that expertise through the data processing system. For example, a user with expertise in appropriate datasets for use in a particular operation may be granted access privileges to create or modify dataset groups scoping for use by explicitly listed users, users with specific roles, or users in groups within the enterprise performing those operations. When other users perform those operations by selecting datasets from dataset groups within which their persona is within scope, the system may automatically limit their dataset choices to those previously specified by the user with expertise in the data.

[0108] Regardless of how access is performed, the data processing system 104 may provide a user interface for creating or modifying dataset groups, performing searches to return dataset groups, and / or selecting datasets from dataset groups. Examples of such user interfaces are provided in the following sections.

[0109] A typical user interface for selecting logical datasets based on groups. Dataset groups may be available for use when selecting one or more datasets to perform data access-related actions. For example, a search interface may be presented in relation to the selection of datasets to use when performing an action, and dataset groups may be included in the search results.

[0110] As an example, an application for execution by a data processing system can be configured to access a specific dataset based on user input. A dataset cart can be used to simplify this selection process. In embodiments where the application is configured as a dataflow graph, the dataset component of the dataflow graph can be configured as a data source for performing read operations. The configuration may involve searching for datasets and selecting the appropriate dataset. The search can be simplified by including a dataset cart in the search results. For example, datasets that match a search query within a dataset cart are not presented separately as search results. Rather, the search results may be limited by presenting the dataset cart.

[0111] Figure 2A shows GUI 800 in a programming environment that can use a dataset cart to assist users in selecting datasets to configure an application. In this example, a user (such as user 112a or 113a in Figure 1A) can specify components of an executable dataflow graph and the interconnections between those components through GUI 800. These components may represent one or more input sources, one or more output sources, and one or more operations performed on the data from the inputs to generate the outputs. Components representing inputs and / or output sources may be configurable by the user. Configuration may involve specifying datasets to be used for inputs or outputs. Configuring these components may involve user input, first selecting a dataset cart, and then selecting datasets within the selected dataset cart.

[0112] Figure 2A shows a simple graph, and for the sake of illustration simplicity, some of the information that may be displayed and interface elements associated with the display components have been omitted. In this example, the user specifies component 804 to process the input dataset. Component 804 may represent, for example, the action of applying data quality rules to the selected input dataset.

[0113] Component 802 represents a data source containing input datasets. Component 802 has interface elements that the user can access to configure the component, such as by first selecting a dataset cart and then selecting datasets within that cart to be used as input data sources. Component 806 represents an output component, which the user can configure to specify an output dataset that can be created to hold the data created in the operation represented by component 804, for example.

[0114] As shown in Figure 2A, component 802 includes user interface elements that allow the user to interact with a selection tool for selecting a dataset. These interface elements may include a field 812, which is shown here as a drop-down menu box. In the state shown in Figure 2A, the user has selected a value in field 812 that indicates the user wishes to select a dataset from the dataset catalog. Link 810 is another user interface element through which the user can enter a command to proceed to the next step in the selection process, which is to select a dataset from the available dataset catalog options.

[0115] In response to the user selection in link 810, the data processing system can generate and present the GUI 890 shown in Figure 2B to the user. Figure 2B shows the interface of the selection tool for selecting a dataset, which in this example is invoked as part of the process of selecting a dataset to constitute component 802 of the data flow graph in Figure 2A. Within GUI 890, available catalog datasets matching the user's selection of their source type are presented, as described above in relation to Figure 2A.

[0116] GUI 890 presents a dataset cart in section 855 containing datasets available for selection. If datasets not present in the dataset cart are available for selection, those datasets may also appear in List 895. List 895 of GUI 890 specifically includes dataset carts created through GUI 400 in Figure 8A (e.g., "Best Cart Ever").

[0117] In this example, search results are presented to preserve the hierarchy of datasets. The icons presented next to elements in List 895 indicate whether an element is a dataset cart or a dataset. For example, an element depicted with a "folder" icon 897 next to it could be a dataset cart, while an element with a different icon 898, shown here as a file icon, could be a dataset. Graphical user interface elements for navigation are provided to allow the user to traverse the hierarchy, for example, by showing or hiding the contents of a group of datasets represented by a "folder" icon. In the example in Figure 2B, GUI 890 includes a graphical user interface element 896 for navigation. By selecting element 896, GUI 890 toggles whether to show or hide datasets contained in a dataset cart (e.g., logical datasets). In this way, the user can identify and select icons at the appropriate hierarchical level.

[0118] Although Figure 2B shows only two hierarchical levels, in some scenarios a group may contain further groups, and if a dataset cart containing further dataset carts is expanded, the user may be presented with an interface having internal groups associated with user interface elements, and the user may also be given the option to expand within the internal groups. In this way, multiple hierarchical levels may be exposed. Regardless of the number of hierarchical levels presented to the user, the user can navigate across the hierarchical levels to reveal the datasets available for selection and then select the desired dataset.

[0119] In addition, users can provide input to obtain additional information about a dataset or group of datasets displayed through the interface. For example, the GUI 900 in Figure 3 depicts a state in which a user is manipulating user interface elements to control the dataset selection tool so that a specific dataset cart containing the “Loyalty Data” dataset cart 920 is expanded and the set of logical datasets contained in that dataset cart is revealed. The GUI 900 allows the user to obtain additional information about a specific logical dataset by selecting a logical dataset 930 in the GUI 900. For example, in response to a user request to view additional information about a logical dataset, a popup GUI 910 may be presented.

[0120] GUI 910 provides additional user interface elements that users can interact with to obtain additional information about datasets. Selecting the "Information" tab in GUI 910 presents basic information about the logical dataset, such as the datastore associated with the logical dataset, the type of datastore or storage, the path to the datastore and / or the physical dataset within the datastore, a link to the corresponding entry in the dataset catalog, and / or other information. Selecting the "Browse" tab in GUI 910 presents physical data associated with the logical dataset, such as the data in the corresponding physical dataset. Selecting the "Record Format" tab in GUI 910 presents recording format information about the dataset (e.g., recording format information for the logical dataset and / or the logical entities of the logical dataset). Selecting the "Profile" tab in GUI 910 generates profile information, such as the relationship between the logical dataset and / or other dataset carts defined in the system. Users can browse any or all of this information to evaluate whether the dataset is suitable for their desired use.

[0121] Other mechanisms (such as a search interface) can be used to limit the number of dataset carts and / or datasets presented to the user as candidates for selection. Returning to Figure 2B, GUI 890 allows the user to enter a search query. GUI 890 may include a graphical user element 892 for the user to enter the search query. In this example, the search query is specified as text. The user can specify words entered into the repository to describe a dataset, the names of fields contained within the dataset, and / or other metadata stored for the dataset. For example, Figure 4A depicts the search results for the search query "Roy". The data processing system can perform a search based on the query and generate search results that include a list of dataset carts and / or logical datasets selected by the data processing system based on the query. In this example, the search query matches the titles of datasets in two dataset carts, and the list of datasets available for selection through GUI 1000 is limited to dataset carts containing these two matching datasets.

[0122] Regardless of how List 895 (Figure 2B) is specified, the selection tool may present a user interface that allows the user to make a selection from the list. In this example, the user interface elements for selection are separate from the user interface elements for navigation. Such a configuration allows for a contextually appropriate hierarchy level of dataset groups in selecting entries in List 895. In scenarios where the action of selection is performed on a single dataset, the user interface elements for selection may only be operational when the user indicates a selection of individual datasets. In scenarios where a selection of a group of datasets is appropriate, the user interface elements for selection may be operational when the user indicates a dataset cart. When a group or a single dataset is appropriate for the action, the user interface elements for selection may be operational when a group or a single dataset element is indicated. In the example in Figure 2A, where the user is selecting a single dataset to make up a component of a graph, the selection tool may limit the selection to a hierarchy level that indicates individual datasets.

[0123] As shown in Figure 2B, the “Loyalty” dataset is specified as the target of selection. This can be achieved by selecting GUI element 898 followed by GUI element 845, thereby presenting the “Loyalty” dataset in section 899 of GUI 890. The selection of GUI element 870 causes the dataset identified in section 899 to be returned by the selection tool as a user selection for use when performing data access operations. The user can, for example, specify a dataset that appears in the list in section 899. From there, the user can retrieve information about that dataset as described above and make a final decision on whether to select the specified dataset. Other user interface elements, including an interface element labeled “Clear” that removes the dataset specified in section 899, or an interface element “Cancel” that terminates the selection process without making a selection, allow the user to modify the specified dataset before the selection tool returns its selection.

[0124] In this example, the search interface is significantly simpler than the search interface in Figure 10A, presenting fewer fields for specifying search criteria. Even with the simpler search interface, the results are limited to those within the dataset cart and / or other contexts of the search that encompass the user, so the results may be just as relevant or more relevant than what the user can find through the interface in Figure 10A.

[0125] The value of simplifying the selection process can be seen in relation to Figure 2C, which shows additional information and user interface elements that may exist even for the simple example in Figure 2A. Figure 2C shows GUI 875 in a programming environment where dataset selection can be performed. In this example, a user (such as user 112a or 113a in Figure 1A) can specify components of an executable dataflow graph and the interconnections between those components through GUI 875. For example, a user can specify a component to perform validation or to apply data quality rules to data. The dataflow graph may include a component 882 that indicates the dataset to be used. A component can be configured to identify which dataset to use for the data access operation associated with that component.

[0126] Figure 2C shows a scenario in which operation 884 involves the execution of data quality rules on the selected data source. Component 886 of the data flow graph may represent the output of the validation operation.

[0127] As shown in Figure 2C, a user interface element associated with component 882 allows the user to select a dataset (such as loyalty.dat) whose content is to be validated. These interface elements may include a field 888, which indicates that the user has selected a value that limits the data source to be selected to those registered in the dataset catalog 107 (Figure 1C). Link 889 is another user interface element that the user can call to enter further search criteria.

[0128] Selecting link 889 can trigger a selection tool to present a user interface, such as GUI 890 described above in relation to Figure 2B, through which the user can select a dataset. In this example, the "Loyalty" dataset is depicted as the selected dataset in component 882 of Figure 2C. Despite the increased complexity of the interface in Figure 2C compared to that in Figure 2A, this dataset is selected through a simple process of selecting it via the dataset selection tool.

[0129] A similar simple process can be used to specify multiple datasets to perform the same operation. For example, a graph like the one shown in Figure 2A, which applies validation rules, can be configured to apply those validation rules to multiple datasets. Figure 2D shows a GUI 800 in operation, where component 802 is configured to represent multiple datasets. In this example, the configuration is achieved by user input in field 812', which indicates the selection of a catalog dataset cart as the source type.

[0130] Regardless of the source type used to construct a component representing data input or output, the data selection tool can receive user input to select a dataset or a group of datasets. In scenarios where a dataset is selected in a context where the operation may be performed on multiple datasets, the data selection tool can be made to select the entire dataset cart. The selection of the dataset cart can be performed as described above in relation to Figure 2B, but user interface element 845 can operate when the dataset group is shown in Listing 855. Figure 2E provides an exemplary user interface for selecting a group of datasets.

[0131] The selection of a dataset group as the target of an action can serve as a command to the data processing system to perform the action on each dataset in the selected dataset cart. For example, the action may include executing data quality rules on each dataset in the dataset cart or other types of processing on the content of each dataset.

[0132] In the example in Figure 2E, GUI 811 lists the dataset carts available for selection in section 850. List 815 of GUI 811 includes, in particular, the dataset cart created through GUI 400 in Figure 8A (e.g., "Best Cart Ever"). The user can make a selection from the list. As shown in Figure 2E, the "Best Cart Ever" dataset is specified as the target of selection. This can be achieved by making a selection in GUI element 840 following a selection in GUI element 820, thereby presenting the "Best Cart Ever" dataset cart in section 860 of GUI 811. A selection in GUI element 861 selects the "Best Cart Ever" dataset cart for use when performing a data access operation.

[0133] Therefore, the selection tools described in these examples provide information and user interface elements that enable users to efficiently make selections from a multitude of options.

[0134] The selection interface may include other user interface elements for identifying datasets or groups of datasets for selection. For example, the user interface may accept other search criteria as input so that the user can identify datasets relevant to an action involving accessing one or more datasets or dataset carts. The options presented to the user may be limited to those that match the specified search criteria, whether they are datasets or dataset carts. In the case of dataset carts, the options presented may be limited to those containing datasets that match the search criteria and / or carts that match the specified criteria. Figure 4A illustrates an exemplary graphical user interface 1000, rendered by a data processing system, for enabling a user to select datasets, where the user navigates through a directory of datasets as a first mechanism for limiting the search, and then enters text such as "Roy" appearing in the dataset description as a search query as a second limitation in the search. The user can then select dataset carts and / or datasets from the filtered search results to use when performing an action.

[0135] In this example, the search interface is significantly simpler than the search interface in Figure 10A, even though it offers additional flexibility in specifying what to search for, and presents fewer fields for specifying search criteria. Even with the simpler search interface, the results are limited to those within the dataset cart and / or other contexts of the search that encompass the user, so the results may be just as relevant or more relevant than what the user can find through the interface in Figure 10A.

[0136] Figure 4B illustrates the operational state after a search query has been executed in the exemplary graphical user interface of Figure 4A, as rendered by the data processing system, allowing the user to select a dataset. A list of datasets matching the search query (e.g., search query "Roy") is presented, allowing the user to select one or more datasets as the target of the operation. Search results may be limited to datasets based on the cart scope and the user performing the search.

[0137] Figure 5 illustrates the operational state after a search query has been executed on an exemplary graphical user interface 1100, rendered by a data processing system to allow the user to select a dataset, with the list of datasets limited to those containing a field for storing emails. Search results may be limited to datasets based on the cart scope and the user performing the search. For example, search results may be limited to datasets within the cart for which the user performing the search is within the cart scope.

[0138] To create dataset carts, perform searches, and / or use or select datasets / carts as targets for actions, data processing systems can use various forms of user input to determine user identity. For example, user identity can be determined using user input such as text input using a keyboard, stylus, or other writing instrument (e.g., user identifier and / or password), voice input using a microphone or other device, biometric input (e.g., fingerprints, facial patterns, voice patterns, etc.), and / or other forms of input. Identity information can be used to represent the user's persona.

[0139] Typical user interfaces for grouping logical datasets A data processing system may provide one or more mechanisms that allow users to manage groups of datasets, such as by creating, modifying, or deleting groups. These mechanisms may be dedicated tools contained within the data processing system, or they may be provided through tools or additional user interface options associated with other interfaces that allow users to access dataset information present in the data processing system. For example, an interface that allows users to search for datasets that meet specified criteria may include user interface elements that allow users to provide input to associate datasets included in the search results with dataset groups. Similarly, other interfaces, such as where lineage information is presented, may be extended with user interface elements that allow users to manage dataset groups. These user interface elements may link to computer executable code that accesses and / or modifies stored information about dataset groups.

[0140] Figure 6 shows a graphical user interface (GUI) 200 generated in response to a request to view information about a dataset and / or a group of datasets, which in this example is depicted as a dataset cart. For example, this interface may be the result of a user providing input that acts as a dataset search query and then selecting a specific dataset from the results. GUI 200 presents information about dataset 202. As shown in Figure 6, information about the dataset "loyalty.dat" is presented. Information about dataset 202 may include the type of dataset (e.g., file, directory, table, etc.), the directory to which the dataset belongs, information about the dataset hierarchy to which the dataset belongs, and / or other information. For example, GUI 200 depicts that dataset 202 is a file, belongs to the directory "main", and belongs to at least three dataset hierarchies (e.g., "loyalty program", "retail", and "main"). Hierarchies can be defined or specified by the user of the data processing system 104.

[0141] The interface may also include interface elements to allow for the management of dataset groups. In this example, GUI 200 also includes a list of dataset carts 204 that contain dataset 202. For example, user interface 200 depicts that the dataset carts "Loyalty Data" and "Management Data" contain dataset 202. A request to view information about dataset carts may generate another GUI. For example, selecting a graphical user element 206 representing the "Loyalty Data" dataset cart may generate GUI 300.

[0142] Figure 7 shows an exemplary GUI 300 generated in response to a request to view and / or modify information about a dataset cart 302. However, it should be understood that the data processing system may provide alternative or additional mechanisms that allow the user to invoke an interface for managing dataset carts as shown. In this example, GUI 300 presents information 340 about the “Loyalty Data” dataset cart. The information about dataset cart 302 includes the name of the dataset cart, information describing the dataset cart, the owner of the dataset cart (e.g., the user who created the dataset cart), users who are permitted to modify the dataset cart (e.g., permission to edit or delete the dataset cart), the contents of the dataset cart (e.g., information about the datasets contained in the dataset cart), other dataset carts associated with the dataset cart, logical datasets or logical entities (e.g., those related to the dataset cart), and / or other information. Information about users who are permitted to view dataset carts can be entered by the user by selecting user interface element 304, whether related to viewing information in the repository 120 or related to displaying dataset carts in the results of searches performed by that user. For example, GUI 300 depicts that the dataset cart 302 contains the logical dataset "loyalty.dat" 202 and information 206 about the physical dataset corresponding to that logical dataset. As shown in GUI 300, the dataset cart 302 may also contain information about the physical datasets corresponding to other logical datasets included in the dataset cart. For example, the dataset cart 302 contains logical datasets 310, 312 and information 314, 316 about the physical datasets corresponding to these logical datasets.

[0143] GUI 300 includes interface elements configured to receive input for modifying the dataset cart. Interface element 330 may, for example, present additional screens, when selected by the user, allowing the user to specify which users can read, edit, delete, etc., the dataset cart, as a list of individuals, by role, by group membership, or by other characteristics of the user persona. A current owner can be assigned to the dataset cart. The current owner may have full access to all aspects of the dataset cart. The current owner may initially be the user who created the dataset cart. Subsequently, the current owner of the dataset cart may transfer ownership to another user by selecting graphical user element 355 and indicating the user or role to whom ownership should be transferred.

[0144] In some embodiments, the scope of a dataset cart is tailored to users who have permission to read and / or edit the dataset cart. In other embodiments, a separate scope can be specified for the users who may appear in the results of searches performed on the dataset. A separate mechanism can be provided in an interface, such as GUI 300, to set the scope of a dataset cart. For example, when selected by a user who has permission to edit the dataset, user interface element 304 may render a separate display screen where the user can input a scope, such as identifying a specific user, group, role, etc.

[0145] In addition to or as an alternative, other parameters can be used to define the scope of a dataset cart. For example, time parameters (e.g., time of day, day of the week, month) can be used to define the scope. In such scenarios, the data processing system can implement the time parameter for scope by limiting the selection of datasets and / or dataset carts to be presented to a user searching for datasets to only those datasets or dataset carts that are authorized for use at the time the search is initiated.

[0146] The dataset cart 302 can be updated via the GUI 300. For example, by selecting a graphical user element 320, a user with editing permission can add a dataset to or remove a dataset from the dataset cart 302.

[0147] In some examples, a user (such as user 111a in Figure 1A) can request to view and / or modify information about a dataset or dataset cart via interfaces 200, 300 in order to define or create a dataset cart.

[0148] Figure 8A shows GUI 400 in a state where a new dataset cart can be created. For example, while browsing a user interface displaying information about a dataset, a user (such as user 111a in Figure 1A) can specify that they want to create a dataset cart. In this example, the user is browsing information about the "loyalty.dat" dataset and may then wish to create a new dataset cart containing the "loyalty.dat" dataset. The user can create a new dataset cart by selecting the graphical user element 402. Upon selection of the graphical user element 402, the system may generate a pop-up dialog box 404, in which the user can name the cart (e.g., "Best Ever Cart"), indicate the type of entity being created (e.g., Dataset Cart), and provide a description of the dataset cart.

[0149] By selecting a graphical user element 406, the system may generate a new dataset cart containing the "loyalty.dat" dataset. The system can store a representation of the newly created dataset cart. For example, an entry can be added to repository 120 (Figure 1C) to represent the dataset cart. In some examples, some or all of the dataset cart's properties may initially be assigned default values. For example, a dataset cart may initially be assigned a scope based on the persona of the user who created it. This can be achieved, for example, by setting the scope so that initially only its creator can view the dataset cart. Regardless of how the initial values ​​of the properties are initially assigned, one or more users can subsequently change them. Once the dataset cart record is created, it can be edited, for example, through a user interface as shown in Figure 6 or 7.

[0150] Alternatively, a dataset cart can be updated in other ways once it is created. For example, rather than creating a new dataset cart to hold a dataset, a user may wish to add a dataset to an existing dataset cart. Figure 8B shows a portion 450 of a GUI 400 in which a user (such as user 111a in Figure 1A) can choose to add the "loyalty.dat" dataset to an existing dataset cart. For example, a dropdown menu 455 is a user interface element that, when selected by the user, presents a list of existing dataset carts defined in the data processing system. In embodiments where the dataset cart has scope, the list may be limited to dataset carts that have scope including the user at that time. A dataset can be added to a selected dataset cart by selecting a specific dataset cart from the list. The system can update the stored representation of the selected dataset cart accordingly.

[0151] Figure 12 shows an exemplary data structure that holds a stored representation of a dataset cart (i.e., stores information about the dataset cart). For each dataset cart, various pieces of information can be stored. Repository 120 (Figure 1C) may have such a data structure for each dataset cart, for example. As shown in Figure 12, the data structure 1202 for a dataset cart may include several fields, including information such as the dataset cart's name field 1222, an identifier 1224 for the list of datasets contained in the dataset cart, and one or more parameters 1226 associated with the dataset cart. Here, parameter 1226 indicates other information that can be stored, such as text describing the dataset cart, the values ​​of one or more tags used in relation to the dataset cart or in other ways as described herein, or other types of information. In embodiments where the grouping of datasets may be hierarchical, list 1224 may include further dataset groups in place of or in addition to other datasets.

[0152] In addition, access information 1240 can be stored along with information about the dataset cart. This access information may indicate users who have the privilege to access the stored information about the dataset cart. This information may include the dataset cart owner 1228, a list 1230 of users who have permission to read the information about the dataset cart, or a list 1232 of users who have permission to modify the information about the dataset cart. Some or all of this authorization information can be processed by other components of the data processing system to establish the scope of the dataset cart. Alternatively or in addition to this, other information may be included to establish the scope. For example, list 1234 may define groups within the scope of the dataset cart. List 1236 may define the roles of users who have permission to access the dataset cart.

[0153] A data processing system may provide multiple user interfaces that represent datasets and / or dataset groups. Each of these interfaces may be configured to allow users to manage dataset groups, such as by creating new dataset groups or adding datasets to existing dataset groups. User operations on these interfaces may modify the set of dataset groups available in the data processing system, which can be done, such as by adding, deleting, or modifying data structures, such as 1202.

[0154] Figure 9 shows GUI 500, which allows a user (such as user 111a in Figure 1A) to specify datasets to include in a dataset cart. GUI 500 displays lineage information associated with the datasets. The data processing system may present such information for any of several reasons that are not necessarily related to the management of dataset groups. For example, displaying technical lineage allows a user to explore possible sources of errors identified in the dataset's data. Displaying business lineage allows a user to identify groups within an enterprise that may be affected by changes to the dataset. Regardless of the reason for displaying lineage information, a user viewing such information may recognize the need to manage one or more dataset groups so that efficient operation is facilitated by integrating user interface elements that enable dataset group management with the lineage user interface.

[0155] For example, GUI 500 is shown to display lineage information 502 for the "loyalty.dat" dataset. The user can select and manipulate one or more components representing datasets in the displayed lineage information, and specify the datasets represented by those components to be included in the dataset cart. In this example, selecting component 510 displays window 512, through which the user can select graphical user interface element 514. When invoked, graphical user interface element 514 adds the dataset "filtered loyalty" to an existing dataset cart (as shown in Figure 8B) or to a newly created dataset cart (as shown in Figure 8A).

[0156] Datasets to be included in the dataset cart can be selected by the user (e.g., user 111a in Figure 1A) via a search GUI (e.g., GUI 600 shown in Figures 10A-10C). The data processing system may include a dataset search interface that includes a rich combination of search criteria. This user interface may be presented in response to a request to create a new dataset cart, or, after identifying one or more datasets through such a search, the user may specify which particular datasets returned in the search results should be used when managing the dataset cart.

[0157] Through the search interface, the system can identify datasets available for use when performing data access-related operations in the data processing system 104. In some implementations, the search GUI 600 may include graphical user interface elements 602, 604, 606, and 608 for the user to enter a search query. User interface element 602 may be, for example, a text field in which the search results are limited to datasets having names, fields, and / or other relevant metadata containing the entered text.

[0158] Users can provide additional input through other user interface elements to define faceted queries. In such queries, users can specify one or more values ​​for facets that describe a dataset defined in the data processing system. User interface elements can be provided for each facet to allow users to indicate values ​​stored in metadata associated with the dataset defined in the data processing system. The range of values ​​may be limited to values ​​for datasets that meet criteria already specified in the search interface. User interface elements 604, 606, and 608 are examples of user interface elements that allow users to specify facet values. For example, one or more facets may correspond to dataset properties (such as type, owner, hierarchy, whether the dataset is registered in a catalog that associates information for accessing the physical dataset with the logical dataset) and / or other properties.

[0159] Alternatively, or in addition to that, other information can be entered through such a user interface to define the search query.

[0160] The data processing system can perform a search based on a query and generate search results that include a list 610 of datasets selected by the data processing system based on the query. A faceted query may include one or more facets, and the search results can be filtered based on those facets. In the example shown, the list 610 of datasets presented in GUI 600 includes all datasets that contain "loyalty" in their name, field name, or dataset description. Additional facets are shown as being specified to further filter the search results. By selecting facets, the search results can be filtered according to the facets.

[0161] For example, if facet 606 is selected, which indicates whether a dataset is registered in a catalog that associates information for accessing a physical dataset with a logical dataset, the search results are filtered so that only datasets registered in the catalog are presented to the user in the GUI, as shown in the example in Figure 10B. As shown in Figure 10B, GUI 600 presents an updated list 615 of datasets that does not include some items (such as items 620 and 625 from list 610).

[0162] Next, the user can select one or more of the presented datasets to include in the dataset cart. The dataset cart can be created based on the selected datasets. For example, as shown in Figure 10C, the user can select the dataset "loyalty.dat" from the list of datasets 615 to include in the dataset cart. In this example, the input indicating inclusion in the dataset cart can be done in multiple steps. The dataset name in list 615 may form, for example, a user interface element 630. Selecting user interface element 630 may open a window 632 containing information about the dataset associated with element 630. Window 632 may contain a further "Add to Cart" user interface element, and selecting it may open a window 634 containing further user interface elements. The user interface elements in window 634 allow the user to specify an existing dataset cart to which the selected dataset will be added (similar to the selection described in relation to Figure 8B) or to create a new dataset cart (similar to the process described in relation to Figure 8A).

[0163] If the dataset is a logical dataset, the data processing system can identify the physical dataset that corresponds to the logical dataset and include information about the physical dataset in the dataset cart.

[0164] The created dataset cart may be available for use in a program. In some examples, the program may be an application run by a data processing system. In other examples, the program may be a utility for a data processing system (such as a data analysis utility configured to perform data quality analysis).

[0165] Figure 11 illustrates an exemplary graphical user interface 700, rendered by a data processing system, which allows a user (such as user 111a in Figure 1A) to view or modify information related to a group of datasets identified here as a technology group. In a system where dataset grouping is hierarchical, the top-level grouping can be identified by a different name than that used for lower-level groupings. The top-level grouping may be called, for example, a dataset cart. Lower-level groupings may have different names, such as technology groups. For example, Figure 11 shows that the “Rewards” directory 702 is a member of technology group 704, which is named “Consumption Trends,” “tpc Customers,” “tpc_date_dim,” and “Web Sales.” Some or all of the operations described herein for managing dataset carts can be performed to manage technology groupings. This dichotomy means that technology groups can be included in dataset carts, but not the other way around. However, hierarchical systems do not need to have such restrictions.

[0166] Typical operation methods of data processing systems that support groups of logical datasets Figure 13 is a flowchart of an exemplary process 1300 for operating a data processing system capable of working with a large number of datasets. Process 1300 can be performed by the data processing system 104 described with reference to Figure 1C. Alternatively or in addition thereto, process 1300 may include other actions, including actions described elsewhere in this specification in relation to other embodiments.

[0167] In action 1302, process 1300 can identify datasets available for use when performing data access-related operations in the data processing system 104. For example, datasets can be identified by performing a search based on a search query specified via the GUI 600, as shown in Figure 10A.

[0168] Process 1300 can proceed to action 1304, during which time the identified dataset can be presented to a user interface (such as GUI 600 in Figure 10B). For example, Figure 10B depicts some of the search results generated in response to the execution of a search query containing the keyword "loyalty," and facets indicating whether the dataset is registered in a catalog that associates the logical dataset with information for accessing the physical dataset.

[0169] Process 1300 can proceed to action 1306, during which it may receive a selection of one or more datasets from the identified datasets. The user can select one or more identified datasets to include in a group (such as a dataset cart). For example, as shown in Figure 10C, the user can select the dataset "loyalty.dat" from the identified datasets to include in a dataset cart. The dataset can be selected to include in a new dataset cart or an existing dataset cart.

[0170] Process 1300 can proceed to action 1308, during which it can generate and store a representation of a group containing one or more selected datasets. Such a representation is depicted in Figure 12 and includes various pieces of information, such as the group name, information about the datasets contained in the group, parameters associated with the datasets in the group, the group owner and / or scope information associated with the group.

[0171] Process 1300 can proceed to action 1310, during which a decision can be made regarding whether to perform further identification of the dataset. For example, the user can specify additional or different facets for a search query. In response, for example, in action 1302, a different set of datasets can be identified. A dataset can be selected from a different set of datasets, resulting in a new representation of the group or an update of an existing representation of the group.

[0172] Figure 14 is a flowchart of an exemplary process 1400 for operating a data processing system configured to perform actions that access a dataset. Process 1400 can be performed by the data processing system 104, which is described with reference to Figure 1C. Alternatively or in addition thereto, process 1400 may include other actions, including actions described elsewhere in this specification in relation to other embodiments.

[0173] In action 1402, process 1400 may present a user interface configured to allow the user to select one or more datasets or dataset carts for use in conjunction with data access-related operations in the data processing system. Examples of such user interfaces are shown in Figures 2B and 2E.

[0174] Process 1400 can proceed to action 1404, during which it can identify personas associated with users of the data processing system (e.g., users requesting a search for a dataset) and identify scope information associated with datasets and / or groups of datasets (e.g., a dataset cart). The scope information associated with datasets and / or groups of datasets can be defined based on the personas and / or other parameters of the user of the data processing system.

[0175] The process can proceed to action 1406, during which one or more groups of datasets can be automatically identified, at least partially based on the correspondence between the user persona and the automatically identified groups of datasets and associated scope information. For example, Figures 2B and 2E depict lists 815, 895 of datasets and / or dataset carts that can be generated by checking the personal characteristics (e.g., permissions) of a user requesting a dataset search, and the resulting set may be limited to dataset carts and / or datasets that have a scope encompassing that user's personal characteristics.

[0176] The process can proceed to action 1408, during which time an indication of automatically identified groups of datasets can be rendered via the user interface. For example, if the user selects a specific dataset cart in Figure 2E, an indication of the selected dataset cart may be rendered in the second part 860 of the user interface.

[0177] Figure 15 is a flowchart of an exemplary process 1500 for operating a data processing system configured to run a program for accessing a dataset. Process 1500 can be performed by the data processing system 104, which is described with reference to Figure 1C. Alternatively or in addition thereto, process 1500 may include other actions, including actions described elsewhere in this specification in relation to other embodiments.

[0178] In action 1502, process 1500 can receive a search query for a dataset via a user interface for use in conjunction with data access-related operations in the data processing system. An example of such a user interface is shown in Figure 4A.

[0179] Process 1500 can proceed to action 1504, during which it can perform a search based on the search query to generate search results. The search results can be presented to a user interface and may include one or more dataset carts. Each of at least some of the dataset carts may contain one or more of the searched datasets. The datasets and / or dataset carts presented to the user interface can be identified by checking the personal characteristics (e.g., permissions) of the user requesting the dataset search, and the result set may be limited to only the dataset carts and / or datasets that have a scope encompassing the personal characteristics of that user.

[0180] Process 1500 can proceed to action 1506, during which time it can perform actions on each dataset contained in the dataset cart as soon as the dataset cart is selected in the user interface. The user interface can provide an option to select the dataset cart as the target of the action.

[0181] Further details on implementation forms Figure 16 shows an example of a suitable computing system environment 1600 in which the technologies described herein may be implemented. Computing system environment 1600 is merely an example of a suitable computing environment and is not intended to imply any limitations on the scope or functionality of the technologies described herein. Computing environment 900 should not be construed as having any dependencies or requirements related to any one or combination of the components exemplified in the exemplary operating environment 900.

[0182] The technologies described herein can be used in conjunction with a number of other general-purpose or dedicated computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the technologies described herein include, but are not limited to, personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics products, network PCs, minicomputers, mainframe computers, and distributed computing environments encompassing any of the above systems or devices.

[0183] A computing environment can execute computer executable instructions, such as program modules. Generally, program modules include routines, programs, objects, components, and data structures that perform specific tasks or implement specific abstract data types. The techniques described herein may be executed in a distributed computing environment in which tasks are performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may reside in both local and remote computer storage media, including memory storage devices.

[0184] Referring to Figure 16, an exemplary system implementing the techniques described herein encompasses a general-purpose computing device in the form of a computer 1610. The components of computer 1610 may include, but are not limited to, a processing unit 1620, system memory 1630, and a system bus 1621 that connects various system components, including system memory, to the processing unit 1620. The system bus 1621 may be any of several types of bus structures that include a memory bus or memory controller, peripheral bus, and local bus, using any of various bus architectures. For example, and not limited to, such architectures include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Extended ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus, also known as a mezzanine bus.

[0185] Computer 1610 generally encompasses a variety of computer-readable media. Computer-readable media may be any available media accessible by computer 1610 and include both volatile and non-volatile media, removable and non-removable media. For example, and not limited to, computer-readable media may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or other media that can be used to store desired information and can be accessed by computer 1610. Communication media generally encompass all information distribution media that embody computer-readable instructions, data structures, program modules, or other data in modulated data signals, such as carrier waves or other transport mechanisms. The term “modulated data signal” means a signal that has one or more of its characteristic sets, or that has been modified to encode information in a signal. By example, and not limited to, communication media include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the above shall also be included within the scope of computer-readable media.

[0186] System memory 1630 includes computer storage media in the form of volatile and / or non-volatile memory, such as read-only memory (ROM) 1631 and random access memory (RAM) 1632. The Basic Input / Output System 1633 (BIOS), which contains basic routines that help transfer information between elements within the computer 1610 during startup, is generally stored in ROM 1631. RAM 1632 generally contains readily available data and / or program modules currently being manipulated by the processing unit 1620. As an example, and not limited to, Figure 16 shows an operating system 1634, an application program 1635, other program modules 1636, and program data 1634. Figure 37 is shown.

[0187] Computer 1610 may also include other removable / non-removable, volatile / non-volatile computer storage media. As just one example, Figure 16 illustrates a hard disk drive 1641 for reading from or writing to a non-removable, non-volatile magnetic medium, a flash drive 1651 for reading from or writing to removable, non-volatile memory 1652 such as flash memory, and an optical disk drive 1655 for reading from or writing to removable, non-volatile optical disk 1656 such as a CD-ROM or other optical media. Other removable / non-removable, volatile / non-volatile computer storage media that can be used in the exemplary operating environment include, but are not limited to, magnetic tape cassettes, flash memory cards, digital multipurpose disks, digital videotapes, solid RAM, solid ROM, and the like. The hard disk drive 1641 is generally connected to the system bus 1621 via a non-removable memory interface such as interface 1640, and the magnetic disk drive 1651 and optical disk drive 1655 are generally connected to the system bus 1621 via a removable memory interface such as interface 1650.

[0188] The drives and associated computer storage media described above and illustrated in Figure 16 provide storage for computer-readable instructions, data structures, program modules, and other data of the computer 1610. In Figure 16, for example, the hard disk drive 1641 is illustrated as storing the operating system 1644, application programs 1645, other program modules 1646, and program data 1647. Note that these components may be the same as or different from the operating system 1634, application programs 1635, other program modules 1636, and program data 1637. The operating system 1644, application programs 1645, other program modules 1646, and program data 1647 are given different numbers here to illustrate that they are at least different copies. An actor can input commands and information into the computer 1610 by input devices such as the keyboard 1662 and a pointing device 1661, commonly called a mouse, trackball, or touchpad. Other input devices (not shown) may include microphones, joysticks, gamepads, satellite dishes, scanners, etc. These and other input devices are often connected to the processing unit 1620 by a user input interface 1660 coupled to the system bus, but may also be connected by other interfaces and bus structures such as parallel ports, game ports, or Universal Serial Bus (USB). A monitor 1691 or other type of display device is also connected to the system bus 1621 via an interface such as a video interface 1690. In addition to the monitor, the computer may also include other peripheral output devices such as speakers 1697 and printers 1696, which can be connected through an output peripheral interface 1695.

[0189] Computer 1610 can operate in a networked environment using logical connections to one or more remote computers, such as remote computers 1680. Remote computers 1680 may be personal computers, servers, routers, network PCs, peer devices, or other common network nodes, and generally, although only the memory storage device 1681 is illustrated in Figure 16, many or all of the above-mentioned elements in relation to computer 1610 are included. The logical connections depicted in Figure 16 include local area networks (LANs) 1671 and wide area networks (WANs) 1673, but may also include other networks. Such networking environments are common in offices, enterprise-scale computer networks, intranets, and the internet.

[0190] When used in a LAN networking environment, computer 1610 is connected to LAN 1671 via a network interface or adapter 1670. When used in a WAN networking environment, computer 1610 generally includes a modem 1672 or other means for establishing communication over WAN 1673, such as the Internet. The modem 1672, which may be internal or external, may be connected to the system bus 1621 via an actor input interface 1660 or other appropriate mechanism. In a networked environment, program modules, or parts thereof, written in relation to computer 1610 may be stored in a remote memory storage device. As an example, and not limited to, Figure 16 illustrates a remote application program 1685 residing in memory device 1681. The network connections shown are illustrative, and it will be understood that other means may be used to establish communication links between computers.

[0191] The techniques described herein are not limited to any particular implementation and can be implemented in any of many ways. Examples of specific implementations are provided herein for illustrative purposes only. Furthermore, the aspects of the techniques described herein are not limited to the use of any particular technique or combination of techniques; therefore, the techniques disclosed herein may be used individually or in any suitable combination.

[0192] As described herein, several aspects of the technology have been explained, but it should be understood that various changes, modifications, and improvements are possible.

[0193] For example, an example is provided in which a dataset group contains multiple datasets. A data processing system like the one described herein can be implemented to support groups having a single dataset in some scenarios, and / or to support null groups without any datasets in other scenarios.

[0194] As another example, a scenario is provided in which the result set from which the user can make selections includes a group of datasets. The user can select a dataset group, and subsequently, the content of the dataset group may be presented to the user for further selection. A scenario is described in which the user selects a dataset contained within that dataset group. In some scenarios, a dataset group may contain other dataset groups. By selecting a dataset group contained within the group, the content of the selected dataset group is presented to the user, and the process of selecting from the content of that dataset group may be repeated. Such a recursive process can be repeated recursively at multiple levels.

[0195] Furthermore, examples are provided in which the dataset selection tool receives user input to specify only a single dataset by guiding the user through one or more screens of the user interface until the user reaches a screen where the desired dataset is presented. In variants of the data processing system described herein, the user can navigate through user interface screens and select multiple datasets, and the selection tool is used in operations where multiple datasets are specified.

[0196] Furthermore, the dataset cart is described as having a scope based on the user persona. Other characteristics that can be evaluated during use may be used to define the scope. Time can be used for the scope, for example. For example, by scoping dataset groups based on the day of the week, it is possible to access datasets returned in a search for the day they are most recent, updated on a specific day of the week.

[0197] As yet another example, scope has been described as something that limits the number and enhances the relevance of the group of datasets returned in response to a search query. In some embodiments, scope can be added individually to datasets so that the datasets returned in response to a search query are limited based on the scope at the time of the search.

[0198] As yet another example, a dataset group is described as having a scope. Scope can be achieved by storing and accessing scope information associated with the dataset group. In a data processing system, components can be given scope (not necessarily limited to dataset groups). For example, certain tools can be scoping, restricting their use to users with personas within that scope. In such embodiments, the scope information for a dataset group can be set and used in the same way as the scope information for other components.

[0199] In a further variation, the search results for datasets can be limited to dataset carts that themselves match the search query or dataset carts that contain datasets that match the search criteria. In some embodiments, the search results may include dataset carts containing datasets that match the criteria, and datasets that match the search criteria but are not assigned to any dataset cart. Individual datasets can be presented, but the search results can be limited by presenting datasets hierarchically so that datasets incorporated into dataset carts or other groupings are not shown individually.

[0200] Furthermore, examples are provided in which user input specifies the source type, thereby distinguishing between contexts where the selection should be a single dataset and contexts where it should be a group of datasets. This context can be determined by other means, such as automatically. If the context is determined automatically, it may be based on a computerized analysis of the actions to be performed on the selected one or more datasets.

[0201] As a further example of possible variations of the disclosed embodiments, it is described that a user writes an application that specifies access to a logical dataset. In some embodiments, the user may be a human user. In other embodiments, the user may be a program with artificial intelligence (AI). For example, the AI ​​may derive a data processing algorithm by processing a dataset that can be applied to other datasets.

[0202] Such changes, modifications, and improvements are intended to be part of this disclosure and to be within the spirit and scope of this disclosure. Furthermore, while the advantages of the technology described herein are shown, it should be understood that not all embodiments of the technology described herein will encompass all described advantages. Some embodiments may not implement any of the features described herein as advantageous, and in some cases, one or more of the described features may be implemented to obtain further embodiments. Accordingly, the above description and drawings are merely examples.

[0203] The above embodiments of the technology described herein may be implemented in any of many ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code can run on any suitable processor or group of processors, whether provided on a single computer or distributed across multiple computers. Such processors may be implemented as integrated circuits and have one or more processors in an integrated circuit component that includes commercially available integrated circuit components known in the industry by names such as CPU chips, GPU chips, microprocessors, microcontrollers, or coprocessors. Alternatively, the processor may be implemented in custom circuitry such as ASICs, or in semi-custom circuitry resulting from the configuration of programmable logic devices. As a further alternative, the processor may be part of a larger circuit or semiconductor device, whether commercial, semi-custom, or custom. As a specific example, some commercial microprocessors have multiple cores such that one or a subset of multiple cores can constitute a processor. However, the processor can be implemented using circuitry of any suitable format.

[0204] Furthermore, it is understood that a computer may be embodied in any of many forms, such as a rack-mount computer, a desktop computer, a laptop computer, or a tablet computer. In addition, a computer may be incorporated into a device that is not generally considered a computer but has appropriate processing power, including a personal digital assistant (PDA), a smartphone, or any other appropriate portable or fixed electronic device.

[0205] Furthermore, a computer may have one or more input and output devices. These devices can be used, in particular, to present a user interface. Examples of output devices that can be used to provide a user interface include a printer or display screen for a visual representation of the output, and a speaker or other sound-generating device for an audible representation of the output. Examples of input devices that can be used for the user interface include a keyboard, as well as pointing devices such as a mouse, touchpad, and digitizer tablet. As another example, a computer may receive input information by speech recognition or in other audible formats.

[0206] Such computers can be interconnected by one or more networks of any appropriate form, encompassing a corporate network or a local area network or wide area network such as the Internet. Such networks may be based on any appropriate technology, may operate according to any appropriate protocol, and may encompass wireless networks, wired networks, or fiber optic networks.

[0207] Furthermore, the various methods or processes outlined herein may be encoded as software executable for one or more processors using any one of various operating systems or platforms. In addition, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and may be compiled as executable machine code or intermediate code to be executed against a framework or virtual machine.

[0208] In this regard, embodiments of the technology described herein may be embodied as computer-readable storage media (or more computer-readable media) (e.g., computer memory, one or more floppy disks, compact discs (CDs), optical discs, digital video discs (DVDs), magnetic tape, flash memory, field-programmable gate arrays or circuit configurations in other semiconductor devices, or other tangible computer storage media) encoded with one or more programs that, when executed on one or more computers or other processors, perform the methods for carrying out the various embodiments described above. As is evident from the above examples, computer-readable storage media can retain information for a time sufficient to provide computer-executable instructions in a non-temporary form. Such one or more computer-readable storage media may be portable so that one or more programs stored therein can be loaded onto one or more different computers or other processors to carry out the various embodiments of the technology described above. In this specification, the term “computer-readable storage media” encompasses only non-temporary computer-readable media that can be considered as products (i.e., manufactured goods) or machines. Alternatively or additionally, embodiments of the technology described herein may be embodied in computer-readable media other than computer-readable storage media, such as propagating signals.

[0209] The terms “program” or “software” are used herein in a general sense to refer to any type of computer code or set of computer executable instructions or set of processor executable instructions that can be used to program a computer or other processor to implement various aspects of the techniques described herein. In addition, according to certain aspects of these embodiments, it will be understood that, when executed, one or more computer programs that perform the methods of the techniques described herein do not need to reside on a single computer or processor, but may be distributed modularly across a number of different computers or processors to implement various aspects of the techniques described herein.

[0210] Computer executable instructions can take many forms, such as program modules, which are executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. Generally, the functionality of program modules may be combined or distributed as desired in various embodiments.

[0211] Furthermore, the data structure may be stored in a computer-readable medium in any appropriate form. For the sake of simplicity in illustration, the data structure may be shown having fields that are related by location within the data structure. Such relationships can also be achieved by assigning locations in the computer-readable medium that convey the relationships between fields to the storage of the fields. However, relationships between information in the fields of the data structure may be established using any appropriate mechanism, including the use of pointers, tags, or other mechanisms for establishing relationships between data elements.

[0212] Various aspects of the technology described herein may be used individually, in combination, or in various arrangements not specifically described in the embodiments described above, and therefore, in their application, are not limited to the details and arrangements of components described above or illustrated in the drawings. For example, an aspect described in one embodiment may be combined in any way with an aspect described in another embodiment.

[0213] Furthermore, the techniques described herein may be embodied as the methods exemplified herein, including by reference to Figures 13-15. Acts performed as part of any method may be ordered in any appropriate manner. Thus, embodiments may be constructed in which the acts are performed in an order different from that illustrated (this may include performing several acts simultaneously, even if they are shown as sequential acts in the embodiments for explanation).

[0214] Furthermore, certain actions are described as being performed by an "actor" or "user." It is understood that an "actor" or "user" does not necessarily have to be a single individual, and in some embodiments, actions attributed to an "actor" or "user" may be performed by a team of multiple individuals and / or by an individual in combination with computer-aided tools or other mechanisms.

[0215] The use of ordinal terms such as “first,” “second,” and “third” in a claim to modify a claim element does not in itself imply a temporal order in which one claim element takes precedence, precedence, or sequence or method of acting over another claim element, and these claim elements are distinguished by being used merely as labels to distinguish one claim element having a certain name from another element having the same name (except for the use of ordinal terms).

[0216] Furthermore, the expressions and terms used herein are for illustrative purposes only and are not intended to be limiting. The use herein of “including,” “comprising,” “having,” “containing,” “involving,” and their variations means that they encompass the items and their equivalents listed thereafter, as well as any additional items. [Explanation of symbols]

[0217] 1, 2, 3, 4 Dataset Cart 16 Program Data 100 IT Systems 102-1, 102-2, 102-3 Datastore 104 Data Processing System 105 Dataset Multiplexer 106-1, 106-2, 162-3 Applications 107 Dataset Catalog 111a, 112a, 113a User 120 dataset group repositories 162-3 Application 200 Graphical User Interfaces (GUI) 202 datasets 204 Dataset Cart 206 Graphical User Elements 300 interfaces 302 Dataset Cart 304 User Interface Elements 310, 312 logical datasets Information 314, 316, 340 320 Graphical User Elements 330 Interface Elements 355, 402 Graphical User Elements 404 Pop-up Dialog Box 406 Graphical User Element 455 Drop-down menu 502 Lineage Information 510 Components 512 windows 514, 602, 604, 606, 608, 820, 840, 845, 861, 870, 896, 898 Graphical User Interface Elements Lists 610, 615, 815, 895 620, 625 items 630 User Interface Elements 632, 634 windows 700 Graphical User Interfaces 702 Directory 704 Technical Group 900 Computing Environments 920 Dataset Cart 930 logical datasets 1000, 1100 Graphical User Interface 1600 Computing System Environment 1610 Computer 1620 Processing Unit 1621 System Bus 1630 System Memory 1631 Dedicated memory (ROM) 1632 Random Access Memory (RAM) 1633 Basic Input / Output System 1634, 1644 Operating Systems 1635, 1645 Application Programs 1636, 1646 Program Modules 1637, 1647 Program Data 1640, 1650 Interface 1641 Hard Disk Drive 1651 Flash Drive 1651 Magnetic Disk Drive 1652 Non-volatile memory 1655 Optical Disc Drive 1656 Non-volatile optical disc 1660 User Input Interface 1660 Actor Input Interface 1661 Pointing device 1662 keyboard 1670 adapter 1671 Local Area Network (LAN) 1672 Modem 1673 Wide Area Network (WAN) 1680 Remote Computer 1681 Memory Storage Devices 1681 memory devices 1685 Remote Application Program 1690 Video Interface 1691 Monitor 1695 Output Peripheral Interface 1696 Printer 1697 Speaker

Claims

1. A method for enabling efficient operation of a data processing system in an environment using multiple datasets by allowing selection of a group of datasets to perform operations on each of the multiple datasets in the group, The system receives search queries via a user interface to find datasets for use in conjunction with operations related to data access in the data processing system, Presenting the search results to the user interface based on the search query, including presenting one or more groups of datasets, wherein each of at least some of the groups of datasets includes one or more of the search datasets, and presenting the results includes presenting one or more groups of datasets, Receiving an operation on a first group of datasets of one or more groups of datasets presented to the user interface via the user interface, wherein the user interface is configured to provide an option to select the first group of datasets as the target of the operation related to data access via the user interface. As soon as the first group of datasets from the one or more groups of datasets presented in the user interface is selected, the operation is performed in each of the one or more datasets included in the first group of datasets. Methods that include...

2. The user interface provides an option to expand the first group of datasets to enable the user to select one or more datasets from the first group of datasets as targets for the operations related to data access, The method according to claim 1, wherein, as soon as one or more datasets from the first group of datasets are selected, the operation is performed in each of the one or more datasets from the first group of datasets.

3. The method according to claim 1 or 2, wherein each of the one or more groups of the dataset presented in the user interface has a correspondence between a persona associated with a user who entered the search query via the user interface and a scope associated with the one or more groups of the dataset.

4. The method according to claim 3, wherein the search results exclude datasets that do not have metadata associated with the user's persona.

5. The method according to any one of claims 1 to 4, wherein performing the operation in each of the one or more datasets includes performing data quality rules in each of the one or more datasets.

6. A method for enabling efficient operation of a data processing system in an environment using multiple datasets, by enabling a first user to form a dataset group and present the dataset group to a second user in order to make a selection relating to configuring an operation to access one or more datasets, The first user receives input from the first user, through one or more first user interfaces, to select one or more datasets from a large number of datasets to associate with one group of a large number of datasets, To store representations of the aforementioned large number of groups in the dataset, Presenting a second user interface configured for use in conjunction with the operation of accessing one or more datasets, wherein the second user has a persona and the datasets have a scope at least partially based on the user's persona, Automatically identifying one or more groups of a dataset based at least partially on the correspondence between the persona associated with the second user of the data processing system and the scope associated with one or more automatically identified groups of the dataset, and Rendering indications of one or more automatically identified groups of the dataset in the second user interface. To present the second user interface, which includes the following: Methods that include...

7. The ability to store representations of the aforementioned large number of groups For each of the numerous groups in the dataset, information about one or more users who have permission to access that group is stored. The method according to claim 6, including the method described in claim 6.

8. The one or more first user interfaces include a dataset search interface that includes a faceted search interface, The method according to claim 6 or 7, wherein the facets of the faceted search interface are based on metadata values ​​associated with the numerous datasets.

9. The method according to any one of claims 6 to 8, wherein the one or more first user interfaces include a user interface for displaying the lineage of a dataset.

10. The method according to any one of claims 6 to 9, wherein the one or more first user interfaces include a user interface for displaying metadata related to one of the numerous datasets.

11. Receiving input from the second user through the second user interface, specifying one of the one or more automatically identified groups, Based on the input received from the second user, the operation is performed for each of the numerous datasets within the selected group. The method according to any one of claims 6 to 10, further comprising:

12. The method according to any one of claims 6 to 11, wherein the operation constitutes an application for execution by the data processing system.

13. The method according to any one of claims 6 to 12, wherein the automatic identification of one or more groups of datasets, at least in part, based on the correspondence between the persona associated with the second user of the data processing system and the scope associated with the one or more automatically identified groups of datasets, includes selecting one or more groups of datasets that the second user is authorized to access.

14. A method according to any one of claims 6 to 13, wherein rendering the indication for one or more automatically identified groups includes rendering a graphical user interface element representing a group of datasets for each of the one or more automatically identified groups, The system receives a selection of rendered graphical user interface elements representing a group of datasets via the second user interface, and renders a number of datasets representing the group onto the second user interface based on the selection. Methods that further include this.

15. A method for enabling efficient operation of a data processing system in an environment using multiple datasets by presenting a group of datasets for a user of the data processing system to make a selection in relation to configuring an operation to access one or more datasets, Presenting a user interface configured to allow the user to select one or more datasets for use in conjunction with the operation of accessing one or more datasets, wherein the user has a persona and the datasets have a scope that is at least partially based on the user's persona. Automatically identifying one or more groups of a dataset based at least partially on the correspondence between the persona associated with the user of the data processing system and the scope associated with one or more automatically identified groups of the dataset, and Rendering indications of one or more automatically identified groups of the dataset in the user interface. Presenting a user interface that includes Methods that include...

16. The system receives user input through the user interface that specifies one of the one or more groups, Based on the received input, render the indication of the datasets within the selected group. The method according to claim 15, further comprising:

17. The system receives user input through the user interface that specifies one of the one or more groups, Based on the received input, the operation described above is performed for each of the numerous datasets within the selected group. The method according to claim 15 or 16, further comprising:

18. Automatically identifying one or more groups of datasets The user interface receives search queries for the dataset, To generate search results, perform a search based on the aforementioned search query. The method according to any one of claims 15 to 17, further comprising:

19. The method according to any one of claims 15 to 18, wherein the operation comprises configuring an application for execution by the data processing system.

20. The method according to any one of claims 15 to 19, wherein the automatic identification of one or more groups of datasets, at least in part, based on the correspondence between the persona associated with the user of the data processing system and the scope associated with the one or more automatically identified groups of datasets, includes selecting one or more groups of datasets that the user is authorized to access.

21. A method according to any one of claims 15 to 20, wherein rendering the indication for one or more automatically identified groups includes rendering a graphical user interface element representing a group of datasets for each of the one or more automatically identified groups, The system receives a selection of rendered graphical user interface elements representing a group of datasets, and renders a number of datasets from the group onto the user interface based on the selection. Methods that further include this.

22. A method for enabling efficient operation of a data processing system in an environment using multiple datasets by forming a group of datasets, Rendering one or more first user interfaces identified by numerous datasets, Receiving user input through one or more first user interfaces to select another identified dataset to associate with one of a number of groups of datasets, To store representations of the aforementioned large number of groups in the dataset Methods that include...

23. The ability to store representations of the aforementioned large number of groups For each of the numerous groups in the dataset, information about one or more users who have permission to access that group is stored. The method according to claim 22, including the method described in claim 22.

24. Rendering a second user interface associated with a user configuration of the data processing system in order to perform operations related to data access, wherein the rendering includes a dataset selection portion of the second user interface. The method according to claim 22 or 23, further comprising: A method for rendering the second user interface, comprising presenting a representation of one or more of the numerous groups of the dataset to the selected portion of the dataset.

25. Selecting one or more groups of the numerous groups of the dataset to be presented in the second user interface based on the user persona. The method according to claim 24, further comprising:

26. The second user interface includes a user interface in the program development environment. The method according to claim 24, wherein the operation related to data access includes configuring a component of a program under development to access a dataset or a group of datasets.

27. The method according to any one of claims 22 to 26, wherein the one or more first user interfaces include a dataset search interface.

28. The dataset search interface includes a faceted search interface. The method according to claim 27, wherein the facets of the faceted search interface are based on metadata values ​​associated with the numerous datasets.

29. The method according to any one of claims 22 to 28, wherein the one or more first user interfaces include a user interface for displaying the lineage of a dataset.

30. The method according to any one of claims 22 to 29, wherein the one or more first user interfaces include a user interface for displaying metadata related to one of the numerous datasets.

31. A method for enabling efficient operation of a data processing system in an environment using multiple datasets, Means for rendering one or more first user interfaces identified from a dataset, Means for receiving user input through one or more first user interfaces to select one or more identified datasets to associate with one group of a number of groups of datasets, means for storing representations of the aforementioned number of groups in a dataset Methods that include...

32. Means for rendering a second user interface associated with a user configuration of the data processing system in order to perform operations related to data access, wherein the second user interface includes a dataset selection portion. The method according to claim 31, further comprising: A method for rendering the second user interface, wherein the means includes presenting a representation of one or more of the numerous groups of the dataset in the selected portion of the dataset.

33. Means for selecting one or more groups of the numerous groups of a dataset to be presented to the second user interface, based on the user persona. The method according to claim 32, further comprising:

34. A method for creating a dataset group in a data processing system capable of operating with a large number of datasets, comprising at least one hardware processor, Identifying a set of datasets available for use when performing an operation by the data processing system, wherein the operation relates to data access by the data processing system. Presenting the identified set of the dataset to the first user interface, The first user interface receives a user selection of one or more datasets from the identified set of presented datasets, To store in a representation of a group containing one or more selected datasets. Methods that include...

35. Identifying the set of datasets available for use when performing data access-related operations in the data processing system, The system receives a search query via a user interface that specifies one or more values ​​of facets describing the numerous datasets defined in the data processing system, Performing a search based on the search query to generate search results, wherein the search results include the set of datasets available for use when performing the operation. The method according to claim 34, including the method described in claim 34.

36. The method according to claim 35, wherein the search query includes a faceted search query, and the faceted search query includes one or more facets for filtering the search results.

37. The method according to claim 36, wherein one or more of the facets include a facet indicating whether the dataset is registered in a catalog that associates information for accessing a physical dataset with a logical dataset.

38. The user interface for receiving the search query includes a number of fields for receiving user input that identifies values ​​for one or more facets, The method according to claim 36 or 37, wherein the plurality of fields include fields for receiving values ​​of logical, physical and / or operational metadata associated with the plurality of datasets.

39. The method according to any one of claims 34 to 38, wherein the operations related to data access constitute a component of an application performed by the data processing system.

40. Receiving a command to update the group via a second user interface, wherein the command includes a request to add one or more datasets to the group or a request to remove one or more datasets from the group. The method according to any one of claims 34 to 39, further comprising the above.

41. To present metadata relating to the identified set of datasets in the dataset in response to user input requesting metadata related to the dataset via the first user interface. The method according to any one of claims 34 to 40, further comprising:

42. The aforementioned group is the second group, The method according to any one of claims 34 to 41, wherein receiving the user selection of one or more datasets includes receiving a selection of a first group of datasets previously defined such that the second group includes a hierarchical grouping of datasets.

43. The method according to any one of claims 34 to 42, wherein storing the representation of the group includes storing scope information for the group.

44. The method according to claim 43, wherein the scope information includes the identification of one or more users who have permission to access the group.

45. The method according to claim 43, wherein the scope information includes the identification of one or more roles that have permission to access the group.

46. Rendering a second user interface associated with a user configuration of the data processing system in order to perform the operations related to data access, wherein the rendering includes a dataset selection portion of the second user interface. A method according to any one of claims 34 to 45, further comprising rendering the second user interface by presenting a representation of the group comprising the selected one or more datasets to the dataset selection portion.

47. At least one computer hardware processor, A non-temporary computer-readable medium that stores processor-executable instructions that, when executed by the at least one computer hardware processor, cause the at least one computer hardware processor to perform the method according to any one of claims 1 to 46, A data processing system that includes this.

48. A non-temporary computer-readable medium comprising, when executed by at least one computer hardware processor, a processor-executable instruction causing the at least one computer hardware processor to perform the method according to any one of claims 1 to 46.

Images