A distributed histogram computation framework using data flow sketches and samples.

Abstract

The method for performing distributed histogram computation within a framework utilizing data flow sketches and samples is performed by the system and devices. The distributions of a large dataset are scanned and processed once by a computation pool without sorting to generate local sketches and value samples for each distribution. The local sketches and samples are used to construct a local histogram on which cardinality estimates for generating distributed queries against the distributions are obtained. The local statistics of the distributions are also merged and integrated to construct a global histogram representing the entire dataset. The global histogram is used to determine cardinality estimates for generating input queries against the entire dataset. Adding new data to a dataset or distribution involves scanning the new data, generating new statistics from it, and then merging it with existing statistics to obtain a new global histogram.

Description

Background Technology

[0001] Histograms are used by query optimizers to estimate cardinality for generating efficient query plans in commercial database systems and data warehouses. Constructing these histograms involves capturing the number of aggregates of tuples, as well as unique values ​​within a range (corresponding to histogram buckets), frequent tuple values, quantiles, and, in some cases, frequency differences between adjacent values. One approach is to maintain a hash table of every value in the dataset and its frequency; however, for large datasets, the memory required for this approach is prohibitively high. Instead, histograms for large datasets are typically constructed by sorting the data and processing the values ​​in the sorted order. Using this approach, quantiles are calculated by keeping track of the total number of observed tuples, and the number of unique values ​​to track becomes negligible. The cost of constructing the histogram under this approach is primarily read-based, especially the sorting of the data, which often involves external memory for sorting large tables. Using sorting on the data during histogram creation means creating only a single-column histogram when scanning multi-column tables. This is a significant drawback depending on the number of columns and table layout (i.e., column-based layout versus row-based layout).

[0002] Building a global histogram for a large, “sharded” dataset traditionally involves constructing multiple histograms across individual shards (or distributions) of the entire dataset and then merging those histograms using a custom merging algorithm. However, histogram merging is a highly lossy process, and many key properties of the underlying data distribution can be lost during the merging process. Furthermore, rescanning the entire table on a single compute node to generate a global histogram may not be feasible, as tables in a data warehouse can be arbitrarily large. Summary of the Invention

[0003] This summary is provided to introduce selected concepts in a simplified form, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter.

[0004] The method for performing distributed histogram computation within a framework utilizing data flow sketches and samples is performed by the system and devices. The distributions of a large dataset are scanned and processed by a computation pool without sorting to generate local sketches and value samples for each distribution. The local sketches and samples are used to construct a local histogram on which cardinality estimates are obtained for query plans that generate local steps of a distributed query run against the distribution. Local statistics of the distributions are also merged and integrated to construct a global histogram representing the entire dataset. The global histogram is used to derive cardinality estimates when generating distributed plans for queries against the entire dataset. Adding new data to a dataset or its distribution involves only scanning the new data, generating new statistics from it, and then merging them with existing statistics to obtain a new global histogram.

[0005] Further features and advantages, as well as the structure and operation of various examples, will be described in detail below with reference to the accompanying drawings. Note that these ideas and techniques are not limited to the specific examples described herein. Such examples are presented herein for illustrative purposes only. Other examples will be apparent to those skilled in the art based on the teachings contained herein. Attached Figure Description

[0006] The accompanying drawings, which are incorporated herein and form part of the specification, illustrate embodiments of the present application and, together with the specification, further serve to explain the principles of the embodiments and enable those skilled in the art to make and use the embodiments.

[0007] Figure 1 A block diagram of a system for distributed histogram computation using a framework of data flow sketches and samples, according to an example embodiment, is shown.

[0008] Figure 2 A block diagram of a computational system for distributed histogram computation in a framework of data flow sketches and samples, according to an example embodiment, is shown.

[0009] Figure 3 A block diagram of a computational system for distributed histogram computation in a framework of data flow sketches and samples, according to an example embodiment, is shown.

[0010] Figure 4 A data structure for distributed histogram computation in a framework utilizing data flow sketches and samples, according to an example embodiment, is shown.

[0011] Figure 5 A data structure for distributed histogram computation in a framework utilizing data flow sketches and samples, according to an example embodiment, is shown.

[0012] Figure 6A data structure for distributed histogram computation in a framework utilizing data flow sketches and samples, according to an example embodiment, is shown.

[0013] Figure 7 A system flowchart for distributed histogram computation in a framework utilizing data flow sketches and samples, according to an example embodiment, is shown.

[0014] Figure 8 A flowchart is shown for distributed histogram computation in a framework utilizing data flow sketches and samples, according to an example embodiment.

[0015] Figure 9 A flowchart is shown for distributed histogram computation in a framework utilizing data flow sketches and samples, according to an example embodiment.

[0016] Figure 10 A flowchart is shown for distributed histogram computation in a framework utilizing data flow sketches and samples, according to an example embodiment.

[0017] Figure 11 A block diagram of an example computing device that can be used to implement the embodiments is shown.

[0018] The features and advantages of the embodiments will become more apparent from the detailed description set forth below when taken in conjunction with the accompanying drawings, in which similar reference numerals identify corresponding elements throughout. In the drawings, similar reference numerals generally indicate identical, functionally similar, and / or structurally similar elements. The first appearance of an element in the drawings is indicated by the leftmost digit(s) of the corresponding reference numeral(s). Detailed Implementation

[0019] I. Introduction

[0020] The following detailed description discloses numerous embodiments. The scope of this patent application is not limited to the disclosed embodiments, but also includes combinations of the disclosed embodiments and modifications thereof.

[0021] References to "an embodiment," "an embodiment," "an exemplary embodiment," etc., in the specification indicate that the described embodiment may include specific features, structures, or characteristics, but each embodiment may not necessarily include all specific features, structures, or characteristics. Furthermore, these phrases do not necessarily refer to the same embodiment. Additionally, when a specific feature, structure, or characteristic is described in connection with an embodiment, whether explicitly described or not, it is assumed that implementing such a feature, structure, or characteristic in conjunction with other embodiments is within the knowledge of those skilled in the art.

[0022] In this discussion, unless otherwise stated, adjectives such as “substantially,” “generally,” and “approximately” that modify one or more features of embodiments of this disclosure are understood to mean that the condition or feature is defined within an acceptable tolerance range for operation of the embodiment for the intended application.

[0023] Furthermore, it should be understood that spatial descriptions (e.g., "above," "below," "upward," "left," "right," "downward," "top," "bottom," "vertical," "horizontal," etc.) are used herein for illustrative purposes only, and actual implementations of the structures and figures described herein may be arranged in space in any orientation or manner. Additionally, the figures may not be provided to scale, and the orientation or organization of elements in the figures may vary in the embodiments.

[0024] In the context of the embodiments herein, for example, for a relational database, a “dataset” is stored as a “table” of data, and these terms—“table” and “dataset”—may be used interchangeably in the specification.

[0025] Many exemplary embodiments are described below. Note that any section / section headings provided herein are not intended to be limiting. Embodiments are described throughout this document, and any type of embodiment may be included under any section / section. Furthermore, embodiments disclosed in any section / section may be combined in any way with any other embodiments described in the same section / section and / or different sections / sections.

[0026] Part II below describes an example embodiment for distributed histogram computation within a framework utilizing data flow sketches and samples. Part III below describes example mobile and computing devices that can be used to implement the features of the embodiments described herein. Part IV below describes additional examples and advantages, and Part V provides some concluding remarks.

[0027] II. Example Implementation of Distributed Histogram Computation within a Framework Utilizing Data Flow Sketches and Samples

[0028] The method for performing distributed histogram computation within a framework utilizing data flow sketches and samples is performed by systems and devices. The various embodiments described herein relate to database systems and computing centers, which can be cloud-based, such as, but not limited to, data warehouses, systems for “big data” analytics, enterprise systems, other database systems storing very large datasets, and so on. The described embodiments are applicable to any type of database server implementation, such as, but not limited to, SQL Server from Microsoft Corporation in Redmond, Washington.

[0029] To handle queries on large datasets, the query optimizer in a database server relies on column-level statistics to generate efficient query plans. Statistics are generated internally by the query optimizer for columns (or a set of columns) in a table, either by explicitly running a statistics generation command or by enabling such an option on the database. The statistical objects include histograms, which represent the distribution of data across the columns(s) they are associated with. A histogram approximates a one-dimensional data distribution T by dividing it into bins (or buckets or cells) and storing information for each bin: the values ​​forming the bin boundaries, the total number of tuples within the bin, and the number of unique values ​​within the bin. The query optimizer uses such histograms to estimate the cardinality of the intermediate result set during query execution. These cardinality estimates enable the query optimizer to generate high-quality query plans.

[0030] For reference, in describing the embodiments of this paper, given a (single-column) data distribution over a value range D, distributed histogram computation within a framework utilizing data flow sketches and samples allows for the construction of a histogram H using a single scan and limited memory. A data distribution labeled as a multiset T has values ​​v appearing in T, where v ∈ T. A set of values ​​v in T can be labeled as V = {v1,...,v...} l}. For a value v∈D, frq(v) is used to indicate the number of times v appears in T; if Therefore, frq(v) = 0.

[0031] As noted above, the various embodiments relate to analytics services built for enterprise data warehousing and "big data" analytics, such as, but not limited to, those from Microsoft Corporation in Redmond, Washington. Synapse. This type of analysis involves various data warehouse, database server, and database manager features and is envisioned as employing a cloud-first, highly scalable architecture to enable fast and efficient querying of petabytes (PB) of data. Implementations include a control node that acts as the query endpoint, responsible for generating a distributed plan for the input query and distributing work among a pool of compute nodes (hereinafter also referred to as “processing nodes”). Each compute node is responsible for processing data from one or more distributions (i.e., portions or distributions of a dataset) from a large distributed table. The control node transforms the input query into a set of queries, and each such query can be run in parallel on each distribution in a dependent order. In the described implementation, each node, including the control node and compute nodes, is configured to run / execute its own instance of a database server.

[0032] In this context, column-level statistics encompass both local and global components of the end-to-end lifecycle of distributed requests in data warehouses or other database implementations. At the global level (e.g., on the control node), a histogram representing the distribution of data across the entire table(s) is needed, and the database server instance at the control node uses this global histogram to optimize queries for parallel processing. On each compute node, a histogram representing the data mapped to the distribution(s) specific to that compute node is needed for processing the corresponding distribution(s).

[0033] Traditional global histogram construction involves building a Max-Diff histogram on each compute node and then merging those histograms using a custom merging algorithm on the control node. However, as noted above, histogram merging is highly lossy and loses many key properties of the underlying data distribution during this process. Furthermore, rescanning and sorting the entire table on a single node to generate a global histogram is resource-constrained, as tables in a data warehouse can be arbitrarily large.

[0034] Therefore, the embodiments described herein depict a process for computing global and local histograms involving scanning the entire dataset only once, which can be performed in parallel on a large number of computing nodes. Furthermore, the described embodiments are easily scaled up with the addition of more computing nodes and / or data, and the global histogram is constructed by merging statistical summaries (described below) without loss of information. That is, the statistics utilized in the embodiments are incrementally mergeable, where the union of statistics built individually on the two datasets has equivalent accuracy to similar statistics built on the union of the two datasets. The embodiments in this document also provide that computations already performed are not partially or entirely redone, and the histogram is constructed without performing a sorting of the datasets, a resource-intensive operation, especially for large datasets, even when using sampling.

[0035] Given that histograms represent an approximation of the data distribution seen within a table, and to avoid the large sorting overhead associated with existing solutions, the embodiments provide constructing histograms based on statistical summaries of large tables obtained with significantly less overhead. For example, obtaining fully random row-level samples is significantly faster because no sorting of the large dataset is required, and multi-column histograms can be constructed using a single scan of the data. Furthermore, the embodiments include generating and storing statistical summaries, including specialized sketches (such as Count-Min sketches and HyperLogLog sketches), to detect “heavy-hitter” values, or values ​​that constitute a significant portion of the entire dataset, as well as the number of unique values ​​in the dataset, so that these numbers are likely to be estimated more accurately than using only random samples of the data. Moreover, these statistical summaries are mergable because the summation of the union of two datasets can be calculated from two separate summaries computed for each individual dataset (or its distribution), while still giving the same error guarantee relative to the summation size, as if the resulting summation were computed from scratch on the union of the two sets. In other words, when more data is added to the table (e.g., a new partition or distribution comes online), the statistical summary for the entire dataset can be kept up-to-date by scanning only the new data and then merging the resulting statistical summary with the statistical summary previously calculated for the older data.

[0036] Therefore, the embodiments provide histogram calculations that do not use sorting of the underlying data—but only scanning—to save processing and memory resources and avoid problems such as temporary database memory / storage overflows that can occur when sorting large datasets. Furthermore, since the summary statistics calculated while scanning the dataset are mergeable, they can be incrementally updated as new data is added, while only the added data is scanned, further improving processing and memory efficiency. This provides very significant savings in the context of append-only databases. Moreover, the described framework is applicable to various types of histograms, including but not limited to equal-width, equal-depth, end-biased, etc., and is not limited to a specific histogram type.

[0037] Therefore, the embodiments described herein provide a solution to the problem of constructing histograms for datasets, including very large datasets, by performing distributed histogram computation within a framework that utilizes data flow sketches and samples. These and other embodiments will be described in more detail below in conjunction with the accompanying drawings and in subsequent sections / subsections.

[0038] Systems, devices, and apparatuses can be configured in various ways for distributed histogram computation within a framework that utilizes data flow sketches and samples. For example, it will now be described... Figure 1 . Figure 1A block diagram of a system 100 according to an example embodiment is shown, which is configured to perform distributed histogram computation within a framework utilizing data flow sketches and samples.

[0039] like Figure 1 As shown, system 100 includes user equipment 102 and data warehouse 104. In an embodiment, user equipment 102 and data warehouse 104 communicate with each other via network 112. It should be noted that in various embodiments, there are different numbers of user equipment and / or data warehouses. Furthermore, according to an embodiment, Figure 1 Any combination of the systems and / or components shown exists in system 100.

[0040] In an embodiment, network 112 includes a variety of communication links connecting devices, platforms, and hosts / servers, such as, but not limited to, the Internet, wired or wireless networks and their components, peer-to-peer networks, local area networks, enterprise networks, cloud networks, and so on.

[0041] User equipment 102 in various embodiments is any combination of any number, type, or computing device or computing system, including terminals, personal computers, laptops, tablets, smartphones, personal digital assistants, servers(s), game consoles, etc., including internal / external storage devices, which are used to perform the functions / operations described herein for interacting with databases, data warehouses, and / or database servers, such as providing queries on processed datasets computed via distributed histograms in a framework utilizing data flow sketches and samples. In embodiments, user equipment 102 also includes additional components (not shown for brevity and clarity), including but not limited to components and subcomponents of other devices and / or systems described herein.

[0042] Data warehouse 104 includes one or more server computers, computing systems, and / or computing devices, which in embodiments may include one or more distributed or “cloud-based” servers, as described in further detail below. In embodiments, in addition to or instead of cloud-based servers, data warehouse 104 also includes one or more local servers. Various systems / devices described herein, such as data warehouse 104, are configured to provide services utilized by user equipment 102, such as business services and / or applications involving data warehousing, database hosting, etc. For example, data warehouse 104 includes a control node 108, one or more compute nodes 110 (hereinafter referred to as “compute nodes 110”), and one or more datasets 106 (hereinafter referred to as “datasets 106”), which are configured to perform functions for these services / applications, as described in further detail below.

[0043] In this embodiment, dataset 106 includes databases, etc., which can be very large datasets such as those used for "big data" analytics and / or data warehousing. It is envisioned herein that, in this embodiment, one or more datasets in dataset 106 are petabyte-scale or larger. Dataset 106 can include any type of data, including relational data, organized as a table with data columns.

[0044] Control node 108 includes one or more server computers, server systems, and / or computing devices, such as local servers in addition to or replacing cloud-based servers. As noted above, control node 108 includes instances of database server applications, as described below. Control node 108 is configured to act as a query endpoint for input queries, generating a distributed plan for the input query based on distributed histogram computation within a framework utilizing data flow sketches and samples, and partitioning the work for query processing among computing nodes 110. That is, according to an embodiment, control node 108 is configured to transform the input query into a set of queries that are run in parallel via computing nodes 110 for each distribution of the dataset.

[0045] Each compute node 110 includes one or more server computers, server systems, and / or computing devices, such as local servers(s) other than or in place of cloud-based servers. As noted above, compute node 110 includes instances of database server applications, as described below. Compute node 110 is configured to process data from one or more distributed large tables while performing query execution tasks assigned by control node 108. In embodiments, compute node 110 is configured to perform such tasks for distributed histogram computation within a framework utilizing data flow sketches and samples, as described herein.

[0046] As noted above, in this embodiment, data warehouse 104 includes one or more distributed or “cloud-based” servers. That is, data warehouse 104 may be a network or “cloud” implementation for applications and / or services associated with hosting databases and / or data warehouses in a network architecture / cloud platform. According to this embodiment, the cloud platform includes a set of networked computing resources, including servers, routers, etc., which are configurable, shareable, provide data security, and are accessible via a network such as the Internet. Cloud applications / services used to host databases and / or data warehouses are configured to run on these computing resources for entities accessing the applications / services locally and / or over the network, typically on top of an operating system running on the resources. As noted above, the cloud platform is configured to support multi-tenancy, where cloud platform-based software serves multiple tenants, each tenant including one or more users who share common access to certain software services and applications of the cloud platform, as noted herein. Furthermore, the cloud platform is configured to support a hypervisor implemented to run hardware, software, and / or firmware of virtual machines (simulating computer systems, including operating systems) for tenants. Virtual machine monitors provide tenants with a virtual operating platform.

[0047] Although a data warehouse (data warehouse 104) is shown in system 100 for purposes of description and illustration and is not limiting, alternative implementations for database hosting are also contemplated herein in various embodiments.

[0048] Now for reference Figure 2 According to an example embodiment, a block diagram of system 200 is shown for distributed histogram computation within a framework utilizing data flow sketches and samples. System 200, as illustrated and described in the exemplary figures, is configured to... Figure 1 This is an embodiment of the control node 108 in the data warehouse 104 of system 100. That is, system 200 is illustrated as being configured to perform operations of the control node. System 200 is described below.

[0049] System 200 includes computing system 202, which is any type of server or computing system, as mentioned elsewhere herein or otherwise known, including but not limited to cloud-based systems, local servers, distributed network architectures, part of a data warehouse, etc. Figure 2 As shown, computing system 202 includes one or more processors (“processors”) 204, one or more memories and / or other physical storage devices (“memory”) 206, and one or more network interfaces (“network interfaces”) 222. Computing system 202 also includes a distributed query processor 208 and a database server application 210 (hereinafter referred to as “DB server 210”).

[0050] Processor 204 and memory 206 can be any type of processor circuitry / system and memory that will be understood by those skilled in the art(s) related to this disclosure and / or those who benefit from it. Processor 204 and memory 206 can each include one or more processors or memories, different types of processors or memories (e.g., multiple caches for query processing, temporary database memory, etc.), remote processors or memories, and / or distributed processors or memories. Processor 204 can be a multi-core processor configured to execute more than one processing thread simultaneously. Processor 204 can include circuitry configured to execute computer program instructions, such as, but not limited to, embodiments of distributed query processor 208 and / or DB server 210, including one or more of its components as described herein, which can be implemented as computer program instructions as described herein.

[0051] Memory 206 includes volatile storage portions such as random access memory (RAM) and / or persistent storage portions such as hard disk drives, non-volatile RAM, etc., for storing or being configured to store computer program instructions / code for performing distributed histogram computations within a framework utilizing data flow sketches and samples as described herein, as well as storing other information and data described herein, including, but not limited to, statistical summaries, sketches, sampling, counting, histograms, their storage, their global versions, etc., in various embodiments. For example, for a dataset generated according to embodiments herein and used by query optimizer 222 to estimate cardinality, computing system 202 also includes global statistics 224, which may be stored in memory 206, and may include global histograms, row counts, etc. In embodiments, global statistics 224 may be stored in storage devices external to computing system 202 and may be shared / used by other systems (e.g., multiple pools of compute / processing nodes as described herein, etc.). In some embodiments, memory 206 also includes datasets (such as... Figure 1 The dataset 106 is stored in a storage device, while in other embodiments, the dataset is stored in an external storage system accessible via a network interface.

[0052] Network interface 226 can be any type or number of wired and / or wireless network adapters, modems, etc., configured to enable system 200, including computing system 202, to communicate intra-system with its components and to communicate with other devices and / or systems over a network, such as in the case of computing system 202 and... Figure 1 Communication between other devices, systems, and hosts in system 100, including systems that store datasets, user devices, computing nodes, etc., via networks such as network 112.

[0053] According to an embodiment, system 200 also includes additional components (not shown for brevity and clarity), including but not limited to components and sub-components of other devices and / or systems described herein, as well as those described below. Figure 9 Those described.

[0054] In this embodiment, the distributed query processor 208 includes a deserializer 212, a directed acyclic graph (DAG) generator 214, and a workload manager 216. The distributed query processor 208 is configured to reliably orchestrate query task DAGs at the petabyte level across a large number (e.g., thousands) of compute nodes and is configured to provide implicit fault tolerance guarantees. The deserializer 212 is configured to deserialize requests received from the DB server 210 for distributed query statistical processing / computation of a dataset distribution and to provide its output to the DAG generator 214, which is then configured to generate DAGs, such as static DAGs that outline the process dependencies and operations for performing distributed statistical computations within a framework utilizing data flow sketches and samples. The DAGs from the DAG generator 214 are provided to the workload manager 216, which is configured to assign compute nodes (as further described below) to perform distributed statistical computations.

[0055] In this embodiment, DB server 210 is any database server application, and DB server 210 includes a data definition language (DDL) engine 218 and a statistics manager 220. DDL engine 218 is configured to execute statements, for example, based on input queries, to create, modify, and delete tables, indexes, and relationships in a dataset. In this embodiment, DDL engine 218 is configured to execute statements to create, modify, and / or delete tables, lists, arrays, and / or other data structures that organize statistical summaries, sketches, dataset sampling, row counts, etc., which are generated when performing functions described herein for distributed histogram calculations within a framework utilizing data flow sketches and samples.

[0056] The statistics manager 220 is configured to generate requests for performing distributed statistical computations and provide such requests to the distributed query processor 208. It also receives results returned from task processing on the compute nodes, such as success / failure indicators, global histograms, etc. In some embodiments, requests for performing distributed statistical computations are generated based on commands or requests received by the statistics manager 220. In other embodiments, as part of the control node, the statistics manager 220 of the DB server 210 is configured to periodically generate requests for performing distributed statistical computations when a dataset is created or modified, when a query is received for a dataset on which distributed statistical computation has not yet been performed, and / or similar situations. In embodiments, the statistics manager 220 may include a portion of the DDL engine 218.

[0057] Query optimizer 222 is configured to generate a query plan for executing an input query against a dataset, based at least on the input query generated according to global statistics 224 (e.g., a global histogram) and cardinality estimates. In embodiments, query optimizer 222 is also configured to subsequently optimize the query plan globally and / or locally based on updated statistics. For example, when a new distribution of the dataset comes online, updated global and / or local histograms can be constructed, and query optimizer 222 can be configured to utilize these updated histograms to optimize newly generated query plans or optimize new queries.

[0058] In some embodiments, the distributed query processor 208 and the database server 210 may be implemented in different physical or logical server hosts that are communicatively linked to perform distributed histogram computations within a framework utilizing data flow sketches and samples. In such embodiments, the distributed query processor 208 may include a separate server node, while the database server 210 includes a control node, as described herein.

[0059] exist Figure 3 In the example embodiment, a block diagram of system 300 is shown for distributed histogram computation within a framework utilizing data flow sketches and samples. System 300, as illustrated and described in the exemplary figures, is configured to... Figure 1 This is an embodiment of a computing node in computing node 110 of computing node 104 in system 100. That is, system 300 is illustrated as being configured to perform operations of the computing node. System 300 is described below.

[0060] System 300 includes computing system 302, which is any type of server or computing system, as mentioned elsewhere herein or otherwise known, including but not limited to cloud-based systems, local servers, distributed network architectures, part of a data warehouse, etc. Figure 3 As shown, computing system 302 includes one or more processors (“processors”) 204, one or more memories and / or other physical storage devices (“memory”) 206, and one or more network interfaces (“network interfaces”) 222. Computing system 202 also includes a database server application 308 (hereinafter referred to as “DB server 308”).

[0061] Processor 304 and memory 306 can be any type of processor circuit / system(s) and memory(s) described herein and / or understood by those skilled in the art(s) who benefit from this disclosure, and can be similarly configured as described above. Figure 2 The processor 204 and memory 206 are described. Processor 304 may include circuitry configured to execute computer program instructions, such as, but not limited to, embodiments of DB server 308, including one or more of its components as described herein, which may be implemented as computer program instructions as described herein. Memory 306 stores or is configured to store computer program instructions / code for computation of distributed histograms within a framework of data flow sketches and samples as described herein, as well as other information and data described in this disclosure, including, in various embodiments, including but not limited to, statistical summaries, sketches, sampling, counting, histograms, their storage, local versions, etc.

[0062] For example, for a dataset generated according to embodiments herein, computing system 302 also includes local statistics 328 that can be stored in memory 306, which may include local histograms, etc., and is used to estimate the cardinality used by query optimizer 314 to generate query plans. In embodiments, local statistics 328 may be stored in storage devices external to computing system 302 and may be shared / used by other systems (e.g., multiple pools of compute / processing nodes as described herein, etc.). In the illustrated embodiments, sketch store 322, sample store 324, and row count store 326 are also maintained or persisted by computing system 302, for example, in memory 306. In embodiments, sketch store 322, sample store 324, and / or row count store 326 include data structures such as lists, tables, arrays, etc., which store their respective associated data (e.g., Count-Min sketches, HyperLogLog sketches, and random dataset value samples) as described in more detail below.

[0063] Network interface 330 can be any type or number of wired and / or wireless network adapters, modems, etc., and its configuration is similar to Figure 2The network interface 226 of system 200 enables system 300 and computing system 302 to communicate with their components within the system, and to communicate with other devices and / or systems over a network, such as... Figure 1 As shown in system 100.

[0064] According to an embodiment, system 300 also includes additional components (not shown for brevity and clarity), including but not limited to components and sub-components of other devices and / or systems described herein, as well as those described below. Figure 9 Those described.

[0065] In this embodiment, DB server 308 is any database server application, and DB server 308 includes DDL engine 310, statistics manager 312, and query optimizer 314. In this embodiment, DDL engine 310 is configured similar to Figure 2 The DDL engine 218 is included. The DDL engine 310 is also configured to receive distributed queries and distributed statistical calculations, or distributed query statistical processing tasks, from the distributed query processor, as described above for... Figure 2 The system 200 describes this. In an embodiment, a distributed query statistics processing task is performed for distributed histogram computation within a framework utilizing a data flow sketch and samples. Query optimizer 314 is configured to generate a query plan for executing an input query (e.g., a distributed query) against a dataset or its distribution, based at least on the input query generated according to local statistics 328 (e.g., a local histogram) and cardinality estimates. In an embodiment, query optimizer 314 is also configured to subsequently optimize the query plan (e.g., locally) based on updated statistics. For example, when a new distribution of the dataset comes online or an existing distribution is added to a new distribution of the dataset, an updated local histogram can be constructed, and query optimizer 314 can be configured to utilize these updated histograms to optimize an existing query plan.

[0066] The statistics manager 312 is configured as a compute node to generate requests / commands for performing distributed statistical computations, and to generate distributed query statistical processing tasks, such as those from... Figure 2 The distributed query processor 208 receives this. In an embodiment, the statistics manager 312 may include a portion of the DDL engine 310. The statistics manager 312 is configured to initiate and / or perform operations to generate sketches and samples associated with the dataset or its distribution, which are used to generate local histograms stored as local statistics 328, as described in more detail below.

[0067] As illustrated, and within the context of computing nodes, the DB server 308 of computing device 302 includes a data transmitter 316, a sketch merger 318, and a sample merger 320. Data transmitter 316 is configured to broadcast row counts for the distribution(s) of the dataset(s) processed by computing device 302 for distributed statistical computation. This broadcast is provided to each other computing node assigned by the control node to perform the distributed statistical computation to be utilized in the sample merging operation and is persistently stored in a row count repository 326. Data transmitter 316 is also configured to provide sketches (such as Count-Min and HyperLogLog sketches) from each computing node to a single computing node for consolidation prior to merging. Sketch merger 318 is configured to merge sketches generated via the statistics manager 312 of computing device 302 and sketches from each other computing node assigned by the distributed query processor for the distribution of the dataset. In this embodiment, sketches such as Count-Min and HyperLogLog sketches are consolidated in a sketch repository 322. Sample merger 320 is configured to merge samples collected for computing device 302 and each other computing node in sample repository 324. In some embodiments, the sketches, samples, and / or row counts described herein may be stored in the same repository, for example, as columns in a single table, such that sketch repository 322, sample repository 324, and / or row count repository 326 comprise a single repository with multiple columns, or are configured as any combination of these repositories. Furthermore, in some embodiments, one type of data, such as row counts, may be stored as a column in another storage type, such as sketch repository 322.

[0068] In embodiments, such as Figure 2 The distributed query processor 208 of system 200 (e.g., in the control node) is also configured to assign a specific computing node to a single computing node illustrated in system 300 to perform sketch merging and sample merging. Thus, each other computing node in the pool of computing nodes assigned for distributed statistical computation of the dataset distribution is configured to provide their respective sketches and samples to the designated computing node for these operations, for example via data transmitter 316.

[0069] Now go to Figure 4According to an example embodiment, a data structure 400 is provided for computation using a distributed histogram within a framework of data flow sketches and samples. Data structure 400 is illustrated as a structure for organizing data associated with a Count-Min sketch. The Count-Min sketch comprises a random mapping of the space (e.g., distribution) of all values ​​in the underlying data domain to “d” cells in the Count-Min sketch, such that all subsequent data operations are deterministic based on this fixed mapping. Given this random mapping, data structure 400 is shown as a two-dimensional array or frequency table of counters with a width “w” and a depth “d”: count max [1;1]...count max [d; w], along with a random sample of values ​​from the dataset or distribution, is used to identify “hit-hit” values, which in statistics are values ​​that repeat with a very high frequency in the dataset. For example, for each value, a lookup is performed in the Count-Min sketch for the estimated frequency of a given value, and values ​​exceeding a certain threshold are identified as “hit-hit” values. While this process may identify false positives, it will not identify false negatives. Furthermore, many “d” hash functions h1,...,h are used. d :D→{1,...,.w} is randomly and uniformly selected from the 2-general hash family. In the embodiment, the sketch counter is initialized to 0, and when scanning the data in the dataset or distribution, for each value 402, the following condition is satisfied as seen during the scan: v∈T, for all hash functions h i (v) i＝1…d Execute iteration and counter max [i,h i [v] is increased by 1. Once initialized, the frequency of the value v can be estimated using a Count-Min plot, as shown in Equation 1:

[0070]

[0071] It should also be noted here that two Count-Min sketches from two different datasets or distributions can be merged to identify the heavy hitter in the merged dataset or distribution, and the statistical guarantee relative to the structure size is the same as that for determining a single Count-Min sketch on the union of the underlying datasets.

[0072] Figure 5A data structure 500 for distributed histogram computation within a framework utilizing a data flow sketch and samples, according to an example embodiment, is illustrated. Data structure 500 is illustrated as a structure for organizing data associated with a HyperLogLog (HLL) sketch. The HLL sketch comprises probabilistic data, and data structure 500 is shown as being generated by scanning the data in the dataset once. The HLL sketch represented in data structure 500 is used to find an approximation DV of the unique value |V| in the dataset with high accuracy. In other words, the HLL sketch provides an observation of the cardinality of a dataset or distribution of uniformly distributed random numbers. This is estimated by calculating the maximum number of leading zeros in the binary representation of each number in the set. If the maximum number of observed leading zeros is n, then the estimate of the number of distinct elements in the set is 2n. n The HLL sketch utilizes a hash function applied to each element in the original dataset or distribution to obtain a uniformly distributed set of random numbers with the same cardinality as the original dataset or distribution. The cardinality of this set of random numbers can then be estimated. In the HLL sketch, the cardinality of the set is estimated by splitting the multiple set of uniformly distributed random numbers into multiple subsets, calculating the maximum number of leading zeros in the numbers of each subset, and combining these estimates for each subset using a harmonic mean. The generation of the HLL sketch involves three main operations: an addition operation to add new elements to the set, a counting operation to obtain the cardinality of the set, and a merging operation to obtain the union of two sets. This is done by storing the cardinality of the set in an array M with a counter or register 502 of size m. Figure 5 Data structure 500 in the diagram illustrates the data in the HLL sketch, and counters or registers 502 are set to 0 in their initial state.

[0073] The append operation involves hashing the input data value v using a hash function h to obtain the first q bits (e.g., q is log2(m)) and the address of the register to be modified. Using the remaining r bits of this value, ρ(w) is calculated, which returns the position of the leftmost "1" in the r bits, i.e., how many leading zeros plus 1s there are in the r bits. The new value of the register is then set to the maximum value between the register's current value and ρ(w) shown in Equation 2:

[0074] x:=h j (v);

[0075] j:=1+ <x1,...,x b >2;

[0076] w:=x q+1 x q+2 ,...;

[0077] M[j] := max(M[j], ρ(w)). Equation 2

[0078] The counting operation involves calculating the harmonic mean of m registers 502 and using a constant to derive an estimate E of the count, as shown in Equation 3:

[0079]

[0080]

[0081] E = α m m 2 Z; Equation 3

[0082] Where n is the cardinality of M, then each subset M j It will have n / m elements, and max x∈Mj ρ(x) is an approximation of log₂(n / m), denoted as "B". B The harmonic mean is mZ, which should be close to n / m. Therefore, m 2 Z should be approximately n. Then, a constant α is introduced. m To correct the problems existing in m due to hash collisions 2 The system multiplication bias in Z. In an embodiment, α can be obtained based on the value of m. m An approximation of that, and in some embodiments with very large cardinal numbers, it is conceivable that when When the base is close to the size of register 502, linear counting can be used.

[0083] The merge operation for HLL(HLL1, HLL2) involves determining the maximum value of each pair of registers 502 (j:1..m) for HLL. union [j] = max(HLL1[j], HLL2[j]). It should also be noted here that two HLL sketches from two different datasets or distributions can be merged to determine how many unique values ​​exist in the merged dataset or distribution, and the statistical guarantees relative to the structure size are the same as those for determining a single HLL sketch on the union of the underlying datasets.

[0084] Figure 6A data structure 600 for distributed histogram computation within a framework utilizing data flow sketches and samples, according to an example embodiment, is shown. Data structure 600 is shown as a structure for organizing data related to sampling values ​​of a dataset or distribution. Data structure 600 is shown as a table, list, array, etc., where V has a size S, i.e., data structure 600 stores S entries 602. Size S can be any value suitable for design considerations and / or any value used to achieve an accurate sample representation of the dataset or distribution. In embodiments, S can be, or can be approximately, 100k tuples of a distribution of a large dataset. In embodiments, Bernoulli sampling is used when the size S of the dataset and / or distribution is known, while in other embodiments, reservoir sampling is used when S is unknown. In the case where the dataset is distributed, samples from each distribution are collected independently in parallel threads and merged to form a single global sample.

[0085] In some embodiments, random numbers, randomly and uniformly selected and associated with each of the dataset / distribution values, are used for sampling. In one example where S is 100k, dataset / distribution values ​​associated with a minimum of 100k random numbers (or a maximum in different embodiments, etc.) can be selected and stored for data structure 600 to achieve random and uniform sampling of dataset / distribution values.

[0086] As noted herein, the embodiments relate to distributed histogram computation within a framework utilizing data flow sketches and samples. For example, as described above, based on a Count-Min sketch, an HLL sketch, and a random sample, the statistics manager 312 is configured to generate a histogram for a dataset or its distribution. That is, using a single scan of the dataset or distribution, statistical summaries are sufficient to subsequently compute the histogram based solely on these summaries. When more data is added (e.g., a new partition or distribution comes online), only the new data is scanned, and then the resulting statistical summaries are subsequently merged with the already computed statistical summaries. In this approach, the underlying statistical summaries must be mergeable, i.e., a statistical summary S(D,∈) is computed that summarizes the dataset or distribution D with error guarantee ∈ from any two summaries S(D1,∈) and S(D2,∈), where S... in Indicates the union of multiple sets.

[0087] In this embodiment, as previously noted, the underlying statistical summaries used are a Count-Min sketch for calculating the “hit-hit” or ultra-high frequency values, an HLL sketch for estimating the number of unique values, and a uniformly random sample for estimating quantiles. While the MaxDiff(V,F) histogram utilizes knowledge of the maximum frequency difference between adjacent values ​​to determine the bin boundaries of the histogram, the embodiments herein alternatively use a variant of the histogram partitioning technique, which maintains the k most common values ​​in T within the singleton (i.e., single-value) bin. This can be well approximated using a combination of samples and Count-Min sketches, and equal-depth partitioning can also be used to partition the remaining range of values ​​for the histogram. Equal-depth partitioning utilizes the calculation of quantiles, which are approximated using the uniformly random sample described herein. In other words, the embodiment provides a statistics manager to generate histograms as a combination of terminal bias histograms and equal-depth histograms, wherein more than one fraction θ constituting a dataset or distribution T is stored in their own singleton bins of the histograms, and the remaining values ​​are partitioned into bins via equal-depth partitioning, because statistical summaries calculated during a single scan of the data are used to estimate the statistical data aggregation required to construct such histograms with high accuracy.

[0088] As described herein, the construction of histograms based on samples and statistical summaries (e.g., sketches) is performed in the embodiments as follows. Determining the singleton bins for ultra-high frequency values ​​includes: determining the ultra-high frequency or "hit" value itself and its frequency. Furthermore, in order to partition the remaining bins, after removing ultra-high frequency values ​​from consideration, the quantile distribution over the remaining dataset or distribution is calculated, and then, for each non-singleton bin, the total number of tuples contained therein and the number of unique values ​​contained in each non-singleton bin are calculated.

[0089] For all values ​​v∈V that are determined to be ultra-high frequency values ​​for T (i.e., for a given θ≤1, e.g., ), frq(v)≥θ·|T| values ​​v), perform iterations on all unique values ​​v in S, and use the generated estimate based on the Count-Min sketch to perform an evaluation on whether The evaluation. Since the property of the Count-Min sketch is: for all values ​​v, This ensures that the test succeeds for all ultra-high frequency (“striker”) values. The set of all strikers found in this way is denoted by H. It is important to note that to ensure all strikers are represented at least once in S, the sample size for the associated dataset or distribution is chosen to be sufficiently large. For example, |S| = θ -1•10 Ensure that each heavy attacker is included in S at least once, with a probability greater than 99.99%. For each heavy attacker v found, construct a histogram box containing only v, i.e., a singleton box, and It is used to estimate the number of tuples in the box, and because it is a singleton box, the box contains a unique value. The number of hitters found in this way is denoted as h. h .

[0090] To partition the residual values ​​into m histogram buckets of equal depth, the embodiment first removes all values ​​corresponding to the hitter from the sample S, and labels the resulting sample S'. Next, the total number of tuples in S' is estimated to be... The quantiles in S' are used to estimate the m corresponding quantiles across the entire data distribution, and the bin boundaries are placed at the quantile values. The total number of tuples in a bin is estimated as... Furthermore, the number of unique values ​​in a given box b is obtained by counting the number of unique values ​​in S' that fall into b and amplifying that estimate. That's how it's estimated.

[0091] Therefore, according to the described embodiments, a histogram from which an accurate cardinality estimate can be derived can be generated.

[0092] Now will describe Figure 7 and Figure 8 . Figure 7 A system flowchart 700 for distributed histogram computation within a framework utilizing data flow sketches and samples, according to an example embodiment, is shown. Figure 8 Its flowchart 800 is shown.

[0093] Figure 7 System flowchart 700 is Figure 2 System 200 and Figure 3 An embodiment of system 300. As illustrated and described in the exemplary figure, system flowchart 700 is configured as an embodiment of a control node represented by a distributed query processor 208 and a DB server 210, and computing nodes represented by computing nodes 716a to 716n. Figure 3 (An embodiment of computing device 302). That is, system flowchart 700 is illustrated as being configured to perform operations on a control node and one or more computing nodes. System flowchart 700 is described below.

[0094] System flowchart 700 exemplarily illustrates the lifecycle of a statistics request that begins at a control node. As noted herein, such a task can be initiated by launching a distributed statistical computation of data columns in a dataset or distribution based on an input query or command. In step 702, the input query or command is received by the DDL engine 218 of DB server 210. DDL engine 218 is configured to provide a command or task initiation instruction to the statistics manager 220 of DB server 210 in step 704 to begin the distributed statistical computation. In some embodiments, statistics manager 220 is configured to initialize the distributed statistical computation, also as described herein.

[0095] In any scenario, the statistics manager 220 provides a request for statistics to the distributed query processor 208 in step 706. The distributed query processor 208 is configured to reliably orchestrate the query task DAG at petabyte level on any number of compute nodes in the pool, and provides implicit fault tolerance guarantees. The request provided in step 706 includes information that enables the distributed query processor 208 to create the desired statistics based on a static DAG generation framework and / or causes the creation of the desired statistics, including but not limited to the name of the table, the column containing the statistics to be generated, the number of distributions in the table, etc.

[0096] Distributed query processor 208 is configured to convert a statistics creation request into a query task DAG and execute it. Deserializer 212 is configured to deserialize the request in step 706 into a representation suitable for input into DAG generator 214, and provides that representation in step 708. DAG generator 214 generates a DAG, such as a static DAG, that outlines the procedural dependencies and operations for performing the distributed query statistics processing task from compute node 716a to compute node 716n. In step 710, the DAG generated by DAG generator 214 is provided to workload manager 216. Workload manager 216 is configured to manage the execution of vertices in the query task DAG in the order of dependencies from compute node 716a to compute node 716n. Workload manager 216 is also configured to assign a compute node as a designated node that will perform the merging and integration of sketches and samples generated by each compute node in the pool. In this embodiment, the distributed portion of the task graph assigned by the workload manager is based on a DAG and includes one or more portions executed in parallel by compute nodes 716a to 716n. In step 712, the distributed portion of the task is provided to compute nodes 716a to 716n, and the resulting statistics are persisted.

[0097] Now for reference Figure 8Flowchart 800 illustrates an embodiment of the DAG generated by DAG generator 214. Flowchart 800 is also an embodiment of step 714 of system flowchart 700.

[0098] As noted, in this embodiment, the distributed workflow for computing column statistics is constructed as a Directed Acyclic Graph (DAG) of operators or tasks. Each vertex of the DAG encapsulates the work or processing performed on one or more compute nodes, while edges capture priority constraints, including data flow dependencies. A vertex can also be considered a task running on one or more compute nodes. In this embodiment, the number of compute nodes scheduling a vertex is controlled by the vertex's Partition Parallelism (DOPP). The vertex's Output Degree (OD) identifies how many compute nodes receive the information generated during the execution of a vertex on a single compute node. Flowchart 800 is described below.

[0099] Flowchart 800 includes multiple vertices, which are described in a dependency order. Vertex 802 represents the result of... Figure 7 The first task performed by compute nodes 716a to 716n is as follows: Vertex 802 is responsible for creating local statistics across compute nodes 716a to 716n, such as Count-Min and HLL sketches, value samples, and row counts for each compute node. The statistical representations generated on each compute node are mapped to the data in the distribution of each particular node (i.e., local), and these statistics are used to generate local histograms associated with the corresponding distributions, as described herein. The Count-Min and HLL sketches and row counts computed during the generation of local histograms are persistently stored in local sketch repositories and local row count repositories—e.g., sketch repository 322 and row count repository 326 of system 300 (compute nodes), and random samples of values ​​are persistently stored in sample repositories—e.g., sample repository 324. In embodiments, these repositories are local tables created in the temporary storage database space of the respective compute nodes. The generated local histograms are stored in local statistics repositories, such as local statistics 328. The local histogram is then later deployed to the query optimizer, such as query optimizer 314 of system 300, which is configured to utilize the local histogram at each compute node to estimate the cardinality in the generation of a query plan for a distributed task of input queries against a dataset distribution. The partition parallelism (DOPP) of vertex 802 is N, meaning the task runs on all compute nodes: compute nodes 716a to 716n in parallel. The OD of vertex 802 is 0, meaning there is no data transfer between compute nodes. In this embodiment, each other vertex of flowchart 800 depends on the completion of the task executed at vertex 802.

[0100] The flowchart proceeds from vertex 802 to vertices 804 and 806, but not to vertex 808, because this vertex depends on the completion of the task performed in vertex 804. For vertex 804, each compute node (compute nodes 716a to 716n) transmits or broadcasts the row count column in its local repository to each other compute node, for example, via data transmitter 316 of system 300. That is, the value for each compute node is written to the local row count repository on each other node in vertex 804. When summed together, the combined row counts provide the total size D of the distributed table for the dataset. Therefore, vertex 804 allows each compute node to independently derive the value of D. In the embodiment, the integrated row count information is stored in the row count repository and is used by the sample merger 320 when merging samples, as described herein, and is also used for histogram construction operations described below and further herein. The DOPP of this vertex is N, and so is the OD.

[0101] In vertex 806, the sketch is integrated at a single compute node, for example via data emitter 316, and in an embodiment, the integrated sketch is then merged at a single compute node, for example via sketch merger 318 of system 300. All rows from sketch repositories across all compute nodes (compute nodes 716a to 716n) are integrated into a global sketch repository (which can be a local table that is part of the sketch repository of a single compute node (e.g., a compute node specified by workload manager 216). A specified compute node can be randomly selected to perform the task of computing the final global histogram. In an embodiment, the global sketch repository has the same schema as any local sketch repository, but contains the union of all rows from all local sketch repositories of compute nodes 716a to 716n.

[0102] As noted above, one property of the statistical summaries maintained during the scanning of a dataset or distribution is that they are mergingable. This property is utilized when data is added to an existing dataset, distribution, table, etc. That is, only summary statistics for new data are built. These new summary statistics are then merged with the existing summary statistics, rather than rescanning the entire dataset. Specifically, merging different summaries uses the following steps. Count-Min sketches with the same depth d and corresponding widths w that are multiples of each other can be generated, and thus the merged Count-Min sketch is determined by adding the counters in cells with the same index. For HLL sketches, in this embodiment, these summaries are generated using the same number of bits used to determine the internal buckets and the same hash function, and thus the merged HLL sketch is calculated by the maximum counter stored in each bucket in the merged HLL sketch that has the same index.

[0103] Vertex 806 can be scheduled in parallel with vertex 804. Vertex 806 has a DOPP of N and an OD of 1 (i.e., to the specified compute node).

[0104] After completing the task at vertex 804, flowchart 800 continues to vertex 808. At vertex 808, samples of values ​​from the distribution are integrated, for example via data transmitter 316, and in an embodiment, the integrated samples are then merged at a single compute node, for example via sample merger 320 of system 300. When scheduled, at vertex 808, each compute node transmits sampled values ​​from its associated distribution to a global sample repository created on a designated compute node, as described similarly for vertex 806. In an embodiment, each processing node may provide a statistically generated (e.g., randomly) set of fewer than all sampled values ​​for integration, or each processing node may provide all sampled values. In an embodiment, the integrated row count in the row count repository (performed in vertex 804) may be used to determine the degree of filtering at the source. The global sample repository may be part of sample repository 324 in system 300. Therefore, each compute node from compute nodes 716a to 716n has a size of S. i A random sample, which is obtained by scanning a size of D i The dataset is used to construct the sample. The probability of each value appearing in this sample is S. i / D i Once vertex 808 is executed, a global sample of size S is generated on the specified computing node, which accepts elements with a probability of S / D, where...

[0105] According to an embodiment, Bernoulli samples are merged (on unique partitions of the data) by resampling all tuples in the sample at corresponding rates. For example, to merge samples S1 and S2 sampled at rates α1 and α2 respectively into a Bernoulli sample S with a sampling rate α (α≤min{α1,α2}), at a rate... Samples are taken from S1, and correspondingly at a rate Samples are taken from S2, and then the union of the results is taken. If there are many samples to be merged, and the size of the underlying dataset, distribution, table, etc., is initially known, the sample number of each sample in the merged sample can be generated using random variables derived from the corresponding probability distributions. This prior knowledge greatly speeds up the merging process because it only requires (randomly) selecting the corresponding number of values ​​from each sample.

[0106] According to an embodiment, during the initial reservoir sampling process, reservoir samples are merged by associating each tuple with a random number between [0,1]. For each sample thus obtained, the tuple associated with the number of samples of the minimum value (e.g., 100k) is retained. Merging is then completed by retaining the number of samples of the tuple associated with the minimum value across all reservoir samples.

[0107] Vertex 808 has a DOPP of N and an OD of 1 (to the specified compute node).

[0108] After the tasks performed at vertices 806 and 808 are completed, the flowchart continues to vertex 810. This final vertex is scheduled on a designated compute node. In vertex 808, in an embodiment, the merging of the integrated sketch and the integrated samples can be performed instead of at vertices 806 and 808 (based on vertex 810 with a DOPP of 1). Furthermore, in vertex 808, as described herein, a histogram construction algorithm is run on the global sample and sketch repository to produce a final global histogram that represents the data distribution across the entire table in the dataset. Vertex 810 has a DOPP of 1 and an OD of 0.

[0109] Now go back to refer to Figure 7 As per system flowchart 700, step 712 is accomplished by storing / retaining local statistics, global statistics, global histograms, etc., that can be retrieved by the distributed query processor 208 from compute nodes 716a to 716n at a local repository (e.g., in local statistics 328) on one or more processing nodes. In step 714, the global histogram / statistics are retrieved by the distributed query processor 208 and provided from the distributed query processor 208 to the DB server 210 for storage in global statistics 224 in association with the dataset to which the statistical task is performed—in an embodiment, the DB server 210 may retrieve the global histogram / statistics directly. The global histogram is then later deployed to a query optimizer, such as query optimizer 222 of system 200, which is configured to utilize the global histogram to estimate the cardinality in the generation of query plans for input queries against the dataset. In this embodiment, it enables other compute node pools and / or other control nodes besides compute nodes 716a to 716n to also access any stored / persistent local and / or global histograms / statistics.

[0110] In this embodiment, the data shuffling capability of a DB server instance running on compute nodes can be used to move data between these nodes to complete tasks for vertices with an OD greater than zero (e.g., an OD of 1). That is, the embodiment provides the ability for any compute node to transfer relational data to a DB server instance running on any other compute node.

[0111] Furthermore, in some embodiments, the row count data collected at each node is deleted at the end of the workflow described for system flowchart 700 and / or flowchart 800. Similarly, in some embodiments, sketches and samples collected during the above process are also deleted, while in other embodiments, sketches and samples are persistently stored for future use to enable incremental statistical maintenance when new data is added to datasets, distributions, tables, etc.

[0112] Figure 9 A flowchart 900 for distributed histogram computation within a framework utilizing data flow sketches and samples, according to an example embodiment, is shown. Further structural and operational examples will be apparent to those skilled in the art based on the following description. According to an embodiment, flowchart 900 can be executed to generate a global histogram for a dataset without sorting the data in the dataset. Flowchart 900 regarding... Figure 2 System 200 and Figure 3 System 300, and Figure 7 Flowchart 700 and Figure 8 The flowchart 800 is described below.

[0113] In the embodiment, steps 902, 904, and / or 906 are performed by, for example, Figure 2 System 200 and / or Figure 7 The distributed query processor 208 and similar control nodes in flowchart 700 are used to execute the query.

[0114] Flowchart 900 begins at step 902. In step 902, a directed acyclic graph (DAG) of the statistical query task associated with the dataset is generated. For example, as described in this paper regarding... Figure 2 and Figure 7 As described, according to the embodiment, deserializer 212 can be configured to deserialize requests for distributed statistical computations on datasets and / or distributions received from DB server 210, and provide its output to DAG generator 214, which is further configured to generate DAGs, such as statistical query task DAGs used to construct histograms. A statistical query task DAG as shown in flowchart 800 can be generated in step 902.

[0115] In step 904, the directed acyclic graph (DAG) of the statistical query task is divided into multiple distributed queries, each associated with a distribution of the dataset. For example, the statistical data query task DAG generated in step 902 above is partitioned by the workload manager 216 of the distributed query processor 208 for assignment to one or more compute nodes, as shown in flowchart 700, and by... Figure 3 The system 300 embodies this.

[0116] In step 906, multiple distributed queries are assigned to multiple processing nodes. For example, the workload manager 216 of the distributed query processor 208 is configured to assign partitions of the statistical query task DAG about the distribution of the dataset on compute nodes 716a to 716n for distributed processing of the statistical query task DAG when determining the global histogram of the dataset.

[0117] In this embodiment, steps 908, 910, 912, 914, and / or step 916 are performed by multiple computing nodes (such as...). Figure 3 System 300 and / or Figure 7 In flowchart 700, the following is used Figure 8 The statistical query task DAG represented in the figure is executed by computing nodes 716a to 716n in a distributed manner. One or more of steps 908, 910, 912, 914 and / or 916 are executed by one or more computing nodes 716a to 716n in parallel or at least partially parallel with each other.

[0118] In step 908, each of the multiple processing nodes generates a data structure that stores first data associated with the corresponding frequency of a value, second data associated with the number of unique values, and third data associated with random sampling of these values. For example, computing nodes 716a to 716n (which are assigned by the control node to the respective parts of the partitioned statistical query task DAG in step 906) are configured to generate a data structure that stores first data associated with the corresponding frequency of a value, second data associated with the number of unique values, and third data associated with random sampling of these values ​​corresponding to the distribution of the dataset for each of the computing nodes 716a to 716n. Such a data structure and data can correspond to the above. Figure 4 , Figure 5 and / or Figure 6 Those described herein. In an embodiment, step 908 may correspond to... Figure 8 Step 802, and can be made by Figure 3 The statistics manager 312 of system 300 is executed in whole or in part.

[0119] In step 910, each of the plurality of processing nodes constructs a histogram based on the first, second, and third data of the distribution processed by each of the plurality of processing nodes. For example, as described herein, without performing data sorting in the corresponding distribution, computing nodes 716a to 716n can each be configured to generate local histograms of their respective distributions based on the first, second, and third data. In an embodiment, step 910 may correspond to Figure 8 Step 802, and can be made by Figure 3 The statistics manager 312 of system 300 is executed in whole or in part.

[0120] In step 912, each of the plurality of processing nodes transmits a count value indicating the number of rows in the corresponding distribution processed by each of the plurality of processing nodes to each of the other processing nodes. That is, each assigned computing node from computing nodes 716a to 716n provides its corresponding row count to each of the other computing nodes for use in subsequent query processing. In an embodiment, step 912 may correspond to... Figure 8 Step 804, and can be made by Figure 3 The statistics manager 312 and / or data transmitter 316 of the system 300 are executed in whole or in part.

[0121] In step 914, a first processing node among multiple processing nodes merges a data structure containing first data generated by the multiple processing nodes and a data structure containing second data generated by the multiple processing nodes. For example, in an embodiment, one of computing nodes 716a to 716n may be assigned by workload manager 216 to perform the merging for the first data and for the second data in step 914. Merging the first data and merging the second data distributed from each computing node provides a single set of merged first data and merged second data that are utilized in the global histogram constructing the entire dataset. In an embodiment, step 914 may correspond to Figure 8 Step 806, and can be made by Figure 3 The sketch merger 318 of system 300 is executed in whole or in part.

[0122] In step 916, a second processing node among multiple processing nodes merges a data structure containing third data generated by the multiple processing nodes. For example, one of the computing nodes 716a through 716n can be assigned by workload manager 216 to perform the merging for the third data in step 916. Merging the third data distributed from each computing node provides a single set of merged third data that is utilized in the global histogram constructing the entire dataset. In an embodiment, step 916 may correspond to Figure 8 Step 808, and can be made by Figure 3 The sample merger 320 of system 300 is executed in whole or in part.

[0123] In step 918, a third processing node among multiple processing nodes constructs a global histogram of the dataset based on the merged first data, merged second data, and merged third data. For example, as described herein, one of the computing nodes 716a through 716n can be assigned to perform this task by workload manager 216 and can be configured to perform the construction of a global histogram of the entire dataset represented by all distributions based on the merged first data, merged second data, and merged third data. In an embodiment, step 916 may correspond to... Figure 8 Step 810. According to an embodiment, the global histogram constructed in step 918 is deployed to... Figure 2 The query optimizer 222 of the DB server 210 in the system 200 is used to generate a query plan for the input query of the dataset by utilizing the cardinality estimate from the constructed global histogram.

[0124] In an embodiment, the same processing / compute nodes from compute node 716a to compute node 716n may execute steps 914, 916, and / or 918.

[0125] Figure 10 A flowchart 1000 for distributed histogram computation within a framework utilizing data flow sketches and samples, according to an example embodiment, is shown. Further structural and operational examples will be apparent to those skilled in the art based on the following description. According to an embodiment, flowchart 1000 can be executed to generate local histograms of multiple distributions of a dataset without performing sorting of the data in the distributions. Flowchart 1000 regarding... Figure 3 System 300 and Figure 4 Data Structures 400 Figure 5 Data Structures 500 and Figure 6 The data structure 600 is described as follows.

[0126] Flowchart 1000 begins at step 1002. In step 1002, information associated with the dataset is determined based on a scan of the dataset. For example, Figure 3 The DB server 308 of system 300 is configured to determine dataset / distribution information by scanning the data therein, for example, via DDL engine 310 and / or statistics manager 312. DB server 308 is configured to scan data rows and / or columns of the dataset / distribution. During data scanning, data structures can be generated and updated using the information determined in the scan, as described below.

[0127] In step 1004, a first data structure is generated based on this information. This first data structure stores first data associated with the corresponding frequencies of values ​​in the dataset. For example, a DB server 308, via a DDL engine 310 and / or a statistics manager 312, is configured to generate a first data structure that stores information related to the frequencies of values ​​in the dataset / distribution, as described above. Figure 4 The data structure described in 400.

[0128] In step 1006, a second data structure is generated based on this information. This second data structure stores second data associated with the number of unique values ​​in the dataset. For example, a DB server 308, such as via a DDL engine 310 and / or a statistics manager 312, is configured to generate a second data structure that stores information related to the number of unique or singular values ​​in the dataset / distribution, as described above. Figure 5 The data structure described in 500.

[0129] In step 1008, a third data structure is generated based on this information. This third data structure stores third data associated with random samples of values ​​in the dataset. For example, a DB server 308, such as via a DDL engine 310 and / or a statistics manager 312, is configured to generate a third data structure that stores information associated with random samples of values ​​in the dataset / distribution, as described above. Figure 6 The data structure described in 600.

[0130] In step 1010, a subset of values ​​in the dataset is determined based at least on the first data, each value in the subset having a corresponding frequency that satisfies or exceeds a threshold. In an embodiment, a DB server 308, for example via a DDL engine 310 and / or a statistics manager 312, is configured to identify one or more hitters, i.e., values ​​in the dataset / distribution that repeat at an extremely high frequency, where the frequency is represented by a threshold. That is, when a value is repeated a specified number of times in the dataset / distribution, it is considered an extremely high-frequency value and is placed in its own bin, as noted herein. In an embodiment, third data associated with a random sampling of values ​​in the dataset, as described above in step 1008, is also used to determine the subset of values ​​in step 1010, i.e., to determine values ​​in the dataset / distribution that repeat at an extremely high frequency.

[0131] In step 1012, a histogram is constructed based on the first, second, and third data. This histogram includes individual bins for each value in a subset of the dataset, and additional bins are included in the histogram by assigning values ​​to additional bins via quantile distributions for each other value in the dataset not included in a subset. These additional bins have equal-depth partitioning. For example, a local histogram for the distribution can be constructed by the statistics manager 312 in step 1012.

[0132] In step 1014, the histogram is deployed to the query optimizer, which is configured to generate a query plan based on the cardinality estimate associated with the histogram. For example, a local histogram constructed by the statistics manager 312 in step 1012 is deployed to... Figure 3 The query optimizer 314 of the DB server 308 in the system 300 generates a query plan for the input query based on the distribution of the dataset by utilizing cardinality estimates from the constructed local histogram.

[0133] III. Example Computing Device Implementation

[0134] The embodiments described herein can be implemented in hardware or in combination with software and / or firmware. For example, the embodiments described herein can be implemented as computer program code / instructions configured to execute in one or more processors and stored in a computer-readable storage medium. Alternatively, the embodiments described herein can be implemented as a hardware logic / circuit system.

[0135] As indicated herein, the described embodiments include, but are not limited to, those described herein. Figure 1 System 100 in Figure 2 System 200 and Figure 3The system 300 and any of its components and / or subcomponents, as well as any data structures, and the operations and flowcharts / flowcharts described herein and / or other examples described herein may be implemented in hardware or in any combination with software and / or firmware, including computer program code implemented as being configured to execute in one or more processors and stored in a computer-readable storage medium, or implemented as hardware logic / circuit systems, such as those implemented together on a system-on-a-chip (SoC), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), trusted platform module (TPM), etc. An SoC may include an integrated circuit chip that includes one or more of the following: a processor (e.g., a microcontroller, microprocessor, digital signal processor (DSP), etc.), memory, one or more communication interfaces, and / or other circuitry and / or embedded firmware that performs its functions.

[0136] The embodiments described herein can be implemented in one or more computing devices similar to mobile systems and / or computing devices in static or mobile computer embodiments, including one or more features of the mobile systems and / or computing devices described herein, as well as alternative features. The description of the computing devices provided herein is for illustrative purposes and is not intended to be limiting. As will be known to those skilled in the art, embodiments can be implemented in other types of computer systems.

[0137] Figure 11 Exemplary implementations of computing device 1100 in which embodiments may be implemented are depicted. For example, the embodiments described herein may be implemented in one or more computing devices or systems similar to computing device 1100 or multiple instances of computing device 1100 in stationary or mobile computer embodiments, including one or more features and / or alternative features of computing device 1100. The description of computing device 1100 provided herein is provided for illustrative purposes and is not intended to be limiting. As will be known to those skilled in the art, embodiments may be implemented in other types of computer systems, servers and / or clusters, etc.

[0138] like Figure 11As shown, computing device 1100 includes one or more processors, referred to as processor circuitry 1102, system memory 1104, and bus 1106, which couples various system components, including system memory 1104, to processor circuitry 1102. Processor circuitry 1102 is electrical and / or optical circuitry implemented as a central processing unit (CPU), microcontroller, microprocessor, and / or other physical hardware processor circuitry in one or more physical hardware circuitry elements and / or integrated circuit devices (semiconductor material chips or dies). Processor circuitry 1102 can execute program code stored in a computer-readable medium, such as program code for operating system 1130, program code for application program 1132, program code for other program 1134, etc. Bus 1106 represents one or more of several types of bus architectures, including memory bus or memory controller, peripheral bus, accelerated graphics port, and processor or local bus using any of the various bus architectures. System memory 1104 includes read-only memory (ROM) 1108 and random access memory (RAM) 1110. The Basic Input / Output System 1112 (BIOS) is stored in ROM 1108.

[0139] The computing device 1100 also includes one or more of the following drives: a hard disk drive 1114 for reading from and writing to a hard disk drive; a disk drive 1116 for reading from and writing to a removable disk drive 1118; and an optical disk drive 1120 for reading from and writing to a removable optical disk drive 1122, such as a CD-ROM, DVD-ROM, or other optical media. The hard disk drive 1114, disk drive 1116, and optical disk drive 1120 are connected to the bus 1106 via a hard disk drive interface 1124, a disk drive interface 1126, and an optical drive interface 1128, respectively. The drives and their associated computer-readable media provide non-volatile storage for computer-readable instructions, data structures, program modules, and other data for use with a computer. Although hard disks, removable disks, and removable optical disks have been described, other types of hardware-based computer-readable storage media, such as flash memory cards, digital video disks, RAM, ROM, and other hardware storage media, may also be used to store data.

[0140] Multiple program modules may be stored on a hard disk, magnetic disk, optical disk, ROM, or RAM. These programs include an operating system 1130, one or more application programs 1132, other programs 1134, and program data 1136. Application programs 1132 or other programs 1134 may include, for example, computer program logic (e.g., computer program code or instructions) for implementing the embodiments described herein, such as, but not limited to, [other components]. Figure 1System 100 in Figure 2 System 200 and Figure 3 The system 300 and any of its components and / or subcomponents, as well as any data structures, and the operations of the flowcharts / flowcharts described herein, including portions thereof, and / or other examples described herein.

[0141] Users can input commands and information into computing device 1100 using input devices such as keyboard 1138 and pointing device 1140. Other input devices (not shown) may include microphone, joystick, game controller, satellite dish, scanner, touchscreen and / or touchpad, voice recognition system for receiving voice input, gesture recognition system for receiving gesture input, and so on. These and other input devices are often connected to processor circuitry 1102 via serial port interface 1142 coupled to bus 1106, but may also be connected via other interfaces such as parallel port, game port, or Universal Serial Bus (USB).

[0142] Display screen 1144 is also connected to bus 1106 via an interface such as video adapter 1146. Display screen 1144 may be external to computing device 1100 or incorporated within computing device 1100. Display screen 1144 may display information and serve as a user interface for receiving user commands and / or other information (e.g., via touch, finger gestures, virtual keyboard, etc.). In addition to display screen 1144, computing device 1100 may include other peripheral output devices (not shown), such as speakers and printers.

[0143] Computing device 1100 is connected to network 1148 (e.g., the Internet) via an adapter or network interface 1150, modem 1152, or other means for establishing communication on the network. Modem 1152 can be internal or external, and it can be connected to bus 1106 via serial port interface 1142, such as... Figure 11 As shown, it can also be connected to bus 1106 using another interface type, including a parallel interface.

[0144] TPM 1154 may be connected to bus 1106 and may be any TPM embodiment, as will be understood by those skilled in the art who benefit from this disclosure. For example, TPM 1154 may be configured to perform one or more functions or operations of a TPM used in various embodiments herein.

[0145] As used herein, the terms “computer program media,” “computer-readable media,” “computer-readable storage media,” and “computer-readable storage device,” etc., are used to refer to physical hardware media. Examples of such physical hardware media include hard disk drives 1114, removable disks 1118, removable optical discs 1122, such as RAM, ROM, flash memory cards, digital video disks, compact disks, MEM, nanotechnology-based storage devices, and other types of physical / tangible hardware storage media (including…). Figure 11 Hard disks associated with other physical hard disk media, such as memory 1120. Such computer-readable media and / or storage media are distinct from and do not overlap with communication media and propagation signals (excluding communication media and propagation signals). Communication media embody computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves. The term "modulated data signal" means having one or more signal characteristics that are set or altered in a way to encode information in a signal. By way of example and not limitation, communication media include wireless media such as acoustic, RF, infrared, and other wireless media, as well as wired media. Embodiments also relate to such communication media that are separate from and do not overlap with embodiments relating to computer-readable storage media.

[0146] As noted above, computer programs and modules (including application 1132 and other programs 1134) may be stored on a hard disk, magnetic disk, optical disk, ROM, RAM, or other hardware storage media. Such computer programs may also be received via network interface 1150, serial port interface 1142, or any other interface type. When executed or loaded by an application, such computer programs enable computing device 1100 to implement the features of the embodiments discussed herein. Therefore, such computer programs represent the controller of computing device 1100.

[0147] The embodiments also relate to computer program products comprising computer code or instructions stored on any computer-readable medium or computer-readable storage medium. Such computer program products include hard disk drives, optical disk drives, storage device packages, memory sticks, memory cards, and other types of physical storage hardware.

[0148] IV. Other Examples and Advantages

[0149] As described, systems and devices embodying the techniques described herein can be configured and enabled in various ways to perform their respective functions for distributed histogram computation within a framework utilizing data flow sketches and samples. In embodiments, one or more steps or operations of any flowchart and / or flowchart described herein may not be performed. Furthermore, steps or operations other than those in any flowchart and / or flowchart described herein may be performed, or steps or operations other than those in any flowchart and / or flowchart described herein may be performed in lieu of those described herein. Additionally, in examples, one or more operations of any flowchart and / or flowchart described herein may be performed out of order, instead of sequentially, or partially (or completely) simultaneously with each other or with other operations.

[0150] As described herein, systems, devices, components, etc., configured to perform functions and / or operations in embodiments are also contemplated to perform such functions and / or operations.

[0151] The embodiments described herein enable high-speed statistical collection during the scanning of underlying data while maintaining low memory footprint. This achieves processing efficiency while utilizing the types of statistics described herein, even with multiple threads scanning data in parallel. In other words, the collection speed and low memory footprint achieved through the described embodiments cannot be achieved using other types of statistics. Furthermore, the embodiments described herein implement fast histogram construction, including when additional data partitions are brought online incrementally, without requiring a complete (re)scan of the data. In other words, the statistics utilized in the embodiments are incrementally mergeable, where the union of statistics built individually on two datasets has equivalent accuracy to similar statistics built on the union of the two datasets. Moreover, since not all histogram types can be easily computed based on the statistical types described herein—for example, the MaxDiff(V,F) histogram cannot be accurately computed in this way—the embodiments described herein also provide the generation of hybrid histogram variants, supported by streaming statistics, while still retaining end-to-end performance comparable to existing histograms used in previous implementations of DB servers. In other words, the embodiments described herein utilize a unique combination of data flow sketches and samples for distributed histogram computation at both the global and distributed levels of a database system. This provides robust query plans based on the generated histograms and resource efficiency that were previously unavailable for software-based services, query hosts, or data warehouses, not to mention the specific embodiments described herein.

[0152] The additional examples and embodiments described in this section may be applied to the examples disclosed in any other part or section of this disclosure.

[0153] The embodiments described herein provide systems, apparatus, and methods for performing distributed histogram computation within a framework utilizing data flow sketches and samples. For example, a system is described herein. According to an embodiment, the system is enabled and configured for performing distributed histogram computation within a framework utilizing data flow sketches and samples. The system includes a control node and multiple processing nodes. The control node is configured to generate a directed acyclic graph of statistical query tasks associated with a dataset and to assign multiple distributed queries to the multiple processing nodes, each distributed query being associated with a distribution of the dataset and generated by partitioning the directed acyclic graph of statistical query tasks. One or more of the multiple processing nodes operate in parallel, at least partially for each distribution, and the multiple processing nodes are configured to: generate a data structure by each of the multiple processing nodes storing first data associated with the corresponding frequency of values, second data associated with the number of unique values, and third data associated with random sampling of these values, and to construct a histogram by each of the multiple processing nodes based on the first, second, and third data of the distribution processed by each of the multiple processing nodes. The first processing node among multiple processing nodes is configured to merge first data and second data generated by multiple processing nodes, and the second processing node among multiple processing nodes is configured to merge third data generated by multiple processing nodes. The third processing node among multiple processing nodes is configured to construct a global histogram of the dataset based on the merged first data, merged second data, and merged third data.

[0154] In one embodiment of the system, one or more of the plurality of processing nodes are at least partially parallel on each distribution, and the plurality of processing nodes are configured such that each of the plurality of processing nodes transmits a count value to each of the other processing nodes after the construction, the count value indicating the number of rows in the corresponding distribution processed by each of the plurality of processing nodes.

[0155] In one embodiment of this system, the first processing node, the second processing node, and the third processing node are the same processing node.

[0156] In one embodiment of the system, after constructing the global histogram, the control node is configured to: generate a query plan for an input query of the dataset, the query plan having a cardinality estimate based at least on the global histogram; generate multiple distributed queries, each associated with a distribution of the dataset, based at least on the input query and the estimated cardinality; assign the multiple distributed queries to one or more of a plurality of processing nodes; and return query results based on the execution of the multiple distributed queries.

[0157] In one embodiment of the system, the control node is configured to receive input queries before constructing the global histogram.

[0158] In one embodiment of the system, the construction of the global histogram is performed without sorting the dataset and is not based on sorting the dataset.

[0159] In one embodiment of the system, one or more of the plurality of processing nodes operate in parallel at least partially for each distribution, and in order to construct a histogram, the plurality of processing nodes are configured to: determine a subset of values ​​in the distribution based on at least first data, each value in the subset having a corresponding frequency that satisfies or exceeds a threshold, and be configured to construct a histogram that includes a separate bin for each value in the subset, and to construct the histogram that includes a plurality of additional bins having equal-depth partitioning in the histogram by assigning values ​​to additional bins via quantile distributions for each other value in the distribution that is not included in the subset.

[0160] This document also describes a method executed by a computing system. According to an embodiment, the method is used for distributed histogram computation within a framework utilizing data flow sketches and samples. The method includes, executed by a control node: generating a directed acyclic graph of statistical query tasks associated with a dataset, dividing the directed acyclic graph of statistical query tasks into multiple distributed queries, each associated with a distribution of the dataset, and assigning the multiple distributed queries to multiple processing nodes. The method further includes: performing, at least partially in parallel, operations on each distribution by one or more of the multiple processing nodes: generating a data structure by each of the multiple processing nodes, the data structure storing first data associated with the corresponding frequency of values, second data associated with the number of unique values, and third data associated with random sampling of these values; constructing a histogram by each of the multiple processing nodes based on the first data, second data, and third data of the distribution processed by each of the multiple processing nodes; transmitting a count value to each of the other processing nodes, the count value indicating the number of rows in the corresponding distribution processed by each of the multiple processing nodes; merging the data structure having the first data generated by the multiple processing nodes and the data structure having the second data generated by the multiple processing nodes by the first processing node; and merging the data structure having the third data generated by the multiple processing nodes by the second processing node. The method further includes: constructing a global histogram of the dataset by a third processing node based on the merged first data, merged second data, and merged third data.

[0161] In one embodiment of the method, the first processing node, the second processing node, and the third processing node are the same processing node.

[0162] In one embodiment, the method further includes: after constructing the global histogram, the control node performs: generating a query plan for an input query of the dataset, the query plan having a cardinality estimate based at least on the global histogram; and generating a plurality of distributed queries, each associated with a distribution of the dataset, based at least on the input query and the estimated cardinality.

[0163] In one embodiment, the method further includes: after generating a plurality of distributed queries that are respectively associated with the distribution of the dataset, the control node performs: assigning the plurality of distributed queries to one or more of a plurality of processing nodes; and returning query results based on the execution of the plurality of distributed queries.

[0164] In one embodiment, the method further includes receiving an input query before constructing a global histogram.

[0165] In one embodiment of the method, the construction of the global histogram is performed without sorting the dataset and is not based on sorting the dataset.

[0166] This paper also describes another method performed by a computing system. According to an embodiment, this method is used for distributed histogram computation within a framework utilizing data flow sketches and samples. The method includes: performing histogram construction by determining information associated with the dataset based on a scan of the dataset; generating a first data structure based on this information, the first data structure storing first data associated with corresponding frequencies of values ​​in the dataset; generating a second data structure based on this information, the second data structure storing second data associated with the number of unique values ​​in the dataset; generating a third data structure based on this information, the third data structure storing third data associated with random sampling of these values ​​in the dataset; and constructing a histogram based on the first, second, and third data. The method also includes deploying the histogram to a query optimizer configured to generate a query plan based on a cardinality estimate associated with the histogram.

[0167] In one embodiment of the method, the construction of the histogram is performed without sorting the dataset and is not based on sorting the dataset.

[0168] In one embodiment, the method further includes: determining a subset of values ​​in the dataset based on first data, each value in the subset having a corresponding frequency that satisfies or exceeds a threshold, and constructing a histogram includes: including a separate bin for each value in the subset in the histogram.

[0169] In one embodiment, the method further includes constructing additional bins with equal-depth partitioning in the histogram by assigning values ​​to additional bins via a quantile distribution for each other value in the dataset that is not included in a subset.

[0170] In one embodiment of the method, the dataset includes one of a plurality of distributions of a larger dataset, histogram construction is performed for each of the plurality of distributions, and the cardinality estimate is a cardinality estimate of the larger dataset and is also based on the histogram of each of the other distributions among the plurality of distributions.

[0171] In one embodiment of the method, the first data structure represents a minimum count sketch, the second data structure represents a superlog log sketch, and the third data structure represents a list of data values ​​from the dataset and corresponding associated random values.

[0172] In one embodiment of the method, after constructing the histogram, additional data is added to the dataset. The method further includes: determining additional information associated with the additional data based on a scan of the additional data; generating a fourth data structure based on the additional information, the fourth data structure storing fourth data associated with the corresponding frequencies of values ​​in the additional data; generating a fifth data structure based on the additional information, the fifth data structure storing fifth data associated with the number of unique values ​​in the additional data; generating a sixth data structure based on the additional information, the sixth data structure storing sixth data associated with random sampling of these values ​​in the additional data; merging the first data structure with the fourth data structure; merging the second data structure with the fifth data structure; and merging the third data structure with the sixth data structure; constructing an updated histogram based on the merging; and deploying an updated cardinality estimate of the dataset to the query optimizer based on the updated histogram.

[0173] The embodiments described herein also provide a computer-readable storage medium storing program instructions that, when executed by one or more processing devices, perform a method for distributed histogram computation within a framework utilizing the data flow sketches and samples described herein.

[0174] V. Conclusion

[0175] While various embodiments of the disclosed subject matter have been described above, it should be understood that they are presented by way of example only and not as limitations. Those skilled in the art will understand that various changes in form and detail may be made therein without departing from the spirit and scope of the embodiments as defined by the appended claims. Therefore, the breadth and scope of the disclosed subject matter should not be limited by any of the exemplary embodiments described above, but should be defined solely by the appended claims and their equivalents.

Claims

1. A system comprising: A control node, which is hardware-based and configured to: Generate a directed acyclic graph for statistical query tasks associated with the dataset; as well as Multiple distributed queries are assigned to multiple hardware-based processing nodes, each of which is associated with the distribution of the dataset and generated by partitioning the statistical query task into a directed acyclic graph. One or more of the plurality of processing nodes are configured to operate at least partially in parallel for each distribution, and the plurality of processing nodes are configured as follows: Each of the plurality of processing nodes generates a data structure, which stores first data associated with the corresponding frequency of a value, second data associated with the number of unique values, and third data associated with random sampling of the value. Each of the plurality of processing nodes constructs a histogram based on the first data, the second data, and the third data, which are distributed and processed by each of the plurality of processing nodes. The first processing node among the plurality of processing nodes merges the first data generated by the plurality of processing nodes and the second data generated by the plurality of processing nodes; as well as The third data generated by the plurality of processing nodes is merged by the second processing node among the plurality of processing nodes; as well as The third processing node among the plurality of processing nodes is configured to construct a global histogram of the dataset based on the merged first data, the merged second data, and the merged third data.

2. The system of claim 1, wherein one or more of the plurality of processing nodes are at least partially parallel on each distribution, and the plurality of processing nodes are configured to: Each of the plurality of processing nodes transmits a count value to each of the other processing nodes after the construction, the count value indicating the number of rows in the corresponding distribution processed by each of the plurality of processing nodes.

3. The system according to claim 1, wherein the first processing node, the second processing node and the third processing node are the same processing node.

4. The system of claim 1, wherein after constructing the global histogram, the control node is configured as follows: Generate a query plan for an input query against the dataset, the query plan having a cardinality estimate based at least on the global histogram; Based at least on the input query and the estimated cardinality, generate multiple distributed queries, each associated with the distribution of the dataset. The plurality of distributed queries are assigned to one or more of the plurality of processing nodes; as well as Based on the execution of the multiple distributed queries, the query results are returned.

5. The system of claim 4, wherein the control node is configured to: The input query is received before the global histogram is constructed.

6. The system of claim 1, wherein the construction of the global histogram is performed without sorting the dataset, and the construction of the global histogram is not based on sorting the dataset.

7. The system of claim 1, wherein one or more of the plurality of processing nodes are at least partially parallel for each distribution, and the plurality of processing nodes are configured to: A subset of values ​​in the distribution is determined based at least on the first data, each value in the subset having a corresponding frequency that satisfies or exceeds a threshold; and The histogram is constructed such that each value in the subset is represented by a separate bin, and the histogram is further constructed to include a plurality of additional bins having equal-depth partitioning in the histogram by assigning values ​​to additional bins via quantile distribution for each other value in the distribution that is not included in the subset.

8. A method performed by a computing system, the method comprising: Executed by the control node: Generate a directed acyclic graph for statistical query tasks associated with the dataset; The directed acyclic graph of the statistical query task is divided into multiple distributed queries, each associated with the distribution of the dataset. as well as The multiple distributed queries are assigned to multiple processing nodes; The process is executed, at least partially in parallel, by one or more of the plurality of processing nodes on each distribution: Each of the plurality of processing nodes generates a data structure, which stores first data associated with the corresponding frequency of a value, second data associated with the number of unique values, and third data associated with random sampling of the value. Each of the plurality of processing nodes constructs a histogram based on the first data, the second data, and the third data distributed by each of the plurality of processing nodes; Each of the plurality of processing nodes transmits a count value to each of the other processing nodes, the count value indicating the number of rows in the corresponding distribution processed by each of the plurality of processing nodes; A data structure containing the first data generated by the plurality of processing nodes and a data structure containing the second data generated by the plurality of processing nodes are merged by a first processing node among the plurality of processing nodes; as well as A data structure containing the third data generated by the plurality of processing nodes is merged by a second processing node among the plurality of processing nodes; as well as A third processing node among the plurality of processing nodes constructs a global histogram of the dataset based on the merged first data, the merged second data, and the merged third data.

9. The method according to claim 8, wherein the first processing node, the second processing node, and the third processing node are the same processing node.

10. The method of claim 8, further comprising: After the global histogram is constructed, the control node performs the following: Generate a query plan for an input query against the dataset, the query plan having a cardinality estimate based at least on the global histogram; as well as Based at least on the input query and the estimated cardinality, multiple distributed queries are generated, each associated with the distribution of the dataset.

11. The method of claim 10, further comprising: After the generation of the plurality of distributed queries, each associated with the distribution of the dataset, is performed by the control node: The plurality of distributed queries are assigned to one or more of the plurality of processing nodes; as well as Based on the execution of the multiple distributed queries, the query results are returned.

12. The method of claim 10, further comprising: The input query is received before the global histogram is constructed.

13. The method of claim 8, wherein the construction of the global histogram is performed without sorting the dataset, and the construction of the global histogram is not based on sorting the dataset.

14. A method performed by a computing system, the method comprising: The histogram is constructed using the following methods: Information associated with the dataset is determined based on a scan of the dataset; A first data structure is generated based on the information, and the first data structure stores first data associated with the corresponding frequencies of values ​​in the dataset; A second data structure is generated based on the information, and the second data structure stores second data associated with the number of unique values ​​in the dataset; A third data structure is generated based on the information, and the third data structure stores third data associated with random sampling of the values ​​in the dataset; as well as The histogram is constructed based on the first data, the second data, and the third data; as well as The histogram is deployed to a query optimizer, which is configured to generate a query plan based on a cardinality estimate associated with the histogram.

15. The method of claim 14, wherein constructing the histogram is performed without sorting the dataset and is not based on sorting the dataset.

16. The method of claim 14, further comprising: Based on the first data, a subset of the values ​​in the dataset is determined, wherein each value in the subset has a corresponding frequency that satisfies or exceeds a threshold; Constructing the histogram includes including a separate bin for each value in the subset within the histogram.

17. The method of claim 16, further comprising: Multiple additional bins are constructed by assigning values ​​to additional bins via a quantile distribution for each other value in the dataset that is not included in the subset, the multiple additional bins having equal-depth partitioning in the histogram.

18. The method of claim 14, wherein the dataset comprises one of a plurality of distributions of a larger dataset; The construction of the histogram is performed for each of the plurality of distributions; and The cardinality estimate is a cardinality estimate of the larger dataset and is also based on the histogram of each of the plurality of distributions.

19. The method of claim 14, wherein the first data structure represents a minimum count sketch, the second data structure represents a superlog log sketch, and the third data structure represents a list of data values ​​from the dataset and corresponding associated random values.

20. The method of claim 14, wherein after constructing the histogram, additional data is added to the dataset, the method further comprising: Additional information associated with the additional data is determined based on the scanning of the additional data; A fourth data structure is generated based on the additional information, the fourth data structure storing fourth data associated with the corresponding frequencies of the values ​​in the additional data; A fifth data structure is generated based on the additional information, the fifth data structure storing fifth data associated with the number of unique values ​​in the additional data; A sixth data structure is generated based on the additional information, the sixth data structure storing sixth data associated with random samples of the values ​​in the additional data; The first data structure is merged with the fourth data structure, the second data structure is merged with the fifth data structure, and the third data structure is merged with the sixth data structure. The updated histogram is constructed based on the merging; as well as Based on the updated histogram, the updated cardinality estimate of the dataset is deployed to the query optimizer.

Images