Method and system for shuffle-free reclustering of clustered tables

Abstract

A method (400) for shuffle-free reclustering of a clustered table (159) includes receiving first and second groups (158) of clustered data blocks (152) ordered by cluster key values (154). The range of cluster key values for one or more data blocks in the second group overlaps the range of cluster key values for data blocks in the first group. The method also includes generating split points (310) for dividing the first and second groups of clustered data blocks into a third group. Each data block in the third group includes a range of cluster key values that does not overlap any other data block in the third group. Each split point defines an upper or lower bound of the range of cluster key values for a data block in the third group.

Description

Technical Field

[0001] This disclosure relates to shuffle-free re-clustering of cluster tables. Background Technology

[0002] As cloud storage becomes increasingly popular, clustering data structures (such as column-oriented data stores) are being used more and more to reduce query costs and improve query performance by clustering data into non-overlapping blocks. Because many tables are so large that they need to be partitioned across many different servers, data blocks are often sorted by clustering keys to locate related data. Each data block consists of a series of clustering key values. Typically, the range of clustering key values ​​associated with each data block does not overlap with any other data blocks within the clustered data block. When new data is appended to a clustered data block, the range of clustering key values ​​in the new data block will often overlap with the original data block, and the data blocks must be sorted again to maintain performance. Summary of the Invention

[0003] One aspect of this disclosure provides a method for shuffling-free re-clustering of a clustering table. The method includes receiving, at data processing hardware, a first set of clustered data blocks sorted by clustering key values. Each clustered data block in the first set may include a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the first set. The method also includes receiving, at data processing hardware, a second set of clustered data blocks sorted by clustering key values. Each clustered data block in the second set may include a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the second set. The method further includes generating one or more split points by the data processing hardware to divide the first and second sets of clustered data blocks into a third set of clustered data blocks. The method also includes dividing the first and second sets of clustered data blocks into the third set of clustered data blocks by the data processing hardware using the one or more generated split points. Each clustered data block in the third group of clustered data blocks may include a corresponding range of cluster key values ​​that do not overlap with any of the ranges of cluster key values ​​of other clustered data blocks in the third group of clustered data blocks. Each of the one or more generated split points defines an upper or lower bound for the corresponding range of cluster key values ​​for one of the clustered data blocks in the third group of clustered data blocks.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, a columnar database table stores the first and second groups of clustered data blocks. The number of the one or more split points generated may be based on the number of data blocks in the first and second groups of clustered data blocks and the size of each data block. In some examples, the first and second groups of clustered data blocks are divided into a third group of clustered data blocks without performing any shuffling operation on the data blocks in the first and second groups of clustered data blocks.

[0005] In some examples, the method further includes identifying which clustering key values ​​in the first and second sets of clustered data blocks fall between adjacent split points, and for each clustered data block in the third set of clustered data blocks, merging the identified clustering key values ​​falling within the corresponding adjacent split points. At least one clustered data block in the third set of clustered data blocks may include a portion of a corresponding range from one of the data blocks in the first or second set of clustered data blocks that does not overlap with any of the corresponding ranges of other clustered data blocks in the other set of clustered data blocks in the first or second set of clustered data blocks.

[0006] Additionally, one cluster data block in the third group may include a portion of a corresponding range of one data block from the first or second group and one data block from another group of cluster data blocks within the first or second group. At least one cluster data block in the third group may include a portion of a corresponding range of two data blocks from the first or second group and one data block from another group of cluster data blocks within the first or second group.

[0007] Optionally, generating the one or more split points includes determining a plurality of quantiles for the first and second group of clustered data blocks, and each of the one or more split points corresponds to a different quantile among the plurality of quantiles. In some examples, the method further includes: determining, by the data processing hardware, a first sum of data values ​​associated with the first and second group of clustered data blocks; and determining, by the data processing hardware, a second sum of data values ​​associated with a third group of clustered data blocks. The method may also include verifying, by the data processing hardware, that the first sum equals the second sum. In some examples, the corresponding range of clustering key values ​​of one or more clustered data blocks in the second group of clustered data blocks overlaps with the corresponding range of clustering key values ​​of at least one clustered data block in the first group of clustered data blocks.

[0008] Another aspect of this disclosure provides a system for shuffling-free re-clustering of clustering tables. The system includes memory hardware in communication with data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include receiving, at the data processing hardware, a first set of clustered data blocks sorted by clustering key values. Each clustered data block in the first set of clustered data blocks includes a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the first set of clustered data blocks. The operations also include receiving a second set of clustered data blocks sorted by the clustering key values. Each clustered data block in the second set of clustered data blocks includes a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the second set of clustered data blocks. The operations further include generating one or more split points for dividing the first and second sets of clustered data blocks into a third set of clustered data blocks. The operation also includes dividing the first and second groups of clustered data blocks into a third group of clustered data blocks using the one or more generated split points. Each clustered data block in the third group may include a corresponding range of cluster key values ​​that does not overlap with any of the ranges of cluster key values ​​of other clustered data blocks in the third group, and each of the one or more generated split points defines an upper or lower bound for the corresponding range of cluster key values ​​for one of the clustered data blocks in the third group.

[0009] Implementations of this disclosure may include one or more of the following optional features. In some implementations, a columnar database table stores a first group and a second group of clustered data blocks. The number of the one or more split points generated is based on the number of data blocks in the first and second groups of clustered data blocks and the size of each data block. Without performing any shuffling operation on the data blocks in the first and second groups of clustered data blocks, the first and second groups of clustered data blocks are divided into a third group of clustered data blocks.

[0010] In some examples, the operation includes identifying which clustering key values ​​in the first and second sets of clustered data blocks fall between adjacent split points, and for each clustered data block in the third set of clustered data blocks, merging the identified clustering key values ​​falling within the corresponding adjacent split points. At least one clustered data block in the third set of clustered data blocks may include a portion of a corresponding range from one of the data blocks in the first or second set of clustered data blocks that does not overlap with any of the corresponding ranges of other clustered data blocks in the other set of clustered data blocks in the first or second set of clustered data blocks.

[0011] Additionally, one clustered data block in the third group may include a portion of a corresponding range of one data block from the first or second group and one data block from another group of clustered data blocks within the first or second group. At least one clustered data block in the third group may include a portion of a corresponding range of two data blocks from the first or second group and one data block from another group of clustered data blocks within the first or second group.

[0012] Optionally, generating the one or more split points may include determining a plurality of quantiles for the first and second group of clustered data blocks, and each of the one or more split points corresponds to a different quantile among the plurality of quantiles. In some examples, the operation includes determining a first sum of data values ​​associated with the first and second group of clustered data blocks and determining a second sum of data values ​​associated with a third group of clustered data blocks. The operation may also include verifying that the first sum equals the second sum. In some examples, the corresponding range of cluster key values ​​of one or more clustered data blocks in the second group of clustered data blocks overlaps with the corresponding range of cluster key values ​​of at least one clustered data block in the first group of clustered data blocks.

[0013] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will become apparent from the description, the drawings, and the claims. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of an exemplary system for shuffling-free re-clustering of cluster tables.

[0015] Figure 2A This is a schematic diagram of the first and second clustered data blocks within the clustering key value range.

[0016] Figure 2B It has multiple dividing points Figure 2A A schematic diagram of the first and second clustered data blocks within the clustering key value range.

[0017] Figure 2C yes Figure 2A A schematic diagram of the first and second groups of clustered data blocks, indicating partitions based on split points.

[0018] Figure 2D It is to merge to form a third group of clustered data blocks. Figure 2A A schematic diagram of the first and second clustered data blocks.

[0019] Figure 2EThis is a schematic diagram of the first and second groups of data from Figure 2, based on multiple quantile divisions.

[0020] Figure 3 This is a flowchart illustrating an exemplary arrangement of operations for a method of re-clustering a clustering table without shuffling.

[0021] Figure 4 This is a schematic diagram of an exemplary computing device that can be used to implement the systems and methods described herein.

[0022] The same reference numerals in the various figures indicate the same elements. Detailed Implementation

[0023] Distributed storage (i.e., cloud storage) is increasingly being used to store large-scale tables. It is not uncommon for tables to be several terabytes (TB) or even petabytes (PB) in size and contain millions of entries (i.e., data blocks). Clustering data structures (e.g., column data stores) are increasingly used to reduce query costs and improve query performance by clustering data into non-overlapping data blocks. For clusters of data blocks, the data blocks are typically ordered by a clustering key, where each data block includes a range of clustering key values. Typically, the range of clustering key values ​​associated with each data block does not overlap with any other data blocks within the clustered data block. When new data is appended to clustered data blocks, the range of clustering key values ​​for the new data block will often overlap with the original data blocks, and the data blocks must be re-clustered to maintain optimal clustering. This is typically achieved through data shuffling, which involves writing some or all of the data to a new location, which is computationally expensive and slow.

[0024] This document describes a data block re-clusterer that re-clusters data without requiring shuffling. The data block re-clusterer receives a first group and a second group of clustered data blocks sorted by clustering key values. The data block re-clusterer generates one or more split points to divide the first and second groups of clustered data blocks into a third group of clustered data blocks. The data block re-clusterer uses the one or more split points to divide the first and second groups of clustered data blocks into the third group of clustered data blocks.

[0025] Now for reference Figure 1In some implementations, exemplary system 100 includes remote system 140. Remote system 140 may be a single computer, multiple computers, or a distributed system (e.g., a cloud environment) having scalable / elastic computing resources 144 (e.g., data processing hardware) and / or storage resources 142 (e.g., memory hardware). Data storage 146 (i.e., remote storage device 146) may overlay storage resources 142 to allow scalable use of storage resources 142 by clients or one or more of computing resources 144. Data storage 146 includes block data storage 150, which is configured to store multiple data blocks 152, 152a-n within groups 158, 158a-n of clustered data blocks 152. Data storage 150 may store any number of groups 158 of clustered data blocks 152 at any point in time. In some examples, clustered data blocks are stored within a columnar database table or a clustering table 159.

[0026] Each group of clustered data blocks is sorted by clustering key values ​​154, 154a-n. For example, in clustering table 159 (i.e., clustering data blocks 152 of one or more groups 158), one or more columns of table 159 are selected to represent clustering keys, where each row of table 159 has a corresponding clustering key value 154. The data in clustering table 159 is organized around clustering keys to, for example, to co-locate related data, since the large table 159 is typically divided into multiple data blocks 152 stored on multiple different servers. Each data block 152 in the clustering data blocks 152 of the group 158 includes a range of clustering key values ​​154 that does not overlap with any of the ranges of clustering key values ​​154 of other data blocks 152 in the same group 158.

[0027] Remote system 140 is configured to receive table data 14. For example, remote system 140 receives table data 14 from user equipment 10 associated with a corresponding user 12 communicating with remote system 140 via network 112. User equipment 10 can correspond to any computing device, such as a desktop workstation, laptop workstation, or mobile device (i.e., a smartphone). User equipment 10 includes computing resources 18 (e.g., data processing hardware) and / or storage resources 16 (e.g., memory hardware). In another example, remote system 140 receives table data 14 from a different table stored on data storage 150 or from another remote system 140.

[0028] In some implementations, the remote system 140 generates a first set of 158a clustered data blocks 152 from table data 14 to form a clustered table 159. The remote system 140 organizes the table data 14 based on a clustering key 15 and divides the table data 14 into multiple clustered data blocks 152, wherein each clustered data block 152 includes a corresponding range of clustering key values ​​154 that does not overlap with any of the ranges of clustering key values ​​154 of other clustered data blocks 152 in the first set of 158a clustered data blocks 152. That is, each clustered data block 152 stores a portion of the table data 14 within the clustered table 159. The first set 158a is stored at the data block data storage 150.

[0029] In some examples, remote system 140 (e.g., from user equipment 10) receives additional table data 14 to add to clustering table 159. Remote system 140 generates a second set 158b of clustering data blocks 152 from the additional table data 14. Each clustering data block 152 in the second set 158b includes a corresponding range of clustering key values ​​154 that does not overlap with any of the ranges of clustering key values ​​154 of other clustering data blocks 152 in the second set 158b. However, the corresponding range of clustering key values ​​154 of one or more clustering data blocks 152 in the second set 158b may overlap with the corresponding range of clustering key values ​​154 of at least one clustering data block 152 in the first set 158a. That is, at least one data block 152 in the second set 158b may have a range of clustering key values ​​154 that overlaps with the range of clustering key values ​​154 of data blocks 152 in the clustering table 159 of the first set 158a.

[0030] Remote system 140 executes data block re-clusterer 160 to re-cluster data blocks 152 of the first group 158a and the second group 158b. As discussed in more detail below, the split point generator 170 of data block re-clusterer 160 receives the first group 158a and the second group 158b and generates one or more split points 310, 310a-n. Figure 2B This is used to divide the clustered data blocks 152 of the first and second groups 158a and 158b into clustered data blocks 152 of the third group 158c. Each split point 310 defines the upper or lower bound of the corresponding range of cluster key values ​​154 of one of the clustered data blocks 152 in the third group 158c. The split point generator 170 passes the first and second groups 158a and 158b and the one or more split points 310 to the data block divider 180.

[0031] Data block partitioner 180 uses one or more generated split points 310 to divide the clustered data blocks 152 of the first and second groups 158a, 158b into clustered data blocks 152 of the third group 158c. Each clustered data block 152 in the third group 158c includes a corresponding range of clustering key values ​​154 that does not overlap with any of the ranges of clustering key values ​​154 of other clustered data blocks 152 in the third group 158c. That is, data block partitioner 180 (using split points 310) re-clusters the data blocks 152 of the first and second groups 158a, 158b such that the ranges of clustering key values ​​154 in any of the data blocks 152 no longer overlap. Without performing any shuffling operation on data blocks 152 in the first and second groups 158a and 158b, the data block partitioner 180 divides the clustered data blocks 152 of the first and second groups 158a and 158b into clustered data blocks 152 of the third group 158c, thereby maintaining the performance characteristics of the clustering table without the associated costs of shuffling data. The data block partitioner 180 stores the data blocks 152 of the third group 158c into the data storage 150.

[0032] Now for reference Figure 2A Figure 200a illustrates exemplary clustered data blocks 152 of a first group 158a and a second group 158b plotted along the x-axis of clustering key values ​​154. The first group 158a consists of data blocks 152a-d, while the second group 158b consists of data blocks 152e-g. Each data block 152a-g includes a range 210, 210a-g of clustering key values ​​154. While the ranges 210 within each respective group 158a, 158b do not overlap, there is overlap between the ranges 210 of data blocks 152 across groups 158a, 158b. For example, the range 210e of data block 152e overlaps with the ranges 210a, 210b of data blocks 152a, 152b. Therefore, simply including all data blocks 152a-g into a single group 158 would result in a performance penalty due to overlap.

[0033] Now for reference Figure 2B Figure 200b illustrates the results plotted using cluster key value 154. Figure 2A Examples include the first group 158 and the second group 158b. Here, the split points 310 generated by the split point generator 170 divide some of the data blocks 152a-g. For example, split point 310a divides data block 152a, while split point 310b divides data block 152a (in the first group 158a) and data block 152e (in the second group 158b). Similarly, split point 310c divides data block 152b, split point 310d divides data block 152c, and split point 310e divides data block 152d and data block 152e.

[0034] Now for reference Figure 2C and 2D In some implementations, the data block partitioner 180 identifies which clustering key values ​​154 in the clustered data blocks 152 of the first group 158b and the clustered data blocks 152 of the second group 158b fall between adjacent split points 310. For each clustered data block 152 in the clustered data blocks 152 of the third group 158c, the data block partitioner 180 merges the identified clustering key values ​​154 that fall within the corresponding adjacent split points 310. For example, Figure 200c indicates the portions of data blocks 152a-g that fall within adjacent split points 310a-e. Here, Figure 200c illustrates how adjacent split points 310a, 310b divide data block 152a into portions 152aa, 152ab, 152ac, and data block 152e into portions 152ea, 152eb. Figure 2C Similarly: data block 152b is divided into parts 152ba and 152bb; data block 152c is divided into parts 152ca and 152cb; data block 152d is divided into parts 152da and 152db; and data block 152g is divided into parts 152ga and 152gb. Note that data block 152f is not divided into any parts because no split point 310 passes through data block 152f.

[0035] Each pair of adjacent split points 310 and ending split points 310a, 310e forms a range 410, 410a-f of clustering key values ​​154. In some examples, the data block partitioner 180 reads only the rows within each partition 410 and (based on the partitions 410 read) writes each data block 152 of the third group 158c to the data block data store 150. In some examples, the data block partitioner 180 reads only one or more columns including the clustering key values ​​154 instead of the entire cluster table 159 to significantly reduce the total amount of data read. Optionally, only portions of overlapping data blocks 152 are read by the data block partitioner 180. For example, when generating data blocks 152 of the third group 158c, the data block partitioner 180 does not read data block 152f because no split point 310 intersects with data block 152f.

[0036] like Figure 2DAs shown in Figure 200d, each data block 152h-n of the third group 158c is formed by merging partitions of groups 158a and 158b within the same adjacent partition point pair 310. Here, because partition point 310a is the leftmost partition point 310, partition point 310a does not have an adjacent partition point 310 to its left, and therefore part 152aa of data block 152a forms data block 152h of group 158c alone. Adjacent partition points 310a and 310b enclose parts 152ab and 152ea, which are merged to form data block 152i. Similarly, adjacent partition points 310b and 310c define parts 152ac, 152ba, and 152eb and merge them to form 152j. Likewise, adjacent partition points 310c and 310d define parts 152bb, 152ca and data block 152f and merge them to form data block 152k. Adjacent split points 310d and 310e define portions 152cb, 152da, and 152ga, which are then merged to form data block 152m. Since split point 310e is the rightmost split point 310, portions 152db and 152gb of the data block are merged to form data block 152n.

[0037] Therefore, in some examples, at least one clustered data block 152 (e.g., data block 152h) in the third group 158c includes a portion (e.g., portion 152aa) of a corresponding range 210 from one of the clustered data blocks 152 in the first group 158a or the second group 158b, which does not overlap with any of the corresponding ranges of other clustered data blocks in the other group of clustered data blocks 152 in the first or second group 158a, 158b. In some implementations, at least one clustered data block 152 in the third group 158c includes a portion (e.g., portion 152ab) of a corresponding range 210 from one of the clustered data blocks 152 in the first or second group 158a, 158b and one of the data blocks 152 from the other group of clustered data blocks 152 in the first or second group 158a, 158b (e.g., portion 152ea). Optionally, at least one clustered data block 152 in the third group 158c includes a portion (e.g., portions 152ac, 152ba) of a corresponding range 210 of two data blocks 152 from the first or second group 158a, 158b and one data block 152 from the other clustered data block 152 from the first or second group 158a, 158b (e.g., portion 152eb). That is, the split point 310 can divide the data block 152 into any number of portions, and the data block divider 180 merges any number of portions or data blocks 152 from the first group 158a or the second group 158b into the data block 152 of the third group 158c.

[0038] Now for reference Figure 2EIn some implementations, the split point generator 170 generates one or more split points 310 by determining multiple quantiles 610, 610a-n of the clustered data blocks 152 of the first and second groups 158a, 158b. A quantile is a cut point that divides a distribution range into intervals, where each interval has an equal or approximately equal distribution. For example, as shown in Figure 200e, given a range of cluster key values ​​154 defined by the minimum cluster key value 154MIN and the maximum cluster key value 154MAX (in this example, determined by the minimum and maximum cluster key values ​​154 of the data blocks 152a-g of groups 158a, 158b), the first, second, and third quantiles 610a-c divide the range 620 of cluster key values ​​158 into four subranges 612a-d. The first range 612a represents 25% (i.e., one-quarter) of the distribution of cluster key value 154, while each of the other ranges 612b-c also represents 25% of the distribution of cluster key value 154. Each quantile 610 can represent the location of a split point 310, and therefore the number of quantiles 610 is equal to the number of split points 310. That is, each of the one or more split points 310 corresponds to a different quantile 610 among the plurality of quantiles 610.

[0039] The split point generator 170 can determine any number of quantiles 610 (and thus split points 310). The split point generator 170 can determine the number of the generated one or more split points 310 based on the number of data blocks 152 in the clustered data blocks 152 of the first and second groups 158a, 158b, and the size of each of the data blocks 152. In some examples, each data block 152 is a configurable size (e.g., 32MB to 256MB), and the split point generator 170 determines the number of quantiles 610 by dividing the total size of the first group 158a and the second group 158b by the configured data block size. In the example shown, the split point generator 170 determines three quantiles 610a-610c (corresponding to three split points 310f-h) to divide the range 620 of clustering key values ​​154 into four subranges 612a-d, each corresponding to data blocks 152h-k of the clustered data blocks 152 of the third group 158c.

[0040] In some examples, the split point generator determines one or more quantiles 610 of data blocks 152 in the first and second groups 158a, 158b based on sampling of the data in data block 152. That is, sampling the data allows the split point generator 170 to determine quantiles 610 in a more efficient and scalable manner due to the potentially huge size of the clustering table 159. In some implementations, the split point generator 170 uses weighted sampling to approximate one or more quantiles of data blocks 152 in the clustered data blocks 152 of the first group 158a and the second group 158b. Alternatively, the split point generator 170 can use other means such as ordered codes to generate split points 310. Ordered codes provide a byte encoding of a typed sequence of items. The resulting bytes can be compared in lexicographical order to produce the same ordering as an item-by-item comparison of the original sequence. That is, ordered codes have the property that comparing ordered codes produces the same result value as an item-by-item comparison.

[0041] Optionally, after dividing data block 152 into a third group 158c, the data block partitioner 180 determines a first sum of data values ​​associated with clustered data blocks 152 in the first and second groups 158a and 158b, and a second sum of data values ​​associated with clustered data blocks 152 in the third group 158c. The data block partitioner verifies that the first sum equals the second sum. That is, to ensure that no data corruption occurs during the partitioning process, the data block partitioner 180 verifies that the values ​​associated with the first and second groups 158a and 158b (e.g., the sum of the number of rows in clustering table 159) are the same as the corresponding values ​​in the third group 158c. These values ​​will match if no data has been corrupted or misplaced. The total number of rows in the third group 158c should be equal to the sum of the total number of rows in the first group 158a and the total number of rows in the second group 158b.

[0042] This example illustrates a block re-clusterer 160 performing shuffle-free re-clustering on two groups of clustered data blocks 152. However, this is merely exemplary, and any number of groups can be re-clustered simultaneously. In some examples, the corresponding range of clustering key values ​​154 for clustered data blocks 152 in the second group 158b does not overlap with the corresponding range of clustering key values ​​154 for clustered data blocks 152 in the first group 158a. In this case, the block re-clusterer 160 can merge the data blocks without generating a split point 310.

[0043] Figure 3This is a flowchart illustrating an exemplary arrangement of the operation of a method 300 for shuffling-free re-clustering of a clustering table. Method 300 includes, at operation 302, receiving clustered data blocks 152 of a first group 158a sorted by clustering key values ​​154 at a data processing hardware 144. Each clustered data block 152 in the first group 158a includes a corresponding range 210 of clustering key values ​​154 that does not overlap with any of the ranges 210 of clustering key values ​​154 of other clustered data blocks 152 in the first group 158a.

[0044] At operation 304, method 300 includes receiving, at data processing hardware 144, clustered data blocks 152 of a second group 158b ordered by clustering key values ​​154. Each clustered data block 152 in the second group 158b includes a corresponding range 210 of clustering key values ​​154 that does not overlap with any of the ranges 210 of clustering key values ​​154 of other clustered data blocks 152 in the second group 158b. The corresponding range 210 of clustering key values ​​154 of one or more clustered data blocks 152 in the second group 158b overlaps with the corresponding range 210 of clustering key values ​​154 of at least one clustered data block 152 in the first group 158a.

[0045] Method 300 includes, at operation 306, generating one or more partition points 310 by data processing hardware 144 to divide clustered data blocks 152 of the first and second groups 158a, 158b into clustered data blocks of the third group 158c. At operation 308, method 300 includes, by data processing hardware 144, using the one or more generated partition points 310, dividing clustered data blocks 152 of the first and second groups 158a, 158b into clustered data blocks 152 of the third group 158c. Each clustered data block 152 in the third group 158c includes a corresponding range 210 of clustering key values ​​154 that does not overlap with any of the ranges 210 of clustering key values ​​154 of other clustered data blocks 152 in the third group 158c. Each of the one or more generated split points 310 defines an upper or lower limit of the corresponding range 210 of the clustering key value 154 of one of the clustering data blocks 152 in the third group 158c.

[0046] Figure 4This is a schematic diagram of an exemplary computing device 400 that can be used to implement the systems and methods described in this document. The computing device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely exemplary and are not intended to limit the implementation of the inventions described and / or claimed in this document.

[0047] Computing device 400 includes a processor 410, a memory 420, a storage device 430, a high-speed interface / controller 440 connected to the memory 420 and a high-speed expansion port 450, and a low-speed interface / controller 460 connected to a low-speed bus 470 and the storage device 430. Each of the components 410, 420, 430, 440, 450, and 460 is interconnected using various buses and may be mounted on a common motherboard or otherwise suitably mounted. The processor 410 is capable of processing instructions for execution within the computing device 400, including instructions stored in the memory 420 or on the storage device 430, to display graphical information for a graphical user interface (GUI) on an external input / output device such as a display 480 coupled to the high-speed interface 440. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and memory types, may be suitably used. Moreover, multiple computing devices 400 may be connected, with each device providing a portion of the necessary operation (e.g., as a server library, blade server group, or multiprocessor system).

[0048] Memory 420 stores information non-transitorily within computing device 400. Memory 420 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transitory memory 420 may be a physical device for temporarily or permanently storing programs (e.g., instruction sequences) or data (e.g., program state information) for use by computing device 400. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0049] Storage device 430 provides mass storage for computing device 400. In some embodiments, storage device 430 is a computer-readable medium. In various embodiments, storage device 430 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory or other similar solid-state storage device, or device array, including devices in a storage area network or other configuration. In other embodiments, a computer program product is tangibly embodied as an information carrier. The computer program product contains instructions that, when executed, perform one or more methods such as those described above. The information carrier is a computer or machine-readable medium, such as memory 420, storage device 430, or memory on processor 410.

[0050] High-speed controller 440 manages bandwidth-intensive operations of computing device 400, while low-speed controller 460 manages lower bandwidth-intensive operations. This allocation of responsibilities is merely exemplary. In some embodiments, high-speed controller 440 is coupled to memory 420, display 480 (e.g., via a graphics processor or accelerator), and high-speed expansion port 450 which can accept various expansion cards (not shown). In some embodiments, low-speed controller 460 is coupled to storage device 430 and low-speed expansion port 490. Low-speed expansion port 490, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), can be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices such as switches or routers, for example, via a network adapter.

[0051] As shown in the figure, the computing device 400 can be implemented in several different forms. For example, the computing device 400 can be implemented as a standard server 400a or multiple times in a set of such servers 400a, as a laptop computer 400b, or as part of a rack server system 400c.

[0052] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can be included in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive and transmit data and instructions from and to a storage system, at least one input device, and at least one output device.

[0053] A software application (i.e., a software resource) can refer to computer software that enables a computing device to perform tasks. In some examples, a software application can mean "application," "app," or "program." Exemplary applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0054] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages ​​and / or in assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0055] The processes and logical flows described in this specification can be executed by one or more programmable processors—also known as data processing hardware—that perform functions by executing one or more computer programs to manipulate input data and generate output. The processes and logical flows can also be executed by special-purpose logic circuitry, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors of any kind of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more storage devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, one or more mass storage devices for storing data, such as magnetic disks, magneto-optical disks, or optical disks, to receive data from or transfer data thereto, or both. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices like EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or incorporated into dedicated logic circuitry.

[0056] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a display device having, for example, a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touchscreen to display information to the user, and optionally on a computer having a keyboard and a pointing device such as a mouse and trackball through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form including sound, speech, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0057] Several embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are also within the scope of the appended claims.

Claims

1. A method comprising: The data processing hardware receives a first set of clustered data blocks sorted by clustering key values, each of the first set of clustered data blocks including a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the first set of clustered data blocks; The data processing hardware receives a second set of clustered data blocks sorted by the clustering key values, each of the second set of clustered data blocks including a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the second set of clustered data blocks; The data processing hardware generates one or more segmentation points, which are used to divide the first group of clustered data blocks and the second group of clustered data blocks into a third group of clustered data blocks. as well as The data processing hardware uses one or more generated split points to divide the first group of clustered data blocks and the second group of clustered data blocks into the third group of clustered data blocks. Each clustered data block in the third group of clustered data blocks includes a corresponding range of cluster key values ​​that does not overlap with any range of cluster key values ​​of other clustered data blocks in the third group of clustered data blocks. Each of the one or more generated split points defines an upper or lower limit of the corresponding range of cluster key values ​​of one of the clustered data blocks in the third group of clustered data blocks.

2. The method of claim 1, wherein, The columnar database table stores the first group of clustered data blocks and the second group of clustered data blocks.

3. The method of claim 1, wherein, The number of the one or more split points generated is based on the number of data blocks in the first group of clustered data blocks and the second group of clustered data blocks, as well as the size of each data block in the data blocks.

4. The method of claim 1, wherein, Without performing any shuffling operation on the data blocks in the first group of clustered data blocks and the second group of clustered data blocks, the first group of clustered data blocks and the second group of clustered data blocks are divided into the third group of clustered data blocks.

5. The method of claim 1, wherein, Dividing the first group of clustered data blocks and the second group of clustered data blocks into the third group of clustered data blocks includes: Identify which clustering key values ​​in the first and second sets of clustered data blocks fall between adjacent split points; and For each cluster data block in the third group of cluster data blocks, the identified cluster key values ​​that fall into the corresponding adjacent split points are merged.

6. The method of claim 1, wherein, At least one clustered data block in the third group of clustered data blocks includes a portion of a corresponding range from one of the data blocks in the first group of clustered data blocks or the second group of clustered data blocks that does not overlap with any corresponding range of other clustered data blocks in the other group of clustered data blocks in the first group of clustered data blocks or the second group of clustered data blocks.

7. The method of claim 1, wherein, At least one clustered data block in the third group of clustered data blocks includes a portion of a corresponding range of one of the data blocks from the first group of clustered data blocks or the second group of clustered data blocks and one of the data blocks from another group of clustered data blocks from the first group of clustered data blocks or the second group of clustered data blocks.

8. The method according to claim 1, wherein, At least one clustered data block in the third group of clustered data blocks includes a portion of the corresponding range of two data blocks from the first group of clustered data blocks or the second group of clustered data blocks and one data block from another group of clustered data blocks from the first group of clustered data blocks or the second group of clustered data blocks.

9. The method according to claim 1, wherein, Generating the one or more split points includes: determining multiple quantiles of the first group of clustered data blocks and the second group of clustered data blocks.

10. The method according to claim 9, wherein, Each of the one or more dividing points corresponds to a different quantile among the plurality of quantiles.

11. The method of claim 1, further comprising: The data processing hardware determines a first sum of data values ​​associated with the first group of clustered data blocks and the second group of clustered data blocks; The data processing hardware determines a second sum of data values ​​associated with the third group of clustered data blocks; as well as The data processing hardware verifies that the first sum equals the second sum.

12. The method according to any one of claims 1 to 11, wherein, The corresponding range of cluster key values ​​of one or more cluster data blocks in the second group of cluster data blocks overlaps with the corresponding range of cluster key values ​​of at least one cluster data block in the first group of cluster data blocks.

13. A system comprising: Data processing hardware; as well as Memory hardware communicating with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Receive a first group of clustered data blocks sorted by clustering key values, each of the first group of clustered data blocks including a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the first group of clustered data blocks; Receive a second group of clustered data blocks sorted by the clustering key values, each of the clustered data blocks in the second group of clustered data blocks including a corresponding range of clustering key values ​​that does not overlap with any range of clustering key values ​​of other clustered data blocks in the second group of clustered data blocks, wherein the corresponding range of clustering key values ​​of one or more clustered data blocks in the second group of clustered data blocks overlaps with the corresponding range of clustering key values ​​of at least one clustered data block in the first group of clustered data blocks; Generate one or more split points, said one or more split points being used to divide the first group of clustered data blocks and the second group of clustered data blocks into a third group of clustered data blocks; and The first group of clustered data blocks and the second group of clustered data blocks are divided into the third group of clustered data blocks using one or more generated split points. Each clustered data block in the third group of clustered data blocks includes a corresponding range of cluster key values ​​that does not overlap with any range of cluster key values ​​of other clustered data blocks in the third group of clustered data blocks. Each of the one or more generated split points defines an upper or lower limit of the corresponding range of cluster key values ​​of one of the clustered data blocks in the third group of clustered data blocks.

14. The system according to claim 13, wherein, The columnar database table stores the first group of clustered data blocks and the second group of clustered data blocks.

15. The system according to claim 13, wherein, The number of the one or more split points generated is based on the number of data blocks in the first group of clustered data blocks and the second group of clustered data blocks, as well as the size of each data block in the data blocks.

16. The system according to claim 13, wherein, Without performing any shuffling operation on the data blocks in the first group of clustered data blocks and the second group of clustered data blocks, the first group of clustered data blocks and the second group of clustered data blocks are divided into the third group of clustered data blocks.

17. The system according to claim 13, wherein, Dividing the first group of clustered data blocks and the second group of clustered data blocks into the third group of clustered data blocks includes: Identify which clustering key values ​​in the first and second sets of clustered data blocks fall between adjacent split points; and For each cluster data block in the third group of cluster data blocks, the identified cluster key values ​​that fall into the corresponding adjacent split points are merged.

18. The system according to claim 13, wherein, At least one clustered data block in the third group of clustered data blocks includes a portion of a corresponding range from one of the data blocks in the first group of clustered data blocks or the second group of clustered data blocks that does not overlap with any corresponding range of other clustered data blocks in the other group of clustered data blocks in the first group of clustered data blocks or the second group of clustered data blocks.

19. The system according to claim 13, wherein, At least one clustered data block in the third group of clustered data blocks includes a portion of a corresponding range of one of the data blocks from the first group of clustered data blocks or the second group of clustered data blocks and one of the data blocks from another group of clustered data blocks from the first group of clustered data blocks or the second group of clustered data blocks.

20. The system according to claim 13, wherein, At least one clustered data block in the third group of clustered data blocks includes a portion of the corresponding range of two data blocks from the first group of clustered data blocks or the second group of clustered data blocks and one data block from another group of clustered data blocks from the first group of clustered data blocks or the second group of clustered data blocks.

21. The system according to claim 13, wherein, Generating the one or more split points includes: determining multiple quantiles of the first group of clustered data blocks and the second group of clustered data blocks.

22. The system according to claim 21, wherein, Each of the one or more dividing points corresponds to a different quantile among the plurality of quantiles.

23. The system according to claim 13, wherein, The operation further includes: Determine the first sum of data values ​​associated with the first group of clustered data blocks and the second group of clustered data blocks; Determine the second sum of data values ​​associated with the third group of clustered data blocks; and Verify that the first sum equals the second sum.

24. The system according to any one of claims 13 to 23, wherein, The corresponding range of cluster key values ​​of one or more cluster data blocks in the second group of cluster data blocks overlaps with the corresponding range of cluster key values ​​of at least one cluster data block in the first group of cluster data blocks.

Images