Method and system for performing inner joins on sharded data sets

Abstract

In a distributed database, inner joins of very large data sets are performed by distributing smaller jobs to multiple computing devices. For each key, the values ​​in the first input are divided into a first set of blocks, and the values ​​in the second input are divided into a second set of blocks. For each key, the first set of blocks is replicated by the number of blocks in the second set of blocks, and the second set of blocks is replicated by the number of blocks in the first set of blocks. Each replicated block is assigned a block key, which includes the key and additional information identifying the replicated block. Each pair of replicated blocks with matching block keys is distributed to one of the multiple computing devices. The results of the inner join are received from the multiple computing devices.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. patent application No. 15 / 928,286 filed on March 22, 2018, the disclosure of which is incorporated herein by reference. Background Art

[0003] In a distributed database, a large data set can be split into multiple shards and stored separately on multiple computing devices in a distributed system. One way to collect information from more than one distributed shard is to perform an inner join of two shards. An inner join is an operation that combines two input tables (each of which is a collection of key-value pairs) to produce a new result table that contains all possible combinations of values ​​from the two input tables for each matching key. Summary of the invention

[0004] The present disclosure provides: receiving a request to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs; determining, by one or more processors, a block size that is a target number of values ​​to be divided into blocks; dividing, by one or more processors, the values ​​in the first input into a first set of blocks for each key; dividing, by one or more processors, the values ​​in the second input into a second set of blocks for each key; replicating, by one or more processors, the first set of blocks by the number of blocks in the second set of blocks for each key to obtain a first set of replicated blocks; The method comprises the steps of: replicating the second set of blocks by the number of blocks in the first set of blocks having the same key to obtain a second set of replicated blocks; assigning, by one or more processors, a block key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block key comprising a key (e.g., a key regarding which block has been replicated or a key corresponding to the value of the block) and additional information to identify the replicated block; distributing, by one or more processors, each pair of replicated blocks having matching block keys to one of a plurality of computing devices for processing; and receiving a result of an inner join from each of the plurality of computing devices, the result of the inner join comprising at least one value pair from one of the pair of replicated blocks.

[0005] Each key-value pair in the first key-value pair set and the second key-value pair set is a pair of a key and a value. The first key-value pair set may include key-value pairs of multiple keys. Thus, for example, partitioning the values ​​in the first input for each key may include partitioning the values ​​corresponding to the same key into different blocks, such that each block includes a subset of the values ​​corresponding to the key. Similarly, the second key-value pair set may also include key-value pairs of multiple keys, which may be partitioned in the same manner.

[0006] The first key-value pair set may include one or more keys that are also present in a key-value pair in the second key-value pair set, and vice versa. Thus, the first and second key-value pair sets may have key-value pairs that have the same key.

[0007] When distribution of each pair of replicated blocks with matching block keys is complete, the replicated block pair may include a first replicated block that has been replicated with respect to a block in the first set of blocks and a second replicated block that has been replicated with respect to a block in the second set of blocks. In some cases, a first replicated block and a second replicated block have matching block keys if the block key of the first replicated block is equal to the block key of the second replicated block.

[0008] Determining the block size may be based on at least one of a total number of the plurality of computing devices, processing capabilities of the plurality of computing devices, or uniformity in terms of the number of values ​​per block. The block size may be a minimum block size for which the block key mapping does not cause any of the plurality of computing devices to run out of operating memory.

[0009] At least one of the partitioning of the values ​​in the first input or the partitioning of the values ​​in the second input may be based on a uniform distribution.

[0010] The block key may include a first index that is assigned in sequence when the values ​​in the first input are divided into a first set of blocks and the values ​​in the second input are divided into a second set of blocks. For example, the first index may be additional information that identifies the duplicate block. For example, the first index may be viewed as an index that identifies blocks divided for the same key. For example, when the division of the values ​​in the first input is done for the corresponding key, each block of values ​​obtained for the corresponding key may be sequentially assigned with a corresponding index, which is then the first index. Similarly, when the division of the values ​​in the second input is done for the corresponding key, each block of values ​​obtained for the corresponding key may be sequentially assigned with a corresponding index, which is then the first index.

[0011] The block key may include a second index that is assigned in sequence when the first block set and the second block set are copied. For example, the second index may be additional information that identifies the copied block. For example, when the copying is done with respect to the first block set, a corresponding number is assigned to each copy until the number is equal to the number of blocks in the second block set of the same key. Similarly, when the copying is done with respect to the second block set, a corresponding number is assigned to each copy until the number is equal to the number of blocks in the first block set of the same key.

[0012] Thus, as explained above, the block key may include the first index and / or the second index, for example as additional information identifying the replicated block.

[0013] The method may further include sending a block key map including all block keys to each of the plurality of computing devices.

[0014] The replicated block pairs may be distributed to the plurality of computing devices based on at least one of a block size, a total number of replicated block pairs, a total number of the plurality of computing devices, or a processing capability of the plurality of computing devices.

[0015] The method may further include: estimating an approximate number of blocks required for each key of at least one of the first input or the second input by sampling at least one of the first input or the second input at intervals of each block size. The method may further include: grouping key-value pairs of at least one of the first input or the second input by key; determining a remaining number of values ​​for each key based on an assumption that each of the approximate number of blocks contains a block size number of values; and adjusting the approximate number of blocks required for one or more keys based on the remaining number of values ​​for the key. The approximate number of blocks required for one or more keys may be adjusted when the remaining number of values ​​for the key is greater than a predetermined threshold. The method may further include: determining that one or more keys are omitted during sampling, and setting the approximate number of blocks required for the one or more keys omitted during sampling.

[0016] At least one of the first input or the second input may be stored as fragments on the plurality of selected computing devices. The method may further include: collecting at least one of the fragments of the first input or the fragments of the second input from the plurality of selected computing devices onto a single pre-processing computing device.

[0017] The result of the inner join may include a cross product of at least one of the pairs of replicated blocks.

[0018] The present disclosure further provides a system, the system comprising a computing device adapted to communicate with a plurality of computing devices in a distributed computing environment, the computing device comprising one or more processors, the one or more processors being configured to: receive a request to perform an inner join on a first input comprising a first set of key-value pairs and a second input comprising a second set of key-value pairs; determine a block size, the block size being a target number of values ​​to be partitioned into blocks; partition the values ​​in the first input into a first set of blocks for each key; partition the values ​​in the second input into a second set of blocks for each key; replicate the first set of blocks for each key by the number of blocks in the second set of blocks for the same key to obtain a first set of replicated blocks; replicate the second set of blocks for each key by the number of blocks in the first set of blocks for the same key to obtain a second set of replicated blocks; assign a block key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block key comprising a key and additional information to identify the replicated block; distribute each pair of replicated blocks having matching block keys to one of a plurality of computing devices for processing; and receive a result of the inner join from each of the plurality of computing devices, the result of the inner join comprising at least one value pair from one of the pairs of replicated blocks.

[0019] The one or more processors may be further configured to determine a block size based on at least one of a total number of computing devices, a processing capability of the multiple computing devices, or a uniformity in terms of the number of values ​​per block. The one or more processors may be further configured to distribute each pair of replicated blocks based on at least one of the block size, a total number of pairs of replicated blocks, a total number of computing devices, or a processing capability of the multiple computing devices. The one or more processors may be further configured to estimate an approximate number of blocks required for each key of at least one of the first input or the second input by sampling at least one of the first input or the second input at intervals of each block size.

[0020] The present disclosure further provides a non-transitory computer-readable storage medium having instructions stored thereon, which when executed by one or more processors causes the one or more processors to perform a method, the method comprising: receiving a request to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs; determining a block size, the block size being a target number of values ​​to be divided into blocks; dividing the values ​​in the first input into a first set of blocks for each key; dividing the values ​​in the second input into a second set of blocks for each key; replicating the first set of blocks by the number of blocks in the second set of blocks for each key by the same key to obtain a first set of replicated blocks; replicating the second set of blocks by the number of blocks in the first set of blocks for each key by the same key to obtain a second set of replicated blocks; assigning a block key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block key comprising a key and additional information to identify the replicated block; distributing each pair of replicated blocks having matching block keys to one of a plurality of computing devices for processing; and receiving a result of the inner join from each of the plurality of computing devices, the result of the inner join comprising at least one value pair from one of the pairs of replicated blocks. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a block diagram illustrating an example system in accordance with various aspects of the present disclosure.

[0022] Figure 2 is a block diagram illustrating an example system in accordance with various aspects of the present disclosure.

[0023] Figure 3 is a flow chart according to various aspects of the present disclosure.

[0024] Figures 4A to 4D is a pictorial diagram according to various aspects of the present disclosure.

[0025] Figure 5 is a flow chart according to various aspects of the present disclosure. DETAILED DESCRIPTION

[0026] Overview

[0027] The technology generally relates to a method and system for performing inner joins on very large and highly fragmented data sets. An inner join is an operation that combines two input tables, each of which is a set of key-value pairs, to produce a new result table that contains all possible combinations of values ​​from the two input tables for each matching key. Inner joins of very large data sets may take some systems a significant amount of resources and time (e.g., weeks) to compute. In some cases, the operation may fail before completion, which may result in the need to rerun the inner join, consuming even more resources and time.

[0028] Inner joins can be particularly challenging if the number of values ​​for one or more keys in either input table is too large to fit in memory, if both input tables have arbitrarily large numbers of keys and values ​​per key, or if the number of values ​​per key in either input table varies greatly from key to key. For example, the first input table can have thousands of keys. Some keys may have only a few values, while others may have thousands, and one key in particular may even have 1,000,000,000 values, meaning that all values ​​for that particular key will not fit in 32GB of memory and therefore cannot be processed on such a computing device. If the second input table is also arbitrarily large in both the number of keys and the number of values, and / or lacks uniformity from key to key, inner joins can become a very daunting task.

[0029] One example method of efficiently performing an inner join is to perform it as many smaller jobs distributed to multiple computing devices, each of which involves a block of values ​​from each of the two input tables. For example, the first input table may contain 5,000 keys, each with a maximum of 1,000,000 values; the second input table may contain 3,000 keys, each with a maximum of 2,000,000 values. In this example, the input tables may be divided into blocks of a block size of 1,000 values, so that a single computing device in the system may be able to trivially process operations involving two blocks of that size (one from each input table). Then, for each key of the first input table, approximately 1,000 values ​​may be divided into each block in the first block set. Similarly, for each key of the second input table, approximately 1,000 values ​​may be divided into each block in the second block set. Then, for each key, the first block set may be replicated by the number of blocks in the second block set for the same key, resulting in a first set of replicated blocks. Likewise, the second block set may be copied by the number of blocks in the first block set with the same key, thereby obtaining a second copy block set.

[0030] Each replicated block may be assigned a block key that includes not only the corresponding keys of all values ​​in the replicated block, but also additional information that distinguishes the replicated block from other replicated blocks from the same input table for that particular key. Each pair of replicated blocks (one pair from each input table) with the same block key may be assigned to one of a plurality of computing devices for processing. The results of the inner join may be received from a plurality of computing devices.

[0031] The method may further include estimating the number of blocks required for each key. For example, the approximate number of blocks required for each key of the first input and / or the second input may be estimated by sampling at intervals of each block size. For each key, the value of the remaining number is counted by assuming that each of the approximate number of blocks contains a value of the block size number. The approximate number of blocks required may be adjusted based on the counted value of the remaining number. In addition, the approximate number of blocks required for one or more keys omitted during sampling may be set.

[0032] The block key assigned to each replicated block may include a first index that may be assigned when a block in the set of blocks is created. The block key may further include a second index that may be assigned when the set of blocks is replicated. Additionally, the first and second indexes of the replicated blocks from the second input table may be assigned in a manner opposite to the first and second indexes of the replicated blocks from the first input table. A block key mapping of all block keys of the first and second sets of replicated blocks may be sent to each of the plurality of computing devices involved in processing the inner join.

[0033] Each of the plurality of computing devices may process a pair of replicated blocks. Alternatively, one or more of the plurality of computing devices may process more than one pair of replicated blocks. The distribution to the plurality of computing devices may depend on a number of factors, such as the block size, the total number of replicated block pairs, the total number of the plurality of computing devices, and the processing power of each of the plurality of computing devices. Additionally, the distribution to the plurality of computing devices may further incorporate dynamic work rebalancing techniques.

[0034] The block size may be determined based on the total number of the plurality of computing devices to process the inner join. Additionally or alternatively, the block size may be determined based on the processing capabilities of each of the plurality of computing devices. Thus, when allocating data for processing, the known capabilities of the plurality of computing devices may be used. The block size may be determined as a minimum size at which the block key mapping does not cause any of the plurality of computing devices to exhaust operating memory. Additionally, the block size may be further determined based on a dynamic work rebalancing technique.

[0035] The first input table and / or the second input table may be stored in a distributed manner. Each of the plurality of computing devices may output the result it computes to a file. Alternatively, all results for each key may be collected so that all inner join results for a single key are located on a single computing device.

[0036] The features described herein can provide efficient computation of inner joins of very large and highly sharded data sets. These features can work efficiently even if the number of keys and the number of values ​​per key of the two inputs to the inner join are arbitrarily large, and even if the number of values ​​for different keys does not follow a particular pattern. These features may even work well if the values ​​for any particular key in the input are too large to fit in memory. Thus, operating in the manner described herein can reduce processing time from days or weeks to a time frame of hours or even less. Further, the technique can be easily extended to perform inner joins on more than two input tables.

[0037] Example System

[0038] Figure 1 An example distributed system for performing inner joins is illustrated. Multiple computing devices (such as servers 140, 150, 160) can communicate with each other, for example, via a network 130. Servers 140, 150, 160 can further communicate with multiple client computing devices (such as clients 110, 120). Servers 140 to 160 can control the storage of data in one or more databases. For example, as shown, each server 140 to 160 is associated with a data center 142, 152, 162. Each data center 142, 152, 162 can include multiple computing devices to store data. In a distributed database, the data items of the database can be sharded to multiple different fragments (such as fragments 146, 156, 166), and each fragment can be copied to multiple computing devices (such as servers) at a data center (e.g., data center 142) or multiple data centers (such as data centers 142, 152 and 162). For example, a large data set may have hundreds of millions of keys, a first input from the example large data set may be several terabytes (TB) in size and have millions of values ​​for some keys, and a second input from the example large data set may also be several TB in size and have millions of values ​​for some keys. The example first input and the second input may be sharded into thousands or tens of thousands of shards, such as shards 146, 156, 166 stored in a distributed system, respectively. An inner join performed on such an example first input and the second input may produce a result that is hundreds of TB in size, with some keys having trillions of value pairs.

[0039] Inner joins performed on multiple fragments of a large data set can be performed in a distributed manner, for example, as smaller jobs involving smaller subsets of data distributed to multiple computing devices. For example, for the example large data set above with hundreds of millions of keys, thousands of computing devices can be used to perform inner joins. In this regard, for example, using the example method described in detail below, one or more computing devices or servers (such as servers 140, 150, 160) in a distributed system can be configured to create smaller jobs involving smaller subsets of data and distribute the smaller jobs to other computing devices in the distributed system. For example, using the example method described in detail below, these one or more servers can also be configured to receive and process the smaller jobs distributed to them.

[0040] Each server 140, 150, 160 can store and perform actions for multiple shards and / or shard replicas. All replicas of a single shard form a group, for example, group 148 contains all replicas of one shard, group 158 contains all replicas of another shard, and group 168 contains all replicas of yet another shard. The shard replicas can be synchronized using a consensus protocol (such as the Paxos protocol). Although some shards may be replicas of other shards, some shards may be dependent on other shards for a reason. For example, data written to data center 142 (e.g., shard 146 or group 148) may affect data stored in data center 152 (e.g., shard 156 or group 158). The distributed database can implement a protocol (such as Paxos) to provide consensus across the system. Each server has a local clock 144, 154, 164. In some systems, the consensus on the data centers 142, 152, 162 is maintained by servers 140, 150, 160, which wait for a period of time to pass before issuing a write transaction. In other systems, the wait period may instead be imposed on one or more other devices, moving in time to a different action by the server, or to a client device attempting to read the written data.

[0041] Although only a few data centers with several servers, clients, fragments, and groups are shown, any number of data centers may be included in a distributed database, where each data center may contain multiple servers (which may communicate with multiple clients), fragments, and groups. Likewise, although each server 140, 150, 160 is shown as being associated with its own data center, it should be understood that in other examples, a server may be associated with one or more smaller databases. For example, a database may include multiple servers.

[0042] Each of the clients 110, 120 is shown as having an application 112, 122 and a client library 114, 124, although it should be understood that additional features of the client devices may also exist. Any of the clients 110, 120 may write data to the distributed database by sending data to one of the servers 140, 150, 160 via the network 130. Although only a few clients are shown, it should be understood that a large number of client devices may communicate with the distributed database via the network 130.

[0043] The data centers 142, 152, 162 may be located at a considerable distance from each other. For example, the data centers may be located in various countries around the world. Each data center 142, 152, 162 may include multiple storage devices, such as hard drives, random access memories, disks, disk arrays, tape drives, or any other type of storage devices. The data centers 142, 152, 162 may implement any of a number of architectures and technologies, including but not limited to direct attached storage (DAS), network attached storage (NAS), storage area network (SAN), Fiber Channel (FC), Fiber Channel over Ethernet (FCoE), hybrid architecture networks, etc. In addition to storage devices, the data center may also include many other devices, such as cables, routers, etc. The data centers 142, 152, 162 may be virtualized environments.

[0044] Figure 2 2 is a diagram of an example distributed system for performing inner joins. As shown, example system 200 may include multiple servers 210 and 270 coupled to network 250. Servers 210 and 270 may be located in different data centers, for example, server 210 may be Figure 1 Server 140 at data center 142 in FIG. 1 , and server 270 may be Figure 1 Alternatively, servers 210 and 270 may be located at the same data center, for example, server 210 may be Figure 1 140 at the data center 142 in FIG. 140, and server 270 may be another server (not in FIG. 140) also at the data center 142. Figure 1 Therefore, Figure 1 Each of the servers 140, 150, 160 may be configured as Figure 2 The system may further include a client 260 capable of communicating with the servers 210 and 270 via the network 250.

[0045] The server 210 may include a processor 220, a memory 230, a clock 235, and other components typically found in a general-purpose computer. The memory 230 may store information accessible by the processor 220, including instructions 232 that may be executed by the processor 220. The memory may also include data 234 that may be retrieved, manipulated, or stored by the processor 220. The memory 230 may be a type of non-transitory computer-readable medium that is capable of storing information accessible by the processor 220, such as a hard drive, a solid-state drive, a tape drive, an optical storage device, a memory card, a ROM, a RAM, a DVD, a CD-ROM, a writeable and read-only memory. The processor 220 may be a well-known processor or other lesser-known types of processors. Alternatively, the processor 220 may be a dedicated controller, such as an ASIC.

[0046] Instruction 232 can be a set of instructions (such as computing device code) directly executed by processor 220 or a set of instructions (such as script) indirectly executed. In this regard, the terms "instruction", "step" and "program" can be used interchangeably in this article. Instruction 232 can be stored in an object code format to be directly processed by processor 220, or stored in other types of computer languages, which include a set of scripts or independent source code modules, which are interpreted or pre-compiled as needed. The functions, methods and routines of instructions are explained in more detail in the aforementioned examples and the following example methods. Instruction 232 may include any example features described herein. For example, instruction 232 may include: dividing the input value into blocks, copying the blocks into the necessary number of copies, assigning block keys to the copied blocks, distributing the copied blocks to multiple computing devices in the distributed system for processing, and receiving the results of the inner connection.

[0047] According to the instructions 232, data 234 can be retrieved, stored or modified by the processor 220. For example, although the system and method are not limited to a specific data structure, the data 234 can be stored in a computer register, stored in a relational database as a table with multiple different fields and records or an XML document. The data 234 can also be formatted in a computer readable format, such as but not limited to binary values, ASCII or Unicode. Moreover, the data 234 can include information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other memories (including other network locations) or information that functions are used to calculate related data. The data 234 can store a portion of the input table or the entire input table. In addition, the data 234 can also store blocks containing partition values ​​of the input table, block key mappings of replicated blocks, and results of inner joins.

[0048] although Figure 2The processor 220 and memory 230 are functionally illustrated as being within the same block, but the processor 220 and memory 230 may actually include multiple processors and memories that may or may not be stored within the same physical housing. For example, some instructions 232 and data 234 may be stored on a removable CD-ROM, while other instructions and data may be stored within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from the processor 220 but still accessible to the processor 220. Likewise, the processor 220 may actually include a collection of processors that may or may not operate in parallel.

[0049] Servers 210 and 270 may be at a node of network 250 and may be able to communicate directly and indirectly with other nodes of network 250. For example, servers 210 and 270 may include a network server that may be able to communicate with client 260 via network 250, such that it transmits information to a client application using network 250. Servers 210 and 270 may also include many computers, such as a load-balanced server farm, that exchange information with different nodes of network 250 for receiving, processing, and transmitting data to client devices. In this case, the client computer will typically be located at a different node of network 250 than the computers that make up servers 210 and 270. Although in Figure 2 Only a few servers 210 , 270 are depicted in FIG. 2 , but it should be appreciated that a typical system may include a large number of connected servers, each located at a different node of the network 250 .

[0050] Similar to servers 210 and 270, each client 260 can be configured with a processor 262, memory 263, instructions 264, and data 267. Each client 260 can be a personal computer, intended to be used by a person with all the internal components typically found in a personal computer, such as a central processing unit (CPU), a CD-ROM, a hard drive, and a display device 265 (e.g., a monitor with a screen, a projector, a touch screen, a small LCD screen, a television, or another device, such as an electrical device that can be operable to display information processed by the processor 262), a speaker, a modem and / or a network interface device, a user input component 266 (such as a mouse, keyboard, touch screen, or microphone), and all components for connecting these elements to each other. Moreover, computers according to the systems and methods described herein can include devices capable of processing instructions and transmitting data to and from people and other computers (including general purpose computers, PDAs, tablet computers, mobile phones, smart watches, network computers lacking local storage capabilities, set-top boxes for televisions, and other networked devices).

[0051] The client 260 may include an application interface module 268. The application interface module may be used to access services available to a server (such as, servers 210 and 270). The application interface module may include subroutines, data structures, object classes, and other types of software components for allowing the server and client to communicate with each other. In one aspect, the application interface module 268 may be a software module that may be operated in conjunction with several types of operating systems known in the art. For example, the client 260 may be connected to a structured query language (SQL) database server that may be operated in conjunction with the application interface module 268 to save and retrieve information data. The memory 263 coupled to the client 260 may store data 267 accessed by the application interface module 268. The data 267 may also be stored on a removable medium that may be connected to the client 260, such as a disk, a tape, an SD card, or a CD-ROM.

[0052] Servers 210 and 270 and client 260 may be capable of direct and indirect communication, such as through network 250. For example, using Internet sockets, client 260 can be connected to services operating on remote servers 210 and 270 through the Internet protocol suite. Servers 210 and 270 can set up listening sockets that can accept initiated connections for sending and receiving information. Network 250 and intermediate nodes can include various configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, one or more company-specific private networks using communication protocols, Ethernet, WiFi (e.g., 802.81, 802.81b, g, n or other such standards) and HTTP and various combinations of the foregoing. Devices capable of transmitting data to and from other computers can facilitate such communications, such as modems (e.g., dial-up, cable or optical fiber) and wireless interfaces.

[0053] although Figure 2The server 210 and the client 260 are shown as separate blocks, each containing its own processor and memory, but the operations described herein may involve a single computing device or many computing devices, such as in a "cloud". For example, various operations described herein as involving a single computing device (e.g., a single central processing unit (CPU) in a single server) may involve multiple computing devices (e.g., multiple processors in a load-balanced server farm). Similarly, memory components located at different locations can store different parts of the instructions 232 and together form a medium for storing instructions. In some examples, the client 260 can act as a thin client, wherein the server 210 performs all or nearly all operations that are not directly related to receiving information via the user input component 266 and the display device 265 and providing information to the user. Various operations described herein as being performed by a computing device can be performed by a virtual computing device. By way of example, the instruction 232 can be specific to a first type of server, but the related operations can be performed by a second type of server running a hypervisor that simulates the first type of server. The operation can also be performed by a container, for example, a computing environment that does not rely on an operating system bound to a specific type of hardware.

[0054] Example Method

[0055] With respect to the above example system, an example method is now described. This method may be performed using any of the above system, modifications thereof, or a variety of systems with different configurations. It should be understood that the operations involved in the following method do not have to be performed in the exact order described. Instead, various operations may be processed in a different order or simultaneously, and operations may be added or omitted.

[0056] Figure 3 An example flow chart 300 is provided for efficiently performing an inner join on two very large inputs on a distributed system. Figures 4A to 4D A simplified graphical representation of an example flowchart 300 is shown. Example features of flowchart 300 may be implemented on one or more computing devices (e.g., Figure 1 One or more computing devices may be executed on one or more processors (such as, Figure 2 The example features of flowchart 300 are executed on a processor 220 of the ... Figure 2 Various data items are stored in memory 230 of the processor. For example, flowchart 300 may be stored in instruction 232, for example, while input tables and various data items (such as blocks, replicated blocks, block keys, and results of inner joins) may be stored in data 234, for example.

[0057] Reference Figure 3In block 310, a request is received to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs. The storage device may be one or more computing devices such as Figure 1 One or more of the servers 140, 150, 160) from a client (such as, Figure 1 The request may be received by one or more processors (such as Figure 2 The request is received at the processor 220 of the processor. The first input and the second input may be stored at one or more computing devices, such as stored at Figure 1 At one or more of the servers 140, 150, 160. For example, the first input and the second input may be stored in the data 234.

[0058] Figure 4A A visual diagram involving two input tables and corresponding data is shown for illustration purposes. A first input table 410 contains shirt colors that multiple people like, and a second input table 420 contains pants colors that multiple people like. The names of the people in each input table are keys, and the shirt colors / pants colors are values. For example, [Anne - Red] is a key-value pair from the first input table 410, and [Bob - Pink] is a key-value pair from the second input table 420. Therefore, for the key "Anne", there are five values ​​in the first input table 410, and there are three values ​​in the second input table 420. Although there are values ​​in each of the first input table 410 and the second input table 420 for some keys, such as the key "Anne" and the key "Bob", there is only one value in one of the two input tables 410, 420 for some other keys, such as the key "Charles" and the key "David". The first and second input tables 410, 420 are shown here as truncated, but each can be a very large table, for example, the first input table 410 can contain 5,000 keys, each key has a maximum of 1,000,000 values; the second input table 420 can contain 3,000 keys, each key has a maximum of 2,000,000 values.

[0059] although Figure 4A The first input table 410 and the second input table 420 in the example of FIG. 4 appear to be sorted by key, but alternatively, the first input table 410 and the second input table 420 may be unordered key-value pairs. Figure 4A As shown, the first input table 410 and the second input table 420 each have only two columns, however, one or both input tables may have multiple columns. As an example, the first input table may have a third column "shirt style" and a fourth column "shirt size". In this case, the key-value pair from the first input table may be, for example, [Anne - red, golf, large].

[0060] Refer again Figure 3 In block 320, a block size is determined as a target number of values ​​to be divided into blocks. Then, in block 330, for each key, the values ​​in the first input are divided into a first set of blocks. In block 340, for each key, the values ​​in the second input are divided into a second set of blocks. The block size may be determined by one or more computing devices, such as, Figure 1 One or more of the servers 140, 150, 160. The block size may be determined by one or more processors, such as, Figure 2 The values ​​of the first input and the second input may be divided by one or more computing devices, such as, Figure 1 One or more of the servers 140, 150, 160. The first input and the second input may be divided by one or more processors, such as, Figure 2 The blocks in the first and second block sets may be stored at one or more computing devices, for example, stored at Figure 1 At one or more of the servers 140, 150, 160. For example, the blocks in the first and second block sets may be stored at Figure 2 The data is 234.

[0061] Figure 4A A representation of the steps of using blocks 320 to 340 with a smaller block size of 2 is shown. According to the first input table 410, for the key "Anne", two values ​​are divided into blocks <1, 1, Anne>, two values ​​are divided into blocks <2, 1, Anne>, and one value is divided into blocks <3, 1, Anne>, and these three blocks constitute a first block set 412 for the key "Anne". For the key "Bob", two values ​​are divided into blocks <1, 1, Bob>, and one value is divided into blocks <2, 1, Bob>, and these two blocks constitute a first block set 414 for the key "Bob". Similarly, according to the second input table, for the key "Anne", two values ​​are divided into blocks <1, 1, Anne>, and one value is divided into blocks <1, 2, Anne>, and these two blocks constitute a second block set 422 for the key "Anne". For the key "Bob", two values ​​are divided into <1, 1, Bob>, and this one block constitutes a second block set 424 for the key "Bob". The steps of blocks 320 through 340 are iterated for each key until all values ​​across all keys in both input tables are partitioned into chunks.

[0062] Although a very small block size is chosen for illustrative purposes, the block size may be determined based on a number of factors discussed further below. For example, partitioning the inputs (which may contain millions or billions of values, respectively) into small blocks of a block size of 1,000 values ​​may allow each computing device in the distributed system to trivially process an operation involving two such blocks (one from each input table). For example, an inner join operation on two blocks of a block size of 1,000 values ​​may take on the order of several megabytes (MB) of operating memory and on the order of several seconds to process, and therefore, some computing devices may trivially process it. Similar to the block size of the two examples, if the block size is 1,000, then for each key of the first input table, approximately 1,000 values ​​are partitioned into each block in the first block set, and for each key of the second input table, approximately 1,000 values ​​are partitioned into each block in the second block set. For example, if there are 99,900 values ​​for a particular key from the first input table, they can be divided into 100 blocks (first block set), and for that particular key, if there are 200,005 values ​​from the second input table, they can be divided into 200 blocks (second block set).

[0063] The values ​​may be partitioned into each block in the block set in any of a number of ways. Up to the block size number of values ​​may be partitioned into each block until there are no more values ​​for the key. For example, if the block size is 1,000 and there are 99,900 values ​​in a particular key, then 1,000 values ​​may be partitioned into the first block, the next 1,000 values ​​may be partitioned into the second block, and so on until the last 900 values ​​are partitioned into the 100th block. Additionally or alternatively, the number of blocks required for each key may be first determined, for example, by counting the total number of times each key appears in the input table. Then, for each key, the blocks in the block set are set in a uniform distribution, and then the values ​​for each key are randomly assigned to the blocks. For example, if the block size is 1,000 and there are 99,900 values ​​in a particular key, then it is first determined that 100 blocks are required in the block set, and then the 100 blocks are set in a uniform distribution. Next, the 99,900 values ​​can be randomly partitioned into 100 blocks each, so that the average number of values ​​per block in the set of blocks is roughly equal to the block size. The partitioning can be parallelized so that more than one value can be assigned to one or more blocks at the same time.

[0064] Reference Figure 3 In block 350, for each key, the first set of blocks is replicated by the number of blocks in the second set of blocks for the same key to obtain a first set of replicated blocks. In block 360, for each key, the second set of blocks is replicated by the number of blocks in the first set of blocks for the same key to obtain a second set of replicated blocks. The blocks in the first set of blocks and the second set of blocks may be stored in memory by one or more computing devices (such as, Figure 1 The first set of blocks and the second set of blocks may be replicated by one or more processors (such as Figure 2 The replicated blocks in the first and second sets of replicated blocks may be stored at one or more computing devices, for example, at Figure 1 At one or more of the servers 140, 150, 160. For example, the replicated blocks in the first and second sets of replicated blocks may be stored at Figure 2 The data is 234.

[0065] from Figure 4A continue, Figure 4B Shows the Figure 4A A visual diagram of the data from the two input tables is provided for illustration purposes. Figure 4A and 4B , because the second block set 414 of the key "Anne" has two blocks, the first block set 412 of the key "Anne" is replicated into two copies (indicated by the dotted box), thereby obtaining the first replicated block set 416 of the key "Anne", which has a total of six blocks. Similarly, because the first block set 412 of the key "Anne" has three blocks, the second block set 414 of the key "Anne" is replicated into three copies (indicated by the dotted box), thereby obtaining the second replicated block set 426 of the key "Anne", which has a total of six blocks. Therefore, for the key "Anne", the first and second replicated block sets 416, 426 both have six replicated blocks.

[0066] For another example with a larger number, if the second block set has 200 blocks for a particular key, the first block set for that particular key can be replicated as 200 copies (the first replicated block set); and if the first block set has 100 blocks for that particular key, the second block set for that particular key can be replicated 100 times (the second replicated block set). Thus, each of the two input tables has the same number of replicated blocks for that particular key, which is 20,000.

[0067] Reference Figure 3 In block 370, a block key is assigned to each replicated block, the block key including not only the corresponding block of all values ​​in the replicated block, but also additional information identifying the replicated block. The block key may be assigned by one or more computing devices, such as, Figure 1 One or more of the servers 140, 150, 160. Block keys may be assigned by one or more processors, such as, Figure 2 The block key may be stored at one or more computing devices, for example, at Figure 1At one or more of the servers 140, 150, 160. For example, the block key may be stored at Figure 2 The data is 234.

[0068] Reference Figure 4A and 4B , the block key is shown in brackets “<>”. In this example, the block key assigned to each block includes a first index, a second index, and a key corresponding to the value in the block in the format of <first index, second index, key>. A first index can be assigned to each block in the block set to distinguish the values ​​divided into the block. Therefore, for the key “Anne”, first indexes 1, 2, and 3 are assigned to each block in the first block set 412, respectively. A second index can be assigned to a replicated block set to distinguish one replicated block from other copies in the replicated block set. Therefore, for the key “Anne”, second indexes 1 and 2 are assigned to each replicated block in the first replicated block set 416, respectively.

[0069] The block key also includes the corresponding keys of all values ​​in the replicated block to further distinguish the replicated block from replicated blocks of other keys. Therefore, <1, 1, Anne> and <1, 1, Bob> can be distinguished from each other. In this way, each replicated block of all keys of the input table has a unique block key. For another example with a large number, if the value of a specific key is to be divided into 100 blocks and then replicated into 200 copies, a first index from 1 to 100, a second index from 1 to 200, and a specific key can be assigned to each replicated block.

[0070] Reference Figure 4A and 4B , the first and second indexes of the second replicated block set 426 of the block "Anne" are assigned in a manner opposite to the first and second indexes of the first replicated block set 416 of the key "Anne". For the second replicated block set 426 of the key "Anne", the second index is assigned to each block in the second block set to distinguish the value divided into the block, and the first index is assigned to each replicated block in the second replicated block set to distinguish one replicated block from other copies in the second replicated block set. Therefore, for the key "Anne", the second indexes 1, 2 are assigned to each block in the second block set 422, respectively, and the first indexes 1, 2, 3 are assigned to each replicated block in the second replicated block set 426, respectively. In this way, each replicated block in the first replicated block set 416 has a corresponding replicated block in the second replicated block set 426 with a matching key, and no two pairs of replicated blocks are identical copies of the blocks.

[0071] If the first index and the second index are not assigned in an opposite manner, the same value may be processed more than once, while some other values ​​will not be processed at all. To illustrate this undesirable situation, <1, 1, Anne> and <1, 2, Anne> of the first replication block set 416 are copies of each other, while <1, 1, Anne> and <2, 1, Anne> of the second replication block set 426 are copies of each other. Therefore, <1, 1, Anne> of the first replication block set 416 matches <1, 1, Anne> of the second replication block set 426, and if the first and second indexes are not assigned in an opposite manner, <1, 2, Anne> of the first replication block set 416 will match <2, 1, Anne> of the second replication block set 426, which means that the resulting two replication block pairs are exact copies of each other, and therefore the same value will be processed twice. Further, if the first index and the second index are not assigned in the opposite manner, then <3, 1, Anne>, <3, 2, Anne> of the first set of replicated blocks 416 will not have any replicated blocks in the second set of replicated blocks 426 with matching block keys. Therefore, reversing the first index and the second index is a way to ensure that all replicated blocks are paired and no two pairs of replicated blocks are identical copies of a block. Figure 4B , by matching each replicated block from the first set of replicated blocks 416 with one from the second set of replicated blocks 426 having the same block key, the result is that the key "Anne" 430 has six replicated block pairs.

[0072] The block key mapping for all block keys of the first and second sets of replicated blocks can be sent to each of the multiple computing devices involved in processing the inner join. The block key mapping itself can be very large, depending on how many replicated blocks are created for the input. In this example, the mapping portion for the key "Anne" only contains six block keys for the set of replicated blocks. However, for example, another key of the input table may have 20,000 replicated blocks, which means there are 20,000 block keys for that key alone. The size of the block key mapping for an input of a given size depends on the selected block size (an example of which will be discussed further below). For example, for an input table of a given size, if the block size is increased, the number of blocks required to partition all values ​​in the input table will be reduced, thereby reducing the size of the block key mapping.

[0073] Block keys can also be used to partition values ​​into sets of blocks in the random manner described above. For example, after determining that the number of blocks required for a particular key in the first input table is 100, 100 blocks can be created. Block keys are then assigned to each of the 100 blocks by randomly assigning a first index from 1 to 100, a second index of 1 (the first copy), and the particular key. The 100 blocks are then set up in a uniform distribution based on their first indexes. Next, each time a value for that particular key appears in the first input table, that value is randomly assigned a first index from 1 to 100 and partitioned into the corresponding blocks.

[0074] Despite Figure 4A and 4B In the example of , the format of the block key is <first index, second index, key>, but the block key can also be any of a variety of formats, as long as each replicated block of all keys of the input table has a unique block key, each replicated block in the first set of replicated blocks has a corresponding replicated block in the second set of replicated blocks with a matching block key, and no two pairs of replicated blocks are identical copies of the blocks. For example, the first index and the second index can be combined into an index that contains the same type of information as the first index and the second index described above. For another example, each block key can be an encrypted number containing the same type of information as the first index, the second index and the key described above. For yet another example, the block key of the first set of replicated blocks can be different from the block key of the second set of replicated blocks, but matching of replicated block pairs is still allowed. For example, instead of assigning the first index and the second index in the opposite manner as described above, matching can be performed in the opposite manner, so that the replicated blocks are paired by matching the first index of the first set of replicated blocks with the second index of the second set of replicated blocks, and matching the second index of the first set of replicated blocks with the first index of the second set of replicated blocks.

[0075] Reference Figure 3 , in block 380, each pair of replicated blocks (one from each input table) with matching block keys is published to one of the plurality of computing devices for processing. Alternatively, the pairs may be distributed only to available computing devices. For example, such as Figure 1 One of the computing devices such as server 140 may determine such as Figure 1 The servers 150 and 160 of the distributed system have sufficient operating memory to process one or more pairs of replicated blocks, and then the server 140 distributes the one or more pairs of replicated blocks to the servers 150 and 160 for processing. The replicated block pairs can be processed by one or more processors (such as Figure 2 The replicated block pairs may be stored at one or more computing devices, for example, stored at Figure 1 At one or more of the servers 140, 150, 160. For example, the replicated block pairs may be stored at Figure 2The data is 234.

[0076] Reference Figure 4B , key "Anne" 430 has six replicated block pairs, each pair having a matching block key. Each of the six replicated block pairs of key "Anne" 430 can be distributed to computing devices for processing. Each of the plurality of computing devices can process one pair. Alternatively, one or more of the plurality of computing devices can process more than one pair. For example, the six replicated block pairs of key "Anne" 430 can be sent to six different computing devices, each processing one pair; or to three computing devices, each processing two pairs; or to two computing devices, one processing two pairs and the other processing four pairs; or all to one computing device, and so on.

[0077] The distribution of replicated block pairs can be based on any of a number of factors. The distribution can depend on the block size, the total number of replicated block pairs, the total number of multiple computing devices in the distributed system, and the processing power of each of the multiple computing devices. For example, for two large inputs, a block size of 1,000 is selected, resulting in a total of 2,000,000 replicated block pairs, there are 500 computing devices in the distributed system, and each of the 500 computing devices has 32GB of operating memory, the 2,000,000 pairs can be distributed to the 500 computing devices, each of which has 4,000 pairs to process. However, if there are 2,000,000 computing devices in the distributed system, the 2,000,000 pairs can be evenly distributed to all 2,000,000 computing devices, each processing one pair; if there are more than 2,000,000 computing devices in the distributed system, some computing devices will not be able to process any pairs. Thus, for an input of a given size, a smaller block size may have the added benefit of ensuring that all computing devices in the distributed system are distributed with one or more pairs of replicated blocks for processing, thereby using the resources of the distributed system more efficiently. Further, if the 500 computing devices in the distributed system do not have identical operating memory, then the 2,000,000 pairs may be distributed such that computing devices with larger operating memory will process more pairs than computing devices with smaller memory, and computing devices with insufficient operating memory to process even a single block pair of block size 1,000 will not distribute any pairs.

[0078] This distribution can be combined with dynamic work rebalancing techniques. Continuing from the example above, if 2,000,000 replicated block pairs are initially sent to 500 computing devices, each computing device has 4,000 pairs to process. For any of a number of reasons, some computing devices may process the job faster than other computing devices, some computing devices may process much slower than other computing devices, and some computing devices may be stuck or fail. Thus, the 500 computing devices can be monitored, and if it is determined that one or more computing devices are processing very slowly (or stuck / failed), one or more computing devices can send some or all pairs to computing devices that are processing quickly (or at least processing normally), or send some or all pairs to idle computing devices.

[0079] Reference Figure 3 In block 390, the result of the inner join is received from a plurality of computing devices. The result may include a cross product or a portion of a cross product of one or more pairs of replicated blocks. The cross product on the replicated block pair is obtained by traversing all values ​​in one replicated block through each value of another replicated block (as illustrated below), thereby obtaining all possible combinations of value pairs from the replicated block pairs. The result may include all cross products collected from all replicated block pairs of a key or all cross products collected from any replicated block pairs of all keys, or any subset thereof. The result may be further formatted and / or encoded in any of a variety of ways. The result of the inner join may also be collected into a file for output or further processing.

[0080] Reference Figure 1 , such as Figure 1 One of the computing devices such as server 140 of FIG. 140 may receive some or all value pairs of the cross product for each pair of replicated blocks that it previously distributed to servers 150 and 160 in block 380. For example, one or more processors (such as Figure 2 The result of the inner join can be stored in Figure 2 The data is 234.

[0081] As another alternative, the results of the inner join of all pairs of replicated blocks for each key may be collected separately on one or more computing devices. For example, the cross products of all pairs of replicated blocks for the first key may be collected and stored in Figure 1 On the server 140, the cross products of all pairs of replicated blocks of the second key may be collected and stored in Figure 1 on server 150, and so on.

[0082] from Figure 4B continue, Figure 4C Shows the Figure 4A A visual diagram of the data from the two input tables is provided for illustration purposes. Figure 4C In the example of , each of the six pairs of replicated blocks for key "Anne" 430 is processed. The processing shown here involves taking cross products of the values ​​in the replicated block pairs. For example, the cross product of [red, yellow] and [black, gray] is four value pairs [red-black, red-gray, yellow-black, yellow-gray]. The result of the inner join may include all or some subsets of the value pairs in the cross product of key "Anne" 440 (for example, only the value pairs involving "red", the first five value pairs, every other value pair, etc.). All or some subsets of the value pairs in the cross products of other keys (for example, key "Bob") may also be part of the inner join results. The results of the inner join can be further formatted and / or encoded in any of a variety of ways. The results of the inner join can also be collected into a file for output or further processing.

[0083] from Figure 4C continue, Figure 4D Shows the Figure 4A A visual diagram of the data from the two input tables is provided for illustration purposes. Figure 4D , the file here presents the results of the inner join in a table called "inner join result table" 450. Instead of a table, the results of the inner join from each of the multiple computing devices can be presented in other formats or collected for further processing, such as through another operation.

[0084] Figure 5 Another example flowchart 500 is shown for efficiently performing an inner join of two very large inputs on a distributed system, including estimating the number of blocks required for each key. For example, some or all features of flowchart 500 may be performed in conjunction with flowchart 300. Figure 4A The simplified graphical representation shown is also used below to illustrate features of the example flowchart 500. The example features of the flowchart 500 may be executed on one or more computing devices in a distributed system, for example, Figure 1 One or more of the servers 140, 150, 160. One or more computing devices may be on one or more processors (such as, Figure 2 The example features of flowchart 500 are executed on a processor 220 of the ... Figure 2 Various data items are stored in the memory 230 of the instruction 232. For example, the flowchart 500 can be stored in the instruction 232, and the input table and various data items (such as blocks, replicated blocks, block keys, results) can be stored in the data 234.

[0085] In block 510, an approximate number of blocks required for each key is estimated for at least one of the first input and the second input by sampling at least one of the first input and the second input at intervals of each block size. The sampling may be performed by one or more computing devices such as Figure 1 The sampling may be performed by one or more processors (such as, Figure 2 The first input and the second input and the sampling result may be stored at one or more computing devices, for example, at Figure 1 At one or more of the servers 140, 150, 160. For example, the first input and the second input and the sampling result can be stored in the data 234.

[0086] Reference Figure 4A , where the block size is 2, the first input table 410 can be sampled at every two key-value pairs for the key. Here, the sampling results for the truncated portion of the first input table 410 shown here are "Anne", "Anne", "Anne", "Bob", "Charles". After passing through the entire first input table 410, a count for each key is obtained, which is the approximate number of blocks required for each key of the first input table. For example, if the count is Anne-3, Bob-1, Charles-123, Jane-4,680, Mary-25,023, etc., these will be the approximate number of blocks required for each key. Although sampling affects accuracy, it can save a lot of time. Here, sampling at every other key-value pair means that only the keys of half of the key-value pairs are actually known, however, it also only takes half the time compared to counting every single key-value pair. For another example, if the block size is 1,000, sampling every 1,000 key-value pairs saves 1,000 times the counting time, which can significantly improve efficiency.

[0087] In block 520, key-value pairs of at least one of the first input and the second input are grouped by key. The grouping may be performed by one or more computing devices such as Figure 1 The packet can be executed on one or more processors (such as, Figure 2 The grouped first input and second input may be stored at one or more computing devices, for example, at Figure 1 At one or more of the servers 140, 150, 160. For example, the grouped first input and the second input may be stored in the data 234.

[0088] Reference Figure 4A, although the key-value pairs in the truncated portion of the first input table 410 appear to have been grouped by key, assume that the key-value pairs in the remainder of the first input table 410 are not grouped by key, for example, the key-value pairs after Charles may appear as [Charles-red], [Jane-blue], [Charles-pink], [Zack-white], [Jane-purple], ..., and so on. The key-value pairs of the first input table 410 are then grouped by key so that all values ​​with the same key are grouped together. Note that the order in which the keys appear does not matter, for example, all values ​​for the key "Zack" may appear before all values ​​for the key "Jane".

[0089] In block 530, the remaining number of values ​​for each key is determined using the assumption that each of the approximate number of blocks contains the block size number of values. The remaining number of values ​​may be determined by one or more computing devices, such as, Figure 1 The remaining number of values ​​may be determined by one or more processors, such as, Figure 2 The remaining number of values ​​may be stored at one or more computing devices, for example, at Figure 1 At one or more of the servers 140, 150, 160. For example, the remaining number of values ​​may be stored in data 234.

[0090] Reference Figure 4A , for the first input table 410, the approximate number of blocks for the key "Anne" is 3, and the block size is 2. Considering the values ​​grouped by the key "Anne", since it is assumed that the approximate number of blocks each contain the block size number of values, the first six values ​​for the key "Anne" can be skipped, and the remaining number of values ​​for the key "Anne" is determined to be zero by counting from the seventh value without the key "Anne".

[0091] For another example, the approximate number of blocks for the key "Charles" is 123. Because it is assumed that the approximate number of blocks each contain the block size number of values, the first 236 values ​​of the key "Charles" can be skipped, and by counting from the 237th value, it is determined that the 237th and 238th values ​​are also for the key "Charles", but the 239th value is for another key. Therefore, the remaining number of values ​​for the key "Charles" is determined to be 2.

[0092] In block 540, the approximate number of blocks for one or more keys is adjusted based on the remaining number of values. The approximate number of blocks may be adjusted by one or more computing devices, such as, Figure 1In this regard, the approximate number of blocks may be adjusted by one or more processors, such as, Figure 2 The adjusted approximate number of blocks may be stored at one or more computing devices, for example, at Figure 1 The adjusted approximate number of blocks may be stored in data 234, for example.

[0093] Reference Figure 4A , the remaining number of values ​​for the key "Anne" is zero, therefore, the approximate number of blocks does not need to be adjusted. For another example, the remaining number of values ​​for the key "Charles" is 2, therefore the approximate number of blocks for the key "Charles" can be adjusted from 123 to 124. A predetermined threshold can be set so that the approximate number of blocks for each key is adjusted only when the remaining number of values ​​is above the predetermined threshold. Continuing from the example of "Charles", if the predetermined threshold is 5, but the remaining number of values ​​is 2 (less than 5), then the approximate number of blocks for the key "Charles" will not be adjusted. In this case, the two additional values ​​for the key "Charles" will be divided into one or more of the 123 blocks.

[0094] In block 550, an approximate number of blocks required for one or more keys omitted during the sampling period may be determined. The approximate number of blocks for one or more keys omitted during the sampling period may be determined by one or more computing devices, such as Figure 1 The approximate number of blocks of one or more keys omitted during sampling can be determined by one or more processors (such as, Figure 2 The approximate number of blocks of one or more keys omitted during sampling can be stored at one or more computing devices, for example, stored at Figure 1 At one or more of the servers 140, 150, 160. For example, an approximate number of blocks of one or more keys omitted during sampling may be stored in the data 234.

[0095] Reference Figure 4A, assuming that the next key-value pair after [Charles-Silver] is [Ed-Gray], then the sampling at every other value will omit [Ed-Gray], and if this is the only key-value pair for the key "Ed" in the first input table 410, then there is no approximate number of blocks for the key "Ed". After the key-value pairs of the first input table 410 are grouped by key, it can be determined that although the grouping results from block 520 indicate that the first input table 410 has the key "Ed", the sampling results from block 510 do not have the approximate number of blocks for the key "Ed". For omitted keys for which the approximate number of blocks was not determined during sampling, the approximate number of blocks can be set to 1. Additionally or alternatively, the approximate number of omitted keys for which the approximate number of blocks was not determined during sampling can be set by actually counting the number of values ​​for the omitted keys and dividing the number of values ​​by the block size.

[0096] The estimated number of blocks per key may also be used to change the block size. The estimated size of the block key map may be determined by the approximate number of blocks for all keys. Based on the estimated size of the block key map, one or more computing devices (such as servers 140, 150, 160) may determine that, although the value of the number of block sizes may be suitable for each of servers 140, 150, 160 to process, at least some of servers 140, 150, 160 may run out of operational memory if each of servers 140, 150, 160 must also maintain the block key map in its operational memory. Therefore, in this case, the block size may be increased accordingly.

[0097] Reference Figure 3 At block 320, the block size may be determined by one or more computing devices (such as, Figure 1 One or more of the servers 140, 150, 160), by one or more processors (e.g., Figure 2 The various concepts above have been introduced, and the factors on which the block size can be based will be discussed below. The block size can be determined based on the number of multiple computing devices to process inner joins. For example, if there are a large number of computing devices for processing inner joins, the block size can be set to be smaller so that each computing device can perform a smaller job, thereby speeding up the processing at each of the multiple computing devices. The block size can be determined based on the processing power of each of the multiple computing devices. For example, if each of the multiple computing devices has a large operating memory, the block size can be increased so that the block can be copied less times, thereby saving the total space occupied by all multiple computing devices on the operating memory. Therefore, the known capabilities of multiple computing devices can be used to determine the block size, thereby determining the division of data for processing. The block size can also be determined as the minimum size at which the block key mapping may not cause any of the multiple computing devices to exhaust the operating memory.

[0098] The block size may be determined based on a dynamic work rebalancing technique. Smaller block sizes improve uniformity in terms of the number of values ​​partitioned into blocks, meaning that it is less likely that one or more computing devices will run much slower than other computing devices due to uneven workloads. To illustrate, if the block size is 1,000, each replicated block in the set of replicated blocks may have anywhere from 1 to approximately 1,000 values, which would result in 20,000 replicated block pairs for a key, distributed to 20 computing devices, each processing 1,000 replicated block pairs. Given a chance, some of the 20 computing devices may receive replicated block pairs at the higher end of the range (e.g., block pairs containing approximately 1,000 values), while other computing devices may receive replicated block pairs at the lower end or in the middle of the range. Computing devices that happen to receive more of the larger replicated block pairs will therefore be more likely to stall. In contrast, if the block size is 10, each replicated block in the set of replicated blocks is likely to have anywhere from 1 to about 10 values ​​(much less than 1 to about 1,000 values), and if 2,000,000 replicated block pairs are generated for a key, still distributed to 20 computing devices, each handling 100,000 replicated block pairs, then the chance that any computing device may receive a particularly large replicated block is greatly reduced. Thus, reducing the block size increases the uniformity in terms of the number of values ​​between replicated block pairs, thereby making it less likely that any computing device will be stuck, which can reduce the need for dynamic work rebalancing. Further, dynamic work rebalancing can be done more efficiently when the blocks are smaller and more uniform in terms of the number of values. For example, instead of moving one large replicated block pair with 1,000 values ​​per block from a stalled computing device to another computing device, moving 100 small replicated block pairs with 10 values ​​per block may be much easier and also potentially distributed to more than one computing device (spreading the workload from the stalled computing device instead of dumping the entire workload onto another computing device).

[0099] In each of the foregoing examples, the first input table and / or the second input table (such as, Figure 4A The first input table 410 and the second input table 420 of the embodiment of the present invention may themselves be stored in a distributed manner (sharded on multiple computing devices). For example, the first input may be stored as a table located on more than one computing device (such as Figure 1shards on servers 140, 150, 160). Although the first input table and / or the second input table can be stored as shards on multiple computing devices, some or all of the preprocessing for one or both input tables can be performed locally where the key-value pairs of the input tables are stored. For example, a shard on server 140 that contains a portion of the first input table can have its values ​​locally divided into blocks. The blocks on server 140 can then also be locally replicated to the necessary number of replicas. Further, the replicated blocks on server 140 can be locally assigned block keys. Alternatively, although the first input table and / or the second input table are stored as shards on multiple computing devices, the entire input table can first be collected on a single computing device for preprocessing. For example, located at Figure 1 All fragments of the first input table on servers 140, 150, 160 can first be collected onto a single computing device (such as server 140), and then on the single computing device, the entire first input table can have its values ​​divided into blocks, replicate the blocks to the necessary number of replicas, and assign block keys to the replicated blocks.

[0100] Although the examples described herein involve inner joins of two input tables, the features described herein can be easily extended to inner joins of any number of input tables. For example, to perform an inner join of three input tables, the above example method can be used to first inner join the two input tables as an intermediate table, and then the intermediate table can be inner joined with the third input table to produce a result table, which is the inner join of all three input tables. Alternatively, the above example method can be performed on all three input tables simultaneously. In short, for each key, the values ​​from the three input tables are divided into three block sets, each of the three block sets is replicated by a number equal to the product of the number of blocks in the other two block sets for the same key, a block key is assigned to each replicated block in the three replicated block sets, the replicated blocks from the three replicated block sets are grouped by matching keys, the replicated block groups are distributed to multiple computing devices for processing, and the results of the inner join are received from the multiple computing devices.

[0101] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but may be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the embodiments should be made by way of illustration rather than by way of limitation of the subject matter defined by the claims. In addition, the provision of the examples described herein and the clauses expressed by phrases such as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples; the present invention is not intended to illustrate only one of many possible embodiments. Further, the same reference numerals in different figures may identify the same or similar elements.

[0102] Thus, when summarizing the subject matter of the foregoing and claims, in a distributed database, inner joins of very large data sets are performed by distributing smaller jobs to multiple computing devices. For each key, the values ​​in the first input are divided into a first set of blocks, and the values ​​in the second input are divided into a second set of blocks. For each key, the first set of blocks is replicated by multiple blocks in the second set of blocks, and the second set of blocks is replicated by multiple blocks in the first set of blocks. Each replicated block is assigned a block key that includes a key and additional information to identify the replicated block. Each pair of replicated blocks with matching block keys is distributed to one of the multiple computing devices. The results of the inner join are received from the multiple computing devices.

[0103] According to other aspects, the present disclosure includes the following embodiments:

[0104] Embodiment 1: A computer-implemented method, the computer-implemented method comprising:

[0105] receiving a request to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs;

[0106] determining, by one or more processors, a block size that is a target number of values ​​to be divided into blocks;

[0107] partitioning, by one or more processors, the values ​​in the first input into a first set of chunks for each key;

[0108] partitioning, by one or more processors, the values ​​in the second input into a second set of chunks for each key;

[0109] By one or more processors, for each key, copy the first set of blocks by the number of blocks in the second set of blocks for the same key to obtain a first set of copied blocks;

[0110] By one or more processors, for each key, copy the second set of blocks by the number of blocks in the first set of blocks for the same key to obtain a second set of copied blocks;

[0111] assigning, by one or more processors, a block key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block key comprising a key and additional information to identify the replicated block;

[0112] distributing, by one or more processors, each pair of replicated blocks having matching block keys to one of a plurality of computing devices for processing; and

[0113] A result of the inner join is received from each of the plurality of computing devices, the result of the inner join comprising at least one value pair from one of the pairs of replicated blocks.

[0114] Embodiment 2: The method according to embodiment 1, wherein determining the block size is based on at least one of a total number of the plurality of computing devices, processing capabilities of the plurality of computing devices, or uniformity in terms of the number of values ​​per block.

[0115] Embodiment 3: The method according to embodiment 1 or 2, wherein the block size is the minimum block size for which the block key mapping will not cause any of the plurality of computing devices to exhaust operating memory.

[0116] Embodiment 4. The method of any one of embodiments 1 to 3, wherein at least one of the partitioning of the values ​​in the first input or the partitioning of the values ​​in the second input is based on a uniform distribution.

[0117] Embodiment 5. A method according to any one of embodiments 1 to 4, wherein the block key includes a first index, and the first index is assigned in sequence when the values ​​in the first input are divided into a first set of blocks and the values ​​in the second input are divided into a second set of blocks.

[0118] Embodiment 6. The method according to any one of embodiments 1 to 5, wherein the block key includes a second index, and the second index is assigned in sequence when the first block set and the second block set are copied.

[0119] Embodiment 7. The method according to any one of embodiments 1 to 6, further comprising:

[0120] A block key map including all block keys is sent to each of the plurality of computing devices.

[0121] Embodiment 8. A method according to any one of embodiments 1 to 7, wherein the distribution to the multiple computing devices is based on at least one of the block size, the total number of replicated block pairs, the total number of the multiple computing devices, or the processing power of the multiple computing devices.

[0122] Embodiment 9. The method according to any one of embodiments 1 to 8, further comprising:

[0123] An approximate number of blocks required for each key is estimated by one or more processors for at least one of the first input or the second input by sampling at least one of the first input or the second input at intervals of each block size.

[0124] Embodiment 10. The method according to any one of embodiments 1 to 9, further comprising:

[0125] grouping key-value pairs of at least one of the first input or the second input by key;

[0126] determining a remaining number of values ​​for each key based on an assumption that each of the approximate number of blocks contains the block size number of values; and

[0127] Adjusts the approximate number of chunks required for one or more keys based on the remaining number of values ​​for the keys.

[0128] Embodiment 11. The method of any one of embodiments 1 to 10, wherein the approximate number of blocks required for one or more keys is adjusted when the remaining number of values ​​for the key is greater than a predetermined threshold.

[0129] Embodiment 12. The method according to any one of embodiments 1 to 11, further comprising:

[0130] determining that one or more keys were omitted during the sampling period; and

[0131] Sets the approximate number of blocks required to omit one or more keys during sampling.

[0132] Embodiment 13. The method of any one of embodiments 1 to 12, wherein at least one of the first input or the second input is stored as fragments on a plurality of selected computing devices.

[0133] Embodiment 14. The method according to any one of embodiments 1 to 13, further comprising:

[0134] At least one of the fragments of the first input or the fragments of the second input from the plurality of selected computing devices is collected onto a single pre-processing computing device.

[0135] Embodiment 15. The method of any one of embodiments 1 to 14, wherein the result of the inner join comprises a cross product of at least one of the pairs of replicated blocks.

[0136] Embodiment 16. A system, comprising:

[0137] A computing device adapted to communicate with a plurality of computing devices in a distributed computing environment, the computing device comprising one or more processors configured to:

[0138] receiving a request to perform an inner join on a first input including a first set of key-value pairs and a second input including a second set of key-value pairs;

[0139] determining a chunk size, which is a target number of values ​​to be divided into chunks;

[0140] partitioning the values ​​in the first input into a first set of chunks for each key;

[0141] partitioning the values ​​in the second input into a second set of chunks for each key;

[0142] For each key, copy the first block set by the number of blocks in the second block set for the same key to obtain a first copy block set;

[0143] For each key, copy the second block set by the number of blocks in the first block set for the same key to obtain a second copy block set;

[0144] assigning a block key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block key comprising a key and additional information to identify the replicated block;

[0145] distributing each pair of replicated blocks having matching block keys to one of a plurality of computing devices for processing; and

[0146] A result of the inner join is received from each of the plurality of computing devices, the result of the inner join comprising at least one value pair from one of the pairs of replicated blocks.

[0147] Embodiment 17. The system of embodiment 16, wherein the one or more processors are further configured to:

[0148] The block size is determined based on at least one of a total number of the plurality of computing devices, processing capabilities of the plurality of computing devices, or uniformity in terms of a number of values ​​per block.

[0149] Embodiment 18. The system according to embodiment 16 or 17, wherein the one or more processors are further configured to:

[0150] Each pair of replicated blocks is distributed based on at least one of a block size, a total number of pairs of replicated blocks, a total number of the plurality of computing devices, or a processing capability of the plurality of computing devices.

[0151] Embodiment 19. The system of any one of embodiments 16 to 18, wherein the one or more processors are further configured to:

[0152] An approximate number of blocks required for each key is estimated for at least one of the first input or the second input by sampling at least one of the first input or the second input at intervals of each block size.

[0153] Embodiment 20. A non-transitory computer-readable storage medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform a method comprising:

[0154] receiving a request to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs;

[0155] determining a chunk size, which is a target number of values ​​to be divided into chunks;

[0156] partitioning the values ​​in the first input into a first set of chunks for each key;

[0157] partitioning the values ​​in the second input into a second set of chunks for each key;

[0158] For each key, copy the first block set by the number of blocks in the second block set for the same key to obtain a first copy block set;

[0159] For each key, copy the second block set by the number of blocks in the first block set for the same key to obtain a second copy block set;

[0160] assigning a block key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block key comprising a key and additional information to identify the replicated block;

[0161] distributing each pair of replicated blocks having matching block keys to one of a plurality of computing devices for processing; and

[0162] A result of the inner join is received from each of the plurality of computing devices, the result of the inner join comprising at least one value pair from one of the pairs of replicated blocks.

Claims

1. A computer-implemented method, characterized in that The method comprises: receiving a request to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs; determining, by one or more processors, a block size, the block size being a target number of values ​​to be divided into blocks; partitioning, by the one or more processors, for each key value, values ​​in the first input associated with the corresponding key value into a first set of blocks, the first set of blocks comprising blocks having the determined block size; partitioning, by the one or more processors, for each key value, values ​​in the second input associated with the corresponding key value into a second set of blocks, the second set of blocks comprising blocks having the determined block size; replicating, by the one or more processors, for each key value, the first set of blocks by the number of blocks in the second set of blocks of the same key to obtain a first set of replicated blocks, the first set of replicated blocks comprising replicated blocks that are copies of blocks in the first set of blocks; replicating, by the one or more processors, for each key value, the second set of blocks by the number of blocks in the first set of blocks of the same key to obtain a second set of replicated blocks, the second set of replicated blocks including replicated blocks that are copies of blocks in the second set of blocks; assigning, by the one or more processors, a block-key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block-key comprising the key and additional information identifying the replicated block; distributing, by the one or more processors, each replicated block pair having a matching block-key to a computing device of the plurality of computing devices for processing; and A result of the inner join is received from each computing device of the plurality of computing devices, the result of the inner join comprising at least one value pair from one of the pairs of replicated blocks.

2. The method according to claim 1, wherein: Determining the block size is based on at least one of a total number of the plurality of computing devices, processing capabilities of the plurality of computing devices, or uniformity in terms of a number of values ​​per block.

3. The method according to claim 2, wherein: The block size is the smallest block size for which the block-key mapping does not cause any of the plurality of computing devices to run out of operating memory.

4. The method according to claim 1, wherein: At least one of the partitioning of values ​​in the first input or the partitioning of values ​​in the second input is based on a uniform distribution.

5. The method according to claim 1, wherein: The chunk-key includes a first index that is assigned sequentially when the values ​​in the first input are partitioned into the first set of chunks and the values ​​in the second input are partitioned into the second set of chunks.

6. The method according to claim 1, wherein: The block-key includes a second index that is assigned in sequence when the first set of blocks and the second set of blocks are copied.

7. The method according to claim 1, further comprising: A block-key map including all of the block-keys is sent to each computing device of the plurality of computing devices.

8. The method according to claim 1, wherein: The distribution to the plurality of computing devices is based on at least one of the block size, a total number of replicated block pairs, a total number of the plurality of computing devices, or a processing capability of the plurality of computing devices.

9. The method according to claim 1, further comprising: An approximate number of blocks required for each key value for at least one of the first input or the second input is estimated by the one or more processors by sampling at least one of the first input or the second input at intervals of each block size.

10. The method according to claim 9, further comprising: grouping key-value pairs of at least one of the first input or the second input by key; determining a remaining number of values ​​for each key value based on an assumption that each of the approximate number of blocks contains a block-sized number of values; and The approximate number of blocks required for one or more keys is adjusted based on a remaining number of the values ​​for the keys.

11. The method according to claim 10, wherein: The approximate number of blocks required for one or more keys is adjusted when the remaining number of values ​​for the keys is greater than a predetermined threshold.

12. The method according to claim 9, further comprising: determining that one or more keys were omitted during the sampling period; as well as An approximate number of blocks required to set the one or more keys omitted during the sampling period.

13. The method according to claim 1, wherein: At least one of the first input or the second input is stored as fragments on a plurality of selected computing devices.

14. The method according to claim 13, further comprising: At least one of the fragments of the first input or the fragments of the second input from the plurality of selected computing devices is collected onto a single pre-processing computing device.

15. The method according to claim 1, wherein: The result of the inner join comprises a cross product of at least one of the pairs of replicated blocks.

16. A system, characterized in that The system comprises: A computing device adapted to communicate with a plurality of computing devices in a distributed computing environment, the computing device comprising one or more processors configured to: receiving a request to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs; determining a block size, the block size being a target number of values ​​to be divided into blocks; For each key value, partitioning the value in the first input associated with the corresponding key value into a first set of blocks, the first set of blocks including blocks having the determined block size; For each key value, partitioning the value in the second input associated with the corresponding key value into a second set of blocks, the second set of blocks including blocks having the determined block size; For each key value, replicate the first block set by the number of blocks in the second block set for the same key to obtain a first replicated block set, the first replicated block set comprising replicated blocks that are copies of blocks in the first block set; For each key value, replicate the second block set by the number of blocks in the first block set for the same key to obtain a second replicated block set, the second replicated block set including replicated blocks that are copies of blocks in the second block set; assigning a block-key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block-key comprising the key and additional information identifying the replicated block; distributing each replicated block pair having a matching block-key to a computing device of the plurality of computing devices for processing; and A result of the inner join is received from each computing device of the plurality of computing devices, the result of the inner join comprising at least one pair of values ​​from one of the pairs of replicated blocks.

17. The system of claim 16, wherein: The one or more processors are further configured to: The block size is determined based on at least one of a total number of the plurality of computing devices, processing capabilities of the plurality of computing devices, or uniformity in units of a number of values ​​per block.

18. The system of claim 16, wherein: The one or more processors are further configured to: Each replicated block pair is distributed based on at least one of the block size, a total number of replicated block pairs, a total number of the plurality of computing devices, or a processing capability of the plurality of computing devices.

19. The system of claim 16, wherein: The one or more processors are further configured to: An approximate number of blocks required for each key value is estimated for at least one of the first input or the second input by sampling at least one of the first input or the second input at intervals of each block size.

20. A non-transitory computer-readable storage medium having stored thereon instructions, which when executed by one or more processors cause the one or more processors to perform a method characterized in that The method comprises: receiving a request to perform an inner join on stored data of a first input including a first set of key-value pairs and a second input including a second set of key-value pairs; determining a block size, the block size being a target number of values ​​to be divided into blocks; For each key value, partitioning the value in the first input associated with the corresponding key value into a first set of blocks, the first set of blocks including blocks having the determined block size; For each key value, partitioning the value in the second input associated with the corresponding key value into a second set of blocks, the second set of blocks including blocks having the determined block size; For each key value, replicate the first block set by the number of blocks in the second block set for the same key to obtain a first replicated block set, the first replicated block set comprising replicated blocks that are copies of blocks in the first block set; For each key value, replicate the second block set by the number of blocks in the first block set for the same key to obtain a second replicated block set, the second replicated block set including replicated blocks that are copies of blocks in the second block set; assigning a block-key to each replicated block in the first set of replicated blocks and each replicated block in the second set of replicated blocks, the block-key comprising the key and additional information identifying the replicated block; distributing each replicated block pair having a matching block-key to a computing device of the plurality of computing devices for processing; and A result of the inner join is received from each computing device of the plurality of computing devices, the result of the inner join comprising at least one pair of values ​​from one of the pairs of replicated blocks.

Images