Multi-pass distributed data shuffling

Abstract

This application relates to multi-pass distributed data shuffle. A system and method for repartitioning data in a distributed network. The method can include performing, by one or more processors (210), a first pass of a data set from a plurality of first sources (400) to a plurality of first sinks (410), each first sink collecting data from one or more of the first sources, and performing, by the one or more processors, a second pass of the data set from a plurality of second sources (420) to a plurality of second sinks (430), each of the plurality of first sinks corresponding to one of the plurality of second sources, and each second sink collecting data from one or more of the second sources. Performing the first pass and the second pass causes the data set to be repartitioned such that one or more of the second sinks collects data originating from two or more of the first sources.

Description

[0001] Case Analysis

[0002] This application is a divisional application of Chinese invention patent application 202080071691.8, filed on November 3, 2020.

[0003] Cross-references to related applications

[0004] This application is a continuation-to-file of U.S. Patent Application No. 16 / 672,939, filed November 4, 2019, the disclosure of which is hereby incorporated by reference. Technical Field

[0005] The techniques disclosed herein generally relate to a system for improving the efficiency of shuffle operations involving multiple destinations. In a shuffle, data blocks from multiple sources are redistributed among multiple destinations using a distribution scheme that distributes data blocks from each source to multiple destinations. At the end of the shuffle, each destination may include blocks from more than one source. Background Technology

[0006] The shuffled data is conventionally organized by its source and mapped to the corresponding source for each sink. Figure 1 This is a functional block diagram illustrating an example of a shuffling operation, where data blocks stored at source 10 are shuffled to destination 30. Figure 1 In the example, there are fourteen sources and sixteen destinations. Each destination is mapped to and receives data from four different sources. For example, each of destinations 31 and 32 is mapped to receive shuffled data from sources 11, 12, 13, and 14. Similarly, each of destinations 33 and 34 is mapped to and receives shuffled data that is mapped from sources 11 and 12 to source 13, from source 13 to source 15, and from source 15 to destinations 33 and 34. Figure 1 For the mixed shuffling, there are a total of 64 mappings between the host and the source—four sources for each of the sixteen hosts.

[0007] Conventionally, shuffling operations may require each source to append its data to a common log. Therefore, shuffling operations can easily scale to accommodate additional sources, and the number of operations used to complete the shuffling can increase linearly with the number of sources. However, since a destination receives data from multiple sources and is thus mapped to multiple different sources, each destination must scan all sources from which it can receive data. Therefore, shuffling operations are not easily scaled to accommodate additional destinations, as the number of operations used to complete the shuffling may increase quadratically with the number of destinations. As the amount of data processed in shuffling operations increases, the data may no longer fit within a finite number of destinations, making it necessary to increase the number of destinations to which the data is further subdivided. Summary of the Invention

[0008] One aspect of this disclosure relates to a method for repartitioning data in a distributed network. The method may include: performing a first transfer of a dataset from a plurality of first sources to a plurality of first destinations by one or more processors, each first destination collecting data from one or more of the first sources; and performing a second transfer of the dataset from a plurality of second sources to a plurality of second destinations by one or more processors, each of the plurality of first destinations corresponding to one of the plurality of second sources, and each second destination collecting data from one or more of the second sources. Performing the first and second transfers can repartition the dataset such that one or more second destinations collect data originating from two or more of the first sources.

[0009] In some examples, the number of multiple first residences can be greater than the number of multiple first sources.

[0010] In some examples, each first residence can collect data from two or more first sources.

[0011] In some examples, the number of multiple secondary residences can be greater than the number of multiple secondary sources.

[0012] In some examples, the method may further include performing N passes, where N is a number with a value greater than two. For each given pass, multiple destinations can collect data from one or more sources, each source corresponding to a destination in a previous pass. Performing N passes can repartition the dataset so that one or more Nth destinations collect data originating from two or more of the first sources.

[0013] In some examples, for at least one of N passes, each destination of the pass may collect data from two or more sources of the pass, and each of the two or more sources of the pass may include data from a different source originating from the previous pass.

[0014] In some examples, for at least one of N passes, each destination of the pass can collect data from two or more sources of the pass, and each of the two or more sources of the pass can include data from a different source originating from the previous pass.

[0015] In some examples, at least one pass and at least another pass can be consecutive passes in a set of N passes.

[0016] In some examples, each of the multiple first residences can be a corresponding second source among the multiple second sources. The method may further include determining the completion of the first transfer and initiating the second transfer when the completion of the first transfer is determined.

[0017] In some examples, each of the multiple first residences can be a corresponding second source among the multiple second sources, and the method may further include: specifying each of the multiple first residences and multiple second residences before the first transmission, thereby the specification of the multiple second residences avoids collecting data from the multiple first sources at the multiple second residences during the first transmission; and specifying one or more second residences to be collected from at least one first residence when at least one first residence has completed collection from one or more first sources and before the first transmission is completed.

[0018] In some examples, the dataset can be passed from multiple first sources to multiple first destinations using a first hash function, and from multiple second sources to multiple second destinations using a second hash function that may be associated with the first hash function.

[0019] In some examples, the identifiers of multiple second residences can be consecutive, and for each second residence, the method can include calculating the range of identifiers of the second residence from the second sources collected by the second residence based on the identifier of the second residence.

[0020] In some examples, the method may further include flushing the first-pass shuffle log by one or more processors, and performing a second pass on segments of the dataset that have been flushed from the first-pass shuffle log before flushing is complete.

[0021] In some examples, performing a second transfer may include performing multiple shuffling operations. A first portion of the second source may be included in the first data log, while a second portion of the second source may be included in the second data log.

[0022] In some examples, each shuffle in multiple shuffles can be performed in parallel by one or more processors, and data included in both the first and second data logs can be flushed from the first and second portions of the second source in parallel.

[0023] Another aspect of this disclosure relates to a system for repartitioning data in a distributed network, including one or more processors and one or more storage devices communicating with the one or more processors. The one or more storage devices may include instructions configured to cause the one or more processors to perform a first transfer of a dataset from a plurality of first sources to a plurality of first destinations, each first destination collecting data from one or more of the first sources, and to perform a second transfer of the dataset from a plurality of second sources to a plurality of second destinations, each of the plurality of first destinations corresponding to one of the plurality of second sources, and each second destination collecting data from one or more of the second sources. The first and second transfers can cause the dataset to be repartitioned such that one or more second destinations collect data originating from two or more of the first sources.

[0024] In some examples, each first residence can collect data from two or more first sources, and the number of multiple second residences can be greater than the number of multiple second sources.

[0025] In some examples, instructions can be configured to cause one or more processors to perform N passes, where N is a number with a value greater than two. For each given pass, multiple destinations can collect data from one or more sources, each source corresponding to a destination in a previous pass. Performing N passes can cause the dataset to be further partitioned, such that one or more Nth destinations collect data originating from two or more of the first sources.

[0026] In some examples, for at least one of N passes, each destination of the pass may collect data from two or more sources of the pass, and each of the two or more sources of the pass includes data originating from a different source of the preceding pass. For at least one other of the N passes, each destination of the pass may collect data from two or more sources of the other pass, and each of the two or more sources of the other pass may include data originating from a different source of the preceding pass. At least one pass and at least one other pass may be consecutive passes among the N passes. Attached Figure Description

[0027] Figure 1 This is a block diagram illustrating a priori data distribution scheme.

[0028] Figure 2A This is a block diagram of an example system within a distributed data storage system according to aspects of this disclosure.

[0029] Figure 2B This is a block diagram of an example system and computing device according to aspects of this disclosure.

[0030] Figure 3 This is an example of a multi-pass shuffling flowchart according to aspects of this disclosure.

[0031] Figures 4 to 7 This is a block diagram illustrating an example data distribution scheme according to aspects of this disclosure.

[0032] Figure 8 and Figure 9 This is a block diagram illustrating an example workflow according to aspects of this disclosure. Detailed Implementation

[0033] Overview

[0034] To scale the shuffling operation as the number of hosts increases, this disclosure divides the shuffling operation into multiple passes. In this way, the number of sources read by each host is reduced, thereby alleviating the overhead of the shuffling operation per host. Multiple-pass shuffling operations can be implemented in different ways, and each implementation can improve efficiency in different ways.

[0035] In some implementations, at least one pass in a multi-pass shuffle can be a "destination split". In a destination split, data from each given source is distributed across multiple destinations, but each destination reads data from only one source. Destination splitting in one pass of a multi-pass shuffle can allow other passes of the multi-pass shuffle to be performed using fewer destinations.

[0036] For example, if you want to distribute data from 100,000 sources to 100,000 destinations, you can first pass the data to 1,000 destinations, and then split the data for each of those 1,000 destinations into 100 destinations, resulting in a total of 100,000 destinations after the splits. In the first pass, each of the 1,000 destinations can read data from a certain number of sources. Because this pass is shuffled and each destination reads data from multiple sources, reducing the number of destinations in this pass by a factor of 100 significantly reduces the overhead of the pass. In subsequent splits, each destination reads from only a single source, which also requires significantly less overhead. As a result, the number of sources read by a destination is reduced in both passes.

[0037] Additionally or alternatively, splitting can be performed as the first pass in a multi-pass shuffle. If the shuffle involves data from many sources, splitting the data early may be preferable. For example, if there are 1,000,000 sources to distribute to 2,000,000 destinations, the data from each of the 1,000,000 sources can be split into two destinations each. The resulting 2,000,000 destinations can then be divided into 2,000 groups, each group taking data from 1,000 different sources and shuffling the data to 1,000 destinations. The destinations from the first shuffle can be regrouped and then shuffled in a similar manner, resulting in each original source being distributed relatively thoroughly. This form of grouping and reshuffling is referred to herein as “butterfly shuffle.” As in the previous example, the overhead in each pass is significantly reduced. The overhead of the first pass in a butterfly shuffle is reduced because it is a split, whereby a destination reads from only one source. Because each individual shuffle between 1,000 sources and 1,000 destinations requires less overhead than a shuffle between 1,000,000 sources and 1,000,000 destinations, the overhead in subsequent passes is reduced. In other words, although the total number of shuffles performed in each pass increases linearly, the processing time per shuffle is reduced compared to a single shuffle of 1,000,000 to 1,000,000.

[0038] In the butterfly shuffling example above, the number of independent shuffles performed from one pass to the next remains constant. In other examples of multi-pass shuffling, the number of shuffles performed from one pass to the next can multiply. For example, an initial pass shuffling data from 10,000 sources to 20,000 destinations might be followed by a second pass with two independent shuffles from 10,000 sources to 20,000 destinations. The result of the second pass would be data distributed across 40,000 destinations. Each independent shuffle in the second pass could have its own log of metadata for the shuffled data blocks. This might be preferable for increasing the number of destinations processed by the overall shuffling operation, since each log contains metadata for those destinations where only the individual shuffled blocks are read, rather than all blocks from the entire pass.

[0039] If metadata needs to be flushed from the log during shuffling, the use of multi-pass shuffling can improve the efficiency of the flushing operation. This is because the second pass of multi-pass shuffling can be controlled to distribute only the data that has been flushed from the first shuffling. Furthermore, in some cases, using multi-pass shuffling can completely avoid the need to flush metadata from the log, as reducing the total number of destinations for a given segment also reduces the total amount of metadata stored for that segment.

[0040] In some cases, multi-pass shuffling can be implemented as a single shuffling by deferring the partitioning of destinations in subsequent passes until a later time. For example, in an example of shuffling data from 100,000 sources to 100,000 destinations using 1,000 destinations in the first pass, the partitioning of the remaining 99,000 destinations can be deferred. This prevents the 100,000 sources from writing directly to those 99,000 destinations. In one case, the 99,000 destinations can be blocked until the entire first pass is complete. In another case, the next pass can begin before the first pass ends, but the partitioning scheme of the destinations in the next pass can be changed to prevent sources from writing directly to them.

[0041] The above implementation improves shuffling performance, especially when scaling to tens of thousands of hosts. For example, test results have shown that by performing a first pass on 1,000 hosts and a second split pass on 100,000 hosts, the speed of shuffling approximately 2 TiB of data from 100,000 sources to 100,000 hosts can be more than doubled. This results in a significant reduction in the resources, cost, and time required to perform the shuffling.

[0042] Example System

[0043] Figure 2AThe illustration includes an example system of a distributed computing environment. Multiple data centers 160, 170, and 180 can be communicatively coupled, for example, via a network 150. Data centers 160, 170, and 180 can further communicate with one or more client devices (such as client 110) via network 150. Thus, for example, client 110 can perform operations in the "cloud". In some examples, data centers 160, 170, and 180 can further communicate with a controller 190.

[0044] Each client 110 may be a personal computer or mobile device intended for human use, having all the internal components typically found in a personal computer, such as a central processing unit (CPU), CD-ROM, hard disk drive, and display device (e.g., a monitor with a screen, projector, touchscreen, small LCD screen, television, or another device, such as an electronic device operable to display information processed by processor 120), speakers, modem and / or network interface device, user input device (such as a mouse, keyboard, touchscreen, or microphone), and all components for connecting these elements to each other. Furthermore, a computer according to the systems and methods described herein may include devices capable of processing instructions and transmitting data to and from humans and other computers, including general-purpose computers, PDAs, tablets, mobile phones, smartwatches, network computers lacking local storage, set-top boxes for televisions, and other networked devices.

[0045] Client 110 may include processor 120, memory 130, and other components typically found in general-purpose computers. Memory 130 is capable of storing information accessible by processor 120, including instructions 132 executable by processor 120. Memory may also include data 134 retrievable, manipulated, or stored by processor 120. Memory 130 may be a non-transitory computer-readable medium capable of storing information accessible by processor 120, such as a hard disk drive, solid-state drive, magnetic tape drive, optical storage device, memory card, ROM, RAM, DVD, CD-ROM, writable, and read-only memory. Processor 120 may be a well-known processor or other lesser-known types of processor. Alternatively, processor 120 may be a dedicated controller such as an ASIC.

[0046] Instruction 132 may be a set of instructions that are directly executed by processor 120, such as machine code, or a set of instructions that are indirectly executed by processor 120, such as a script. In this regard, the terms “instruction,” “step,” and “program” may be used interchangeably herein. Instruction 132 may be stored in an object code format for direct processing by processor 120, or in other types of computer languages, including scripts or collections of standalone source code modules that are interpreted on demand or compiled in advance.

[0047] Data 134 can be retrieved, stored, or modified by processor 120 according to instructions 132. For example, although the system and method are not limited to a specific data structure, data 134 can be stored in a computer register, stored in a data store as a structure with multiple different fields and records, or stored in a document or buffer. Data 134 can also be formatted in a computer-readable format such as, but not limited to, binary values, ASCII, or Unicode. Furthermore, data 134 can include information sufficient to identify relevant information, such as numbers, descriptive text, proprietary code, pointers, references to data stored in other memory (including other network locations), or information used by functions to compute relevant data.

[0048] although Figure 2A While processor 120 and memory 130 are illustrated functionally as being within the same block, processor 120 and memory 130 may actually comprise multiple processors and memories that may or may not be stored in the same physical housing. For example, some instructions 132 and data 134 may be stored on a removable CD-ROM, while other instructions and data may be stored within a read-only computer chip. Some or all of the instructions and data may be stored in a location physically remote from processor 120 but still accessible by processor 120. Similarly, processor 120 may actually comprise a collection of processors that may or may not operate in parallel.

[0049] Data centers 160 to 180 can be located at considerable distances from each other. For example, data centers could be located in various countries around the world. Each data center 160, 170, or 180 can include one or more computing devices, such as processors, servers, shards, etc. For example, as... Figure 2A As shown, data center 160 includes computing devices 162 and 164, data center 170 includes computing device 172, and data center 180 includes computing devices 181 to 186. According to some examples, computing devices may include one or more virtual machines running on a host. For example, computing device 162 may be a host that supports multiple virtual machines 166 and 167 running operating systems and applications. Although in Figure 2AThe diagram illustrates only a few virtual machines (166, 167), but it should be understood that any number of virtual machines can be supported by any number of host computing devices. Furthermore, it should be understood that... Figure 2A The configuration shown in the diagram is merely an example, and the computing devices in each of the example data centers 160 to 180 may have various structures and components that may be the same as or different from each other.

[0050] Programs can be executed across these computing devices; for example, some operations may be performed by one or more computing devices in a first data center, while others may be performed by one or more computing devices in a second data center. In some examples, the computing devices in the various data centers may have different capacities. For example, different computing devices may have different processing speeds, workloads, etc. Although only a few of these computing devices are shown, it should be understood that each data center 160, 170, 180 may include any number of computing devices, and the number of computing devices in the first data center may differ from the number of computing devices in the second data center. Furthermore, it should be understood that the number of computing devices in each data center 160 to 180 may vary over time (e.g., as hardware is removed, replaced, upgraded, or expanded).

[0051] In some examples, each data center 160 to 180 may also include numerous storage devices (not shown), such as hard disk drives, random access memory, disks, disk arrays, tape drives, or any other type of storage device. Data centers 160 to 180 can implement any of many architectures and technologies, including but not limited to Direct Attached Storage (DAS), Network Attached Storage (NAS), Storage Area Network (SAN), Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), hybrid architecture networks, etc. In addition to storage devices, data centers may also include many other devices, such as cabling, routers, etc. Furthermore, in some examples, data centers 160 to 180 may be virtualized environments. Moreover, although only a few data centers 160 to 180 are shown, many data centers may be coupled via network 150 and / or additional networks.

[0052] In some examples, controller 190 can communicate with computing devices in data centers 160 to 180 and facilitate program execution. For example, controller 190 can track the capacity, status, workload, or other information of each computing device and use such information to assign tasks. Similar to client 110 described above, controller 190 may include processor 198 and memory 192 including data 194 and instructions 196. Controller 190 can be configured to redistribute or repartition data stored in computing devices in data centers 160 to 180.

[0053] Client 110, data centers 160-180, and controller 190 may be able to communicate directly and indirectly, for example, via network 150. For instance, using Internet sockets, client 110 may connect to a service running on a remote server via Internet Protocol Suite. The server may be configured to accept listening sockets for initiating connections to send and receive information. Network 150 and intermediate nodes may include various configurations and protocols, including the Internet, World Wide Web, Intranet, Virtual Private Network, Wide Area Network, Local Area Network, Private Network using one or more proprietary communication protocols, Ethernet, WiFi (e.g., 702.71, 702.71b, g, n, or other such standards), and RPC, HTTP, and various combinations thereof. This communication may be facilitated by devices capable of transmitting and receiving data to and from other computers, such as modems (e.g., dial-up, cable, or fiber optic) and wireless interfaces.

[0054] Client 110 may request access to data stored in computing devices in data centers 160 to 180. This request may be handled by controller 190 and / or one or more computing devices in data centers 160 to 180. In some examples, the response to the request may involve manipulation that requires data in other ways, such as using the operations described in more detail herein.

[0055] Figure 2B This is a block diagram illustrating an example system having one or more computing devices 200 for performing a shuffling operation according to the present disclosure. The computing devices 200 may be included in a distributed data processing system (such as a computing device in one of data centers 160 to 180) or a controller 190, such as... Figure 2A As shown. Computing device 200 can be configured to run complex queries on large amounts of data by further partitioning the data, such as "JOIN" and "GROUP BY" operations. Such operations can be performed in response to queries. In some examples, queries may be received by a client computing device. In some examples, queries may be required to execute additional instructions or queries received from a client computing device.

[0056] Data in a distributed data processing system can be stored ephemerally, such as in a distributed memory file system or on a disk, or by any combination of both. Data can be transferred from many data sources A1 to A2. N Mixed shuffling into many data rooms B1 to B N For the purpose of tracking data during repartitioning, virtual addresses can be assigned to the source and destination. In some examples, data can be stored in virtual machines (such as those powered by...). Figure 2A In the data center, there are 160 to 180 virtual machines hosted (166 to 167).

[0057] Computing device 200 may include one or more processors 210, servers, shards, units, etc. It should be understood that each computing device may include any number of processors or computing devices, and the number of such devices in a computing device may vary over time (e.g., as hardware is removed, replaced, upgraded, or expanded).

[0058] The computing device 200 may also include a number of storage devices or memories 220, such as hard disk drives, random access memory, disks, disk arrays, tape drives, or any other type of storage device. The computing device 200 can implement any of many architectures and technologies, including but not limited to direct-attached storage (DAS), network-attached storage (NAS), storage area network (SAN), Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), hybrid architecture networks, etc. In addition to storage devices, the computing device 200 may also include many other devices, such as communication devices 250 for enabling input and output between computing devices, such as cabling, routers, etc.

[0059] The memory 220 of each computing device is capable of storing information accessible by one or more processors 210, including data 230 received at or generated by the computing device 200, and instructions 240 executable by one or more processors 210.

[0060] Data 230 may include sources A1 to A1 of the tracking system. N Wataka B1 to B N The shuffling log 232 includes details of the data segments 234 and 236 shuffled during the shuffling operation, such as metadata for individual data segments 238, including information about segment length and commit status for each segment. Conceptually, shuffled data can be represented as a mapping between a source and its corresponding destination.

[0061] As discussed in more detail below, the shuffling operation can be divided into multiple passes from the first pass to the Nth pass. Therefore, the details of data segmentation are discussed later. Figure 2B The example shuffle log 232 is shown as being stored separately. Separate tracking of details for each delivery helps manage the storage of data segment details 234, 236 and metadata 238. For example, even if a subsequent delivery is in progress, once the first delivery has been completed and the data has been stored in its destination log file, the details of the first delivery 234 can be flushed from storage 220. In other cases, there may be overlap between the shuffle log 232 and the destination data log.

[0062] Instruction 240 may include a shuffling control program 242 configured to control the data shuffling operation. Instruction 240 may also include a shuffling log flushing program 244 configured to manage the storage of data segmentation details 234, 236 and metadata 238 of the shuffling log 232. The above examples of stored data and programs are discussed in more detail below.

[0063] Example Method

[0064] Figure 3 This is a flowchart 300 illustrating an example of multi-pass shuffling, where data from multiple sources is shuffled to multiple destinations. The operations in the flowchart can be performed by the processor of computing device 200 (such as...). Figure 2B One or more processors 210) execute the process, the one or more processors 210 interacting with the storage location of the shuffled data (such as source A1 to A...). N Wataka B1 to B N ) communication.

[0065] In box 310, a first transfer of data can be performed. The first transfer may involve moving data from a first source set to a first destination set. Then, in box 320, a second transfer of data can be performed. The second transfer may involve moving data from a second source set, which may correspond to the first destination set, to the second destination set.

[0066] For example, Figure 4 An example of a data distribution scheme and setup is shown, where a shuffling operation is used to shuffle data in a series of two passes. The shuffling operation is divided into two separate passes: a first pass from a first source 400 to a first destination 410, and a second pass from a second source 420 corresponding to the first destination 410 to a second destination 430. In the first pass, each of the first destinations 410 receives data segments from three different first sources in the first source 400. For example, destination 411 receives data from each of sources 401, 402, and 403. In another example, destination 418 receives data from each of sources 401, 402, and 406. Since each of the destinations is mapped to the source from which it receives data, the... Figure 4 In the first pass of the mixed shuffling, there are a total of 24 mappings between the first residence and the first source—three mappings for each of the eight residences. In the second pass, each residence 430 can receive data from one of the second sources 420 corresponding to the second residence 410. For example, each residence 431 and 432 receives data from source 421. For example, each residence 445 and 446 receives data from source 418. Since each residence is mapped to a source from which it receives data only, for Figure 4For the second pass of the shuffle, there are only 16 mappings in total between the second destination and the second source—one mapping for each of the sixteen destinations. Meanwhile, the total number of mappings tracked during the two-pass shuffle is 40—24 mappings in the first pass and 16 mappings in the second pass—which is less than the total of 64 mappings when the same shuffle is performed as a single pass.

[0067] As the number of sources and destinations involved in a shuffling operation increases, so does the benefit of performing multiple passes. For example, to distribute data from 100,000 sources to 100,000 destinations in a single pass, it might be necessary for each destination to read data from many sources (e.g., 10,000 sources). This would require a total of 10,000 passes between sources and destinations. 100,000 = 1 billion mappings. Even if each destination reads from only 1,000 sources, 100 million mappings would still need to be tracked. Introducing a second pass would allow data to be passed from sources to a small initial subset of destinations, such as 1,000 destinations, and then the data for each of the 1,000 destinations could be split into 100 destinations. In the first pass, if each of the 1,000 destinations reads data from 10,000 or 1,000 of the 100,000 sources, the total number of mappings would be 10 million or 1 million, respectively. In the second pass, each of the 100,000 destinations would be mapped to a single source, resulting in a total of 100,000 mappings. Therefore, the total number of mappings tracked in the two-pass shuffling would be 10.1 million mappings when using 10,000 intermediate destinations, or 1.1 million mappings when using 1,000 intermediate destinations. This number of mappings is significantly smaller than the 1 billion or 100 million mappings in a single delivery scenario. Experiments show that a shuffling operation using 1,000 intermediate hosts to shuffle approximately 2 TiB of data can be at least twice as efficient as the same operation performed without any intermediate hosts.

[0068] Figure 4 The example above illustrates such a two-pass shuffle, where the second pass is called a "destination split". In a destination split, data from each given source is distributed across multiple destinations, but each destination reads data from only one source. Destination splits in a pass of a multi-pass shuffle can allow the use of fewer destinations to perform other previous passes of the multi-pass shuffle. As can be seen from the example scenario above, introducing destination splits as a subsequent pass to the shuffle operation can significantly reduce the overhead in previous passes because there are far fewer intermediate destinations that need to be mapped to multiple sources and read data from multiple sources. The introduction of destination splits can also significantly reduce the overhead in the final pass because each of the final destinations is mapped to only one source and reads data from only one source.

[0069] In the example above, a destination in a pass is described as "corresponding" to the source in the next pass. Correspondence can mean that the destination and the source in the next pass are the same, meaning their addresses are identical. Alternatively, the address of the destination can be different from the address of the source in the next pass, but all data in the destination can be moved to the address of the corresponding source in the next pass.

[0070] Multi-pass shuffling can include further operations, such as third passes, fourth passes, etc. Figure 3 In the example of multi-pass shuffling, passes are shown to be performed up to the Nth pass. Specifically, in box 330, the (N-1)th pass of data can be performed. The (N-1)th pass may involve moving data from the (N-1)th source set, which may correspond to the (N-2)th host set, to the (N-1)th host set. Furthermore, in box 340, the Nth pass and the final pass of data can be performed. The Nth pass may involve moving data from the Nth source set, which may correspond to the (N-1)th host set, to the Nth host set.

[0071] The value of N can vary depending on the amount of data to be shuffled, the purpose of the shuffling operation, and the specific type of shuffling operation being performed. For example, in Figure 4 In the case of the destination split shown in the example, having a relatively small number of destination passes (such as two passes) may be sufficient. In other types of shuffling operations, dividing the operation into three or more passes can be beneficial.

[0072] Figure 5 This illustrates an example of a "butterfly" shuffling distribution scheme. In the butterfly algorithm, a finite number of sources are paired with a finite number of destinations, and data from the sources is then shuffled within the paired destinations. Thus, all data is shuffled in each pass but in separate blocks. Introducing multiple butterfly passes to the entire operation allows shuffled data from each individual block to be paired with data from other separately shuffled blocks.

[0073] exist Figure 5In the example, the shuffling operation includes a first pass from a first source 500 to a first destination 510, a second pass from a second source 520 (which corresponds to the first destination 510) to a second destination 530, and a third pass from a third source 540 (which corresponds to the second destination 530) to a third destination 550. In the first pass, each of the first destinations 510 receives a data segment from one of the first sources 500. For example, destination 511 receives data from source 501, destination 513 receives data from source 502, destination 515 receives data from source 503, and destination 517 receives data from source 504. In each subsequent pass, each destination in the pass receives data segments from two sources that have received segments from different sources in the previous pass. For example, in the second transmission, the second host 531 receives data segments from each of the second sources 521 and 522, whereby second source 521 received data from first source 501 in the prior first transmission, and second source 522 received data from first source 502 in the prior first transmission. Host 535 receives data segments from each of the second sources 525 and 526, whereby second source 525 received data from first source 503 in the prior first transmission, and second source 526 received data from first source 504 in the prior first transmission. Similarly, in the third transmission, each of the third hosts 551 and 552 receives data segments from each of the third sources 541 and 542, whereby third source 541 received data from second sources 521 and 522 in the prior second transmission, and third source 542 received data from second sources 552 and 526 in the prior second transmission.

[0074] although Figure 5 The example shows only the first, second, and third passes, but those skilled in the art will readily understand that operation can continue for additional passes. Figure 5 In the example, each pass is shown to have no more than eight sources and eight destinations, which is sufficient for each final destination 550 to include data segments from each of the first sources 501, 502, 503, and 504. As the number of sources and destinations increases, it may be necessary to add more passes to the butterfly shuffle, or increase the complexity of each shuffle operation.

[0075] The use of "butterfly" shuffling is particularly beneficial when distributing data from a large number of sources. For example, to distribute data from 1,000,000 sources to 2,000,000 destinations in a single delivery, each destination might need to read data from many sources. For instance, if each destination will read from 10,000 sources, the shuffling will require a total of 10,000 shuffles between the sources and destinations. 2,000,000 = 20 billion mappings. Even if each host reads from only 1,000 sources, it would still require tracking 2 billion mappings. Even if each host reads from only 8 sources, as in Figure 5 As in the previous example, this arrangement would still require tracking 16 million mappings (8 mappings for each of the 2 million destinations). Introducing an initial split as the first pass within the multi-pass butterfly shuffle operation would allow data to be passed to 2,000,000 destinations initially, with the minimum number of mappings between sources and destinations, i.e., 2,000,000 mappings, or one source per destination. Then, in each subsequent pass, each destination can be mapped to two sources, resulting in a total of 4,000,000 mappings for the second pass, an additional 4,000,000 mappings for the third pass, and the same for any subsequent passes. In summary, for Figure 5 The three passes shown require a total of 10 million source-destination mappings to be tracked, significantly fewer than 10 billion or 1 billion mappings, and still fewer than 16 million mappings required in the single pass scenario described above. Therefore, the overhead in each pass of the shuffling operation can be significantly reduced: the first pass's overhead is reduced because it is split; and the overhead in subsequent passes is reduced because each pass requires less overhead than a single pass shuffling between 1,000,000 sources and 1,000,000 destinations. In other words, although the total number of shuffling operations performed in each pass increases linearly, the total amount of processing used for shuffling is quadratically reduced overall.

[0076] exist Figure 4 In the above examples of sub-shuffling shown, each pass is depicted as comprising a single operation, rather than operations performed independently. In other examples, the number of operations performed in a single given pass of a multi-pass shuffling operation may be more than one. For example, those skilled in the art will recognize that... Figure 5 The individual butterfly shuffles shown may be performed independently of each other, resulting in multiple operations in each of the second and third passes.

[0077] Additionally, in Figure 5 In the example of butterfly shuffling, the number of independent shuffles remains constant from one pass to the next. In other examples, the number of independent operations performed in each pass can change from one pass to the next.

[0078] Figure 6 An example of a "wash-split" distribution scheme involving multiple destination splits is shown, where the number of split operations performed in each pass increases from one pass to the next. Specifically, in Figure 6In the example, the first pass from the first source 600 to the first residence 610 involves shuffling the data from two sources 601 and 602 across four residences 611, 612, 613, and 614. In the second pass from the second source 620 corresponding to the first residence 610 to the second residence 630, each source is split into two residences. For example, the data from source 621 is split between residences 631 and 632, the data from source 622 is split between residences 633 and 634, the data from source 623 is split between residences 635 and 636, and the data from source 624 is split between residences 637 and 638. In the third pass from the third source 640 corresponding to the second residence 630 to the third residence 650, each source is again split into two residences. For example, data from source 641 is split between residences 651 and 652, data from source 642 is split between residences 653 and 654, data from source 643 is split between residences 655 and 656, and data from source 644 is split between residences 657 and 658.

[0079] Furthermore, the shuffling operation of the second pass is split or divided between two independently performed operations, such that the splitting of data in sources 621 and 622 is managed in the first operation 682, and the splitting of data in sources 623 and 624 is managed in the second separate operation 684. Similarly, the destination of each given independent operation of the second pass is split between two separate operations of the third pass. For example, the splitting of data in sources 641 and 642 is managed in the first operation 692, and the splitting of data in sources 643 and 644 is managed in the second separate operation 694. Similarly, the data split in operation 684 of the second pass is then divided between two separate splitting operations 696 and 698 in the third pass.

[0080] exist Figure 6 In the example, the number of split operations from one code to the next is doubled. Therefore, the "split multiple" of this shuffle split is said to be two. In other examples, the number of independent operations can be increased by multiples greater than two (such as three, four, five, eight, ten, etc.).

[0081] While the total number of source-destination mappings cannot be reduced by treating the transit as multiple separate operations, the use of separate operations does allow for multiple shuffle logs to be used for the same transit. This allows the overall size of each shuffle log to remain at a reduced size. As the number of sources and destinations increases and the amount of metadata to be tracked during a shuffle operation increases, it can be beneficial to divide the metadata into separate shuffle logs. For example, and further reference... Figure 6 The processor that controls the operation of the second transfer from the second source 620 to the second destination 630, such as Figure 2BOne or more processors 210 shown in the example can control operations 682 and 684 to be processed sequentially or simultaneously, thereby improving bandwidth and timing control of the shuffling operation.

[0082] For example, an initial pass that shuffles data from 10,000 sources to 20,000 destinations can be followed by a second pass with two independent shuffles from 10,000 sources to 20,000 destinations. The result of the second pass would be data distributed across 40,000 destinations, but the metadata of the data blocks moved in the second pass can be split between the two separate shuffle logs. This can be particularly helpful when the shuffle operation involves many destinations far more than the number of sources, because the data may be sufficiently shuffled in the early passes of the operation before the number of destinations increases, and then split into many destinations according to the independent operations managed by separate shuffle logs. In this way, each shuffle log will contain metadata for those destinations that only read the blocks from its own operation, and not the blocks from other independent operations in the pass.

[0083] Additionally, if operations are processed in parallel or simultaneously, and if one operation completes before another, metadata from the shuffled log can be flushed from the completed operation without waiting for the other separate operation to complete as well. To illustrate, if a single “settlement split” operation containing 100,000 evenly distributed subsidiaries would contain 50 GiB of shuffled log data, the flusher would flush an average of approximately 0.5 MiB of data upon completion of each subsidiary. However, if the same shuffled log data is split across 100 separate “settlement split” operations, each containing 1,000 subsidiaries, the flusher would flush an average of approximately 50 MiB of data upon completion of each subsidiary.

[0084] Figure 7 Showing combinations from Figure 5 butterfly-shaped mixed washing and from Figure 6 Two example data distribution schemes based on the concept of shuffling and splitting. Figure 7 In the example, data from the first source 700 is written to the first residence 710 in the first transmission, data from the second source 720 corresponding to the first residence 710 is written to the second residence 730 in the second transmission, and data from the third source 740 corresponding to the second residence 730 is written to the third residence 750 in the third transmission.

[0085] In each pass, each destination can receive data from two sources. Furthermore, within each pass, destinations can be grouped, whereby each destination group receives data from the same two sources. In this way, the shuffling operation for each destination group and its corresponding two sources can be processed as an independent operation that limits its own shuffling log to metadata of the data written to a given destination group. For example, in the second pass, second destinations 735, 736, 737, and 738 can all receive data from second sources 723 and 724. The shuffling between sources 723 and 724 and destinations 735, 736, 737, and 738 can be independently managed in its own shuffling log, which contains metadata of the data written only to destinations 735, 736, 737, and 738.

[0086] exist Figure 7 In the example, the number of independent operations performed from each pass to the next has increased by a factor of two. That is, the first pass involves two operations, the second pass involves four operations, and the third pass may involve eight operations (not all are shown).

[0087] from Figure 7 The example also shows that each third residence 750 can receive data from each of the first sources 700. For example, each of the third residences 751 may include data from the third source 741, which in turn corresponds to the second residence 735. The second residence 735 may include data from each of its corresponding second sources 723 and 724, which correspond to the first residences 712 and 716, respectively. The first residence 712 may include data from each of the first sources 701 and 702, and the first residence 716 may include data from each of the first sources 703 and 704. Therefore, data in the third residences 751 and 752 can be traced back to each of the first sources 700.

[0088] in other words, Figure 7 The example can be considered like split shuffling because the number of independent operations can multiply from one pass to the next, and it can be considered like butterfly shuffling because it can combine data drawn from different sources in previous passes in a single operation of subsequent passes. This adds another layer of complexity and robustness to the shuffling operation to produce improved results for queries applied to the collected data.

[0089] Figure 4 , Figure 5 , Figure 6 and Figure 7 The examples above demonstrate various types of transfer and shuffling operations that can be performed to alleviate quotas as the amount of data, source, destination, shuffling, or any combination thereof increases. One or more processors, such as Figure 2BThe processor 210 can be programmed to track the progress of the shuffling operation and dynamically determine the type of delivery to be performed based on the tracked progress. Automatic operators or schedulers associated with various operations (such as "join" or "group shuffling") can dynamically decide whether to initiate a multi-delivery shuffling, and if so, what type of operation to apply. For example, the automatic join scheduler can determine whether to initiate a "shuffling split" if data has not yet been consistently distributed to both sides of the join, or a "destination split" if data has been consistently distributed.

[0090] In some examples, in response to a processor determining that a monitored value exceeds a predetermined threshold, one or more processors, such as Figure 2B The processor 210 initiates the shuffling split. The monitoring value can be the number of destinations used in a given pass, whereby the destinations are split into individual operations until the number of destinations in each operation is less than a threshold. Alternatively or additionally, the monitoring value can be the number of source-destination mappings in a given pass, whereby the destinations are split into individual operations until the number of source-destination mappings in each individual operation is less than a threshold.

[0091] The example above demonstrates a multi-transit shuffling where each transit is treated as a separate operation. In other examples, transits can be combined into a single shuffling operation, while maintaining the benefits of reduced overhead and a smaller metadata log. Furthermore, the example above treats the destinations of each transit as separate from each other. In other examples, there may be overlap between the destinations of each transit. For example, in an example shuffling operation where each transit writes to more destinations than the previous transit, the destinations of one transit can be a subset of the destinations of the next transit, and the destinations of the next transit can be a subset of the destinations of subsequent transits, and so on.

[0092] For example, if data is shuffled from 100,000 sources to 100,000 destinations, and if 1,000 destinations are used in the first pass, then those 1,000 destinations could be 1,000 of the 100,000 destinations to be used in the second pass. In such a case, a partitioning scheme can be used to prevent the first source from being written to any of the remaining 99,000 destinations that are not the first destination. One effect of such a partitioning scheme is that it transforms the multiple shuffle operations of the previous example into a single shuffle operation that is essentially from a pre-specified set of sources to a pre-specified set of destinations.

[0093] Figure 8 and Figure 9 An example partitioning scheme that can be used to partition such a single shuffle operation is shown. Figure 8 and Figure 9The examples are limited to the first and second passes, but it should be understood that the schemes outlined herein may be repeated in shuffle operations with more than two passes.

[0094] Figure 8 The example illustrates a "delayed partitioning" scheme, where the first pass 810 is completed before the second pass 820 begins. The first pass 810 may involve one or more processors, such as... Figure 2B One or more processors 210 specify a first source set (812) and a first destination set (814), and block a second destination set (816). When data is written from the first source set to the first destination set (818) according to instructions from one or more processors, the second destination set can remain blocked to receive any shuffled data. This ensures that all shuffled data in the first pass is written only to the first destination set and not to the second destination set. When the first pass set 810 is fully completed, one or more processors can then initiate the operation of the second pass set 820, thereby unblocking the second destination set (822), after which data from the second source set (which may correspond to the first destination set) can be written to the second destination set (824). Once all data has been written to the second destination set, the second pass set is complete.

[0095] Figure 9 An alternative example illustrates a "pipeline deferred partitioning" scheme, where the second pass 920 can begin before the first pass 910 completes. The first pass 910 may involve one or more processors using a pipeline to specify a first source set (812), a first destination set (814), and a second destination set (816). The pipeline can use a partitioning scheme where none of the first sources writes to the destination to which the first destination with deferred partitioning is written. The second pass 920 can begin when writes to any of the first destinations are complete, even if writes to other first destinations are not yet complete. In the second pass 920, the second source corresponding to the completed first destination can write data to the second destination (922) according to the partitioning scheme. As write operations to each first destination complete, write operations to another corresponding second source can begin, and this can continue until the entire second pass 920 is complete.

[0096] In some examples, the initiation of a second pass in a multi-pass shuffle can be conditional upon data already being flushed from the shuffle log of the first shuffle. This adjustment ensures that the shuffle log of a multi-pass shuffle does not consume unnecessary space and can improve the efficiency of the flushing operation. In other examples, using multi-pass to perform the shuffle operation itself avoids the need to flush metadata completely from the shuffle log, because reducing the total number of destinations for any given segment (as done in multi-pass shuffle) will also reduce the total amount of metadata that needs to be stored for each segment.

[0097] In some examples, the delivery of multi-pass shuffle operations can be pipelined. This may increase quota usage for one or more processors, but has the advantage of improved performance. In such examples, shuffles occurring in earlier passes can be given higher priority, such as being given sufficient quota to avoid backlogs in the pipeline. One or more processors can receive instructions from the scheduler to appropriately distribute quotas during pipelined shuffles and passes.

[0098] In some examples, a multi-pass shuffling operation partitioning scheme can further divide the data into all available destinations in one pass, and then compress the data into a subset of available destinations in subsequent passes. Such a partitioning scheme can optimize destination reads in subsequent passes. Specifically, if the destination addresses used are contiguous, and if the shuffling log maps subsets of available destinations to non-overlapping ranges of addresses, then destination lookup operations in subsequent passes can be as simple as for addresses within a given range. As a result, the source-destination mapping may not consume any space, because the source for which a destination is specified can be determined based on the destination's own address, without having to store a separate mapping between destination and source addresses.

[0099] In the examples above, each pass can use a hash partitioning function to direct data from its source to its corresponding destination. The hash partitioning function used in each pass can be associated with the hash function of a previously passed pass. Similarly, in the case of "shuffle split," the individual shuffle operations are thus performed separately in a single pass and split from the common shuffle operations of the previous passes, and each of the individual shuffle operations can use a corresponding hash function associated with the hash function of the previous passes.

[0100] The examples above typically address problems encountered when attempting to run complex queries on data volumes greater than approximately 10 TiB (such as tens of TiB of data). Such data volumes often require the use of more than 10,000 hosts, which introduces scaling difficulties for conventional single-pass shuffling operations. Those skilled in the art will recognize that the advantages of the multi-pass shuffling operations described herein also apply to smaller amounts of data. That is, even for those smaller volumes that could potentially be handled using conventional single-pass shuffling operations, the multi-pass shuffling operations described herein can improve efficiency and reduce the overall cost and overhead of the operation. In fact, some experiments have shown that the overall speedup for a smaller input data volume of approximately 1 TiB can be greater than the speedup for a larger input data volume of approximately 10 TiB.

[0101] Although the techniques described herein have been illustrated with reference to specific embodiments, it should be understood that these embodiments are merely illustrative of the principles and applications of the technology. Therefore, it should be understood that many modifications can be made to the illustrative embodiments and other arrangements can be designed without departing from the spirit and scope of the technology as defined by the appended claims.

[0102] Most of the preceding alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Because these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter as defined by the claims, the preceding description of the embodiments should be understood as illustrative rather than as a limitation on the subject matter defined by the claims. As an example, the preceding operations need not be performed in the exact order described above. Instead, the steps can be performed in a different order, such as the reverse order, or simultaneously. Steps can also be omitted unless otherwise stated. Furthermore, the examples described herein and the provision of phrases such as "such as," "comprising," etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, the examples are intended to illustrate only one of many possible embodiments. Moreover, the same reference numerals in different figures can identify the same or similar elements.

Claims

1. A method for re-partitioning data in a distributed network, the method comprising: In response to a repartition instruction that further partitions the dataset: The first transfer of the dataset to multiple first-dwellings is executed by one or more processors; as well as The dataset is second-passed from the plurality of first-dwellings to the plurality of second-dwellings by the one or more processors, wherein the second-passed-datasets include a plurality of independent shuffles, and wherein each independent shuffle is associated with its own corresponding metadata log.

2. The method of claim 1, wherein each corresponding metadata log includes metadata of those second-night independent shuffles associated only with itself.

3. The method according to claim 2, further comprising: The respective metadata logs of the multiple independent shuffles are managed separately by the one or more processors.

4. The method of claim 3, wherein separately managing the respective metadata logs comprises: Flush the details of a metadata log while an independent shuffle associated with a metadata log is completed and other independent shuffles are still being performed.

5. The method of claim 1, wherein the first transmission includes the plurality of first destinations collecting data from a plurality of first sources.

6. The method of claim 5, wherein performing the first pass and the second pass causes the dataset to be further divided, such that one or more second sources collect data originating from two or more of the first sources.

7. The method of claim 6, wherein the number of the plurality of first residences is greater than the number of the plurality of first sources, and wherein the number of the plurality of second residences is greater than the number of the plurality of first residences.

8. The method according to claim 1, further comprising: Perform N passes, where N is a number with a value greater than two, wherein for each given pass, multiple destinations collect data from one or more of multiple sources, each source corresponding to a destination in a previous pass, wherein performing the N passes causes the dataset to be repartitioned such that one or more Nth destinations collect data originating from two or more of the first sources.

9. The method according to claim 1, wherein the method further comprises: Confirm the completion of the first transmission; as well as The second transmission is initiated when the first transmission is determined to be complete.

10. The method of claim 1, wherein the method further comprises: Prior to the first transmission, each of the plurality of first residences and the plurality of second residences is specified; as well as While at least one first residence has completed collection from one or more of the first sources, and before the first transfer is completed, the one or more processors perform an independent shuffle of the second transfer associated with the at least one first residence.

11. The method of claim 1, wherein the dataset is passed to the plurality of first destinations in the first pass using a first hash function, and the dataset is passed to the plurality of second destinations in the second pass using a plurality of second hash functions, wherein each second hash function is used for a corresponding independent shuffle and is associated with the first hash function.

12. A system for repartitioning data in a distributed network, the system comprising: One or more processors; as well as One or more storage devices communicating with the one or more processors, wherein the one or more storage devices contain a plurality of metadata logs and instructions configured to cause the one or more processors to respond to a partitioning instruction that re-partitions the dataset: Perform the first pass of the dataset to multiple first-night locations; as well as Perform a second pass of the dataset from the plurality of first-nights to the plurality of second-nights, wherein the second pass includes a plurality of independent shuffles, and wherein each independent shuffle is associated with its own corresponding metadata log.

13. The method of claim 12, wherein each corresponding metadata log includes metadata of those second-night independent shuffles associated only with itself.

14. The method of claim 13, wherein the instructions are configured to cause the one or more processors to manage the plurality of metadata logs separately.

15. The method of claim 14, wherein the instructions are configured to cause the one or more processors to separately manage the respective metadata logs by flushing the details of the one metadata log while an independent shuffle associated with one metadata log is completed and other independent shuffles are still being performed.

16. The system of claim 12, wherein each first residence collects data from two or more of the first sources, and wherein the number of the plurality of second residences is greater than the number of the plurality of second sources.

17. The system of claim 16, wherein the instructions are configured to cause the one or more processors to perform N passes, N being a number having a value greater than two, wherein for each given pass, a plurality of destinations collect data from one or more of a plurality of sources, each source corresponding to a destination of a previous pass, wherein performing the N passes causes the dataset to be repartitioned such that one or more Nth destinations collect data originating from two or more of the first sources.

18. The system of claim 17, wherein for at least one of the N transmissions: Each destination of the transmission collects data from two or more sources of the transmission; and Each of the two or more sources in the transmission includes data originating from a different source immediately preceding the previous transmission. Where at least one of the N transmissions is another transmission: Each destination of the transmission collects data from two or more sources of the other transmissions; and Each of the two or more sources in the other transmissions includes data originating from a different source immediately preceding the previous transmission, and The at least one pass and the at least one other pass are consecutive passes among the N passes.

19. The system of claim 12, wherein the instructions are configured to cause the one or more processors to: Prior to the first transmission, each of the plurality of first residences and the plurality of second residences is specified; and Upon completion of collection from one or more of the first sources at at least one first residence, and prior to completion of the first transfer, an independent shuffle of the second transfer associated with the at least one first residence is performed.

20. The system of claim 12, wherein the instructions are configured to cause the one or more processors to perform the first pass using a first hash function and to perform the second pass using a plurality of second hash functions, wherein each second hash function is used for a corresponding independent shuffle and is associated with the first hash function.

21. A method for re-partitioning data in a distributed network, the method comprising: The first shuffle of the dataset is performed by one or more first processors from multiple first sources to multiple first destinations, with each first destination collecting data from one or more of the first sources; The metadata of the first portion of the dataset is tracked by the one or more first processors during the first shuffling; A second portion of the dataset is shuffled from the plurality of first sources to a plurality of second destinations by one or more second processors independent of the one or more first processors, each second destination collecting data from one or more of the first sources; as well as The metadata of the second portion of the dataset is tracked by the one or more second processors during the second shuffling. The first and second shuffles are performed to further divide the dataset, such that one or more first residences and one or more second residences collect data from two or more of the first sources.

22. The method of claim 21, wherein the number of combinations of the plurality of first residences and the plurality of second residences is greater than the number of the plurality of first sources.

23. The method of claim 21, further comprising: Perform N shuffles from the plurality of first sources to N plurality of destinations, where N is a number having a value greater than two, wherein performing the N shuffles causes the dataset to be repartitioned such that one or more of each of the N plurality of destinations collects data from two or more of the first sources.

24. The method of claim 21, wherein the first wash and the second wash are performed in parallel with each other.

25. The method of claim 21, wherein tracking the metadata of the first portion of the dataset includes maintaining a first shuffle log of the first portion of the dataset, and wherein tracking the metadata of the second portion of the dataset includes maintaining a second shuffle log of the second portion of the dataset, wherein the first shuffle log and the second shuffle log are independent of each other.

26. The method of claim 21, wherein each of the plurality of first residences is a corresponding one of the plurality of second sources, and wherein the method further comprises: Confirm the completion of the first rinsing; as well as Upon determining that the first shuffle is complete, one or more subsequent shuffles are initiated, including the first portion of the dataset.

27. The method of claim 26, further comprising: In response to the completion of the first wash, regardless of whether the second wash has been completed, the wash log associated with the first wash is flushed by the one or more first processors.

28. The method of claim 26, further comprising: For at least one subsequent shuffle from two or more of the plurality of second sources to two or more of the plurality of third sources, maintain a third shuffle log that includes the data in the at least one subsequent shuffle.

29. The method of claim 28, wherein the at least one subsequent shuffling comprises at least some of the first portion of the dataset and at least some of the second portion of the dataset.

30. The method of claim 29, wherein the at least one subsequent wash is a butterfly wash.

31. The method of claim 28, wherein the method further comprises: Prior to the first shuffle, each of the plurality of first residences and the plurality of third residences is designated, thereby the designation of the plurality of third residences prevents data from the plurality of first sources from being collected at the plurality of third residences during the first shuffle; as well as When at least one first residence has completed collection from one or more of the first sources, and before the first transfer is completed, one or more third residences are designated to collect from the at least one first residence.

32. The method of claim 31, wherein the dataset is transmitted from the plurality of first sources to the plurality of first destinations using a first hash function, and the dataset is transmitted from the plurality of second sources to the plurality of third destinations using a second hash function associated with the first hash function.

33. The method of claim 28, wherein the identifiers of the plurality of third residences are consecutive, and wherein for each third residence, the method includes calculating a range of identifiers of the second source collected by the third residence from it based on the identifier of the third residence.

34. The method of claim 28, further comprising: The first shuffle log of the first shuffle is flushed by one or more processors; as well as Before completing the rinsing of the first shuffle log, the subsequent shuffling is performed on segments of the first portion of the dataset that have already been rinsed from the first shuffle log.

35. A system for repartitioning data in a distributed network, the system comprising: Multiple processors; as well as One or more storage devices communicating with the plurality of processors, wherein the one or more storage devices contain instructions configured to cause the plurality of processors to: The first part of the dataset is shuffled from multiple first sources to multiple first destinations, where each first destination collects data from one or more of the first sources; Metadata of the first portion of the dataset is tracked during the first shuffle; Perform a second shuffle of the dataset from the plurality of first sources to the plurality of second destinations, wherein each second destination collects data from one or more of the first sources; as well as During the second shuffling, the metadata of the second portion of the dataset is tracked. The first shuffle and the second shuffle cause the dataset to be further divided such that one or more first residences and one or more second residences collect data from two or more of the first sources.

36. The system of claim 35, wherein each first residence and each second residence collects two or more data from the first source, and wherein the number of combinations of the plurality of first residences and the plurality of second residences is greater than the number of the plurality of first sources.

37. The system of claim 35, wherein the instructions are configured to cause the one or more processors to perform N shuffles from the plurality of first sources to N plurality of destinations, where N is a number having a value greater than two, wherein performing the N shuffles causes the dataset to be repartitioned such that one or more of each of the N plurality of destinations collects data originating from two or more of the first sources.

38. The system of claim 35, wherein the instructions are configured to cause the one or more processors to track the metadata of the first portion of the dataset in each of a first shuffle log of the first portion of the dataset and a second shuffle log of the second portion of the dataset, wherein the first shuffle log and the second shuffle log are maintained independently of each other.

39. A system for repartitioning data in a distributed network, the system comprising: Multiple processors; as well as One or more storage devices communicating with the plurality of processors, wherein the one or more storage devices contain instructions configured to cause the plurality of processors to: In the first data shuffling phase, data is received from multiple sources; In the first data shuffling stage, a shuffling operation is performed on the data; The instruction to re-divide the data into multiple files based on the shuffling operation is forwarded from the first data shuffling stage to the second data shuffling stage. as well as In the second data shuffling stage, data from the multiple files is output to multiple destinations. The instructions are configured to cause the one or more processors to begin forwarding some files from the first data shuffling stage to the second data shuffling stage before the first data shuffling stage is completed.

40. The system of claim 39, wherein the instructions are configured to cause the one or more processors to perform two or more independent shuffles during the first data shuffle phase.

41. The system of claim 39, wherein the amount of the plurality of sinks is greater than the amount of the plurality of sources.

42. The system of claim 39, wherein the instructions are configured to cause the one or more processors to perform a second shuffling operation on the data during the second data shuffling phase.

43. The system of claim 39, wherein each of the plurality of files includes a data log representing a portion of the shuffling operation at the first data shuffling phase.

44. The system of claim 43, wherein each of the plurality of files further comprises a shuffling log of at least a portion of the shuffling operation at the first data shuffling phase.

45. The system of claim 39, wherein each of the plurality of files includes data segmentation details and metadata of the shuffling operation.

46. ​​The system of claim 39, wherein the instructions are configured to cause the one or more processors to hold the plurality of files during the first data shuffling phase until the first data shuffling phase is completed.

47. A method for re-partitioning data in a distributed network, the method comprising: Receive data from multiple sources at a computing system that includes one or more processors; The first data shuffling stage is performed by the one or more processors, wherein the first data shuffling stage includes shuffling the data; The one or more processors forward multiple files from the first data shuffling stage to the second data shuffling stage, wherein the multiple files indicate that the data is to be re-divided based on the shuffling operation; as well as The second data shuffling stage is performed by the one or more processors, wherein the second data shuffling stage includes outputting data from the plurality of files to multiple destinations. The method further includes: before the first data shuffling stage is completed, the one or more processors forward some files from the first data shuffling stage to the second data shuffling stage.

48. The method of claim 47, wherein the shuffling operation in the first data shuffling stage comprises two or more independent shuffling operations.

49. The method of claim 47, wherein the amount of the plurality of sinks is greater than the amount of the plurality of sources.

50. The method of claim 47, wherein the second data shuffling stage includes performing a second shuffling operation on the data.

51. The method of claim 47, wherein each of the plurality of files comprises a data log of a portion of the shuffling operation at the first data shuffling stage.

52. The method of claim 51, wherein each of the plurality of files further comprises a shuffling log of at least a portion of the shuffling operation at the first data shuffling phase.

53. The method of claim 47, wherein each of the plurality of files includes data segmentation details and metadata of the shuffling operation.

54. The method of claim 47, further comprising: The multiple files are held by one or more processors during the first data shuffling phase until the first data shuffling phase is completed.

55. The method of claim 47, wherein performing the shuffling operation causes the data to be further divided such that at least one destination collects data originating from two or more sources.

56. The method of claim 47, wherein performing the shuffling operation causes the data to be further divided such that each destination collects data originating from two or more sources.

57. A method comprising: One or more processors perform a first regrouping of datasets from multiple first sources, and perform a first split of the regrouped datasets into multiple first destinations, wherein the number of multiple first destinations is greater than the number of multiple first sources; as well as The one or more processors perform a second regrouping of the dataset in the plurality of first residences, and perform a second split of the regrouped dataset into a plurality of second residences, wherein the number of the plurality of second residences is greater than the number of the plurality of first residences.

58. The method of claim 57, wherein the second regrouping comprises multiple independent washes.

59. The method of claim 58, wherein the plurality of independent washes are performed sequentially.

60. The method of claim 58, wherein the plurality of independent washes are performed simultaneously.

61. The method of claim 58, wherein the first regrouping and first split of the dataset are associated with a first metadata log, and wherein each independent shuffle of the second regrouping of the dataset is associated with its own corresponding second metadata log, wherein each corresponding second metadata log includes metadata of those second destinations associated only with itself by the independent shuffles.

62. The method of claim 61, further comprising: Flush the details of a second metadata log while an independent shuffle associated with a second metadata log is completed and other independent shuffles are still being performed.

63. The method of claim 57, wherein the method further comprises: Prior to the first regrouping, each of the plurality of first dormitories and the plurality of second dormitories is specified; as well as While at least one first host has completed collecting the corresponding portion of the regrouped dataset from one or more of the first sources, and before the first regrouping and splitting is completed, the one or more processors perform an independent shuffling of the corresponding portion of the regrouped dataset received by the at least one first host.

64. The method of claim 57, wherein the number of the plurality of second residences is twice or more than the number of the plurality of first residences.

65. The method of claim 57, wherein in the first split of the dataset, each first residence receives data from at least two first sources, and wherein in the second split of the dataset, each second residence receives data from at least two first residences.

66. The method of claim 65, wherein the first and second residences are divided among a plurality of independent shuffles of the second regrouping and second split of the dataset, wherein each independent shuffle is associated with its own corresponding metadata log, such that each corresponding metadata log includes metadata of only those second residences associated with its own independent shuffles.

67. A system comprising: One or more processors; as well as One or more storage devices communicating with the one or more processors, wherein the one or more storage devices contain instructions configured to cause the one or more processors to: Perform a first regrouping of the datasets in multiple first sources, and perform a first split of the regrouped datasets into multiple first destinations, wherein the number of multiple first destinations is greater than the number of multiple first sources; as well as Perform a second regrouping of the dataset in the plurality of first residences, and perform a second split of the regrouped dataset into a plurality of second residences, wherein the number of the plurality of second residences is greater than the number of the plurality of first residences.

68. The system of claim 67, wherein the second regrouping comprises a plurality of independent washes.

69. The system of claim 68, wherein the instructions are configured to cause the one or more processors to sequentially perform the plurality of independent shuffles.

70. The system of claim 68, wherein the instructions are configured to cause the one or more processors to perform the plurality of independent shuffles simultaneously.

71. The system of claim 68, wherein the first regrouping and first split of the dataset are associated with a first metadata log, and wherein each independent shuffle of the second regrouping of the dataset is associated with its own corresponding second metadata log, wherein each corresponding second metadata log includes metadata of those second swirls associated only with itself.

72. The system of claim 71, wherein the instructions are configured to cause the one or more processors to flush the details of the second metadata log when an independent shuffle associated with a second metadata log is completed while other independent shuffles are still being performed.

73. The system of claim 67, wherein the instructions are configured to cause the one or more processors to: Prior to the first regrouping, each of the plurality of first dormitories and the plurality of second dormitories is specified; and While at least one first residence has completed collecting the corresponding portion of the regrouped dataset from one or more of the first sources, and before the first regrouping and splitting is completed, an independent shuffling of the corresponding portion of the regrouped dataset received by the at least one first residence is performed.

74. The system of claim 67, wherein the number of the plurality of second dwellings is twice or more than the number of the plurality of first dwellings.

75. The system of claim 67, wherein the instructions are configured to cause the one or more processors to: In the first split of the dataset, each first residence is configured to receive data from at least two first sources, and In the second split of the dataset, each second residence is configured to receive data from at least two first residences.

76. The system of claim 75, wherein the first and second destinations are divided among a plurality of independent shuffles of a second regrouping and a second split of the dataset, wherein each independent shuffle is associated with its own corresponding metadata log, such that each corresponding metadata log includes metadata of only those second destinations associated with its own independent shuffles.

Images