Transmission error correction method and system for encoding MapReduce framework

Abstract

This invention discloses a transmission error correction method and system for the encoding MapReduce framework. The method addresses the transmission correctness problem in the Shuffle phase of encoding MapReduce. Based on the multicast characteristics within encoding MapReduce multicast groups, and by fully utilizing the redundant intermediate results inherent in each node, a novel distributed encoding scheme is designed. This scheme encodes and multicasts intermediate files on a multicast group basis, fully utilizing the computational redundancy introduced by encoding MapReduce while significantly reducing the redundant communication load caused by encoding error correction in the Shuffle phase. This results in a reduction of the total execution time of the MapReduce task for transmission error correction.

Description

Technical Field

[0001] This invention belongs to the field of distributed systems, specifically relating to a method and system for encoding MapReduce transmission error correction based on distributed encoding error correction technology. Background Technology

[0002] In the MapReduce model of distributed computing, task execution is divided into three stages: the Map and Reduce computation stages, and the Shuffle operation in the intermediate communication stage. The Map stage processes multiple independently running Map tasks after the large-scale data job is split, generating a large number of intermediate results. These intermediate results are then shuffled in the Shuffle stage before being processed by the Reduce tasks to output the final results. However, the large number of intermediate values ​​shuffled in the Shuffle stage leads to excessive communication load, thus impacting the overall performance of MapReduce. Researchers have found that in Facebook Hadoop clusters, the Shuffle stage accounts for 50% and 70% of the total job execution time in 26% and 16% of computation tasks, respectively. Even in Amazon EC2 clusters, the Shuffle stage can account for 70% of the total job execution time during self-joining processes. In the TeraSorting sorting task, the Shuffle data shuffling time accounts for as much as 98.4% of the total job time. Therefore, reducing the communication load in the Shuffle stage is crucial for improving the overall performance of job execution.

[0003] To address the issue of excessive communication load on compute nodes during the Shuffle phase, researchers have proposed introducing computational redundancy. By XOR-encoding the Map phase results and multicasting them within a multicast group, the communication overhead of the Shuffle phase is significantly reduced. This framework is called Encoded MapReduce, where a substantial reduction in communication load is achieved with relatively small redundant storage and computational costs. However, existing Encoded MapReduce frameworks ignore potential transmission errors during the Shuffle phase, assuming that the entire system transmits data error-free.

[0004] In existing research, index coding technology has been used to address the transmission reliability issue in the Shuffle stage. This technology encodes and transmits exchanged information only in a single computing node. However, when the number of exchanged information packets is small, the overhead of the Shuffle stage can increase by up to two times, resulting in a large amount of transmission redundancy and severely impacting performance. Therefore, a more suitable error correction scheme for encoding the MapReduce framework is needed to reduce the additional overhead caused by error correction. Summary of the Invention

[0005] The purpose of this invention is to propose a method and system for error correction during the Shuffle phase of a MapReduce task, under the constraint of one-time transmission, in the scenario of transmission error correction based on multicast groups. Error correction during the Shuffle transmission phase is achieved by designing a distributed index coding scheme based on multicast groups, and the additional transmission overhead caused by transmission error correction is significantly reduced.

[0006] The present invention also provides a corresponding encoding MapReduce transmission error correction system.

[0007] To achieve the above-mentioned objectives, the present invention adopts the following technical solution:

[0008] A transport error correction method for the encoding MapReduce framework includes the following steps:

[0009] (1) The control node determines the storage redundancy of the input file according to the distributed index encoding and places the file set on the corresponding MapReduce computing node. The storage redundancy indicates the number of nodes on which the same file is computed in the Map stage on multiple different computing nodes.

[0010] (2) The control node divides the entire system into multiple multicast groups based on storage redundancy and the number of computing nodes. The files contained in each multicast group have no overlap. The encoding strategy of the computing nodes contained in the multicast group is determined according to the size of the multicast group.

[0011] (3) Within one or more multicast groups to which it belongs, the computing node performs XOR encoding calculation on the local redundant intermediate value according to the pre-set index encoding strategy, and transmits the obtained original intermediate file and check bit file to other computing nodes in the multicast group through multicast. The original file indicates the file that participates in the inverse XOR decoding calculation.

[0012] (4) The computing node combines the received check bit file with the local redundant intermediate value, and then performs correction and decoding together with the received original file according to the pre-set index encoding strategy to recover the correct original file.

[0013] (5) The compute nodes perform reverse decoding calculations based on the correct original files and the redundant intermediate value files on the compute nodes' local computers to recover the intermediate value files required for the Reduce stage.

[0014] Further, step (1) includes:

[0015] Run a MapReduce test job and use a monitoring program to obtain the runtime T of the Map phase tasks. map and the transmission time T during the Shuffle stages h uffle ;

[0016] Determine the storage redundancy r such that (rT) map +kT s h uffle It can reach the minimum value, where k is a coefficient that varies with r, and the formula for calculating k is:

[0017]

[0018] Where N(r+1,δ) represents the shortest linear coding length for correcting δ errors in r+1 valid bytes, where δ is determined by the quality of system transmission and n indicates the number of computing nodes in the system;

[0019] Based on the determined storage redundancy r, all possible subsets of the working nodes are arranged and combined using r as the base, and the input file set is evenly distributed. At the same time, the file subsets of each arrangement and combination have no overlap. The input files are placed according to the distribution results, and each input file is placed on all computing nodes contained in its respective computing node set.

[0020] Further, step (2) includes:

[0021] Based on the storage redundancy r determined in step (1), the size of the multicast group in the system is determined to be r+1. The multicast group is a subset of working nodes of size r+1. All possible subsets of computing nodes are arranged and combined with r+1 as the base, and the constructed subset is the multicast group.

[0022] Using the size r+1 of the multicast group as the number of significant bits, construct a length for which the Hamming distance is greater than or equal to 2δ+1. The index encoding strategy is determined, and the corresponding generation matrix is ​​constructed.

[0023] Further, step (3) includes:

[0024] Traverse all multicast groups divided in step (2) and perform the following steps:

[0025] (a) For any multicast group, the intermediate value results of any computing node in the multicast group are evenly distributed to other nodes for transmission. Any computing node will be responsible for transmitting the XOR value between r intermediate value results, which is the original file.

[0026] (b) Based on the position of the computing node in the generator matrix, the computing node transmits the part of the check bit file that it needs to transmit. The check bit files are all files after XOR operation between the corresponding original files. The computing node uses the XOR value between the original file and the intermediate value file after removing the missing local part as the check bit file for multicast transmission and sends it to other computing nodes in the multicast group.

[0027] Further, step (4) includes:

[0028] Traverse all multicast groups divided in step (2) and perform the following steps:

[0029] (i) For any multicast group, for any computing node in the multicast group, based on the position of the computing node in the generator matrix, the node receives the part of the check bit file that it needs to receive through the multicast of the multicast group. All missing parts of the check bit file are redundant intermediate value results of the node itself. The XOR value between the missing intermediate values ​​is XORed with the corresponding check bit file to obtain the required check bit file.

[0030] (ii) Any compute node in the multicast group receives a check file in addition to the check file it transmits. It then performs a check against the consistency check matrix of the generated matrices in the original file and the received check file. If there is no transmission error, the current traversal ends. If a transmission error occurs, the process continues.

[0031] (iii) Based on the consistency check matrix, identify the erroneous files among all accepted files, discard the erroneous files, and restore the erroneous files based on the other files in the consistency check matrix to finally obtain the correct original files.

[0032] Further, step (5) includes:

[0033] Traverse all multicast groups divided in step (2) and perform the following steps:

[0034] For any multicast group, any compute node in the multicast group will use the corrected original file or its local intermediate file to restore the missing intermediate file for that compute node.

[0035] A transport error correction system for a MapReduce framework includes a control node and several compute nodes. The control node is configured to: determine the storage redundancy of input files based on distributed index coding; place the file set on the corresponding MapReduce compute nodes, wherein the storage redundancy indicates the number of nodes on which the same file is computed in the Map phase on multiple different compute nodes; and divide the entire system into multiple multicast groups based on the storage redundancy and the number of compute nodes, wherein the files contained in each multicast group have no overlap; and determine the coding strategy of the compute nodes contained in the multicast group based on the size of the multicast group.

[0036] The computing node is configured to: perform XOR encoding calculation on local redundant intermediate values ​​according to a pre-set index encoding strategy within one or more multicast groups to which it belongs, and transmit the resulting original intermediate file and check bit file to other computing nodes in the multicast group via multicast. The original file indicates the file that participated in the inverse XOR decoding calculation.

[0037] After combining the received check bit file with the local redundant intermediate value, it is then corrected and decoded together with the received original file according to the pre-set index encoding strategy to recover the correct original file.

[0038] Based on the correct original file and combined with the redundant intermediate value file on the local compute node, reverse decoding calculation is performed to recover the intermediate value file required for the Reduce stage.

[0039] Furthermore, the control node includes:

[0040] The partitioning unit is used to divide the computing node set, input file set, and multicast group;

[0041] The allocation unit is used to evenly distribute the input file set across n computing nodes;

[0042] The encoding scheme design unit is used to design a distributed encoding computing scheme with error correction capability. The error correction encoding scheme adopts an index encoding scheme for use by computing nodes when encoding.

[0043] Furthermore, the computing node includes:

[0044] The execution unit is used to perform Map phase operations on the input file set allocated locally and Reduce phase operations on the local intermediate value results and the recovered intermediate value results;

[0045] The transmission unit is used to multicast the original file and checksum file to the computing nodes within the multicast group;

[0046] The decoding unit is used to perform correction decoding and recovery decoding on the received original file and checksum file.

[0047] Beneficial effects: This invention addresses the transmission correctness problem in the Shuffle stage of MapReduce encoding. Based on the multicast characteristics within multicast groups in MapReduce encoding, and by fully utilizing the redundant intermediate results of each node, a novel distributed encoding scheme is designed to encode and multicast intermediate files in units of multicast groups. This fully utilizes the computational redundancy brought about by MapReduce encoding, and significantly reduces the redundant communication load brought about by encoding error correction in the Shuffle stage, thereby reducing the total execution time of the MapReduce task for transmission error correction. Attached Figure Description

[0048] Figure 1 This is a flowchart of the MapReduce transmission error correction method of the present invention;

[0049] Figure 2 This is a diagram illustrating the process of determining storage redundancy in this invention;

[0050] Figure 3 This is a schematic diagram of the placement of the input file set in an embodiment of the present invention;

[0051] Figure 4 (a) and Figure 4 (b) is a schematic diagram of the encoding matrix in an embodiment of the present invention;

[0052] Figure 5 This is a schematic diagram of computing node transmission in an embodiment of the present invention;

[0053] Figure 6 This is a schematic diagram of correction decoding and recovery decoding in an embodiment of the present invention. Detailed Implementation

[0054] The technical solution of the present invention will be further described below with reference to the accompanying drawings and specific embodiments.

[0055] Encoded MapReduce is based on the Map-Shuffle-Reduce distributed computing framework. In this framework, the input file set is computed on different computing nodes, and then data is transferred between these nodes to the corresponding Reduce function nodes. The Reduce function then outputs the results. Each computing node performs both Map and Reduce computations. Because the input file set is distributed, intermediate values ​​after the Map phase computation need to be shuffled. Encoded MapReduce can effectively compress the amount of data transmitted, but the massive redundancy introduced by traditional one-time delivery error correction schemes is a huge waste of bandwidth resources. Therefore, a new distributed indexing strategy needs to be designed to reduce the additional overhead caused by error correction during the Shuffle phase.

[0056] like Figure 1 As shown, the present invention proposes a MapReduce transmission error correction method based on distributed coding error correction technology, which includes the following steps:

[0057] Step S1: Determine storage redundancy through a distributed indexing and encoding strategy, and place the input file set on the computing node;

[0058] Step S2: Divide multicast groups according to storage redundancy, and determine the specific distributed coding strategy within the multicast group according to the size of the multicast group.

[0059] Step S3: Encode the intermediate values ​​to form the original file and the checksum file, and then transmit them via multicast.

[0060] Step S4: The computing node corrects and decodes the received data based on the local intermediate values.

[0061] Step S5: The computing node decodes the corrected data to recover the required intermediate value.

[0062] The coded MapReduce framework described in this paper can also be referred to as a coded distributed computing framework. The transmission error correction method / system for the coded MapReduce framework described in this paper can also be referred to as a transmission error correction method / system for the coded distributed computing framework. The " / " in this paper represents "or".

[0063] Figure 2 The process for determining storage redundancy involves a test job, which is a small subset of the input file set to be processed. This small subset can be considered to reflect the overall job's performance to some extent, thus estimating the execution time of each stage. Calculations can determine the optimal value of r for the entire job to run optimally.

[0064] Specifically, determining the storage redundancy of the input file based on distributed index coding includes:

[0065] Set a fixed number of compute nodes and a storage redundancy level r, where the storage redundancy level must be a positive integer not greater than the number of compute nodes. Run a MapReduce test job—a small-scale TeraSort program—in a distributed cluster, and use a monitoring program to obtain the runtime T of the Map phase tasks. map and the transmission time T during the Shuffle stage s h uffle ;

[0066] Determine the storage redundancy r such that (rT) map +kT s h uffle It can reach the minimum value, where k is a coefficient that varies with r, and the formula for calculating k is:

[0067]

[0068] Where N(r+1,δ) represents the shortest linear coding length for correcting δ errors in (r+1) valid bytes, δ is determined by the quality of system transmission and is generally taken as 1, and n indicates the number of computing nodes in the system;

[0069] Based on the determined storage redundancy r, all possible subsets of the working nodes are arranged and combined using r as the base, and the input file set is evenly distributed. The evenly distributed file subsets have no overlap. The input files are placed according to the distribution results, and each input file is placed on all computing nodes contained in its set of computing nodes.

[0070] According to an embodiment of the present invention, the encoding strategy for dividing multicast groups and determining the encoding strategy of computing nodes within a multicast group based on the multicast group size includes:

[0071] Based on the determined storage redundancy r, the size of the multicast group in the system is determined to be (r+1). The multicast group is a subset of working nodes of size (r+1). All possible subsets of computing nodes are arranged and combined with a base of (r+1) to construct the multicast group.

[0072] Using the size of the multicast group (r+1) as the effective number of bits, construct a length for which the Hamming distance is greater than or equal to (2δ+1). The index encoding strategy is determined, and the corresponding generation matrix is ​​constructed.

[0073] Figure 3Consider a system with 4 compute nodes, a storage redundancy of 3, and 12 input files. The multicast group size is the same as the number of compute nodes, so all input file sets reside in the same multicast group. In the system, each file has three redundancies. Each node computes a portion of the files, obtaining intermediate values. After the Map phase, the intermediate values ​​for each file are categorized into four types: squares, circles, triangles, and rhombuses. These intermediate values ​​are the processing requirements for nodes 1, 2, 3, and 4, respectively. Each node's intermediate values ​​are marked with `has`. To process all intermediate values ​​for a specific shape, each node needs to obtain intermediate values ​​from other nodes; `needs` represents the required intermediate values ​​for each node.

[0074] Because the Shuffle phase involves multicasting the XOR-encoded original file, the encoding process for obtaining the original file involves each compute node XOR-encoding the intermediate values ​​that it already has locally but are missing from other compute nodes. For example... Figure 3 It can be seen that any computation node is missing a portion of intermediate values. Figure 3 In this scenario, each node is missing 3 intermediate value files. The missing intermediate values ​​of any node in the multicast group are evenly distributed among other computing nodes for transmission. Each computing node can effectively compress the amount of data transmitted by performing an XOR operation on the intermediate values ​​it needs to transmit.

[0075] According to an embodiment of the present invention, encoding intermediate values ​​to form an original file and a checksum file, and then multicasting them within the corresponding multicast group includes:

[0076] Iterate through all the multicast groups and perform the following steps:

[0077] (a) For any multicast group, the intermediate value results of any computing node in the multicast group are evenly distributed to other nodes for transmission. Any computing node will be responsible for transmitting the XOR value between r intermediate value results, which is the original file.

[0078] (b) Based on the position of the computing node in the generator matrix, it needs to transmit a portion of the check bit file corresponding to it. The check bit file is the file after XOR operation between the corresponding original files. The computing node uses the XOR value between the original file and the intermediate value file after removing the missing local part as the check bit file to multicast and transmit it to other computing nodes in the multicast group.

[0079] According to an embodiment of the present invention, the computing node performs correction decoding on the received data based on existing local intermediate value results, including:

[0080] Iterate through all the multicast groups and perform the following steps:

[0081] (a) For any multicast group, for any computing node in the multicast group, based on the position of the computing node in the generator matrix, the node receives the part of the check bit file that it needs to receive through the multicast of the multicast group. All missing parts of the check bit file are redundant intermediate value results of the node itself. The XOR value between the missing intermediate values ​​is XORed with the corresponding check bit file to obtain the required check bit file.

[0082] (b) Any compute node in the multicast group receives a checksum file in addition to the checksum file it transmits. It then performs a checksum verification based on the consistency checksum matrix generated by the original file and the received checksum file. If there is no transmission error, the current traversal ends. If a transmission error occurs, the process continues.

[0083] (c) Based on the consistency check matrix, the erroneous files among all accepted files can be identified, the erroneous files can be discarded, and the erroneous files can be restored based on the other files in the consistency check matrix, finally obtaining the correct original files.

[0084] According to an embodiment of the present invention, the process of restoring and decoding the corrected result based on existing local intermediate values ​​by the computing node includes:

[0085] Iterate through all the multicast groups and perform the following steps:

[0086] For any multicast group, any compute node in the multicast group will use the corrected original file or its local intermediate file to restore the missing intermediate file for that compute node.

[0087] Figure 4 The matrix shown in (a) is the generator matrix of a shortest linear code that satisfies a coding distance of 4 and an information bit length of 4. Figure 4 (b) is Figure 4 (a) A portion of the matrix, representing a linear code with a coding distance of 3 and an error correction capability of... Combination Figure 3 The four nodes shown are obtained by encoding the original file. The specific operation method is as follows: Figure 4 In the generator matrix shown in (a), the parts in each column that are 1 represent files XORed with the corresponding row numbers. After XORing each column, the four check bits (excluding the information bits) are evenly distributed among the four computing nodes for transmission. Figure 5 As shown, each node transmits an original file plus a checksum file. The checksum file is incomplete and needs to be completed in subsequent steps.

[0088] When a computing node receives the required file, it needs to correct, restore, and decode the received data. For example... Figure 6As shown, taking node 1 as an example, the first step is to complete the checksum data. It can be seen that by completing the latter half of the three checksum files received by the compute node, a complete checksum file can be obtained. These files are all files that compute node 1 itself possesses. After completing the checksum file, compute node 1... Figure 4 (b) The matrix is ​​used for consistency verification. After verification, correction and restoration can yield three correct and complete original files. At this point, the compute node can XOR the original files with the existing redundant intermediate value files locally to decode and restore the three intermediate values ​​that compute node 1 itself actually has, thus enabling it to continue executing the subsequent Reduce stage computation.

[0089] This invention also provides a MapReduce transmission error correction system based on distributed coding error correction technology, including a control node and computing nodes. The control node is used for the allocation and placement of the input file set for subset partitioning of computing nodes, and for dividing multicast groups and designing a distributed index coding scheme that conforms to the characteristics of multicast groups. Each computing node processes the allocated input file set and outputs an intermediate value file set. According to the distributed index coding strategy, the original file and check bit file are generated and multicast within the corresponding multicast group. Each computing node generates the required intermediate value file according to the correction decoding strategy and the recovery decoding strategy and outputs it through the Reduce function.

[0090] According to an embodiment of the present invention, the control node includes:

[0091] The partitioning unit is used to divide the computing node set, input file set, and multicast group;

[0092] The allocation unit is used to evenly distribute the input file set across n computing nodes;

[0093] The encoding scheme design unit is used to design a distributed encoding computing scheme with error correction capability. The error correction encoding scheme adopts an index encoding scheme for use by computing nodes when encoding.

[0094] According to an embodiment of the present invention, the computing node includes:

[0095] The execution unit is used to perform Map phase operations on the input file set allocated locally and Reduce phase operations on the local intermediate value results and the recovered intermediate value results;

[0096] The transmission unit is used to multicast the original file and checksum file to the computing nodes within the multicast group;

[0097] The decoding unit is used to perform correction decoding and recovery decoding on the received original file and checksum file.

[0098] The working principle of the MapReduce transmission error correction system of this invention is as follows: The encoding process includes encoding the original data from the intermediate value, obtaining the check data from the original data, and the receiving node encoding the received check data with the local intermediate value. The decoding process includes obtaining the correct original file after error detection and decoding the corrected original file to obtain the intermediate value required by the computing node, all of which are XOR operations. The XOR operation performs bitwise addition without carry between data. In the MapReduce encoding, the XOR operation can transform multiple values ​​into a single value, thereby compressing the amount of transmitted data. When there is an XOR value, the result of XORing n data values, performing another XOR operation with (n-1) of the data values ​​can restore the required value.

[0099] The input file set is placed using a redundant storage approach, managed by the partitioning and allocation units of the control nodes. For each input file, it is placed on r compute nodes, with each compute node having the same number of input files. This redundant storage ensures that the r compute nodes, after processing the same input file, will produce the same r intermediate value files, creating encoding conditions and providing a foundation for data compression during transmission. Furthermore, this redundant storage method allows the time taken for each compute node to complete the Map phase computations to be as evenly distributed as possible, thus reducing the time wasted due to the "lagging" problem. When r is 1, it means there is no redundant storage, and the normal MapReduce task execution method applies.

[0100] Since the redundant transmission volume caused by error correction is related to the multicast group size, which is determined by the storage redundancy, and different storage redundancy values ​​have a certain impact on the total task execution time, it is necessary to pre-execute a test job to determine the storage redundancy. The test job is a small subset of the task dataset, used to test the computing and data transmission capabilities of the MapReduce cluster. Running the test job requires all computing nodes involved in the task to participate in a comprehensive test of cluster performance. Because the test dataset is small, the test can be completed in a short time to determine the storage redundancy r. This is achieved by recording the Map phase computation time T of the test task. map and the Shuffle stage transfer time T shuffle This is used to estimate the test task runtime for different values ​​of r. The specific estimation formula is (rT) map +kT shuffle ), where k is a coefficient that varies with r, and is calculated as follows:

[0101]

[0102] Where N(r+1,δ) represents the shortest linear coding length for correcting δ errors in r+1 valid bytes, δ is determined by the quality of system transmission and is generally taken as 1, n indicates the number of computing nodes in the system, r takes different values ​​to calculate the estimated value, and finds the value of r that results in the lowest estimated value, and uses this value of r as the storage redundancy.

[0103] During the file set placement phase, the file set is first divided into equal parts. The term "r copies" refers to a permutation of files with a base of r across n computing nodes. The file set is evenly distributed among each permutation, and each computing node in each permutation places its assigned file set. This ensures that each input file has r copies throughout the entire system.

[0104] Based on the determined storage redundancy, the size of the multicast group in the system can be determined. The multicast group size is (r+1), which is the number of compute nodes it contains. Each multicast group contains (r+1) subsets of compute nodes with a cardinality of r. The file sets divided by these subsets are executed within their respective multicast groups. Specifically, when r is n, all files are backed up on all compute nodes without the need for shuffling intermediate values; when r is 1, it is a typical MapReduce execution method, involving point-to-point file transfer between compute nodes.

[0105] Multicast group error correction coding differs from traditional error correction coding strategies. In traditional error correction models, data is sent from a central service node to slave nodes, allowing for unified encoding without the need for data padding at the receiving nodes. Distributed index coding strategies can be implemented using redundant storage on each computing node. For any multicast group, the required number of valid data bits to be transmitted is (r+1), therefore a linear index coding strategy is constructed using (r+1) information bits. For the check bits, due to the storage redundancy of the MapReduce encoding, each check bit data can be used by r compute nodes. Let N(r+1,δ) be the index encoding length for correcting δ errors in r+1 information bits. Theoretically, each compute node needs (N(r+1,δ)-(r+1)) check bits to achieve error correction. Therefore, it can be calculated that ((N(r+1,δ)-(r+1))*(r+1)) check bit data are needed in a multicast group. With a storage redundancy of r, and each check bit data being usable by r compute nodes, the number of check bits required in a multicast group is:

[0106]

[0107] The encoding strategy is completed by the encoding scheme design unit of the control node. A specific example can be found in the above method embodiments. Figure 3 , Figure 4 (a) Figure 4(b) The example described.

[0108] At this point, each compute node in the multicast group multicasts the original file and the checksum file within the group. This is done by the transmission unit of the compute node, while the other compute nodes process the received files using their decoding units. It's important to note that because some files are missing from the transmission nodes, the checksum file, upon receipt, needs to be XORed with the compute node's redundant intermediate values ​​to obtain complete checksum data without affecting the overall error correction capability. Based on the consistency check matrix assigned to each compute node, the compute node can perform error detection and correction on the received data—this is the correction phase. After obtaining the original file through correction, XORing the corresponding existing intermediate values ​​recovers the missing subset of intermediate values ​​from the compute node. Then, the Reduce function is executed to obtain the final output of the MapReduce task.

[0109] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0110] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0111] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0112] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0113] This invention has many specific applications. The above description is only a preferred embodiment of this invention. It should be noted that for those skilled in the art, several improvements can be made without departing from the principle of this invention, and these improvements should also be considered to fall within the protection scope of this invention.

Claims

1. A transmission error correction method for the encoding MapReduce framework, characterized in that, The method includes the following steps: (1) The control node determines the storage redundancy of the input file according to the distributed index encoding and places the file set on the corresponding MapReduce computing node. The storage redundancy indicates the number of nodes on which the same file is computed in the Map stage on multiple different computing nodes. (2) The control node divides the entire system into multiple multicast groups based on storage redundancy and the number of computing nodes. The files contained in each multicast group have no overlap. The encoding strategy of the computing nodes contained in the multicast group is determined according to the size of the multicast group. (3) Within one or more multicast groups to which it belongs, the computing node performs XOR encoding calculation on the local redundant intermediate value according to the pre-set index encoding strategy, and transmits the obtained original file and check file to other computing nodes in the multicast group through multicast. The original file indicates the file participating in the inverse XOR decoding calculation, and the check file is the file after XOR operation between the corresponding original files. (4) The computing node combines the received check bit file with the local redundant intermediate value, and then performs correction and decoding together with the received original file according to the pre-set index encoding strategy to recover the correct original file. (5) The compute nodes perform reverse decoding calculations based on the correct original files and the redundant intermediate value files on the compute nodes' local computers to recover the intermediate value files required for the Reduce stage.

2. The method according to claim 1, characterized in that, Step (1) includes: Run a MapReduce test job and use a monitoring program to obtain the runtime T of the Map phase tasks. map and the transmission time T during the Shuffle stage shuffle ; Determine the storage redundancy r such that (rT) map +kT shuffle It can reach the minimum value, where k is a coefficient that varies with r, and the formula for calculating k is: Where N(r+1,δ) represents the shortest linear coding length for correcting δ errors in r+1 valid bytes, where δ is determined by the quality of system transmission and n indicates the number of computing nodes in the system; Based on the determined storage redundancy r, all possible subsets of the working nodes are arranged and combined using r as the base, and the input file set is evenly distributed. At the same time, the file subsets of each arrangement and combination have no overlap. The input files are placed according to the distribution results, and each input file is placed on all computing nodes contained in its respective computing node set.

3. The method according to claim 2, characterized in that, Step (2) includes: Based on the storage redundancy r determined in step (1), the size of the multicast group in the system is determined to be r+1. The multicast group is a subset of working nodes of size r+1. All possible subsets of computing nodes are arranged and combined with r+1 as the base, and the constructed subset is the multicast group. Using the size r+1 of the multicast group as the number of significant bits, construct a length for which the Hamming distance is greater than or equal to 2δ+1. The index encoding strategy is determined, and the corresponding generation matrix is ​​constructed.

4. The method according to claim 3, characterized in that, Step (3) includes: Traverse all multicast groups divided in step (2) and perform the following steps: (a) For any multicast group, the intermediate value results of any computing node in the multicast group are evenly distributed to other nodes for transmission. Any computing node will be responsible for transmitting the XOR value between r intermediate value results, which is the original file. (b) Based on the position of the computing node in the generator matrix, the computing node transmits the part of the check bit file that it needs to transmit. The check bit files are all files after XOR operation between the corresponding original files. The computing node uses the XOR value between the original file and the intermediate value file after removing the missing local part as the check bit file for multicast transmission and sends it to other computing nodes in the multicast group.

5. The method according to claim 3, characterized in that, Step (4) includes: Traverse all multicast groups divided in step (2) and perform the following steps: (i) For any multicast group, for any computing node in the multicast group, based on the position of the computing node in the generator matrix, the node receives the part of the check bit file that it needs to receive through the multicast of the multicast group. All missing parts of the check bit file are redundant intermediate value results of the node itself. The XOR value between the missing intermediate values ​​is XORed with the corresponding check bit file to obtain the required check bit file. (ii) Any compute node in the multicast group receives a checksum file in addition to the checksum file it transmits. It then performs a checksum verification based on the consistency checksum matrix generated by the original file and the received checksum file. If there is no transmission error, the current traversal ends. If a transmission error occurs, the process continues. (iii) Based on the consistency check matrix, identify the erroneous files among all accepted files, discard the erroneous files, and restore the erroneous files based on the other files in the consistency check matrix to finally obtain the correct original files.

6. The method according to claim 3, characterized in that, Step (5) includes: Traverse all multicast groups divided in step (2) and perform the following steps: For any multicast group, any compute node in the multicast group will use the corrected original file or its local intermediate file to restore the missing intermediate file for that compute node.

7. A transmission error correction system for the MapReduce encoding framework, characterized in that, It includes a control node and several computing nodes. The control node is configured to: determine the storage redundancy of the input file according to the distributed index encoding, and place the file set on the corresponding MapReduce computing node. The storage redundancy indicates the number of nodes on which the same file is computed in the Map phase on multiple different computing nodes. Based on storage redundancy and the number of computing nodes, the entire system is divided into multiple multicast groups, where the files contained in each multicast group have no overlap, and the encoding strategy of the computing nodes contained in the multicast group is determined according to the size of the multicast group. The computing node is configured to: perform XOR encoding calculation on local redundant intermediate values ​​according to a pre-set index encoding strategy within one or more multicast groups to which it belongs, and transmit the resulting original file and check file to other computing nodes in the multicast group via multicast. The original file indicates the file participating in the inverse XOR decoding calculation, and the check file is the file after XOR operation between the corresponding original files. After combining the received check bit file with the local redundant intermediate value, it is corrected and decoded together with the received original file according to the pre-set index encoding strategy to recover the correct original file. Based on the correct original file and combined with the redundant intermediate value file on the local compute node, reverse decoding calculation is performed to recover the intermediate value file required for the Reduce stage.

8. The system according to claim 7, characterized in that, The control node includes: The partitioning unit is used to divide the computing node set, input file set, and multicast group; The allocation unit is used to evenly distribute the input file set across n computing nodes; The encoding scheme design unit is used to design a distributed encoding computing scheme with error correction capability. The distributed encoding computing scheme with error correction capability adopts an index encoding scheme for use by computing nodes when encoding.

9. The system according to claim 7, characterized in that, The computing nodes include: The execution unit is used to perform Map phase operations on the input file set allocated locally and Reduce phase operations on the local intermediate value results and the recovered intermediate value results; The transmission unit is used to multicast the original file and checksum file to the computing nodes within the multicast group; The decoding unit is used to perform correction decoding and recovery decoding on the received original file and checksum file.

Images