A Method for Improving MapReduce Spill Writing Based on MPI-IO

Abstract

The present invention discloses a method for improving MapReduce spill writing based on MPI-IO. The method includes: the Map-side MPI process reads a data set slice from a target file; the Map-side MPI process runs a Map task, performs mapping processing on the data slice and partitions the mapping processing result to obtain partitioned key-value pairs; when it is determined that the size of the mapping processing result exceeds the memory capacity threshold, the Map-side executes a spill writing operation, and parallelly spills the sorted key-value pairs after partitioning to the same disk file to obtain a spill writing processing result; the Reduce-side MPI process pulls the spill writing processing result of the Map-side and uses a Reduce task to perform reduction processing on the key-value pairs to obtain a Reduce processing result; the Reduce-side writes the Reduce processing result to the disk. The present invention aggregates the IO requests of multiple MPI processes by means of parallelly writing a large file, reduces a large amount of file reading and writing, and can also avoid the generation of excessive intermediate files, alleviating the pressure on the metadata server. The present invention can be widely applied to the fields of big data processing frameworks and high-performance computing.

Description

Technical Field

[0001] The present invention relates to the fields of big data processing frameworks and high-performance computing, and particularly to a method for improving MapReduce spill writing based on MPI-IO. Background Art

[0002] Big data processing frameworks are an important research content in the field of high-performance computing. They are used to read, write, distribute, and operate on large-capacity data sets, aiming to efficiently complete parallel processing tasks and ultimately obtain valuable data information. Nowadays, with the explosion of data, the system's requirements for data storage and processing capabilities are getting higher and higher. The computing power and storage capacity required to process massive data have long exceeded the upper limit of a single computer. Using big data processing frameworks on clusters to achieve distributed computing and parallel processing of large-scale data has become the mainstream.

[0003] MapReduce is a parallel computing model and method for large-scale data processing. Technically, it can be regarded as a programming model. It abstracts data processing into the concepts of "Map (mapping)" and "Reduce (reduction)", and provides abstract operations and parallel programming interfaces to simply and conveniently complete the programming and computational processing of large-scale data. The Map function maps data into a set of (key-value) key-value pairs, and it performs specified operations on each element of a conceptually list composed of some independent elements. Map operations can be highly parallel, so they are very useful for applications with high-performance requirements and the needs of the parallel computing field. The Reduce function will summarize the values with the same key in all mapped key-value pairs, and it is often the step to obtain the final result. Since the operations of nodes in a large-scale cluster are relatively independent, the Reduce tasks can also be executed in parallel on the framework.

[0004] In a distributed processing framework based on MapReduce, each element is operated independently. When processing data, the original file where the element is located will not be changed, but the mapping processing results will be written into a new temporary file. In large-scale data processing, when the output results of the Map task are too large, it may cause memory overflow. Therefore, under certain conditions, the data in the buffer needs to be written to the disk, and then this buffer can be reused. This process of writing data from memory to disk is called spill writing. When the traditional MapReduce framework performs the spill writing operation, the process will create a new temporary storage file for the mapping processing results of each data block. Due to the use of the POSIX interface and being restricted by the linux VFS, when creating a temporary file, the directory where it is located will be locked, and data files cannot be created in parallel. Secondly, a large number of temporary files will be generated after each Map operation is executed. Operating on these temporary files frequently makes the metadata server overloaded. In addition, a large number of temporary files also bring a large number of small file reads and writes, and the storage bandwidth cannot be fully utilized, which makes the read and write speed of files also become a bottleneck affecting the framework performance. Summary of the Invention

[0005] The purpose of the present invention is to provide a method for improving the spill writing of MapReduce based on MPI-IO, which overcomes the technical defect that the traditional POSIX API interface cannot read and write files in parallel. By aggregating the IO requests of multiple MPI processes through the method of parallel writing to a large file, a large number of file reads and writes are reduced, and at the same time, the generation of too many intermediate files can be avoided, alleviating the pressure on the metadata server.

[0006] The first technical solution adopted by the present invention is: a method for improving the spill writing of MapReduce based on MPI-IO, including the following steps:

[0007] S1. The MPI process at the Map side reads the dataset slice from the target file;

[0008] S2. The MPI process at the Map side runs the Map task, performs mapping processing on the data slice and partitions the mapping processing results to obtain the partitioned key-value pairs;

[0009] S3. When it is judged that the size of the mapping processing results exceeds the memory capacity threshold, the MPI process at the Map side performs the spill writing operation, and spills the partitioned and sorted key-value pairs to the same disk file in parallel to obtain the spill writing processing results;

[0010] S4. The MPI process at the Reduce side pulls the spill writing processing results at the Map side and uses the Reduce task to perform reduction processing on the key-value pairs to obtain the Reduce processing results;

[0011] S5. The Reduce side writes the Reduce processing results to the disk.

[0012] Furthermore, the step where the Map-side MPI process reads the data set slice from the target file specifically includes:

[0013] S11. Query and cache the metadata of the target data set from the distributed file system;

[0014] S12. The MPI root process calculates the total number of data slices after the target data set is sliced according to the size of the target data set and the predefined data slice size;

[0015] S13. The MPI root process broadcasts the data slice size and the total number of slices to other MPI processes in the group;

[0016] S14. Each MPI process calculates the slice number according to the process number and the total number of MPI processes in the group;

[0017] S15. The MPI process calculates the absolute offset of the read data slice in the target file according to the data slice size and the slice number, and obtains the starting address of each data slice;

[0018] S16. Through the MPI-IO interface, the MPI process starts from the starting address of the data slice allocated to it and reads the data slice from the target data set into the cache in parallel and periodically.

[0019] Furthermore, the step where the Map-side MPI process runs the Map task, performs mapping processing on the data slice and partitions the mapping processing result to obtain the partitioned key-value pairs specifically includes:

[0020] S21. The Map-side MPI process reads the data slice line by line, extracts the data and uses it as the key value;

[0021] S22. Perform a Map operation on each key, map it to a key-value pair, and obtain the mapping processing result;

[0022] S23. Perform a hash process on the key-value pair and disperse the key-value pairs with different key values into r different hash partitions, where r represents the number of Reduce-side nodes executing the Reduce task, to obtain the partitioned key-value pairs.

[0023] Furthermore, the step where it is determined that the size of the mapping processing result exceeds the memory capacity threshold, and the Map side performs the spill write operation to spill the partitioned and sorted key-value pairs to the same disk file in parallel to obtain the spill write processing result specifically includes:

[0024] S31. When it is determined that the size of the mapping processing result exceeds the memory capacity threshold, create r spill write files on the disk;

[0025] S32, each MPI process generates r IO requests to write the key-value pairs of r partitions in the mapping processing result to disk;

[0026] S33, MPI-IO aggregates IO requests from different MPI processes that write to the same overflow file;

[0027] S34, based on the MPI-IO middleware, a POSIX API call is initiated to the system, and according to the process number and the write file offset of the MPI process, the key-value pairs belonging to the same hash partition in different MPI processes are written to different areas of the overflow file;

[0028] S35, return to step S33 until all mappings generated by the Map operation have been written;

[0029] S36. Sort the key-value pairs in the overwrite file according to the key values, and reduce the data with the same key value to obtain the overwrite processing result.

[0030] Furthermore, the Reduce-side MPI process pulls the overflow processing result of the Map side and uses the Reduce task to perform reduction processing on the key-value pair to obtain the Reduce processing result, which specifically includes:

[0031] S41, Reduce-side MPI process receives overflow files from each Map-side through MPI_File_read interface;

[0032] S42, the Reduce side merges the received overflow files and sorts all the key-value pairs according to the key value comparison;

[0033] S43. According to the Reduce task requirements, all values ​​with the same key are summarized to obtain the Reduce processing result.

[0034] Furthermore, the Reduce end writes the Reduce processing result to the disk, which specifically includes:

[0035] S51, perform secondary hashing on the Reduce processing results according to the key value and distribute them to different files;

[0036] S52, the MPI process writes the Reduce processing result into the buffer, and returns to step S51 until it is determined that the memory reaches the threshold;

[0037] S53: if it is determined that the key has been written into the file corresponding to the data, the process goes to step S54; if it is determined that the key has not been written into the file corresponding to the data, the process goes to step S55;

[0038] S54, summarize the existing key and the value of the key to be written, and jump to step S56;

[0039] S55. Insert the key-value pairs into the file in the order of the key values;

[0040] S56. Return to step S51 until the Reduce processing result has been written out.

[0041] The beneficial effects of the method and system of the present invention are as follows: The present invention uses MPI-IO to replace the POSIX interface, aggregates the IO requests of multiple MPI processes, and improves the development efficiency through the MPI interface. Using MPI-IO enables multiple processes to write data into the same file in parallel, which directly avoids the generation of a large number of intermediate files and also eliminates the overhead of merging small files. In addition, this method reduces the memory pressure on the metadata server and also avoids the overwriting of the same parallel file system by a large number of processes. In summary, without introducing additional overhead, the present invention improves the spill performance of the MapReduce framework through the parallel read-write interface of MPI-IO and improves the task processing efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is the flowchart of the steps of a method for improving MapReduce spillage based on MPI-IO according to the present invention;

[0043] Figure 2 is the schematic diagram of the MPI process of the present invention for reading a slice of the data set.

[0044] Figure 3 is the schematic diagram of the present invention for executing the Map task

[0045] Figure 4 is the flowchart of the present invention for using MPI-IO to aggregate the IO requests of multiple MPI processes for file spillage.

[0046] Figure 5 is the schematic diagram of the present invention for writing the data of each MPI process to different regions on the spillage file.

[0047] Figure 6 is the schematic diagram of the Reduce-side MPI process of the present invention for pulling data from the Map side.

[0048] Figure 7 is the flowchart of the present invention for writing the Reduce processing result to the disk. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0049] The following further elaborates the present invention in detail with reference to the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of explanation and do not limit the order between the steps at all. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0050] Taking the WordCount task running on the MapReduce framework as an example, referring to Figure 1 , the present invention provides an improved MapReduce spill writing method based on MPI-IO, and the method includes the following steps:

[0051] S1. The MPI process at the Map side reads the dataset slice from the target file;

[0052] S2. The MPI process at the Map side runs the Map task, performs mapping processing on the data slice and partitions the mapping processing result to obtain the key-value pairs after partitioning;

[0053] Specifically, perform the Shuffle operation on the mapping processing result.

[0054] S3. When it is determined that the size of the mapping processing result exceeds the memory capacity threshold, the Map side performs the spill writing operation, and parallelly spills the sorted key-value pairs after partitioning to the same disk file to obtain the spill writing processing result;

[0055] S4. The MPI process at the Reduce side pulls the spill writing processing result at the Map side and uses the Reduce task to perform reduction processing on the key-value pairs to obtain the Reduce processing result;

[0056] Specifically, replace the POSIX interface with MPI-IO, collect the write requests of multiple MPI processes, and parallelly spill the key-value pair data that are assigned to the same partition after Shuffle to the same disk file.

[0057] S5. The Reduce side writes the Reduce processing result to the disk.

[0058] Further as a preferred embodiment of this method, referring to Figure 2 , for the step that the MPI process at the Map side reads the dataset slice from the target file, it specifically includes:

[0059] S11. The MPI process with process number 0 (root process) queries and caches the metadata of the target dataset from the distributed file system;

[0060] S12. The MPI root process calculates the total number of data slices after the target dataset is sliced according to the predefined data slice size spliteSize and the target dataset size fileSize

[0061] S13. The MPI root process broadcasts the data slice size and the total number of slices to other MPI processes in the group;

[0062] S14. Each MPI process calculates the slice numbers {i, i + m, i + 2*m, i + 3*m, …} based on the process number i and the total number m of MPI processes in the group.

[0063] S15. The MPI process calculates the absolute offset offset = i * spliteSize of the read data slice in the target file according to the data slice size and the slice number, and obtains the starting address p of the data read by the MPI process, where p = head + offset and head is the file header.

[0064] S16. Through the MPI-IO interface, the MPI process starts from the starting address p of the data slice allocated to it and reads the data slice into the cache in parallel and periodically from the target data set.

[0065] In this embodiment, the Map-side MPI root process obtains the metadata of the WordCount file to be processed from the distributed file system, divides the WordCount file into several data slices of 64MB in size, and each MPI process reads the corresponding slice according to its own number. By taking the remainder of the slice number and the MPI process, the number of the read data slice is obtained. For example, if there are 10 MPI processes to read data slices, the 1st process will read slices numbered 1, 11, 21,.... After having the slice number, the offset of the read data on the file can be calculated. For example, the 1st slice corresponds to the space with offsets 0 - 64MB, 640 - 704MB,.... Then, using the MPI-IO interface, multiple MPI processes can read the data file in parallel.

[0066] Further as a preferred embodiment of this method, referring to Figure 3 , the step that the Map-side MPI process runs the Map task, performs mapping processing on the data slice and partitions the mapping processing result to obtain the partitioned key-value pairs specifically includes:

[0067] S21. The Map-side MPI process reads the data slice line by line, extracts the data and uses it as the key value.

[0068] S22. Perform a Map operation on each key, map it to a key-value pair, and obtain the mapping processing result.

[0069] S23. Perform a hash processing on the key-value pair and disperse the key-value pairs with different key values into r different hash partitions, where r represents the number of Reduce-side nodes executing the Reduce task, to obtain the partitioned key-value pairs.

[0070] In this embodiment, each MPI process performs mapping processing on the read 64MB data slice. It reads data by row, and then takes each word in the row as a key and maps it to generate key-value pairs. For example, "how are you" will be divided and mapped into (how,1), (are,1), (you,1), and then hash processing is performed on these key-value pairs. Assume that in the MapReduce framework at this time, the number of Reduce ends is 8, and the key-value pairs will be distributed to 8 partitions according to the hash value of the key. Since all MPI processes use the same hashing method, the same keys will ultimately be assigned to the same partition.

[0071] As a further preferred embodiment of this method, referring to Figure 4 , when it is determined that the size of the mapping processing result exceeds the memory capacity threshold, the Map side performs the spill operation, and the key-value pairs after sorting the partitions are spilled in parallel to the same disk file to obtain the spill processing result. This step specifically includes:

[0072] S31. When it is determined that the size of the mapping processing result exceeds the memory capacity threshold, create r spill files on the disk;

[0073] S32. Each MPI process generates r IO requests to write the key-value pairs of r partitions in the mapping processing result to the disk;

[0074] S33. MPI-IO aggregates the IO requests for writing to the same spill file in different MPI processes;

[0075] S34. Based on the MPI-IO middleware, initiate a POSIX API call to the system, and according to the process number of the MPI process and the write file offset, write the key-value pairs belonging to the same hash partition in different MPI processes to different regions of the spill file;

[0076] Specifically, referring to Figure 5 , writing the key-value pairs belonging to the same hash partition in different MPI processes to different regions of the spill file specifically includes:

[0077] S341. Each MPI process generates r IO requests to write the key-value pairs to r files, where r is the number of Reduce ends. Since the hash space is shared, the jth IO request of different MPI processes writes the key-value pairs to be assigned to the jth partition to the file, so they can be stored in one file.

[0078] S342. According to the number m of Map - side MPI processes, with m as a cycle and according to a fixed data - block size, write the data from m processes into spill files. The MPI process with process number i writes the data into the i - th, (i + m)-th, (i + 2*m)-th... blocks respectively. Assuming the data - block size is blockSize, the process with process number i writes the data into the file space with offsets:

[0079] [(i - 1)*blockSize, i*blockSize], [(i - 1 + m)*blockSize, (i + m)*blockSize], [(i - 1 + 2*m)*blockSize, (i + 2*m)*blockSize]...

[0080] S343. Through MPI - IO, aggregate multiple requests from different MPI processes writing to the same file into a large request, and according to the process number and the partitioning method in S342, write the data from different MPI processes into different regions of the file in parallel.

[0081] S35. Return to step S33 until all the mappings generated by the Map operation have been written;

[0082] S36. Sort the key - value pairs in the spill file according to the keys, and reduce the data with the same key values to obtain the spill - processing result.

[0083] This embodiment is a supplement to the previous example. When using the MPI - IO interface for parallel writing of files, each MPI process sends out the file header, absolute offset, data size, data type, and process status written. These are uniformly managed by the MPI - IO interface. The intervals where the key - value pairs of each process are written can be abstracted as blocks, and the spill file is composed of blocks. Assuming the block size is 64MB, when data is written to the spill file, if the written interval is full, 10 * 64M of space will be newly expanded. The interval with offset 0 - 64MB belongs to process 1, the interval with offset 65 - 128MB belongs to process 2... The interval with offset 577 - 640MB belongs to process 10. Each process writes the data into the corresponding interval, and the position of this interval is calculated through the file header and the absolute offset.

[0084] Further as a preferred embodiment of this method, referring to Figure 6 , the step that the Reduce - side MPI process pulls the spill - processing result of the Map - side and uses the Reduce task to perform reduction processing on the key - value pairs to obtain the Reduce - processing result specifically includes:

[0085] S41. The Reduce - side MPI process receives the spill files from each Map - side through the MPI_File_read interface;

[0086] Specifically, the Map-side MPI process uses the MPI_File_write interface to send the spill files in blocking mode to ensure that the data information and data envelopes have been safely saved. The Reduce-side MPI process uses the MPI_File_read interface to perform inter-process data transfer on the premise that the receive buffer is larger than the sent data, receiving the spill files from each Map-side. The k-th Reduce-side receives the data of the k-th partition from each Map-side.

[0087] S42. The Reduce-side merges the received spill files and sorts all key-value pairs according to key-value comparison. When two keys are the same, their values are placed in a value iterator.

[0088] S43. According to the requirements of the Reduce task, all values with the same key are summarized to obtain the Reduce processing result.

[0089] In this example, the Reduce-side obtains data from the Map-side through the MPI interface. Each Reduce-side receives the key-value pairs of the corresponding partition. For example, the 1st Reduce-side receives the 1st spill files from each Map-side, the 2nd Reduce-side receives the 2nd spill files from each Map-side... The received key-value pairs consist of words and numerical values. After receiving all the key-value pairs, they are sorted according to the key value, and then the values with the same key are summarized. For example, if there are 300 key-value pairs with the word "you" as the key value (you, 1), then these 300 key-value pairs are summarized to obtain (you, 300).

[0090] Further as a preferred embodiment of this method, referring to Figure 7 , the step of the Reduce-side writing the Reduce processing result to the disk specifically includes:

[0091] S51. Perform secondary hashing on the Reduce processing result according to the key value and distribute it to different files;

[0092] S52. The MPI process writes the Reduce processing result to the buffer and returns to step S51 until it is determined that the memory reaches the threshold;

[0093] S53. If it is determined that the key has been written to the corresponding file, jump to step S54; if it is determined that the key has not been written to the corresponding file, jump to step S55;

[0094] S54. Summarize the existing key and the value of the key to be written, and jump to step S56;

[0095] S55. Insert the key-value pairs into the file in the order of the key values;

[0096] S56. Return to step S51 until the Reduce processing result has been completely written.

[0097] In this example, the Reduce side distributes the induction results such as (you, 300) to different files according to the hash of the key values. The induction results are written into the cache. Since the memory space is limited, it is possible that all Key values cannot be placed in the memory at the same time. In this case, the spilled write operation needs to be performed on the key-value pairs after induction. Determine whether the key has been written into the target file to be written. If it has not been written, insert it in the sorted order. If it has been written, for example, there is already a (you, 500) in the disk file, then the two are merged to generate a new key-value pair (you, 800).

[0098] The above is a specific description of the preferred embodiment of the present invention. However, the present invention is not limited to the described embodiments. Those skilled in the art can make various equivalent deformations or substitutions without departing from the spirit of the present invention. These equivalent deformations or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A method for improving MapReduce spill writing based on MPI-IO, characterized in that, The following steps are involved: S1, Map-side MPI process reads data set slices from the target file; S2, the Map-side MPI process runs the Map task, maps the data slices and partitions the mapping results to obtain the partitioned key-value pairs; S3: When it is determined that the size of the mapping processing result exceeds the memory capacity threshold, the Map end performs an overflow operation to overflow the key-value pairs sorted by the partitions in parallel to the same disk file to obtain the overflow processing result; S4, the MPI process on the Reduce side pulls the overflow processing results on the Map side and uses the Reduce task to reduce the key-value pairs to obtain the Reduce processing results; S5, the Reduce side writes the Reduce processing results to disk; The step of determining that the size of the mapping processing result exceeds the memory capacity threshold, the Map end performs an overflow operation, and overflows the key-value pairs sorted by the partitions in parallel to the same disk file to obtain the overflow processing result, specifically includes: S31, determining that the size of the mapping processing result exceeds the memory capacity threshold, and creating r overflow files on the disk; S32, each MPI process generates r IO requests to write the key-value pairs of r partitions in the mapping processing result to disk; S33, MPI-IO aggregates IO requests from different MPI processes that write to the same overflow file; S34, based on the MPI-IO middleware, a POSIX API call is initiated to the system, and according to the process number of the MPI process and the write file offset, the key-value pairs belonging to the same hash partition in different MPI processes are written to different areas of the overflow file; S35, return to step S33 until all mappings generated by the Map operation have been written; S36, sorting the key-value pairs in the overwrite file according to the key values, and reducing the data with the same key value to obtain the overwrite processing result; When using the MPI-IO interface for parallel writing of files, each MPI process sends out the written file header, absolute offset, data size, data type, and process status, which are uniformly managed by the MPI-IO interface.

2. The method for improving MapReduce spill writing based on MPI-IO according to claim 1, wherein The step of the Map-side MPI process reading a data set slice from a target file specifically includes: S11, query and cache the metadata of the target data set from the distributed file system; S12, the MPI root process calculates the total number of data slices after the target data set is divided according to the target data set size and the predefined data slice size; S13, the MPI root process broadcasts the data slice size and the total number of slices to other MPI processes in the group; S14, each MPI process calculates the slice number according to the process number and the total number of MPI processes in the group; S15, the MPI process calculates the absolute offset of the read data slice in the target file according to the data slice size and the slice number, and obtains the first address of each data slice; S16. Through the MPI-IO interface, the MPI processes start from the first addresses of the data slices they have obtained, and read the data slices from the target data set in parallel and periodically into the cache.

3. The method for improving MapReduce spill writing based on MPI-IO according to claim 2, wherein, The Map-side MPI process runs the Map task, performs mapping processing on data slices, and partitions the mapping processing results to obtain key-value pairs after partitioning. This step specifically includes: S21. The Map-side MPI process reads the data slice line by line, extracts the data, and uses it as the key value; S22. Perform a Map operation on each key, map it to a key-value pair, and obtain the mapping processing result; S23. Perform a hash process on the key-value pairs and disperse the key-value pairs with different key values into r different hash partitions, where r represents the number of Reduce-side nodes executing the Reduce task, to obtain the key-value pairs after partitioning.

4. The method for improving MapReduce spill writing based on MPI-IO according to claim 1, characterized in that The step where the Reduce-side MPI process pulls the spill processing result of the Map side and performs a reduction process on the key-value pairs using the Reduce task to obtain the Reduce processing result specifically includes: S41. The Reduce-side MPI process receives the spill files from each Map side through the MPI_File_read interface; S42. The Reduce side merges the received spill files and sorts all the key-value pairs according to key comparison; S43. According to the requirements of the Reduce task, summarize all the values with the same key to obtain the Reduce processing result.

5. The method for improving MapReduce spill writing based on MPI-IO according to claim 4, wherein The step where the Reduce side writes the Reduce processing result to disk specifically includes: S51. Perform a secondary hash on the Reduce processing result according to the key value and distribute it to different files; S52. The MPI process writes the Reduce processing result to the buffer, and returns to step S51 until it is determined that the memory reaches the threshold; S53. If it is determined that the key has been written to the corresponding file, jump to step S54; if it is determined that the key has not been written to the corresponding file, jump to step S55; S54. Summarize the value of the existing key and the key to be written, and jump to step S56; S55. Insert the key-value pairs into the file in the order of the key value; S56. Return to step S51 until the Reduce processing result has been written completely.

Images