Data-driven parallel sorting system and method

Abstract

A data driven parallel sorting method includes distributing input data records to n partitions one by one in a circular manner. Each partition corresponds to a parallel sorting process with an allocated memory chunk sized to store m data records. The method also includes sorting, in parallel, current data records in respective memory chunks in respective partitions. The method also includes in response to distribution of data records of └m / n┘ rounds, circularly controlling one of the n partitions, and writing data records that have been sorted in the memory chunk of the partition into a mass storage as an ordered data chunk, and emptying the memory chunk. The method also includes in response to all data records being distributed, writing data chunks that have been sorted in respective memory chunks into the mass storage, and performing a merge sort on all ordered data chunks in the mass storage.

Description

technical field [0001] The present invention relates to the field of parallel computing, more specifically, the present invention relates to a data-driven parallel sorting system and method. Background technique [0002] The parallel sorting algorithm is an algorithm proposed in order to improve the sorting efficiency after the computer's parallel computing capability has been greatly developed. Parallel sorting is a technique widely used in databases, data extraction, transformation and loading (Extraction-Transformation-Loading, ETL) and other fields. Parallel sorting algorithm is a very typical application of Divide and Conquer. Its principle is to divide the sequence to be sorted into several subsequences, make each subsequence in order, and then merge the ordered subsequences. get a completely ordered sequence. [0003] In parallel sorting, data is distributed into multiple partitions, each partition corresponding to a sorting process, such as a process or thread. Fo...

AI Technical Summary

Problems solved by technology

That is to say, when each partition sorts all the data records in the memory block based on the driver newly allocated to its own data records, each partition will compete for CPU resources at the same time, which causes image 3 The appearance of the peak in

In addition, due to the above reasons, when the sorting in each partition is completed, in order to write the ordered data blocks to the hard disk, each partition will also compete for input and output (IO) resources at the same time, which will also cause waiting and delay.

This makes the use of system resources inefficient

[0008] It has been found that, in general, the system inefficiency of parallel sorting is less pronounced when the size of the memory block is small

Reducing the size of the memory block alleviates the above problems, but has another significant drawback

When the amount of data is very large, small memory blocks are not beneficial, because this will lead to too many data blocks on the hard disk, making the merge sort process to merge all these data blocks together. makes the whole sorting process very slow

Therefore, reducing the size of the memory block does not solve the above problem

Images