Internal data sorting method and device

Abstract

The embodiment of the invention discloses an internal data sorting method and device. By invoking a mmap function, a mapping relation between a file to be processed and a physical memory is built, and a pointer variable is generated; according to the pointer variable, internal sorting of the file to be processed is completed. Through a memory mapping file technique, a big file is mapped into a piece of memory, and therefore internal sorting of data inside the big file is achieved. Additionally, since read-write I / O operation can no longer be executed on the file when a memory mapping file is used for processing files saved in a hard disk, the course can access the files just like accessing the internal memory, so that the file reading and writing frequency is greatly reduced, the program execution time is extremely shortened and the program execution efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a method and device for sorting within data. Background technique [0002] When sorting the data in the file, if the file is small, all the data in the file can be read into the memory, and it can be easily realized by internal sorting methods such as bubble sorting and quick sorting. But if the file size exceeds the memory size, because the memory resources are limited, and the files to be sorted are very large (for example, a 100G file has only 1G of memory), the files to be sorted cannot be loaded into the memory at one time, and only internal sorting is used. Goal achieved. [0003] At present, the existing general solution is to use external sorting, divide the file according to the available memory size, divide the large file into several small files, so that all the divided small files can be loaded into the memory, and then internally sort the small files respect...

AI Technical Summary

Problems solved by technology

When loading a small file into the memory, it first copies the data in the small file from the hard disk to a buffer in the kernel space, and then copies the data to the user space. In fact, two data copies are performed, which reduces the data size. loading efficiency

And for multiple small files, using the external sorting method requires a large number of data exchanges between the internal memory and the external storage, which will consume a long time

Images