A file-level striping system suitable for high concurrent access to large data

Abstract

The invention discloses a file level partitioning system suitable for big data high concurrence access for handling big data application challenges and existing problems in aspects such as high concurrence access of file data in a current parallel file system. In respect of module constitutions, the file level partitioning system mainly comprises a system initial parameter acquisition module, a file feature reading module, a single-file optimal partitioning width computing module, a whole-file partitioning size computing module and a file level partitioning execution module. According to access characters, such as request sizes and file concurrence access degrees, of files, partitioning sizes of the files are automatically set up, file level partitioning support is provided for a high-performance computer and suitable for big data and high concurrence access characters, and then powerful support is provided for efficient concurrent input / output. Therefore, the file level partitioning system suitable for big data high concurrence access has the advantages of having broad application prospects and being capable of bringing prominent economical benefits.

Description

technical field [0001] The invention discloses a file-level striping system suitable for high concurrent access of large data, in particular to a file-level striping system suitable for high concurrent access of large data in a parallel file system. It belongs to the field of computer technology. Background technique [0002] In recent years, big data has become a research hotspot in academia and industry. On the one hand, the scale and application of the Internet, mobile Internet, Internet of Things, and cloud computing have surged, and the interaction between a large number of users and applications has resulted in the generation of huge amounts of data; On the other hand, with the advancement of data acquisition technology, such as satellite remote sensing, sensors, GPS, etc., it also leads to the generation of huge amounts of data every moment; finally, in the fields of scientific research and industry, complex new technologies, new The use of instruments has also led t...

AI Technical Summary

Problems solved by technology

In addition, existing parallel file systems (such as PVFS, etc.) can provide subsystems supporting directory-level striping and system-level striping, but there is still a lack of systems that can support file-level striping

For the parallel file system used by high-performance computers suitable for big data applications, because it needs to face a large number of I / O requests of scientific computing programs with burst characteristics, it also has to face higher file access parallelism. It can be seen that the existing research generally still lacks a file-level striping method and system that can be well applied to high-concurrent data access in high-performance computing, and especially lacks a method and system that can be based on each file's own access characteristics and The load condition of the entire file system to determine the stripe system of the file stripe width, thus providing guarantee for efficient parallel data access

Images