The present invention discloses a
data processing method and device based on a large-scale storage
system, and relates to the field of computer storage technology. The
data processing method based on a large-scale storage
system includes dividing the persistence layer of the large-scale storage
system into a pre-write log module, an index memory cache module and an index persistence module, then writing data to the persistence layer through a write process, and then reading user data from the persistence layer through a read process. In a large-scale storage system based on a write-ahead log, the present invention optimizes the data reading and writing process of the persistence layer, improves the data double-writing problem caused by WAL, has less memory requirements, and does not increase unnecessary write delays due to the independent storage of data and index
metadata; there is no need to significantly modify the WAL design, and for large I / O, only the index can be recorded, while for small I / O, user data can also be recorded in the index at the same time, reducing one data redirection, and supporting the optimization of I / O of different sizes.