Massive small file storage performance optimization method and system based on time series prediction

Abstract

The invention discloses a massive small file storage performance optimization method based on time sequence prediction, and belongs to the field of information storage. The massive small file storageperformance optimization method comprises the following steps: collecting historical file access records with time information to obtain a data set; after the data set is preprocessed into discrete time series data, using a time window to rolling on the data set to generate a training data set, wherein the training data at any t moment takes the data at the t-n-t moment as input data, and takes the data at the t + 1 moment as label data; establishing a time sequence prediction model based on a recurrent neural network, and performing training, verification and testing in sequence by utilizinga training set, a verification set and a test set obtained by dividing the training data set, thereby obtaining a target model; predicting the change trend of the file size by utilizing the target model so as to identify a large file and a small file in the file size; and directly storing the large files, and aggregating and storing the small files based on a time sequence. According to the massive small file storage performance optimization method, the storage performance of massive small files in the distributed storage system can be optimized.

Description

technical field [0001] The invention belongs to the field of information storage, and more specifically relates to a method and system for optimizing storage performance of massive small files based on time series prediction. Background technique [0002] In the context of the rapid development of Internet information technology, data has shown exponential growth. Faced with the ever-growing web applications and mobile applications, as well as the massive small files generated during the user's use. Distributed storage systems basically have performance problems when storing a large number of small files. This type of problem is called the small file problem. There are challenges in a series of issues such as performance. When faced with the problem of massive small files, disk I / O efficiency is low. Traditional disks are more suitable for sequential large file I / O access patterns, but the performance of random small file I / O reads and writes is not good. The time consum...

AI Technical Summary

Problems solved by technology

However, the above methods did not consider the access law of load changing over time when designing the aggregation strategy, that is, ignoring the timing characteristics of the load, making the storage performance (writing and reading performance) of massive small files in the distributed storage system restricted

Images