Mass small file distributed caching method oriented to AI (Artificial Intelligence) training

Abstract

The invention discloses an AI (artificial intelligence) training-oriented distributed caching method for massive small files, which is used for realizing high-performance distributed caching of the massive small files. The method comprises the following steps: firstly, combining small files into chunk according to a rule of fitting Batch Size features in AI training; secondly, analyzing the cache state of chunk, and carrying out double-layer shuffle operation on the small file sequence; and finally, during AI training and data reading, adopting Local Cache short-circuit reading for repeated I / O, and starting asynchronous grouping pre-reading at the Local Cache short-circuit reading time. According to the method, the problem of random reading of massive small files oriented to AI training is solved by efficiently utilizing the cache, and under the scene oriented to AI training, the data access rate and the cache hit rate are remarkably improved, and the iteration time of AI training is shortened.

Description

Technical field [0001] The present invention is directed to smart city, for face recognition, video search, intelligent storage and other scenes, the massive design of small files distributed caching oriented AI training methods to achieve high performance distributed cache massive small files. Background technique [0002] In recent years, with the rapid development of the global socio-economic and science and technology, and large-scale application of AI technology in the landing field of security, strongly boosting the peace smart city development. At the same time, the wisdom of peace in the city, such as face recognition, tracking and other cross-border pedestrian scene of AI technology challenges. AI is usually the number of documents required training mission in millions or even tens of millions of the order, such as Google OpenImages dataset contains 9,000,000 images, Tencent ML-Images dataset contains nearly 1,769 ten thousand images. Usually requires massive scale of te...

AI Technical Summary

Problems solved by technology

The system proposes a scheme to merge massive small files into data blocks for storage, and proposes intra-group shuffle to replace random access for AI training. However, the system still needs to wait for a long time due to cache misses when AI tasks access data blocks for the first time. Disk I / O time affects data access rate

Images