Data access method and device, equipment and storage medium

Abstract

The invention discloses a data access method and device, equipment and a storage medium, and the method comprises the steps: obtaining a training data set for training a target deep learning model, determining a target index file according to the training data set, and carrying out random access of each piece of training data in the training data set based on an index. on this basis, converting operations such as random sorting on training data into operations on index information, namely, in the process of training the target deep learning model in each round, randomly upsetting the sequence of the index information in the target index file to acquire a target index file in which the index information is arranged according to a second sequence and acquire target index file according to the second sequence; and reading target training data corresponding to the target index information from the training data set, and inputting the target training data into the target deep learning model for the round of training. Therefore, only the index information with small volume and a small amount of training data required by each step in training need to be stored in the memory, the use of the memory is remarkably reduced, and the high efficiency of reading the training data can be ensured.

Description

technical field [0001] The present application relates to the field of data processing, in particular to a data access method, device, device and storage medium. Background technique [0002] In recent years, deep learning technology has made great progress, and has achieved performance close to or even surpassing that of humans in many fields such as natural language processing, computer vision, and speech recognition. A deep learning model with high training accuracy and good effect is inseparable from massive training data. [0003] Currently, deep learning models are mainly trained based on the stochastic gradient descent algorithm. In most model training scenarios, it is necessary to first read the entire training data set into the memory, and then randomly shuffle the training data in the training data set before each round of training, and read sequentially in the shuffled training data set Feed a small portion of the training data into the model. [0004] However,...

AI Technical Summary

Problems solved by technology

[0004] However, as the scale of data used for model training becomes larger and larger, this approach gradually becomes unfeasible, because the training data of tens or even hundreds of GB may not be fully loaded into the memory

In addition, only a small part of the entire training data set is actually used in each step of model training, which makes the effective utilization of memory very low and wastes a lot of memory resources

Images