A fast data reading method, device, electronic equipment and storage medium

Abstract

The invention discloses a fast data reading method, device, electronic equipment and storage medium. The method includes: dividing a data set into several picture subsets, performing normalization processing on each picture subset, and aggregating them respectively It is a file; assign a request number to each picture in the data set; by inheriting the abstract class of the data set of the deep learning platform, according to the request number, the file address where each picture is located and the file address described in the file are calculated by hashing. Offset information, so as to obtain the mapping of each picture to the file to which the picture belongs; according to the mapping, quickly read all the pictures in the data set. Aggregating each subset of the pictures into one file reduces the overhead of metadata management for a large number of small samples, optimizes the organizational form of the data set, and greatly improves the reading speed of pictures; when reading pictures, it uses multi-level addresses Mapping replaces the original inefficient random search process in massive images, greatly improving the reading rate.

Description

technical field [0001] The present invention relates to the field of deep learning, in particular to a fast data reading method, device, electronic equipment and storage medium. Background technique [0002] As a method to automatically describe objects, trends and anomalies, deep learning has been widely used in scientific and commercial fields. The specific process of deep learning is: 1. Set the loss function and initialize the model parameters. 2. Randomly read a certain batch of data from the selected training data set, input the model, perform forward propagation, and calculate the loss value. 3. Then use the backpropagation method to pass the corresponding loss value in the opposite direction layer by layer, and calculate the parameter error of each parameter. The model parameters are then updated using the model parameter update optimization method. 4. Repeat steps 2 and 3 until the loss value drops to an acceptable value and the model converges. [0003] During ...

AI Technical Summary

Problems solved by technology

[0004] The purpose of the embodiments of the present invention is to provide a fast data reading method, device, electronic equipment and storage medium to solve the problem of slow data set reading in related technologies

Images