Remote caching device and method for sample loading in model distributed training

Abstract

The invention discloses a far-end caching device and method for sample loading in model distributed training, relates to the technical field of deep learning, and solves the problem that a traditional cache cannot play a role due to failure of a locality principle in a completely random access mode. The method comprises the steps that a client sends a sample request; the server side responds to the sample request; wherein the server side comprises a storage pool used for storing a training sample; the buffer pool is used for outputting training samples according to the sample request and storing the training samples randomly extracted from the storage pool; according to the method, the time delay of training sample loading in the model training process can be reduced, and the memory cache utilization rate of the storage nodes in the model training scene is improved.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a remote caching device and method for sample loading in model distributed training. Background Art

[0002] Deep learning model training often uses various optimizers based on stochastic gradient descent (SGD). Its working paradigm is that in each round of training, small batches of samples are randomly sampled from the entire data set without replacement to train the model once. Only a small batch of samples is used for training in one iteration. On the one hand, it is to speed up the training speed. On the other hand, it is also limited by the capacity of video memory, which cannot support the simultaneous network inference and parameter update by placing a large number of samples in the video memory.

[0003] When the total amount of data in the data set is large, the host's main memory also faces the problem of not being able to accommodate all the data, which is very common in the training scenario of large models. Therefore, the life cycle of the data required in one round of iteration is: the program points out a batch of training samples to be read through random sampling, and then the data is read from the external memory to the main memory, the main memory is transferred to the video memory, and finally the GPU uses the data on the video memory for training. The external memory here can be the local external memory. When the local external memory cannot accommodate a large-scale data set, it can also be a network storage service, such as distributed file storage.

[0004] Before the GPU starts computing, it needs to wait for a series of data IOs. It can only compute after the data arrives at the video memory, which leads to a serious waste of computing resources. Therefore, in order to make full use of computing resources, it is necessary to make computing and data transmission in parallel. While the training calculation of the current batch of data is being performed, the next batch of data is loaded into the main memory. The two processes overlap, so that the calculation of the next batch can start faster. At the same time, the training sample access is required to have low latency, so that there can be a shorter data IO time, ensuring that the computing time covers the data IO time as much as possible, and fully utilizing computing resources.

[0005] Existing deep learning frameworks such as PyTorch and Tensorflow all provide data set loading functions and integrate data pre-fetching mechanisms that are asynchronous with computing, pre-fetching data to the local main memory to achieve overlap between computing and data IO. However, deep learning frameworks are designed for generality and are not optimized for diverse underlying storage. When faced with large-scale data sets, distributed storage is required, which results in a longer IO path, including at least the transmission from remote persistent storage to remote main memory, and the network transmission from storage nodes to local machines.

[0006] The high access latency caused by the long IO path makes it impossible for the strategy of pre-fetching data to the local main memory to cover up the data IO with calculation. Therefore, in order to meet the low latency requirement, the cache should be used to pipeline the several transmission processes from remote persistent storage to local main memory so that they can be performed asynchronously. Traditional caches are designed based on the principle of locality, including temporal locality: if a data item is accessed, it is likely that this data item will be accessed again in the near future; and spatial locality: if a data item is accessed, then the data items adjacent to its address are also likely to be accessed in the near future. Based on the principle of locality, when traditional cache systems read data, they read the adjacent data together and keep them in the cache until they are swapped out.

[0007] However, the data loading process of deep learning training tasks does not have such locality. Once a data item is accessed, it will not be accessed again until all data are accessed once. There is no temporal locality; the access order of data items is completely random, and there is no spatial locality. Summary of the invention

[0008] In order to solve the problems existing in the above-mentioned prior art, the present invention provides a remote cache device and method for sample loading in model distributed training, which solves the problem that the traditional cache cannot function due to the failure of the locality principle in a completely random data access mode.

[0009] A remote cache device for sample loading in model distributed training, comprising:

[0010] Client, sends batch sample request;

[0011] The server responds to sample requests;

[0012] The above-mentioned server includes:

[0013] Storage pool, used to store training samples;

[0014] The buffer pool is used to output training samples according to sample requests and save training samples randomly selected from the storage pool.

[0015] Furthermore, there are multiple servers to ensure the amount of stored data.

[0016] Furthermore, the client includes an access interface for providing the user with a function of obtaining batch random training samples.

[0017] Furthermore, the client access interface is implemented by using a server random selection strategy to determine which server provides each sample, counting the number of training samples requested from each server, and sending a sample request to the server.

[0018] Furthermore, the server-side random selection strategy includes:

[0019] If extraction with replacement is required, the ratio of the number of training samples of each server to the total number of training samples is the selection probability of the server, and the selection probability of the i-th server is p i =cnt i / ∑ k cnt k , 1≤k≤N, N is the number of storage nodes, cnt i is the number of training samples on the i-th server;

[0020] If sampling without replacement is required, the ratio of the number of samples that have not been requested by each server to the total number of samples that have not been requested is the selection probability of the server, and the selection probability of the i-th server is p i =left i / ∑ k left k , 1≤k≤N, N is the number of storage nodes, left i is the number of samples that have not been requested on the i-th server.

[0021] Furthermore, the server includes a sample selection module, and the sample selection module is used to randomly extract training samples from the storage pool to the buffer pool based on an in-situ random sample selection strategy.

[0022] Furthermore, the on-site random sample selection strategy includes:

[0023] Use the no-replacement strategy to randomly select unaccessed training samples from the storage pool and put them into the buffer pool;

[0024] Using the replacement strategy, training samples are randomly selected from all training samples in the storage pool and put into the buffer pool.

[0025] Furthermore, the buffer pool is designed according to a producer-consumer model, and the buffer pool returns training samples according to the number of samples requested by the client, and the server synchronously or asynchronously randomly selects samples from the storage pool based on an in-situ random sample selection strategy to fill the buffer pool.

[0026] Furthermore, the client sets a random seed and sends it to the server. The server performs a random selection strategy and an on-site random sample selection strategy based on the set random seed. When retrieving data, the client uses the random seed to generate a random number sequence to select which server to obtain the training sample from; the server uses seed+id as the random seed to generate the file access order in the storage pool, and id is the number of each server, which is globally unique.

[0027] A remote caching method for sample loading in model distributed training is applicable to a remote caching device for sample loading in model distributed training, comprising:

[0028] Send batch sample requests to the server via the above client;

[0029] Output the training samples to the client through the buffer pool of the server;

[0030] Randomly extract training samples from the above storage pool to the buffer pool.

[0031] Furthermore, when the buffer pool of the server outputs training samples to the client and causes vacancies in the buffer pool, the sample selection module of the server randomly extracts training samples from the storage pool to the buffer pool in a synchronous or asynchronous manner based on the local random sample selection strategy.

[0032] The beneficial effects of the present invention include:

[0033] The present invention designs a remote cache design for sample loading in model distributed training, which transfers the training sample out-of-order operation of the computing node to the storage node, so that the storage node can determine the future training sample access order without the intervention of the computing node, making the cache of the storage node effective. Specifically:

[0034] 1. Compared with the general-purpose cache design, the present invention can improve the memory cache utilization of storage nodes in the model training scenario, reduce the delay of training sample loading during model training, and improve the utilization of computing resources;

[0035] 2. Compared with the disordered training samples in the computing nodes, the computing nodes do not need to cache metadata information of a large number of samples in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 This is a first schematic diagram of a remote cache device for loading samples in a distributed model training involved in an embodiment of the present application.

[0037] Figure 2 This is a second schematic diagram of a remote cache device for loading samples in a distributed model training involved in an embodiment of the present application.

[0038] Figure 3 This is a third schematic diagram of a remote cache device for loading samples in a distributed model training involved in an embodiment of the present application.

[0039] Figure 4 This is a fourth schematic diagram of a remote cache device for loading samples in a distributed model training involved in an embodiment of the present application.

[0040] Figure 5This is a schematic diagram of the server operation involved in the embodiment of the present application.

[0041] Figure 6 This is a schematic diagram of the client operation involved in the embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0043] A remote cache device for sample loading in distributed model training, such as Figure 1 As shown, including:

[0044] Client 100, sends a batch sample request;

[0045] Server 200, responds to sample request;

[0046] The server 200 includes:

[0047] Storage pool 201, used to store training samples;

[0048] The buffer pool 202 is used to output training samples according to sample requests and store training samples randomly extracted from the storage pool 201 .

[0049] The server 200 is located at a storage node on a remote storage side, and the client 100 is located at a computing side on a user's local computer.

[0050] The client 100 module is responsible for applying to multiple storage nodes for batch random training samples, and the server 200 is responsible for maintaining the buffer pool 202 and responding to sample requests from the client 100 .

[0051] At the beginning of the deep learning training task, the application 400 on the computing node for implementing the training task sends an initialization instruction to the cache device. After receiving the initialization instruction, the client 100 sends an initialization request to the server 200. The initialization request includes training task information, such as the sample set contained in the training data set (which can be the directory name where the data set is located). After receiving the initialization request, the server 200 initializes the buffer pool 202 for the training task: including allocating memory; initializing the metadata list: according to the training sample set information given by the client 100, extracting the sample metadata contained in this storage node (which can be any data that can be indexed to the content of the training sample, such as the size and offset of the sample data, and the sample id); randomly extracting a batch of training samples from the storage pool 201 to the buffer pool 202, and then the server 200 sends a response message of initialization completion to the client 100, informing the client 100 of the number of training samples on the storage node.

[0052] In another embodiment, if Figure 3 As shown, there are multiple servers 200, specifically a first server 210, a second server 220, a third server 230, etc., to ensure the amount of stored data while having a higher fault tolerance rate, thereby ensuring smooth data calls.

[0053] In another embodiment, the server 200 further includes a sample selection module 203 for implementing a local random sample selection strategy, such as Figure 2 shown.

[0054] In another embodiment, if Figure 4 As shown, the client 100 includes an access interface 101, which is used to provide the computing program with the function of obtaining batch random training samples. The implementation method of the access interface 101 is: using the server random selection strategy to determine which server provides each random sample, counting the number of training samples requested from each server, and sending a sample request to the server 200. Specifically: Taking the in-place random file selection strategy using the no-replacement strategy as an example, the client 100 module maintains a sample quantity vector cnt of each storage node and a vector left of the number of unaccessed samples of each storage node, which is used to record the number of remaining unaccessed samples of each storage node in the current round of training sample access cycle, where the sample access cycle is a round of training, and all training samples in the data set need to be traversed in this cycle.

[0055] The server 200 records the number of training files c contained in the storage node for each training task, and maintains a memory buffer queue, namely a buffer pool 202, a metadata list, and a remaining sample count l. The memory buffer queue is used to store sample data to be accessed in the future; the metadata list is used to record the location information of the training samples in the data set; and the remaining file count l is used to record the number of samples that have not been cached in the current sample access cycle.

[0056] When the client 100 requests a batch of random training samples, the ratio of the number of samples that have not been requested by each server to the total number of samples that have not been requested is used as the server selection probability to determine which server each random sample comes from. The probability of selecting the i-th storage node server is left i / ∑ k left k , 1≤k≤N, N is the number of storage nodes. Then count the number of training samples requested to each server, and send multiple sample requests {request(i,n i )},request(i,n i ) indicates that n is requested from the i-th storage node i training samples.

[0057] When the i-th storage node receives the i When a random sample is requested, the server 200 on the storage node takes n from the head of the buffer queue. i The training samples are sent to the client 100 as a response, and random training samples are added to the end of the buffer queue, and the remaining sample count l is updated to ln i .

[0058] When the remaining sample count l of the server 200 is 0, l is reset to the sample number c; when all bits of the unvisited sample number vector left recorded by the client 100 are 0, if the training is not completed, left is reassigned to cnt, indicating the beginning of a new round of training sample access cycle.

[0059] In another embodiment, the in-situ random sample selection strategy adopts a replacement strategy;

[0060] The client 100 module maintains a file quantity vector cnt for each storage node;

[0061] The server 200 module records the number c of training samples contained in the storage node for each training task, and maintains a memory buffer queue, namely a buffer pool 202 and a metadata list.

[0062] The memory buffer queue is used to store sample data that needs to be accessed in the future; the metadata list is used to store the location information of the training samples in the data set.

[0063] When the deep learning training task starts, the client 100 module notifies the server 200 module of each storage node and provides the sample set contained in the training data set (which can be the directory name where the data set is located).

[0064] The server 200 module of each storage node initializes the memory buffer queue for the training task: including allocating memory; initializing the metadata list: extracting the sample metadata contained in this storage node (which can be any data that can be indexed into the sample content, such as sample data size and offset, sample id) according to the training sample set information provided by the client 100; recording the sample quantity c as the number of data set samples contained in this storage node.

[0065] When the client 100 requests a batch of random training samples, a storage node is randomly selected for each sample in the batch, with the ratio of the number of samples of each node to the total number of samples as probability. The probability of selecting the i-th storage node is cnt i / ∑ k cnt k , 1≤k≤N, N is the number of storage nodes. Then count the number of training samples requested to each server, and send multiple sample requests {request(i,n i )},request(i,n i ) indicates that n is requested from the i-th storage node i training samples.

[0066] When the i-th storage node receives the i When a random sample is requested, the server 200 on the storage node takes n from the head of the buffer queue. i The training samples are sent to the client 100 as a response, and random training samples are added to the tail of the buffer queue.

[0067] In another embodiment, if Figure 5 As shown, the buffer pool 202 is a lock-free circular buffer queue, which avoids lock contention and ensures that the memory location occupied by the buffer pool 202 is fixed, thereby further improving the efficiency of data processing. Specifically, at the current moment, a batch of training samples are taken from the head of the circular buffer queue as the buffer pool 202 according to the number of samples requested by the client 100 to the server 200 and transmitted to the client 100, and at the same time, a batch of training samples are randomly selected from the persistent storage medium that stores a large number of training samples as the storage pool 201 asynchronously and placed at the end of the circular buffer queue.

[0068] In another embodiment, a preferred method for selecting batch random training samples is: generate a random integer r from 1 to 1, exchange the rth and lth metadata in the metadata list, and the file corresponding to the lth metadata in the metadata list after the exchange is the selected random training sample. Repeat the above process until the specified number of samples are taken.

[0069] In another embodiment, in order to make the random access order of files on the storage system repeatable (reproducible), the client 100 module sets a random seed seed when initializing the training task and broadcasts it to each storage node. When fetching data, the client 100 uses the random seed seed to generate a random number sequence to determine which storage node each random sample comes from; the server 200 uses seed+id as a random seed to generate the sample access order within the storage node, and id is the number of each storage node, which is globally unique.

[0070] In another embodiment, the access interface 101 is a sample(n) function, where n is a parameter indicating the number of random samples to be obtained. Figure 6 shown.

[0071] The local application 400 initiates a data call instruction. After receiving the call instruction, the client 100 calls the interface function sample(n); sample(n) selects the probability p1…p N Randomly select which storage node each sample is provided by, and count how many training samples need to be requested from storage node 1...storage node N respectively; determine to randomly obtain n1 and n3 training samples from storage node 1 and storage node 3 respectively based on the random selection; issue corresponding control instructions, i.e., sample requests, to storage node 1 and storage node 3 respectively; storage node 1 and storage node 3 take out n1 and n3 training samples, a total of n training samples, from the head of the memory buffer queue, i.e., their respective buffer pools 202, and feed them back to the client 100, and randomly select a batch of training samples from the persistent storage medium and add them to the tail of the memory buffer queue; the client 100 returns the received training samples to the application 400 through the interface function sample(n).

[0072] A remote caching method for sample loading in model distributed training is applicable to a remote caching device for sample loading in model distributed training, comprising:

[0073] The client 100 sends a batch sample request to the server 200;

[0074] Outputting the training samples to the client 100 via the buffer pool 202 of the server 200;

[0075] Training samples are randomly extracted from the storage pool 201 to the buffer pool 202 .

[0076] In another embodiment, when the buffer pool 202 of the server 200 outputs training samples to the client 100 and causes vacancies in the buffer pool 202, the sample selection module 203 of the server randomly extracts a batch of training samples from the storage pool 201 to the buffer pool 202 in a synchronous or asynchronous manner based on an on-site random sample selection strategy.

[0077] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A remote cache device for sample loading in model distributed training, characterized in that: include: Client, sends batch sample request; The server responds to sample requests; The above-mentioned server includes: Storage pool, used to store training samples; The buffer pool is used to output training samples according to sample requests and save training samples randomly selected from the storage pool.

2. According to claim 1, a remote cache device for sample loading in model distributed training is characterized in that: There are multiple servers.

3. The remote cache device for sample loading in model distributed training according to claim 2, characterized in that: The client includes an access interface for providing users with the function of obtaining batch random training samples, using the server random selection strategy to determine the server that provides the samples, counting the number of training samples requested from each server, and sending sample requests to the client.

4. The remote cache device for sample loading in model distributed training according to claim 3, characterized in that: The server-side random selection strategy includes: If extraction with replacement is required, the ratio of the number of training samples of each server to the total number of training samples is the selection probability of the server; If sampling without replacement is required, the ratio of the number of samples that have not been requested by each server to the total number of samples that have not been requested is the selection probability of the server.

5. The remote cache device for sample loading in model distributed training according to claim 1, characterized in that: The server includes a sample selection module, and the sample selection module is used to randomly extract training samples from the storage pool to the buffer pool based on an in-situ random sample selection strategy.

6. A remote cache device for sample loading in model distributed training according to claim 5, characterized in that: The in-situ random sample selection strategy includes: Use the no-replacement strategy to randomly select unaccessed training samples from the storage pool and put them into the buffer pool; Using the replacement strategy, training samples are randomly selected from all training samples in the storage pool and put into the buffer pool.

7. A remote cache device for sample loading in model distributed training according to claim 5 or 6, characterized in that: The buffer pool is designed according to the producer-consumer model. The buffer pool returns training samples according to the number of samples requested by the client. The server synchronously or asynchronously randomly selects samples to fill the buffer pool using an in-situ random sample selection strategy.

8. A remote cache device for sample loading in model distributed training according to any one of claims 1 to 6, characterized in that: The client sets a random seed and sends it to the server.

9. A remote caching method for sample loading in model distributed training, applicable to a remote caching device for sample loading in model distributed training, comprising: Send batch sample requests to the server via the above client; Output the training samples to the client through the buffer pool of the server; Randomly extract training samples from the above storage pool to the buffer pool.

10. A remote caching method for sample loading in model distributed training according to claim 9, characterized in that: When the buffer pool of the server outputs training samples to the client and causes the buffer pool to be empty, the sample selection module of the server randomly extracts training samples from the storage pool to the buffer pool in a synchronous or asynchronous manner based on the local random sample selection strategy.

Images