A device and method for remote caching of sample loading in model distributed training

Abstract

This invention discloses a remote caching device and method for loading samples in distributed model training, relating to the field of deep learning technology. It addresses the problem that the failure of the locality principle in a completely random access mode renders traditional caching ineffective. The invention includes a client that sends sample requests and a server that responds to sample requests. The server includes a storage pool for storing training samples and a buffer pool for outputting training samples according to sample requests and storing training samples randomly selected from the storage pool. This invention can reduce the latency of loading training samples during model training and improve the memory cache utilization of storage nodes in model training scenarios.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and specifically to a remote caching device and method for loading samples in distributed model training. Background Technology

[0002] Deep learning model training often employs various optimizers based on Stochastic Gradient Descent (SGD). The working paradigm involves repeatedly and randomly sampling small batches of samples from the entire dataset without replacement in each training iteration, training the model once per iteration. Using only a small batch of samples per iteration is done to accelerate training and also because GPU memory is limited, preventing the simultaneous loading of large numbers of samples for network inference and parameter updates.

[0003] When the total amount of data in the dataset is very large, the host's main memory also faces the problem of not being able to hold all the data, which is very common in training scenarios for large models. Therefore, the lifecycle of the data required in one iteration is as follows: the program randomly samples and points out a batch of training samples to be read, then the data is read from external storage to main memory, transferred from main memory to GPU memory, and finally the GPU uses this data on GPU memory for training. The external storage here can be local external storage, or when local external storage cannot hold a large-scale dataset, it can be a network storage service, such as distributed file storage.

[0004] Before computation can begin, GPUs must wait for a series of data I / O operations to arrive in video memory, leading to a significant waste of computing resources. Therefore, to fully utilize computing resources, computation and data transfer need to be performed in parallel. While training the current batch of data, the next batch should be loaded into main memory, with these two processes overlapping to allow the next batch of computation to begin faster. Simultaneously, low latency access to training samples is required to ensure short data I / O times, guaranteeing that computation time covers data I / O time as much as possible, thus fully utilizing computing resources.

[0005] Existing deep learning frameworks such as PyTorch and Tensorflow provide dataset loading capabilities and integrate asynchronous data prefetching mechanisms to prefetch data into local main memory, thereby overlapping computation and data I / O. However, deep learning frameworks are designed for generality and are not specifically optimized for diverse underlying storage. When dealing with large-scale datasets, distributed storage is required, which introduces longer I / O paths, including at least the transfer from remote persistent storage to remote main memory and the network transfer from storage nodes to the local machine.

[0006] The high access latency caused by long I / O paths makes the strategy of simply prefetching data to local main memory insufficient to mask the data I / O computation. Therefore, to meet the low latency requirement, caching should be used to pipeline the several transfer processes of data from remote persistent storage to local main memory, making them asynchronous. Traditional caches are designed based on the principle of locality, including temporal locality: if a data item is accessed, it is likely to be accessed again in the near future; and spatial locality: if a data item is accessed, data items adjacent to it are also likely to be accessed in the near future. Based on the principle of locality, traditional caching systems read data along with neighboring data and keep them in the cache until they are evicted.

[0007] In contrast, the data loading process for deep learning training tasks does not exhibit such locality. Once a data item is accessed, it will not be accessed again until all data has been accessed once, thus exhibiting no temporal locality. Furthermore, the access order of data items is completely random, exhibiting no spatial locality. Summary of the Invention

[0008] To address the problems existing in the prior art, this invention provides a remote caching device and method for sample loading in distributed model training, which solves the problem that the failure of the locality principle under completely random data access mode causes traditional caching to fail.

[0009] A remote caching device for sample loading in distributed model training includes:

[0010] The client sends a batch sample request;

[0011] The server responds to the sample request;

[0012] The aforementioned server-side components include:

[0013] Storage pool, used to store training samples;

[0014] The buffer pool is used to output training samples according to sample requests and to store training samples randomly drawn from the storage pool.

[0015] Furthermore, there are multiple servers. This ensures the amount of data stored.

[0016] Furthermore, the client includes an access interface for providing users with the function of obtaining batches of random training samples.

[0017] Furthermore, the client access interface is implemented by using a server-side random selection strategy to determine which server provides each sample, counting the number of training samples requested from each server, and sending sample requests to the server.

[0018] Furthermore, the server-side random selection strategy includes:

[0019] If sampling with replacement is required, the ratio of the number of training samples to the total number of training samples for each server is the selection probability of that server, and the selection probability of the i-th server is p. i =cnt i / ∑ k cnt k 1≤k≤N, where N is the number of storage nodes, cnt i Let be the number of training samples on the i-th server;

[0020] If sampling without replacement is required, the ratio of the number of unrequested samples to the total number of unrequested samples on each server is the selection probability of that server, and the selection probability of the i-th server is p. i =left i / ∑ k left k , 1≤k≤N, where N is the number of storage nodes, left i Let be the number of samples that have not yet been requested on the i-th server.

[0021] Furthermore, the server includes a sample selection module, which is used to randomly extract training samples from the storage pool to the buffer pool based on an on-site random sample selection strategy.

[0022] Furthermore, the on-site random sample selection strategy includes:

[0023] Using a no-replacement policy, unaccessed training samples are randomly selected from the storage pool and added to the buffer pool.

[0024] Using a replacement strategy, training samples are randomly selected from all training samples in the storage pool and added to the buffer pool.

[0025] Furthermore, the buffer pool is designed according to the producer-consumer model. The buffer pool returns training samples according to the number of samples requested by the client. The server synchronously or asynchronously selects samples from the storage pool to fill the buffer pool based on the local random sample selection strategy.

[0026] Furthermore, the client sets a random seed and sends it to the server. The server uses the set random seed to perform a random selection strategy and an on-site random sample selection strategy. When retrieving data, the client uses the random seed to generate a random number sequence to select which server to obtain training samples from; the server uses seed + id as the file access order in the random seed generation storage pool, where id is the unique identifier for each server.

[0027] A remote caching method for sample loading in distributed model training, applicable to a remote caching device for sample loading in distributed model training, comprising:

[0028] The aforementioned client sends batch sample requests to the server;

[0029] Training samples are output to the client through the buffer pool on the server side.

[0030] Training samples are randomly drawn from the above storage pool to the buffer pool.

[0031] Furthermore, when the server-side buffer pool outputs training samples to the client, resulting in a shortage in the buffer pool, the aforementioned server-side sample selection module randomly extracts training samples from the storage pool to the buffer pool in a synchronous or asynchronous manner based on the local random sample selection strategy.

[0032] The beneficial effects of this invention include:

[0033] This invention designs a remote cache for sample loading in distributed model training. It transfers the out-of-order operation of training samples from computation nodes to storage nodes, allowing storage nodes to determine the future access order of training samples without computation node intervention. This makes the storage node's cache effective. Specifically:

[0034] 1. Compared with general-purpose cache designs, this invention can improve the memory cache utilization of storage nodes in model training scenarios, reduce the latency of loading training samples during model training, and improve the utilization of computing resources.

[0035] 2. Compared to the disordered training samples on the computing node, the present invention eliminates the need for the computing node to cache a large amount of sample metadata information. Attached Figure Description

[0036] Figure 1 This is a first schematic diagram of a remote caching device for loading samples in distributed training of a model, according to an embodiment of this application.

[0037] Figure 2 This is a second schematic diagram of a remote caching device for loading samples in distributed training of a model, according to an embodiment of this application.

[0038] Figure 3 This is a third schematic diagram of a remote caching device for loading samples in distributed training of a model, according to an embodiment of this application.

[0039] Figure 4 This is a fourth schematic diagram of a remote caching device for sample loading in distributed training of a model, according to an embodiment of this application.

[0040] Figure 5This is a schematic diagram illustrating the operation of the server in an embodiment of this application.

[0041] Figure 6 This is a schematic diagram illustrating the operation of the client in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0043] A remote caching device for sample loading in distributed model training, such as Figure 1 As shown, it includes:

[0044] Client 100 sends a batch sample request;

[0045] The server returned a 200 response to the sample request.

[0046] Among them, the aforementioned server 200 includes:

[0047] Storage pool 201 is used to store training samples;

[0048] Buffer pool 202 is used to output training samples according to sample requests and to store training samples randomly drawn from storage pool 201.

[0049] The server 200 is located on a storage node on the remote storage side, and the client 100 is located on the computing side on the user's local machine.

[0050] The client 100 module is responsible for requesting batches of random training samples from multiple storage nodes, while the server 200 is responsible for maintaining the buffer pool 202 and responding to sample requests from the client 100.

[0051] At the start of a deep learning training task, the application 400 on the computing node, used to implement the training task, sends an initialization command to the cache device. Upon receiving the initialization command, the client 100 sends an initialization request to the server 200. This initialization request includes training task information, such as the set of samples contained in the training dataset (which may be the directory name where the dataset is located). Upon receiving the initialization request, the server 200 initializes the buffer pool 202 for the training task: this includes allocating memory; initializing the metadata list: based on the training sample set information provided by the client 100, extracting the sample metadata contained in this storage node (which may be any data that can be indexed to the training sample content, such as the size and offset of the sample data, and the sample ID); randomly selecting a batch of training samples from the storage pool 201 into the buffer pool 202; and then the server 200 sends an initialization completion response to the client 100, informing the client 100 of the number of training samples on the storage node.

[0052] In another embodiment, such as Figure 3 As shown, there are multiple servers 200, including a first server 210, a second server 220, a third server 230, etc., which ensures the amount of data stored, while also having a higher fault tolerance rate and ensuring smooth data retrieval.

[0053] In another embodiment, the server 200 further includes a sample selection module 203 for implementing an on-site random sample selection strategy, such as... Figure 2 As shown.

[0054] In another embodiment, such as Figure 4 As shown, the client 100 includes an access interface 101, used to provide the computing program with the function of obtaining batch random training samples. The access interface 101 is implemented as follows: a server-side random selection strategy is used to determine which server provides each random sample, the number of training samples requested from each server is counted, and a sample request is sent to the server 200. Specifically: taking the local random file selection strategy using a non-replacement strategy as an example, the client 100 module maintains a sample count vector cnt for each storage node and a vector left for the number of unaccessed samples for each storage node, used to record the number of remaining unaccessed samples for each storage node in the current round of training sample access. The sample access period is one round of training, during which all training samples in the dataset need to be traversed.

[0055] Server 200 records the number c of training files contained on the storage node for each training task, and maintains a memory buffer queue (buffer pool 202), a metadata list, and a remaining sample count l. The memory buffer queue is used to store sample data that will be accessed in the future; the metadata list is used to record the location information of training samples within the dataset; and the remaining file count l is used to record the number of samples that have not yet been cached in the current sample access cycle.

[0056] When client 100 requests a batch of random training samples, the selection probability of each server is determined by the ratio of the number of samples not yet requested by each server to the total number of unrequested samples. The probability of selecting the i-th storage node server is left. i / ∑ k left k 1 ≤ k ≤ N, where N is the number of storage nodes. Then, the number of training samples requested from each server is counted, and multiple sample requests are sent to the server cluster: {request(i, n...} i )}, request(i,n i This indicates a request for n from the i-th storage node. i Training samples.

[0057] When the i-th storage node receives the request n i When a request for a random sample is received, server 200 on the storage node retrieves n samples from the head of the buffer queue. i One training sample is sent to the client as a response, and random training samples are simultaneously added to the tail of the buffer queue, updating the remaining sample count l to ln. i .

[0058] When the remaining sample count l of server 200 is 0, l is reset to the sample count c; when all bits of the unaccessed sample count vector left of client 100 are 0, if training has not ended, left is reassigned to cnt, representing the start of a new training sample access cycle.

[0059] In another embodiment, the on-site random sample selection strategy employs a strategy with replacement;

[0060] The client module 100 maintains a vector cnt representing the number of files on each storage node;

[0061] The server-side module 200 records the number of training samples c contained on the storage node for each training task, and maintains a memory buffer queue, namely buffer pool 202, and a metadata list.

[0062] The memory buffer queue is used to store sample data that will be accessed in the future; the metadata list is used to store the location information of training samples within the dataset.

[0063] At the start of the deep learning training task, the client module 100 notifies the server module 200 of each storage node and provides the set of samples contained in the training dataset (which may be the directory name where the dataset is located).

[0064] Each storage node's server-side module 200 initializes a memory buffer queue for the training task, including allocating memory; initializing a metadata list: based on the training sample set information provided by the client 100, extracting the sample metadata contained in this storage node (which can be any data that can be indexed to the sample content, such as sample data size and offset, sample ID); and recording the number of samples c as the number of dataset samples contained in this storage node.

[0065] When client 100 requests a batch of random training samples, a storage node is randomly selected for each sample in the batch, with the probability being the proportion of samples from each node to the total number of samples. The probability of selecting the i-th storage node is cnt. i / ∑ k cnt k 1 ≤ k ≤ N, where N is the number of storage nodes. Then, the number of training samples requested from each server is counted, and multiple sample requests are sent to the server cluster: {request(i, n...} i )}, request(i,n i This indicates a request for n from the i-th storage node. i Training samples.

[0066] When the i-th storage node receives the request n i When a request for a random sample is received, server 200 on the storage node retrieves n samples from the head of the buffer queue. i One training sample is sent to the client as a response, and at the same time, random training samples are added to the tail of the buffer queue.

[0067] In another embodiment, such as Figure 5 As shown, the buffer pool 202 is a lock-free circular buffer queue, which avoids lock contention and ensures that the memory location occupied by the buffer pool 202 is fixed, thereby further improving the efficiency of data processing. Specifically, at the current moment, a batch of training samples is taken from the head of the circular buffer queue, which serves as the buffer pool 202, according to the number of samples requested by the client 100 from the server 200, and transmitted to the client 100. At the same time, a batch of training samples is randomly selected asynchronously from the persistent storage medium, which serves as the storage pool 201 and stores a large number of training samples, and placed into the tail of the circular buffer queue.

[0068] In another embodiment, a preferred method for selecting batch random training samples is as follows: generate a random integer r between 1 and 1, swap the r-th and l-th metadata in the metadata list, and the file corresponding to the l-th metadata in the metadata list after the swap is the selected random training sample. Repeat the above process until a specified number of samples are retrieved.

[0069] In another embodiment, to ensure that the random access order of files on the storage system is repeatable (reproducible), the client 100 module sets a random seed when initializing the training task and broadcasts it to each storage node. When retrieving data, the client 100 uses the random seed to generate a random number sequence to determine which storage node each random sample comes from; the server 200 uses seed + id as the random seed to generate the sample access order within the storage node, where id is the globally unique number of each storage node.

[0070] In another embodiment, the access interface 101 is a sample(n) function, where n is a parameter specifying the number of random samples to be obtained, as detailed below. Figure 6 As shown.

[0071] The local application 400 initiates a data call command. After receiving the call command, the client 100 calls the interface function sample(n); sample(n) is based on the selection probabilities p1…p N Randomly select which storage node provides each sample, and calculate how many training samples need to be requested from storage node 1... storage node N respectively; based on the random selection, determine that n1 and n3 training samples are randomly obtained from storage node 1 and storage node 3 respectively; issue corresponding control commands, i.e. sample requests, to storage node 1 and storage node 3 respectively; storage node 1 and storage node 3 take out n1 and n3 training samples (a total of n) from the head of their respective memory buffer queues (i.e., their respective buffer pools 202) and send them back to client 100, and randomly select a batch of training samples from the persistent storage medium and add them to the tail of the memory buffer queue; client 100 returns the received training samples to application 400 through the interface function sample(n).

[0072] A remote caching method for sample loading in distributed model training, applicable to a remote caching device for sample loading in distributed model training, comprising:

[0073] The client 100 sends a batch sample request to the server 200.

[0074] Training samples are output to the client 100 through the buffer pool 202 of the server 200;

[0075] Training samples are randomly drawn from the aforementioned storage pool 201 into the buffer pool 202.

[0076] In another embodiment, when the buffer pool 202 of the server 200 outputs training samples to the client 100, resulting in a shortage in the buffer pool 202, the sample selection module 203 of the server described above randomly selects a batch of training samples from the storage pool 201 to the buffer pool 202 in a synchronous or asynchronous manner based on the local random sample selection strategy.

[0077] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. A device for remote caching of sample loading in model distributed training, characterized in that, Comprise: Client, set in the computing side, sends batch sample request, the sample request is the number of training samples requested by each server; Server, set in the storage node of the remote storage side, responds to the sample request; Wherein, the above-mentioned server comprises: Storage pool, for storing training samples; Buffer pool, for outputting training samples according to sample request, saving randomly extracted training samples from storage pool; The client includes an access interface for providing the function of obtaining batch random training samples to the user, using the server random selection strategy to determine the server providing the sample, counting the number of training samples requested from each server, and sending the sample request to the server; The server has multiple; The sample selection module of the server is used for randomly extracting training samples from the storage pool to the buffer pool based on the local random sample selection strategy.

2. The remote caching device for sample loading in model distributed training of claim 1, wherein, The server random selection strategy includes: If you want to extract with replacement, the ratio of the number of training samples of each server to the total number of training samples is the selection probability of the server; If you want to extract without replacement, the ratio of the number of samples not requested by each server to the total number of samples not requested is the selection probability of the server.

3. The remote caching device for sample loading in model distributed training of claim 1, wherein, The local random sample selection strategy includes: Using the non-replacement strategy, randomly select the training samples not accessed from the storage pool to the buffer pool; Using the replacement strategy, randomly select training samples from all training samples in the storage pool to the buffer pool.

4. The remote caching device for sample loading in model distributed training of any of claims 1-3, wherein, The buffer pool is designed according to the producer-consumer model, and the buffer pool returns training samples according to the number of samples requested by the client, and the server synchronously or asynchronously fills the buffer pool with random samples according to the local random sample selection strategy.

5. The remote caching device for sample loading in model distributed training of any one of claims 1-3, wherein, The client sets a random seed seed and sends it to the server; When taking data, the client uses the random seed seed to generate a random number sequence to select which server to get training samples from; The server uses seed+id as the random seed to generate the file access order in the storage pool, and id is the number of each server, which is globally unique.

6. A remote caching method for sample loading in model distributed training, suitable for the remote caching device for sample loading in model distributed training of any one of claims 1-3, comprising: Sending batch sample request to server by the above-mentioned client; Outputting training samples from the buffer pool of the above-mentioned server to the client; Randomly extracting training samples from the above-mentioned storage pool to the buffer pool.

7. The method of claim 6, wherein, When the buffer pool of the server outputs training samples to the client resulting in vacancies in the buffer pool, the sample selection module of the above-mentioned server randomly extracts training samples from the storage pool to the buffer pool based on the local random sample selection strategy in a synchronous or asynchronous manner.

Images