A distributed training data loading method and device based on separation architecture

Abstract

The present invention discloses a distributed training data loading method and device based on a separation architecture, which belongs to the field of computer technology. The method includes: a remote central scheduler responds to a request sent by a local data loading client, collects data metadata of training samples, and performs load balancing calculation to generate a data loading plan and a data consumption plan; a remote data loading server obtains a data loading plan to load and preprocess data, obtains preprocessed distributed training data, and pre-takes the next batch of data; a local data loading client requests a designated remote data loading server to obtain preprocessed distributed training data according to the data consumption plan and caches it locally. The present invention not only improves the efficiency of distributed training data loading, but also ensures the balance of computing loads of various computing nodes in distributed training. It is suitable for distributed training scenarios of large-scale deep learning models and has high application value and promotion prospects.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a distributed training data loading method and device based on a separation architecture. Background Art

[0002] The expansion of deep learning models has given many downstream tasks excellent performance. From natural language processing to computer vision, deep learning models are gradually becoming the core force driving technological innovation in these fields. However, as the complexity of models continues to increase, the amount of data processing and computing resources required to train these large basic models have also exploded. How to efficiently manage and utilize these data resources has become one of the key factors affecting the efficiency and performance of large-scale deep learning model training.

[0003] From the perspective of data flow, the training process of large-scale basic models can be roughly divided into two stages: data processing and data execution. In the first stage, data processing, the data loader running on the CPU is responsible for extracting and preprocessing data from the storage medium. These preprocessed data exist in the form of tensors and are the basis for subsequent model training. The data loader transmits the processed tensors to the trainer through the network to prepare for the data execution stage. In the second stage, data execution, after receiving the tensors transmitted by the data loader, the trainer uses a specific accelerator (such as GPU) to execute the model graph and calculate the gradient to update the model weights. A large number of deep learning training frameworks have carried out sophisticated system design and advanced parallel scheme optimization (such as tensor parallelism, pipeline parallelism, sequence parallelism, expert parallelism, context parallelism and data parallelism) in the data execution stage, which improves the training performance. However, the impact of data loader efficiency on large-scale deep learning model training has been seriously overlooked.

[0004] Existing deep learning training frameworks usually use centralized data loaders, that is, the data loader and the training process are deployed on the same computing node. However, this approach will lead to resource competition between the data loading process and the training process. Especially in large-scale distributed training scenarios, this resource competition problem is more serious and becomes one of the key factors restricting training efficiency. In addition, the use of different distributed parallel schemes will lead to redundant reading of distributed training data by the training process, which not only wastes computing resources, but also may lead to data inconsistency and deviation of training results. In addition, different computing nodes usually only have training batch data information from a local perspective, while ignoring the load information between different training micro-batches from a global perspective, which limits the opportunities for data sharing and load balancing, resulting in uneven distribution of data loading tasks among different nodes in a distributed training environment, further affecting training efficiency.

[0005] In summary, the efficiency of data loaders has become one of the key factors restricting the efficiency of distributed training of large-scale deep learning models. In order to improve training efficiency and quality, it is necessary to re-examine the design of data loaders and explore a more efficient and scalable distributed training data loading method. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a distributed training data loading method and device based on a separated architecture. The separated architecture can better isolate data processing and eliminate data stagnation in the distributed training of large-scale deep learning models, and at the same time combine the distributed training data load balancing algorithm to coordinate data reading, thereby improving the distributed training efficiency of large-scale deep learning models.

[0007] In order to achieve the above-mentioned invention object, the technical solution provided by the present invention is as follows:

[0008] In a first aspect, an embodiment of the present invention provides a distributed training data loading method based on a separation architecture, comprising the following steps:

[0009] The remote central scheduler responds to the data consumption plan request sent by the local data loading client, collects the data metadata of the training samples, and performs load balancing calculation based on the data metadata to generate the data loading plan and data consumption plan;

[0010] Obtain the data loading plan generated by the remote central scheduler through the remote data loading server to load and preprocess the distributed training data of the training samples, obtain the preprocessed distributed training data, and pre-fetch the next batch of distributed training data;

[0011] The local data loading client requests the designated remote data loading server to obtain the pre-processed distributed training data according to the data consumption plan generated by the remote central scheduler and caches it locally for use by the main process of the local deep learning model distributed training.

[0012] Preferably, the local data loading client and the main process of the deep learning model distributed training are deployed on the same computing node.

[0013] Preferably, the local data loading client is used as a subprocess of the distributed training main process of the deep learning model, and asynchronously requests a data consumption plan from the remote central scheduler without blocking the operation of the distributed training main process.

[0014] Preferably, a data buffer is set in the remote data loading server, and the data metadata information of the training samples is stored in the data buffer. When the remote central scheduler collects all the data metadata information of the current training batch in the data buffers of all the remote data loading servers, it has the global data metadata information of the current training batch data, and then performs load balancing calculations based on all the acquired data metadata information of the current training batch.

[0015] Preferably, the data metadata information of the training samples includes the word token sequence length and / or the number of pictures of the text training samples and / or picture training samples used in the distributed training of the deep learning model.

[0016] Preferably, the remote central scheduler and the remote data loading server are deployed on different physical computing nodes respectively from the local data loading client, and the remote central scheduler and the remote data loading server do not compete for resources with the main process of the distributed training of the deep learning model.

[0017] Preferably, the performing load balancing calculation based on the data metadata to generate a data loading plan and a data consumption plan includes:

[0018] The remote center scheduler runs the distributed training data load balancing algorithm to calculate the load balancing based on the data metadata to obtain a unique data partitioning method, including: the algorithm input is n The data metadata of the training samples and the local data loading client are divided m data parallel groups; for each training sample, the data metadata , construct a m Tuples are used to represent a way of partitioning data. m Each element in the tuple is a subset, and m The elements of a subset of a tuple are , others m- 1 subset is an empty set; in each cycle, select the two sets with the largest difference between the maximum and minimum values ​​of the sum of the internal subsets m Tuple x and y , first x and y Sort by the sum of the internal subsets, and then sort the y According to the reverse order of the internal subset and then x Merge to get a new m Tuples are added to the data partitioning method set as a new data partitioning method, and x and y Delete from the data partitioning method set; continue iterating in the same way mThe tuples are merged until a final m Tuples are the only way to partition data;

[0019] Develop a data consumption plan based on the unique data partitioning method. The data consumption plan specifies the partitioning of the local data loading client. m The training sample data resources that each data parallel group can obtain;

[0020] According to the data consumption plan, a data loading plan is formulated. The data loading plan specifies the division of remote data loading servers and local data loading clients. m The loading and allocation strategy of training sample data resources corresponding to the data parallel groups.

[0021] In a second aspect, to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a distributed training data loading device based on a separation architecture, which is implemented using the above-mentioned distributed training data loading method based on a separation architecture, including: a remote central scheduler, a remote data loading server and a local data loading client;

[0022] The remote central scheduler is used to respond to the data consumption plan request sent by the local data loading client, collect data metadata information of the training samples, and perform load balancing calculation based on the data metadata to generate a data loading plan and a data consumption plan;

[0023] The remote data loading server is used to obtain the data loading plan generated by the remote central scheduler to perform distributed training data loading and data preprocessing of the training samples, obtain the preprocessed distributed training data, and pre-fetch the next batch of distributed training data;

[0024] The local data loading client is used to request the designated remote data loading server to obtain the pre-processed distributed training data according to the data consumption plan generated by the remote central scheduler and cache it locally for use by the main process of the local deep learning model distributed training.

[0025] In the third aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention also provides an electronic device, including a memory and one or more processors, wherein the memory is used to store a computer program, and the processor is used to implement the above-mentioned distributed training data loading method based on a separated architecture when executing the computer program.

[0026] In a fourth aspect, in order to achieve the above-mentioned purpose of the invention, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned distributed training data loading method based on the separated architecture is implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] (1) The present invention adopts a separate data loader architecture including a remote central scheduler, a remote data loading server and a local data loading client, and offloads data processing and scheduling tasks to a scalable remote data loading server and a remote central scheduler, which is conducive to eliminating the distributed training stagnation phenomenon caused by data loading, preprocessing and scheduling, allowing training nodes to focus on executing computing-intensive tasks, more effectively utilizing computing resources, and improving the distributed training efficiency of large-scale deep learning models.

[0029] (2) The present invention adopts a distributed training data load balancing algorithm to realize the scheduling of distributed training data in the same batch to balance the training load of each training node, thereby reducing the impact of tailgaters in the data parallel group on the distributed training efficiency of large-scale deep learning models. By balancing the load distribution, computing resources are more fully utilized, thereby effectively improving the distributed training efficiency of large-scale deep learning models. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0031] Figure 1 It is a flowchart of a distributed training data loading method based on a separation architecture provided in an embodiment of the present invention;

[0032] Figure 2 It is a schematic diagram of a framework of a distributed training data loading method based on a separation architecture provided in an embodiment of the present invention;

[0033] Figure 3 It is a flow chart of a distributed training data load balancing algorithm provided by an embodiment of the present invention;

[0034] Figure 4 It is a structural diagram of a distributed training data loading device based on a separation architecture provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0036] The inventive concept of the present invention is: in view of the problem of inefficient data loading and uneven distributed training data load in the distributed training of large-scale deep learning models in the prior art, which leads to low distributed training efficiency, the embodiments of the present invention provide a distributed training data loading method and device based on a separated architecture. Through the separated architecture design of the remote central scheduler, the remote data loading server and the local data loading client, the data processing and scheduling work is offloaded to the scalable remote data loading server and the remote central scheduler. At the same time, the data loading plan and the data consumption plan are obtained in the remote central scheduler through the distributed training data load balancing algorithm, so as to efficiently schedule the data consumed by each training process, so that the load of each computing node in the data parallel group is as balanced as possible, and the influence of the slowdown in the data parallel group on the distributed training efficiency of the large-scale deep learning model is reduced, thereby improving the distributed training efficiency of the large-scale deep learning model.

[0037] Figure 1 is a flow chart of a distributed training data loading method based on a separation architecture provided in an embodiment of the present invention, Figure 2 Schematic diagram of the framework of the distributed training data loading method based on the separation architecture provided by the embodiment of the present invention. Figure 1 and Figure 2 As shown, the embodiment provides a distributed training data loading method based on a separation architecture, comprising the following steps:

[0038] S1, responds to the data consumption plan request sent by the local data loading client through the remote central scheduler, collects the data metadata of the training samples, and performs load balancing calculations based on the data metadata to generate data loading plans and data consumption plans.

[0039] In an embodiment, the remote center scheduler and the remote data loading server are respectively deployed on different physical computing nodes with the local data loading client, and the communication and data transmission between them are transmitted through the network, and the remote center scheduler and the remote data loading server do not compete for resources with the main process of the distributed training of the deep learning model. At the same time, the local data loading client and the main process of the distributed training of the deep learning model are deployed on the same computing node. The local data loading client is used as a subprocess of the distributed training main process of the deep learning model, and asynchronously requests a data consumption plan from the remote center scheduler without blocking the operation of the distributed training main process.

[0040] S1.1, the local data loading client sends a data consumption plan request to the remote central scheduler.

[0041] S1.2, the remote center scheduler responds to the request and collects the data metadata of the training samples stored in the data buffer in the remote data loading server. The data metadata of the training samples includes the word token sequence length and / or number of pictures of the text training samples and / or picture training samples used in the distributed training of the deep learning model. These metadata are closely related to the computational load of the deep learning model training. In addition, by setting up a data buffer in the remote data loading server, it is possible to reduce waiting time, improve data processing efficiency, optimize resource utilization, and reduce the risk of data interruption or loss due to hardware failure, network interruption, or unstable power supply during data transmission.

[0042] S1.3, when the remote center scheduler collects all the data metadata of the current training batch in the data buffer of all remote data loading servers, it has the global data metadata of the current training batch data, and then runs the distributed training data load balancing algorithm to perform load balancing calculations to generate the data loading plan required by the remote data loading server and the data consumption plan required by the local data loading client.

[0043] The distributed training data load balancing algorithm is a multi-way digital partitioning problem. The multi-way digital partitioning problem is an NP-Hard problem. There is currently no known polynomial time algorithm that can determine the optimal solution. It can be flexibly adjusted according to actual computing needs, using common algorithms such as dynamic programming, integer linear programming, and greedy algorithms to achieve a balance between real-time and accuracy while facilitating task migration. Figure 3 As shown, in the embodiment, a distributed training data load balancing algorithm based on the Karmarkar-Karp heuristic algorithm is used to obtain a unique data partitioning method.

[0044] S1.3.1, first, specify an evaluation function for calculating load balancing within a grouping method for:

[0045] (1),

[0046] in, is a collection of sub-sets m Tuple, m The size of the tuple is the size of the data parallel group, Is the first j The set of data metadata information of the training samples assigned to the computing nodes. It is j The computing nodes are assigned to i The computational load of each training sample measured by the data metadata, and Represents the maximum and minimum values ​​respectively. It is used to measure the difference between the set with the largest sum of elements and the set with the smallest sum of elements. The smaller the value, the more evenly the computing load of this grouping method is distributed on each computing node.

[0047] S1.3.2, e.g. Figure 3 As shown in Figure 1, the algorithm starts from the "Start" node. First, determine the total number of training samples as , , For the The total number of data parallel groups divided by the local data loading client is , initialize the data partitioning method set Empty Set .

[0048] Initialize index , build Tuple ,in, m Each element in the tuple is a subset, and m The elements of a subset of a tuple are , others m- 1 subset is the empty set. Tuples are added as a way to partition data In. Traverse indivual , determine the index Is it less than If so, in the next cycle, , continue to build the next Tuple and add to until Stop traversal when .

[0049] for Each of Substitute the tuple into formula (1) to find the difference between the maximum and minimum values ​​of the internal subset sum , and According to each Tuple The values ​​are sorted in descending order.

[0050] Get the descending order The first Tuple and the second Tuple .Will x and yArrange them in descending order according to the sum of the internal subsets, and After reversing, each sub-set inside it is The corresponding sub-sets are merged to generate new Tuples as a new way to partition data ,Will x and y from Remove it from Add to middle.

[0051] Continue iterating in the same way m Merge of tuples, judgment during iteration within Is the number of tuples not equal to 1? If not, return and continue to Processing, repeat the above grouping processing and iterative judgment steps, if it is equal to 1, the algorithm ends and returns the remaining final m Tuples as the only way to partition data .

[0052] S1.3.3, based on unique data classification Develop a data consumption plan that specifies the local data loading client's allocation m The training sample data resources that each data parallel group can obtain.

[0053] S1.3.4. Develop a data loading plan based on the data consumption plan. The data loading plan specifies the division of the remote data loading server and the local data loading client. m The loading and allocation strategy of training sample data resources corresponding to the data parallel groups.

[0054] S2, obtains the data loading plan generated by the remote central scheduler through the remote data loading server to perform distributed training data loading and data preprocessing of the training samples, obtains the preprocessed distributed training data, and pre-fetches the next batch of distributed training data.

[0055] S2.1, the remote data loading server obtains the data loading plan generated by the remote central scheduler, and performs distributed training data loading and data preprocessing of the training samples according to the data loading plan to obtain the preprocessed distributed training data. Since the remote data loading server and the local data loading client are deployed on different physical nodes, the data loading and preprocessing work of the remote data loading server will not compete for resources with the distributed training main process.

[0056] S2.2, the remote data loading server pre-fetches the next batch of distributed training data from the local storage medium or distributed file system, which allows the data loading process and the data preprocessing process to be executed in a pipeline manner, ensuring that the I / O operations of data loading and data preprocessing operations overlap, reducing the impact of data I / O operations on system performance, and improving the task execution efficiency of the data loading server.

[0057] S3, through the local data loading client, requests the designated remote data loading server to obtain the pre-processed distributed training data according to the data consumption plan generated by the remote center scheduler and caches it locally for use by the main process of the local deep learning model distributed training.

[0058] In an embodiment, a local data loading client obtains a data consumption plan generated by a remote central scheduler, requests data from a designated remote data loading server according to the data consumption plan, and caches the data locally for use by a distributed training main process.

[0059] In summary, the embodiment of the present invention provides a distributed training data loading method based on a separated architecture. By designing a separated architecture including three roles including a remote central scheduler, a remote data loading server, and a local data loading client, the heavy and I / O-intensive data preprocessing work is offloaded to the remote data loading server for execution, isolating data processing, and utilizing network transmission between nodes to reduce the impact of data loading work on the efficiency of the distributed training main process. The remote central scheduler has a global perspective on the distributed training batch data, and can formulate targeted data balancing algorithms to schedule designated data to specific training processes for consumption, thereby reducing the uneven distribution of data and causing serious impact on the distributed training efficiency of large-scale deep learning models. The local data loading client is a subprocess of the distributed training main process, responsible for asynchronous loading and pre-fetching of remote data, which is conducive to eliminating bubbles between training pipelines caused by data stagnation.

[0060] Based on the same inventive concept, Figure 4 As shown, an embodiment of the present invention further provides a distributed training data loading device 400 based on a separation architecture, including: a remote central scheduler 410, a remote data loading server 420 and a local data loading client 430.

[0061] The remote center scheduler 410 is used to respond to the data consumption plan request sent by the local data loading client, collect data metadata information of the training samples, and perform load balancing calculation based on the data metadata to generate a data loading plan and a data consumption plan.

[0062] The remote data loading server 420 is used to obtain the data loading plan generated by the remote central scheduler to perform distributed training data loading and data preprocessing of the training samples, obtain preprocessed distributed training data, and pre-fetch the next batch of distributed training data.

[0063] The local data loading client 430 is used to request the designated remote data loading server to obtain the pre-processed distributed training data according to the data consumption plan generated by the remote central scheduler and cache it locally for use by the main process of the local deep learning model distributed training.

[0064] Based on the same inventive concept, an embodiment of the present invention also provides an electronic device, including a memory and one or more processors, the memory is used to store a computer program, and the processor is used to implement the above-mentioned distributed training data loading method based on a separated architecture when executing the computer program.

[0065] Based on the same inventive concept, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a computer, the above-mentioned distributed training data loading method based on the separated architecture is implemented.

[0066] It should be noted that the distributed training data loading device based on the separated architecture, the electronic device, and the computer-readable storage medium provided in the embodiments of the present invention all belong to the same inventive concept as the distributed training data loading method based on the separated architecture. The specific implementation process is detailed in the embodiment of the distributed training data loading method based on the separated architecture, which will not be repeated here.

[0067] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A distributed training data loading method based on a separation architecture, characterized in that: The following steps are involved: The remote central scheduler responds to the data consumption plan request sent by the local data loading client, collects the data metadata of the training samples, and performs load balancing calculation based on the data metadata to generate the data loading plan and data consumption plan; Among them, load balancing calculation is performed based on data metadata to generate data loading plan and data consumption plan, including: the remote center scheduler runs the distributed training data load balancing algorithm to perform load balancing calculation based on data metadata to obtain a unique data partitioning method, including: the algorithm input is n The data metadata of the training samples and the local data loading client are divided m data parallel groups; for each training sample, the data metadata , construct a m Tuples are used to represent a way of partitioning data. m Each element in the tuple is a subset, and m The elements of a subset of a tuple are , others m- 1 subset is an empty set; in each cycle, select the two sets with the largest difference between the maximum and minimum values ​​of the sum of the internal subsets m Tuple x and y , first x and y Sort by the sum of the internal subsets, and then sort the y According to the reverse order of the internal subset and then x Merge to get a new m Tuples are added to the data partitioning method set as a new data partitioning method, and x and y Delete from the data partitioning method set; continue iterating in the same way m The tuples are merged until a final m Tuples are used as the only data partitioning method; a data consumption plan is formulated based on the unique data partitioning method, and the data consumption plan specifies the local data loading client partitioning m The training sample data resources that each data parallel group can obtain; formulate a data loading plan based on the data consumption plan, and the data loading plan stipulates the division of the remote data loading server and the local data loading client. m The loading and allocation strategy of training sample data resources corresponding to each data parallel group; Obtain the data loading plan generated by the remote central scheduler through the remote data loading server to load and preprocess the distributed training data of the training samples, obtain the preprocessed distributed training data, and pre-fetch the next batch of distributed training data; The local data loading client requests the designated remote data loading server to obtain the pre-processed distributed training data according to the data consumption plan generated by the remote central scheduler and caches it locally for use by the main process of the local deep learning model distributed training.

2. The distributed training data loading method based on the separation architecture according to claim 1 is characterized in that: Deploy the local data loading client and the main process of deep learning model distributed training on the same computing node.

3. The distributed training data loading method based on the separation architecture according to claim 1 or 2, characterized in that: The local data loading client is used as a subprocess of the distributed training main process of the deep learning model, and asynchronously requests a data consumption plan from the remote central scheduler without blocking the operation of the distributed training main process.

4. The distributed training data loading method based on the separation architecture according to claim 1 is characterized in that: A data buffer is set in the remote data loading server, and the data metadata of the training samples is stored in the data buffer. When the remote central scheduler collects all the data metadata of the current training batch in the data buffer of all remote data loading servers, it has the global data metadata of the current training batch data, and then performs load balancing calculations based on all the data metadata of the current training batch obtained.

5. The distributed training data loading method based on the separation architecture according to claim 1 or 4, characterized in that: The data metadata information of the training samples includes the word token sequence length and / or the number of pictures of the text training samples and / or picture training samples used in the distributed training of the deep learning model.

6. The distributed training data loading method based on the separation architecture according to claim 1 is characterized in that: The remote central scheduler and remote data loading server are deployed on different physical computing nodes from the local data loading client respectively, so that the remote central scheduler and remote data loading server do not compete for resources with the main process of distributed training of deep learning models.

7. A distributed training data loading device based on a separation architecture, implemented using the distributed training data loading method based on a separation architecture according to any one of claims 1 to 6, characterized in that: Includes: remote center scheduler, remote data loading server and local data loading client; The remote central scheduler is used to respond to the data consumption plan request sent by the local data loading client, collect data metadata information of the training samples, and perform load balancing calculation based on the data metadata to generate a data loading plan and a data consumption plan; The remote data loading server is used to obtain the data loading plan generated by the remote central scheduler to perform distributed training data loading and data preprocessing of the training samples, obtain the preprocessed distributed training data, and pre-fetch the next batch of distributed training data; The local data loading client is used to request the designated remote data loading server to obtain the pre-processed distributed training data according to the data consumption plan generated by the remote central scheduler and cache it locally for use by the main process of the local deep learning model distributed training.

8. An electronic device comprising a memory and one or more processors, wherein the memory is used to store a computer program, characterized in that: The processor is used to implement the distributed training data loading method based on the separated architecture as described in any one of claims 1-6 when executing the computer program.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a computer, the distributed training data loading method based on the separated architecture described in any one of claims 1 to 6 is implemented.

Images