Distributed machine learning-oriented full-slice data parallel optimization method and system

Abstract

The invention relates to the technical field of distributed training, in particular to a distributed machine learning-oriented full-slice data parallel optimization method and system. Calculating a total demand video memory in a model training process according to the model parameter quantity and precision; according to the number of GPUs and the number of layers of the model, the model is fragmented; setting an elastic coefficient, and setting a one-to-one mapping priority for each layer; setting a priority queue as a buffer based on the small root heap; forward calculation is carried out on each partition Pi in sequence, and each GPU collects a non-local layer as a cache candidate layer to be input into a cache; according to the current calculation partition Pi, evaluating the parameter use emergency degree, and adjusting the priority; when the cache candidate layer is input into the cache, calculating the residual video memory of the cache, and releasing the existing element model with the minimum priority in the cache according to the residual video memory and the adjusted priority; and after the training is finished, taking the total duration T of all GPU training as reinforcement learning earnings.

Description

Technical Field

[0001] This invention relates to the field of distributed training technology, and in particular to a method and system for parallel optimization of full-slice data for distributed machine learning. Background Technology

[0002] As the number of parameters in machine learning models continues to grow, the training workload of ultra-large models requires multiple devices to share the high computing power and storage demands brought about by massive parameters in a distributed manner. To fully leverage the advantages of distributed systems and improve training efficiency, various parallel methods have been proposed and widely applied in the industry. Among them, Full Slice Data Parallelism (FSDP) is a variant of the traditional Data Parallelism (DP) method, which splits the model and distributes it across multiple GPUs, solving the problem of model size exceeding the storage limit of a single GPU. However, to ensure the integrity of training, FSDP requires a large amount of inter-node communication to move model slice data, with communication overhead exceeding 80% compared to traditional DP methods, posing a significant challenge to system throughput. From a macro perspective, FSDP trades increased communication for greater storage efficiency, and there is a certain degree of convertibility between its storage and communication.

[0003] The relationship between deep learning model size and performance is typically governed by a power law. To achieve better model performance and accuracy, training models with more parameters on larger datasets has become a trend. This massive number of parameters and computational demands presents challenges to hardware infrastructure, leading to the rapid expansion of distributed training facilities from kilocalories to tens of thousands of processors, and now towards large clusters with hundreds of thousands of processors. In this context, planning for massive amounts of data and computation is complex but necessary. A well-designed and effective parallel system can not only improve training efficiency but also save significant amounts of electricity. Therefore, optimizing parallel training methods to improve system throughput while maintaining deep learning model performance has become a pressing issue for the industry.

[0004] If the model size exceeds the memory limit of a single device, the model parameters need to be distributed across multiple nodes. During computation, the required model parameters are retrieved via ensemble communication to address the Out of Memory (OOM) problem. This method is called "Fully Sharded Data Parallel (FSDP)". Using FSDP, the model is first divided into multiple "partitions," the number of which corresponds to the number of GPUs. Each GPU is responsible for storing a parameter segment (not contiguous, not necessarily a complete partition; for a given partition, its parameters may be divided into multiple segments stored by multiple GPUs) and the corresponding model state. The parameter segment is updated by the GPU responsible for that segment during iteration. FSDP divides the forward / backward computation into multiple stages according to each partition. In each stage, GPUs use ensemble communication to collect non-locally stored parameter segments from the partition, reproduce the complete partition, perform computation, and immediately discard the non-local parameters after completion. In this way, each GPU computes the complete model in one iteration, but only the number of parameters the size of one partition needs to be stored in memory at any given time, improving the scalability for training model size.

[0005] In FSDP, there is an issue of insufficient GPU memory utilization. During model training, experts first calculate the amount of GPU memory required by the model to provide sufficient hardware storage. However, current GPU memory specifications are fixed, and the resources provided may exceed actual needs. This results in some GPU memory space remaining underutilized after model deployment, a phenomenon known as "resource overselling." Figure 1 This indicates the memory usage during FSDP training. It's difficult for the model size to perfectly match the given GPU memory capacity, and there may be unused memory addresses of varying lengths (for example, if a training iteration calculates that the model requires 72GB of GPU memory, but an NVIDIA A100 80GB card is used, then 8GB of GPU memory will remain idle during training). FSDP does not have a suitable adaptive method for using the excess GPU memory resources, thus failing to fully utilize the GPU's performance. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a method and system for full-slice data parallel optimization oriented to distributed machine learning, which improves upon existing technologies and increases the system throughput of full-slice data parallel FSDP.

[0007] To achieve the above objectives, the technical solution adopted by this invention is as follows: First, a method for parallel optimization of fully sliced ​​data for distributed machine learning is provided, comprising the following steps:

[0008] S1 memory capacity test:

[0009] Calculate the total memory requirement M during model training based on the number of model parameters and accuracy. req ;

[0010] S2 model partitioning:

[0011] Based on the number of GPUs N and the number of layers L in the model, the model's layers are partitioned into P1, P2, ..., P... i ,…,P N Partition P i The layer in the middle is denoted as L ij Where i = {1, 2, ..., N}, j = {1, 2, ..., L / N};

[0012] S3 configuration priority:

[0013] Set the elasticity coefficient Limit for each layer L ij Setting the priority of one-to-one mapping Pri ij ;

[0014] S4 buffer initialization:

[0015] Within each GPU, a priority queue based on a min-heap is used as a cache B. n During initialization, the caches in each GPU are empty;

[0016] S5 priority is dynamically updated:

[0017] During model training, each partition P is processed sequentially. i Forward computation is performed, and each GPU collects non-local layers as input buffers for candidate cache layers; based on the current computation partition P i Assess the urgency of parameter usage and improve partition P. i+Δ Layer L (i+Δ)j Priority:

[0018] Pri (i+Δ)j =Pri (i+Δ)j +γ / Δ;

[0019] Where γ∈(0,1) represents the urgency of parameter use, and Δ∈[1,N) represents the relationship between any partition and the current computation partition P. i The sequence distance, the smaller Δ is, the better P i+Δ The urgency level of each layer increases as the number of layers increases, and so on, until i + Δ = N;

[0020] When i > N-Limit, consider the parameter utilization of the backpropagation process:

[0021] Pri (N+1-Δ)j =Pri (N+1-Δ)j +γ / Δ;

[0022] S6 Buffer Update:

[0023] When the candidate layer is input into the buffer, calculate the remaining video memory M of the buffer req = M ava - M allo , when the remaining video memory M req < M, release the existing element with the lowest priority in the buffer according to the priority obtained in step S5 until M req ≥ M; where M is the video memory size of the candidate layer, M ava is the inherent video memory size of the GPU, M allo is the allocated video memory size;[[ID=1十八]]

[0024] S7 Revenue Estimation and State Update:

[0025] After the model training ends, perform an α operation in the reinforcement learning state s to shorten the total training duration T of all GPUs.

[0026] Preferably, the number N of GPUs described in step S2 satisfies the following conditions:

[0027] M req ≤ N × M ava - Threshold,

[0028] where M req is the total required video memory of the model during training, M ava is the video memory of each graphics card GPU, and Threshold is the reserved threshold.

[0029] Preferably, the setting process of the priority Pri ij described in step S3 is:

[0030] The initial value of the priority Pri ij is 1;

[0031] When the layer L ij is at the end of the model sequence it belongs to, that is, satisfies , the corresponding priority Pri ij = Pri ij + 1;

[0032] When the layer L ij is at the end of the partition it belongs to, that is, satisfies , the corresponding priority Pri ij = Pri ij + 1.

[0033] Preferably, the reinforcement learning state s described in step S7 is represented as <ID, model, device num , device mem,Threhold,Limit,γ>, represent the training number, model sequence structure, and device configuration, respectively; the α operation is to fine-tune the memory reservation threshold Threshold, elasticity Limit, and urgency level γ, with adjustment granularities of ±0.5 (GB), ±1, and ±0.01, respectively.

[0034] A fully sliced ​​data parallel optimization system for distributed machine learning includes:

[0035] Memory capacity detection module:

[0036] Used to calculate the total memory requirement M during model training based on the number of model parameters and accuracy. req ;

[0037] Model partitioning module:

[0038] This is used to partition the model into layers P1, P2, ..., P based on the number of GPUs N and the number of layers L. i ,…,P N Partition P i The layer in the middle is denoted as L ij Where i = {1, 2, ..., N}, j = {1, 2, ..., L / N};

[0039] Configure priority module:

[0040] Used to set the elastic coefficient Limit, for each layer L ij Setting the priority of one-to-one mapping Pri ij ;

[0041] Buffer initialization module:

[0042] A priority queue based on a min-heap is used as a cache B within each GPU. n During initialization, the caches in each GPU are empty;

[0043] Priority dynamic update module:

[0044] Used during model training, for each partition P sequentially i Forward computation is performed, and each GPU collects non-local layers as input buffers for candidate cache layers; based on the current computation partition P i Assess the urgency of parameter usage and improve partition P. i+Δ Layer L (i+Δ)j Priority:

[0045] Pri (i+Δ)j =Pri (i+Δ)j +γ / Δ;

[0046] Among them, γ∈(0,1) is the parameter for the urgency of use, and Δ∈[1,N) is the sequence distance between any partition and the current calculation partition P i , and the smaller Δ is, the higher the urgency of the layers in P i+Δ , and so on until i + Δ = N;

[0047] When i > N - Limit, consider the parameter utilization in the backpropagation process:

[0048] Pri (N+1-Δ)j = Pri (N+1-Δ)j + γ / Δ;

[0049] Buffer update module:

[0050] When the candidate layer for caching is input into the buffer, calculate the remaining video memory M of the buffer req = M ava - M allo , and when the remaining video memory M req < M, release the element with the smallest priority in the buffer according to the priority obtained in step S5 until M req ≥ M; where M is the video memory size of the candidate layer for caching, M ava is the inherent video memory size of the GPU, and M allo is the allocated video memory size;

[0051] Benefit estimation and state update module:

[0052] After the model training is completed, the total GPU training duration T is used as the reinforcement learning benefit.

[0053] The above technical solutions have the following advantages or beneficial effects: i 43>The present invention aims at the FSDP method for distributed training, focuses on the problem of insufficient utilization of video memory resources, endows FSDP with the ability to adaptively perform storage and communication conversion, thereby improving system throughput. An elastic caching mechanism is established to store some non-local parameter copies, thus omitting the communication overhead required for subsequent transmission of this parameter segment, realizing the conversion of "space for time", and can significantly shorten the training completion time. The present invention reduces the technical threshold for distributed training users, optimizes the adaptability and flexibility of the current FSDP series methods, accelerates the task completion time of the FSDP method, and improves the portability of the parallel framework.

[0055] The above summary is only for the purpose of the specification and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present invention will be readily apparent by reference to the drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments disclosed in the invention and should not be construed as limiting the scope of the invention.

[0057] Figure 1 Memory usage during FSDP training;

[0058] Figure 2 A flowchart illustrating a full-slice data parallel optimization method for distributed machine learning provided by this invention;

[0059] Figure 3 This invention provides a system architecture diagram for a fully sliced ​​data parallel optimization system for distributed machine learning. Detailed Implementation

[0060] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0061] Firstly, this invention provides a full-slice data parallel optimization method for distributed machine learning, the process of which is as follows: Figure 2 As shown, it includes the following steps:

[0062] S1 memory capacity test:

[0063] Calculate the total memory requirement M during model training based on the number of model parameters and accuracy. req For example, training a model with 6 billion parameters requires 6 × 10^6 GPU memory at FP16 accuracy. 9 ×2÷1024÷1024÷1024≈12 (GB). Then, for different optimizers, calculate their corresponding memory requirements. The specific optimizer type needs to be provided. For example, using the ADAM optimizer, the total memory required for parameters, gradients, and optimizer states is 6 × 10^6 GB. 9 ×(2+2+12)÷1024÷1024÷1024≈96(GB).

[0064] S2 model partitioning:

[0065] Based on the number of GPUs N and the number of layers L in the model, the model's layers are partitioned into P1, P2, ..., P... i ,…,P N Each partition contains one or more layers, i.e., satisfying {N≤L|N,L∈N}, partition Pi The layer in the middle is denoted as L ij , where i = {1, 2, ..., N}, j = {1, 2, ..., L / N}.

[0066] The number N of GPUs satisfies the following condition:

[0067] M req ≤N×M ava -Threshold,

[0068] Among them, M req M represents the total memory required by the model during training. ava The threshold is the video memory of each graphics card GPU.

[0069] The model parameters are packaged and mapped onto the corresponding GPUs according to the user-provided wrapping policy. At this stage, wrapping is not done on a layer-by-layer basis; the parameters that make up a layer may be packaged onto different GPUs. Model training consists of multiple epochs, each including steps S3 to S7.

[0070] S3 configuration priority:

[0071] Set the elasticity coefficient Limit for each layer L ij Setting the priority of one-to-one mapping Pri ij ;

[0072] Priority ij The initial value is 1;

[0073] When layer L ij At the end of the model sequence, that is, satisfying At that time, the corresponding priority is Pri ij =Pri ij +1;

[0074] When layer L ij At the end of the partition, that is, satisfying At that time, the corresponding priority is Pri ij =Pri ij +1.

[0075] S4 buffer initialization:

[0076] Within each GPU, a priority queue based on a min-heap is used as a cache B. n, the storage elements are key-value pairs <k, v>, where k represents the priority and value is the initial address of the layer stored in the CPU memory (the buffer is in the form of a min-heap, with multiple copies maintained in each GPU memory, which is dynamic during the training process, and the value needs to be mapped to a unique layer, so a unique copy of the model parameters needs to be maintained in the CPU memory for unique identification). At initialization, the buffers in each GPU are empty; step S4 is performed once in each training round.

[0077] S5 Priority buffer dynamic update:

[0078] During the model training process of each round, forward calculations will be sequentially performed on each partition P i , and each GPU collects non-local layers as candidate layers for the buffer to input into the buffer; according to the current calculation partition P i , evaluate the urgency of parameter usage and increase the priority of the layer L i+Δ in partition P (i+Δ)j :

[0079] Pri (i+Δ)j = Pri (i+Δ)j + γ / Δ;

[0080] where γ ∈ (0, 1) is the urgency of parameter usage, and Δ ∈ [1, N) is the sequence distance between any partition and the current calculation partition P i , the smaller Δ is, the higher the urgency of the layers in P i+Δ , and so on until i + Δ = N;

[0081] When i > N - Limit, consider the parameter utilization in the backpropagation process:

[0082] Pri (N+1-Δ)j = Pri (N+1-Δ)j + γ / Δ;

[0083] S6 Buffer update:

[0084] When the candidate layer for the buffer inputs into the buffer, calculate the remaining video memory M of the buffer req = M ava - M allo , when the remaining video memory M req < M, according to the priority obtained in step S5, release the element with the smallest priority in the buffer until M req ≥ M; where M is the video memory size of the candidate layer for the buffer, M ava is the inherent video memory size of the GPU, and M allo is the allocated video memory size; in each training round, loop steps S5 - S6 until the calculation of each partition is completed.

[0085] S7 Profit Estimation and Status Update:

[0086] After the model training is completed, the α operation is performed in the reinforcement learning state s, which changes the value of a certain constraint and shortens the total training time T of all GPUs.

[0087] The state s in reinforcement learning is represented as <ID,model,device num device mem ,Threhold,Limit,γ>, represent the training number, model sequence structure, and device configuration (number and memory size), respectively. The α operation is to fine-tune the memory reservation threshold Threshold, elasticity limit, and urgency level γ, with adjustment granularities of ±0.5 (GB), ±1, and ±0.01, respectively. Step S7 is performed once in each training round.

[0088] After multiple training epochs, the algorithm fine-tunes the state parameters and gradually converges. Finally, according to the user settings, after the model has gone through multiple epochs, the process ends regardless of whether the state parameters have converged.

[0089] Secondly, the present invention provides a fully sliced ​​data parallel optimization system for distributed machine learning, comprising:

[0090] Memory capacity detection module:

[0091] Used to calculate the total memory requirement M during model training based on the number of model parameters and accuracy. req ;

[0092] Model partitioning module:

[0093] This is used to partition the model into layers P1, P2, ..., P based on the number of GPUs N and the number of layers L. i ,…,P N Partition P i The layer in the middle is denoted as L ij Where i = {1, 2, ..., N}, j = {1, 2, ..., L / N};

[0094] Configure priority module:

[0095] Used to set the elastic coefficient Limit, for each layer L ij Setting the priority of one-to-one mapping Pri ij ;

[0096] Buffer initialization module:

[0097] A priority queue based on a min-heap is used as a cache B within each GPU. n, at initialization, the buffers in each GPU are all empty;

[0098] Priority dynamic update module:

[0099] During the model training process, it is used to perform forward calculations on each partition P in turn. Each GPU collects non-local layers as candidate layers for caching and inputs them into the buffer; according to the current calculation partition P i , evaluate the urgency of parameter usage, and increase the priority of the layer L in partition P i : i+Δ Layer L of (i+Δ)j Priority:

[0100] Pri (i+Δ)j = Pri (i+Δ)j + γ / Δ;

[0101] Among them, γ∈(0,1) is the urgency of parameter usage, and Δ∈[1,N) is the sequence distance between any partition and the current calculation partition P i . The smaller Δ is, the higher the urgency of the layers in P i+Δ , and so on until i + Δ = N;

[0102] When i > N - Limit, consider the parameter utilization in the backpropagation process:

[0103] Pri (N+1-Δ)j = Pri (N+1-Δ)j + γ / Δ;

[0104] Buffer update module:

[0105] When the candidate layer for caching is input into the buffer, it is used to calculate the remaining video memory M of the buffer req = M ava - M allo . When the remaining video memory M req < M, according to the priority obtained in step S5, release the element with the smallest priority in the buffer until M req ≥ M; where M is the video memory size of the candidate layer for caching, M ava is the inherent video memory size of the GPU, and M allo is the allocated video memory size;

[0106] Benefit estimation and state update module:

[0107] After the model training is completed, the total training time T of all GPUs is used as the reinforcement learning benefit.

[0108] In a third aspect, the present invention provides a specific embodiment of a full-slice data parallel optimization method for distributed machine learning, including:

[0109] Suppose we are training a model on a device cluster with 3 available GPUs and 32GB of memory. The model consists of 9 layers, partitioned as follows: layers 1-3, 4-6, and 7-9. Assume the basic parameters are: Threhold = 2, Limit = 4, γ = 0.1.

[0110] Case 1:

[0111] To simplify the demonstration, we assume that the number of parameters in each layer is consistent and assess the hardware capacity. Each layer's parameters occupy 1GB. With the ADAM optimizer, each layer requires 8GB of GPU memory, for a total model memory requirement of 72GB. 72 / 3 < 32-2, so training is feasible. The model layers were deployed on various GPUs, and the results are shown in Table 1.1:

[0112] Table 1.1 GPU Storage Parameter Segment Comparison Table - 1

[0113] GPU 0 1 2 Layer 1，4，7 2，5，8 3，6，9

[0114] Initialize priorities and configurations, determine the partition and sequence positions of each layer, and allocate priorities according to the content described in step S3 of this invention. The correspondence between the model's layers and priorities is shown in Table 1.2:

[0115] Table 1.2 Model Layer Priority Comparison Table - 1

[0116] Priority 1 1 2 1 1 3 2 2 3 Layer 1 2 3 4 5 6 7 8 9

[0117] Training begins, and forward computation is performed on partition P1. At this time, GPU0 collects non-local layers L2 and L3 as candidate cache layers. The cache memory usage is 3*8 + 2*1 = 26GB, and the remaining GPU memory is 32 - 2 - 26 = 4GB, meaning the cache can cache four more non-local layers. The priority is then adjusted according to the content in step S5 of this invention, and the adjustment results are shown in Table 1.3:

[0118] Table 1.3 Model Layer Priority Comparison Table - 2

[0119] Priority 1.025 1.025 2.025 1.15 1.15 3.15 2.15 2.15 3.15 Layer 1 2 3 4 5 6 7 8 9

[0120] Forward computation is performed on partition P2, at which point GPU0 collects non-local layers L5 and L6 as cache candidate layers;

[0121] Forward computation is performed on partition P2, at which point GPU0 collects non-local layers L8 and L9 as cache candidate layers;

[0122] The priority update at this point is shown in Table 1.4:

[0123] Table 1.4 Model Layer Priority Comparison Table - 3

[0124] Priority 1.075 1.075 2.075 1.25 1.25 3.25 2.45 2.45 3.45 Layer 1 2 3 4 5 6 7 8 9

[0125] At this point, the remaining video memory in the cache is less than the video memory size of the cache candidate layer, so the existing elements with the lowest priority in the cache, namely L2 and L5, need to be released.

[0126] After performing the same operation on GPU1 and GPU2, the layers stored in the GPU at this time are shown in Table 1.5:

[0127] Table 1.5 GPU Storage Parameter Segment Comparison Table - 2

[0128]

[0129] Backpropagation then begins, calculating partitions P3, P2, and P1 sequentially without updating priorities. For GPU0 and GPU1, L5 and L4 need to be collected when calculating partition P2, and L2 and L1 need to be collected when calculating partition P1. No collection is required for GPU2. The first iteration ends, and the reinforcement learning algorithm attempts to update the basic parameter Limit = 5.

[0130] Case 2:

[0131] Following the results of Case 1, the basic parameters are as follows: Threhold = 2, Limit = 5, γ = 0.1.

[0132] After the new round of forward computation is completed, the priority is updated as shown in Table 2.1:

[0133] Table 2.1 Model Layer Priority Comparison Table - 4

[0134] Priority 1.075 1.075 2.075 1.25 2.25 3.25 2.45 2.45 3.45 Layer 1 2 3 4 5 6 7 8 9

[0135] The layers stored in the GPU at this time are shown in Table 2.2:

[0136] Table 2.2 GPU Storage Parameter Segment Comparison Table - 3

[0137]

[0138] Unlike Case 1, GPU0 stores all parameters of P2 and P3, so it does not need to be collected during backpropagation, while GPU1 still needs to collect L4 during backpropagation. GPU0 saves communication time, reducing the total training time T, which is a positive gain, and the reinforcement learning algorithm continues to update the basic parameter Limit = 6.

[0139] In the new round, after the forward computation is completed, the priority is updated as shown in Table 2.3:

[0140] Table 2.3 Model Layer Priority Comparison Table - 5

[0141] Priority 1.075 1.075 2.075 2.25 2.25 3.25 2.45 2.45 3.45 Layer 1 2 3 4 5 6 7 8 9

[0142] The layers stored in the GPU at this time are shown in Table 2.4:

[0143] Table 2.4 GPU Storage Parameter Segment Comparison Table - 4

[0144]

[0145] Both GPU0 and GPU2 store all parameters of P2 and P3, so they do not need to be collected during backpropagation. This further reduces the total training time T, resulting in a positive gain, and the basic parameter Limit is updated to 7.

[0146] In the next round, after the forward computation is completed, the priority is updated as shown in Table 2.5:

[0147] Table 2.5 Model Layer Priority Comparison Table - 6

[0148] Priority 1.075 1.075 3.075 2.25 2.25 3.25 2.45 2.45 3.45 Layer 1 2 3 4 5 6 7 8 9

[0149] This situation is consistent with the GPU storage layer in Case 1. We can see that reducing the total duration T to the state in Case 1 results in a negative return. Restoring Limit = 6 leads to state convergence. This case study concludes.

[0150] The above technical solution has the following advantages or beneficial effects: This invention uses a parameter buffer to store model parameter copies in a limited manner, thereby reducing additional communication overhead in subsequent iterations. The scheduling algorithm can determine the release priority of parameter copies based on the existing cached parameters in the current GPU memory and the urgency of the parameters needed later. The parameter buffer can utilize limited GPU memory resources to store an appropriate amount of model parameters that will be called soon, according to a given priority, to avoid potentially high-latency communication needs.

[0151] This invention addresses the issue of insufficient utilization of GPU memory resources in the FSDP method for distributed training. It endows FSDP with the ability to adaptively convert between storage and communication, thereby improving system throughput. An elastic caching mechanism is established to store copies of some non-local parameters, thus eliminating the communication overhead required for subsequent transmission of these parameter segments, achieving a "space-for-time" trade-off, and significantly shortening training completion time. This invention reduces the technical barrier for distributed training users, optimizes the adaptability and flexibility of current FSDP methods, accelerates the task completion time of FSDP methods, and improves the portability of parallel frameworks.

[0152] In the description of this specification, references to terms such as "an embodiment," "a preferred embodiment," "an example," "a specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0153] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more executable instructions for implementing a particular logical function or process, and the scope of the preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as will be understood by those skilled in the art to which embodiments of the invention pertain.

[0154] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0155] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0156] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it includes one or a combination of the steps of the method embodiments. Since the device implementation corresponds to the method implementation, the description of the device implementation is relatively brief, and the relevant description can be found by referring to the description of the method implementation.

[0157] Furthermore, the functional units in the various embodiments of the present invention can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk, or an optical disk, etc.

[0158] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A full-slice data parallel optimization method for distributed machine learning, characterized in that, Includes the following steps: S1 memory capacity test: According to the model parameter quantity and the accuracy, calculate the total required memory M in the model training process req ; S2 model partitioning: According to the number N of GPUs and the number L of layers of the model, the layers of the model are partitioned to obtain P1, P2, …, P i ,…,P N , the layers in the partition P i ij , where i={1, 2, …, N}, j={1, 2, …, L / N}.​ S3 configuration priority: Set the elasticity coefficient Limit for each layer L ij Set the priority Pri of one-to-one mapping ij ; S4 buffer initialization: Each GPU has a priority queue based on a min-heap as a buffer B n At initialization, the buffer in each GPU is empty. S5 priority is dynamically updated: During the model training process, each partition P i is sequentially calculated forward, and each GPU collects the non-local layer as a cache candidate layer input buffer; according to the current calculation partition P i , the urgency of the evaluation parameter is used to improve the priority of the layer L i+Δ of the partition P (i+Δ)j . Pri (i+Δ)j = Pri (i+Δ)j + γ / Δ; Wherein, γ ∈ (0, 1) is a parameter using emergency degree, Δ ∈ [1, N) is the sequence distance between any partition and the current calculation partition P i The smaller Δ is, the higher the emergency degree of the layer in P i+Δ is, and so on until i+Δ=N. When i > N-Limit, consider the parameter utilization of the backpropagation process: Pri (N+1-Δ)j = Pri (N+1-Δ)j + γ / Δ; S6 cache update: When caching the input buffer of the candidate layer, calculate the remaining video memory M of the buffer req = M ava - M allo , when the remaining video memory M req < M, release the existing element with the lowest priority in the buffer according to the priority obtained in step S5 until M req ≥ M; where M is the video memory size of the caching candidate layer, M ava is the inherent video memory size of the GPU, and M allo is the allocated video memory size; S7 Profit Estimation and Status Update: After the model training is completed, perform the α operation in the reinforcement learning state s to shorten the total training time T of all GPUs.

2. The full-slice data parallel optimization method for distributed machine learning according to claim 1, wherein: The number N of GPUs mentioned in step S2 satisfies the following condition: M req ≤N×M ava -Threshold, Wherein, M req is the total demand of the model in the training process, M ava is the memory of each GPU, and Threshold is the reserved threshold.

3. The full slice data parallel optimization method for distributed machine learning of claim 1, wherein: The priority Pri in step S3 ij The setting procedure is: Priority Pri ij with an initial value of 1; When the layer L ij At the end of the sequence of models, i.e. when the condition the corresponding priority Pri ij = Pri ij + 1 ; When the layer L ij At the end of the partition in which it is located, i.e. when the corresponding priority Pri ij = Pri ij + 1.

4. The full slice data parallel optimization method for distributed machine learning of claim 1, wherein: The reinforcement learning state s in step S7 is represented as <ID, model, device num ,device mem , Threhold, Limit, γ> respectively representing training number, model sequence structure and device configuration; the α operation is to fine-tune the reserved threshold value of the display memory Threshold, the elastic amount Limit and the emergency degree amount γ, and the adjustment granularity is ±0.5 (GB), ±1, ±0.01 respectively.

5. A distributed machine learning oriented full-slice data parallel optimization system, characterized in that: include: Memory capacity detection module: A method for calculating total required memory M in a model training process according to model parameter quantity and accuracy req ; Model partitioning module: partitioning layers of the model according to the number N of GPUs and the number L of layers of the model to obtain P1, P2, …, PN i ,…,P N , the layers in the partition P i are recorded as L ij , where i={1, 2, …, N}, j={1, 2, …, L / N}. Configure priority module: For setting the elasticity coefficient Limit, for each layer L ij Setting the priority Pri of the one-to-one mapping ij ; Buffer initialization module: For each GPU, set up a priority queue based on a small root heap as a buffer B n At initialization, the buffer in each GPU is empty; Priority dynamic update module: For model training process, each partition P i is sequentially calculated forward, and each GPU collects non-local layers as cache candidate layer input buffer; according to the current calculation partition P i , the priority of the layer L i+Δ of the partition P (i+Δ)j is improved using the urgency of parameter evaluation: Pri (i+Δ)j = Pri (i+Δ)j + γ / Δ; wherein γ ∈ (0, 1) is a parameter using emergency degree, Δ ∈ [1, N) is the sequence distance between any partition and the current computing partition P i The smaller Δ is, the higher the emergency degree of the layer in P i+Δ is, and so on until i + Δ = N. When i > N-Limit, consider the parameter utilization of the backpropagation process: Pri (N+1-Δ)j = Pri (N+1-Δ)j + γ / Δ; Cache update module: For calculating the remaining video memory M of the buffer when caching the candidate layer input buffer req = M ava - M allo When the remaining video memory M req < M, according to the priority obtained in step S5, release the existing element with the smallest priority in the buffer until M req ≥ M; where M is the video memory size of the caching candidate layer, M ava is the inherent video memory size of the GPU, and M allo is the allocated video memory size; Profit Estimation and State Update Module: The total training time T across all GPUs after model training is completed is used as the reinforcement learning gain.

Images