Pipeline parallel training-oriented scheduling strategy determination method and system

Abstract

The application provides a scheduling strategy determination method and system for flow-to-line parallel training, and belongs to the technical field of cloud computing. The method comprises the following steps: determining a plurality of candidate scheduling strategies according to a training sample set and attribute information of each computing node, wherein the candidate scheduling strategies comprise a sample splitting scheme for uniformly splitting the training sample set into a plurality of training sample subsets and a corresponding maximum K value, and the maximum K value is used to indicate the maximum value of the number of subsets when each computing node performs one forward calculation or backward calculation under the corresponding sample splitting scheme; generating a task image corresponding to each candidate scheduling strategy based on the sample splitting scheme in each candidate scheduling strategy and the association relationship between the computing nodes; and determining a target scheduling strategy with the shortest training duration according to the current network state, the task image corresponding to each candidate scheduling strategy and the maximum K value. The application can improve the training efficiency of the model in the case of network resource preemption.

Description

Technical Field

[0001] This application relates to the field of cloud computing technology, and in particular to a method and system for determining scheduling strategies for parallel training in a flow direction. Background Technology

[0002] With the continuous development of deep learning, the scale of deep learning models is becoming increasingly larger. For training large models, distributed training methods are particularly crucial. Pipeline parallel training, as an important distributed training method, divides the deep learning model into multiple interconnected training stages (referred to as stages) by layer, and then deploys multiple stages on multiple computing nodes. By employing a reasonable scheduling strategy, the computation order of multiple computing nodes is controlled, thereby completing the training task of the deep learning model.

[0003] Currently, when using pipelined parallel training, the training sample set for each round is typically divided into multiple training sample subsets, and each computing node performs training according to a 1F1B scheduling strategy. Here, F (forward) refers to the forward computation stage of deep learning training, and B (backward) refers to the backward computation stage. The 1F1B scheduling strategy means that each computing node performs forward or backward computation on only one training sample subset at a time. Using the 1F1B scheduling strategy results in higher network resource consumption, leading to lower model training efficiency in scenarios with network resource contention.

[0004] Therefore, there is an urgent need to provide a new method for determining scheduling strategies for pipelined parallel training, so as to achieve high model training efficiency even in scenarios where network resources are preempted. Summary of the Invention

[0005] This application provides a method and system for determining scheduling strategies for pipelined parallel training, which can achieve high model training efficiency in scenarios involving network resource contention. The technical solution is as follows:

[0006] Firstly, a method for determining scheduling strategies for pipelined parallel training is provided, which is applied to network resource preemption scenarios. The method includes:

[0007] Based on any batch of training sample set and the attribute information of each computing node, multiple candidate scheduling strategies are determined. The computing nodes are deployed with training stages divided by the deep learning model. The candidate scheduling strategies are used to indicate the training method of each computing node to train the deep learning model based on the training sample set. The candidate scheduling strategies include a sample splitting scheme that evenly splits the training sample set into multiple training sample subsets and the corresponding maximum K value. The maximum K value is used to indicate the maximum number of subsets when each computing node performs a forward or backward computation under the corresponding sample splitting scheme. The maximum K value is greater than or equal to 2.

[0008] Based on the sample splitting scheme and the relationship between each computing node in each candidate scheduling strategy, a task image corresponding to each candidate scheduling strategy is generated.

[0009] Based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy, the target scheduling strategy with the shortest training time is determined from multiple candidate scheduling strategies. The target scheduling strategy is used to control the computation process of each computing node.

[0010] Secondly, a scheduling strategy determination device for pipelined parallel training is provided. This device is applied to network resource preemption scenarios and includes:

[0011] The first determining module is used to determine multiple candidate scheduling strategies based on any batch of training sample sets and the attribute information of each computing node. The computing nodes are deployed with training stages divided by the deep learning model. The candidate scheduling strategies are used to indicate the training method of each computing node to train the deep learning model based on the training sample set. The candidate scheduling strategies include a sample splitting scheme that evenly splits the training sample set into multiple training sample subsets and the corresponding maximum K value. The maximum K value is used to indicate the maximum number of subsets when each computing node performs a forward or backward computation under the corresponding sample splitting scheme. The maximum K value is greater than or equal to 2.

[0012] The generation module is used to generate task images corresponding to each candidate scheduling strategy based on the sample splitting scheme in each candidate scheduling strategy and the correlation between each computing node;

[0013] The second determining module is used to determine the target scheduling strategy with the shortest training time from a variety of candidate scheduling strategies based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy. The target scheduling strategy is used to control the calculation process of each computing node.

[0014] Thirdly, a pipelined parallel training system is provided, the system including a scheduling strategy determination device for pipelined parallel training and multiple computing nodes;

[0015] The scheduling strategy determination device is used to execute the scheduling strategy determination method for pipelined parallel training as described in the first aspect.

[0016] Each computing node is used to receive the target scheduling policy sent by the scheduling policy determination device and to train the model according to the target scheduling policy.

[0017] Fourthly, a computing device is provided, including a processor and a memory; the memory stores at least one piece of program code; the at least one piece of program code is used to be called and executed by the processor to implement the scheduling strategy determination method for pipelined parallel training as described in the first aspect.

[0018] Fifthly, a computer-readable storage medium is provided, wherein at least one computer program is stored therein, and when executed by a processor, the at least one computer program is capable of implementing the scheduling strategy determination method for pipelined parallel training as described in the first aspect.

[0019] In a sixth aspect, a computer program product is provided, the computer program product comprising a computer program, which, when executed by a processor, is capable of implementing the scheduling strategy determination method for pipelined parallel training as described in the first aspect.

[0020] The beneficial effects of the technical solutions provided in this application are:

[0021] When training deep learning models using a pipelined parallel training approach, a fixed 1F1B scheduling strategy is not adopted. Instead, in each training round, the training sample set is split according to the number of samples used in that round, resulting in multiple sample splitting schemes. Each scheme has a different number of subsets and a different number of samples within each subset. Within the maximum storage space of each computing node, for the same sample splitting scheme, the number of samples within each subset is fixed. The more subsets of training samples each computing node can compute in parallel, the higher its computational efficiency. For different sample splitting schemes, the more samples within each subset, the fewer subsets that a computing node can compute in parallel. However, the more subsets that need to be computed in parallel, the fewer communication sessions between computing nodes and the lower the dependence on network state. This application uses the maximum storage space of each computing node as a constraint to search for the corresponding maximum K value for each sample splitting scheme, thereby obtaining multiple candidate scheduling strategies. Then, it obtains the training time of each candidate scheduling strategy under the current network state. This training time, while comprehensively considering the network state and the computing efficiency of each computing node, reduces the impact of the network state on the training efficiency. Thus, when selecting the target scheduling strategy with the shortest training time to control the training process of each computing node, it can also achieve high model training efficiency in the scenario of network resource contention, so that the entire model training process can maintain a high training level. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of a scheduling strategy determination device for pipelined parallel training provided in an embodiment of this application;

[0024] Figure 2 This is a flowchart of a scheduling strategy determination method for pipelined parallel training provided in an embodiment of this application;

[0025] Figure 3 This is a schematic diagram of a memory limit curve provided in an embodiment of this application;

[0026] Figure 4 This is a comparison diagram showing the advantages of a scheduling strategy determination method for pipelined parallel training provided in this application under a preemptive network.

[0027] Figure 5 This is a schematic diagram of a scheduling strategy determination device for pipelined parallel training provided in an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of a pipelined parallel training system provided in an embodiment of this application;

[0029] Figure 7 A structural block diagram of a computing device provided in an exemplary embodiment of this application is shown. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0031] It is understood that the terms "each," "multiple," and "any" used in the embodiments of this application, etc., mean that "multiple" includes two or more, "each" refers to each of the corresponding multiples, and "any" refers to any one of the corresponding multiples. For example, multiple words include 10 words, and "each word" refers to each of the 10 words, while "any word" refers to any one of the 10 words.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0033] Before implementing the embodiments of this application, the terms involved in the embodiments of this application will be explained first.

[0034] Forward computation (F) and backward computation (B) are the two computational stages in deep learning. Forward computation refers to the sequential computation process of processing training samples from input to output. Backward computation refers to the computation process of propagating the results of forward computation from output back to input. Essentially, backward computation uses the chain rule to sequentially calculate the gradients of the parameters of each layer. These calculated gradients are then used for gradient descent on the parameters of each layer, thereby updating the parameters of each layer until the model converges.

[0035] A computing node is a node used to train a deep learning model using a distributed training approach. Each computing node includes at least one computing device, and each computing device has a computing card (such as a GPU (Graphics Processing Unit)). Distributed training methods include data-parallel computing and model-parallel computing. In data-parallel training scenarios, the computing node deploys the complete deep learning model, and each computing node is used to process a portion of the training samples. In model-parallel computing scenarios, the deep learning model is divided into layers, and each computing node deploys a portion of the deep learning model's layers, with each computing node used to compute all the training samples.

[0036] A task graph is a graph that describes the relationships between tasks. A task graph is usually a directed graph, where nodes represent tasks, node values ​​represent the computational cost of the task, directed edges represent dependencies or communication relationships between tasks, and the weights of directed edges represent the communication cost.

[0037] A batch is the set of training samples used in one round of model training, which can be obtained by splitting the entire training sample set. Each batch contains multiple training samples, which are independent of each other and participate in the same round of training simultaneously. When all the training samples in a batch have been computed, it is said that one round of training of the model is completed.

[0038] A micro-batch is a smaller batch obtained by further splitting the training samples in a batch. Simply put, a micro-batch is a subset of a batch; a batch contains multiple micro-batches, and different micro-batches belonging to the same batch participate in the same training round. A micro-batch is the smallest unit in which a computing node performs a single forward or backward computation; that is, a computing node performs forward or backward computations on the training samples in units of micro-batches.

[0039] A bubble is a metric for measuring the computational efficiency of pipelined parallel training. Typically, pipelined parallel training allows multiple computing devices to perform calculations in parallel over time. When a computing device is idle, it becomes a "bubble" in terms of time. By calculating the percentage of bubbles in a given round, the computational efficiency of pipelined parallel training can be determined. Generally, a higher percentage of bubbles indicates lower computational efficiency.

[0040] A stage (training phase) is a computational unit in pipelined parallel training. When using pipelined parallel training, the deep learning model needs to be divided into multiple sequentially connected stages by layers. The training of a deep learning model includes forward computation and backward computation stages. The forward computation stage corresponds to the backward computation stage; that is, the forward and backward computation stages use the same model layers, only the data propagation direction differs. To facilitate control of the forward and backward computation stages, the deep learning model is typically divided for different computation stages, resulting in multiple forward stages and multiple backward stages. Forward stages and backward stages at the same layer within the deep learning model correspond to each other, and these corresponding forward and backward stages are deployed on the same computation node, allowing for scheduling of the forward and backward stages on each computation node.

[0041] The 1F1B scheduling strategy is short for a 1-forward-stage + 1-backward-stage scheduling strategy. The 1F1B strategy means that each computing node deploys one forward stage and one backward stage, and the training task is completed by alternately scheduling these two stages. Since each computing node has one forward stage and one backward stage, and scheduling one forward stage (or backward stage) can only perform forward (or backward) computation on one micro-batch, the 1F1B scheduling strategy limits the number of micro-batches computed per computing node in a single forward or backward computation to one. Under the 1F1B scheduling strategy, the forward stage and backward stage deployed on each compute node are scheduled alternately. For the last compute node, the micro-batch computed by the alternately scheduled forward stage and backward stage belongs to the same micro-batch. That is, after the forward stage on the last compute node performs forward computation on a certain micro-batch, the backward stage on the last compute node is called to perform backward computation on the same micro-batch. For other compute nodes, the micro-batch computed by the alternately scheduled forward stage and backward stage belongs to different micro-batch. That is, after the forward stage on another compute node performs forward computation on a certain micro-batch, when the compute node is called again for computation, the backward stage on that compute node will be called to compute other micro-batches.

[0042] Cloud computing refers to the delivery and usage model of IT (Information Technology) infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. According to operating models, cloud computing is divided into public cloud, private cloud, and hybrid cloud. Public cloud typically refers to a cloud provided by a third-party provider that is usable by users. Public cloud is generally accessible via the Internet and may be free or low-cost. The core attribute of public cloud is shared resource service, that is, providing services in an open public network. With the development of cloud computing technology, more deep learning models are being trained using computing resources provided by public clouds.

[0043] Deep learning is a general term for a class of pattern analysis methods that use deep neural networks to extract and transform features in a supervised or unsupervised manner, utilizing multi-layered nonlinear information. With the continuous development of deep learning, the scale of deep learning models is becoming increasingly larger. To shorten training time and improve training efficiency, pipelined parallel training is typically employed. Pipeline parallel training requires dividing the deep learning model into multiple stages and deploying these stages across multiple computing nodes. In the forward computation process, the computation of the subsequent forward stage depends on the computation results of the previous forward stage; in the backward computation process, the computation of the subsequent backward stage depends on the computation results of the previous backward stage. Therefore, it is necessary to traverse multiple computing nodes and transfer data between them. Thus, the training efficiency of a deep learning model depends not only on the computational efficiency of each computing node (which in turn depends on the uniformity between stages and the amount of data transferred between stages), but also on the network bandwidth between the computing nodes.

[0044] However, in public cloud scenarios, network resources are shared by various computing devices and applications on the cloud, making it impossible to exclusively allocate network resources to individual computing nodes used for training deep learning models. This means that the network bandwidth between computing nodes may be lower than the bandwidth between individual computing cards within each computing node. When data is transmitted across computing nodes, network congestion can occur if other computing devices or applications use network resources, thus slowing down communication between computing nodes. While the 1F1B scheduling strategy requires less data to be cached per computing node, and the cached forward computation results are released promptly after a micro-batch completes its forward computation, alleviating memory pressure on each computing node, the preemptive nature of network resources leads to a large number of buffers on computing nodes during pipeline parallel training, resulting in lower model training efficiency. Furthermore, because network states frequently change, it is impossible to predict the network state throughout the entire model training process; therefore, the 1F1B scheduling strategy cannot achieve high training efficiency.

[0045] To achieve higher training efficiency in scenarios where network resources are scarce, this application improves upon the original 1F1B scheduling strategy by proposing a KFKB scheduling strategy when training deep learning models using a pipelined parallel training method. The KFKB scheduling strategy treats K forward stages or K backward stages as a single scheduling unit (K groups are indivisible) and schedules them alternately. Since the KFKB strategy schedules K forward stages or K backward stages as a group for simultaneous scheduling, each computing node can compute K micro-batches during a single forward or backward computation. Here, K is the grouping value, representing the number of micro-batches that each computing node performs in a single forward or backward computation. K takes values ​​greater than or equal to 1, with different K values ​​corresponding to different scheduling strategies. For example, when K = 1, the KFKB scheduling strategy is a 1F1B strategy; when K = 2, the KFKB scheduling strategy is a 2F2B strategy. By adopting the KFKB scheduling strategy (when K is greater than or equal to 2), each computing node does not need to wait for the data it depends on to be transmitted. It can perform calculations based on data without dependencies or data that has already been prepared. This allows computing nodes to be in a computing state as much as possible, thereby alleviating the dependence on network resources, reducing the bucket of computing nodes, and improving the computing efficiency of computing nodes.

[0046] Based on the proposed KFKB scheduling strategy, the embodiments of this application employ a pipelined parallel training method to train the deep learning model, including the following two processes:

[0047] The first process: Determining the scheduling strategy

[0048] When training a deep learning model in a certain round, multiple candidate scheduling strategies are determined based on the training sample set of that batch and the attribute information of each computing node. Each scheduling strategy includes a sample splitting scheme that evenly splits the training sample set into multiple training sample subsets and the corresponding maximum K value. Then, based on the current network state, the target scheduling strategy with the shortest training time under the current network state is selected.

[0049] The second process: Adjusting the scheduling strategy based on network status.

[0050] Based on the target scheduling strategy determined in the first process, the network state of the current network is monitored in each subsequent training round, and the scheduling strategy of each round is adaptively adjusted according to the current network state, so as to maintain high pipeline performance in network preemption scenarios.

[0051] Figure 1 This application provides a scheduling strategy determination device for pipelined parallel training. The device includes an adaptive grouper, a task image generator, and an adaptive grouping scheduler. The output of the adaptive grouper is connected to the input of the task image generator, and the output of the task image generator is connected to the input of the adaptive grouping scheduler.

[0052] The Adaptive Grouper Pass is used to determine multiple candidate scheduling strategies based on the training sample set and the attribute information of each computing node. The attribute information of the computing nodes includes the number of computing devices included in each node, the storage space of each computing device (e.g., GPUs), and the stages deployed on the computing node. When determining multiple candidate scheduling strategies, the Adaptive Grouper Pass uniformly splits the training sample set according to the number of samples, resulting in multiple sample splitting schemes. Each sample splitting scheme includes the number of subsets of the split training sample set and the number of samples within each subset. Then, based on the storage space of each computing node, it searches for the maximum K value corresponding to each splitting scheme. This maximum K value is greater than or equal to 2. A sample splitting scheme and its corresponding maximum K value are then combined to form a candidate scheduling strategy. Generally speaking, the more samples within a subset of the training sample set under each sample splitting scheme, the smaller the corresponding maximum K value; conversely, the fewer samples within a subset of the training sample set under each sample splitting scheme, the larger the corresponding maximum K value. For example... Figure 1The ratio of MBS (short for micro batch size) to K is inversely proportional. When the ratio of MBS is 6, the ratio of K is 1; when the ratio of MBS is 4, the ratio of K is 2; and when the ratio of MBS is 2, the ratio of K is 4.

[0053] The Task Graph Builder generates task graphs for each candidate scheduling strategy based on the sample splitting scheme and the relationships between computing nodes.

[0054] The Adaptive Grouper scheduler is used to determine the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy. Then, the determined target scheduling strategy is sent to each computing node so that each computing node can train the model according to the target scheduling strategy.

[0055] In this embodiment, the adaptive group scheduler includes a scheduling plan generation module (SchedulePlanner), an automatic adjustment module (AutoTunner), and a coordination module (Coordinator). The input of the scheduling plan generation module is connected to the output of the task image generator, the output of the scheduling plan generation module is connected to the input of the automatic adjustment module, and the output of the automatic adjustment module is connected to the input of the coordination module. The scheduling plan generation module generates a candidate scheduling plan for each candidate scheduling strategy based on the task image and the maximum K value corresponding to each candidate scheduling strategy. This candidate scheduling plan is the training plan for each computing node when executing the model training task according to the corresponding candidate scheduling strategy. The automatic adjustment module simulates the process of each computing node training the model according to the candidate scheduling plan corresponding to each candidate scheduling strategy based on the current network state, to obtain the training time of each candidate scheduling strategy, and then selects the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies. The coordination module sends the target scheduling strategy to each computing node.

[0056] In the embodiments of this application, see Figure 1 The device also includes an Auto Parallel Pass, whose output is connected to the input of an adaptive grouper. The Auto Parallel Pass determines multiple model splitting schemes based on the model information of the deep learning model and the attribute information of each computing device in the cluster. From these multiple model splitting schemes, the Auto Parallel Pass selects the target splitting scheme with the shortest training time. Based on the target splitting scheme, the Auto Parallel Pass determines the attribute information of each computing node.

[0057] In this embodiment, the device further includes a network state monitor. The output of the network state monitoring module is connected to the input of the automatic adjustment module. When training the model based on the next batch of training sample sets, the monitor responds to a monitoring command sent by the automatic adjustment module to monitor the current network state and sends the monitored network state to the automatic adjustment module. Based on the monitored network state, the automatic adjustment module reselects the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies and sends the reselected target scheduling strategy to the coordination module. The coordination module receives the reselected target scheduling strategy and sends it to each computing node, enabling each computing node to train the model according to the reselected target scheduling strategy.

[0058] based on Figure 1 The apparatus shown in this application provides a scheduling strategy determination method for pipelined parallel training. This method is applied to network preemption scenarios. See [link to relevant documentation]. Figure 2 The method flow provided in this application embodiment includes:

[0059] 201. Based on the model information of the deep learning model and the attribute information of each computing device used for model training, determine the attribute information of each computing node.

[0060] The deep learning model's information includes the number of layers, the relationships between layers, and the number of convolutional kernels in each layer. The attribute information of each computing device used for model training includes the number of computing devices, the GPU size of each device, and the network bandwidth between devices. The attribute information of a computing node includes the number of computing devices it contains, the GPU size of each device, and the stages deployed on that node. Based on the deep learning model's information and the attribute information of the computing devices used for model training, the following method can be used to determine the attribute information of each computing node:

[0061] 2011. Based on the model information of the deep learning model and the attribute information of each computing device used for model training, various model segmentation schemes were determined.

[0062] The model segmentation scheme indicates how the deep learning model is segmented and the corresponding computation nodes in the training phase after segmentation. Specifically, the following steps can be taken:

[0063] The first step is to divide the deep learning model into multiple stages based on the model information.

[0064] This step involves dividing the deep learning model according to the number of layers. Since deep learning models have at least two layers, there are multiple ways to divide them, and each method can result in multiple stages.

[0065] Based on the training characteristics of deep learning models, the training phase includes not only forward computation but also backward computation. To facilitate gradient descent based on the results of forward and backward computations for updating model parameters, after sequentially splitting the deep learning model layer by layer to obtain multiple forward stages, the model is then split in reverse order based on these forward stages to obtain multiple backward stages. Each forward stage has a corresponding backward stage, and the layers included in the corresponding forward and backward stages are mirror images of each other. Typically, the corresponding forward and backward stages are deployed on the same computing node, which is beneficial for storing and updating model parameters.

[0066] The second step is to assign a corresponding computing device to each stage based on the attribute information of each computing device used for model training and the stage being segmented, for each segmentation method.

[0067] In one possible implementation, if the number of computing devices is an integer multiple of the number of stages, then the corresponding computing devices can be allocated equally to each stage based on the number of stages. For example, if there are 8 computing devices used for model training, and the stage is divided into 2 stages, then 4 computing devices are allocated to each stage; if the stage is divided into 4 stages, then 2 computing devices are allocated to each stage.

[0068] In another possible implementation, computing devices can be allocated to each stage based on the computational resources required and the storage space of each computing device, such as the size of a GPU. For example, if a stage has a large number of layers and a large amount of computation per layer, it is determined that the stage requires more computational resources. Therefore, more computational resources are allocated to this stage, such as computing devices with larger storage space or more computing devices. For instance, if there are 8 computing devices used for model training, and the stage is divided into 2 stages, where stage 1 requires more computational resources than stage 2, then 5 computing devices can be allocated to stage 1 and 3 computing devices to stage 2.

[0069] Considering that the segmented stage includes a forward stage and a backward stage, and that the proportion of computing resources consumed by each forward stage in the forward computing stage is consistent with the proportion of computing resources consumed by the corresponding backward stage in the backward computing stage, that is, if a forward stage requires more computing resources, then the corresponding backward stage will also require more computing resources. Therefore, when allocating computing devices to each stage, the allocation can be based on the computing resources consumed by each stage in the forward computing stage, or it can be based on the computing resources consumed by each stage in the backward computing stage.

[0070] The third step is to group the computing devices corresponding to each stage under each partitioning method into a computing node, and then use each partitioning method and the computing nodes corresponding to each stage under the corresponding partitioning method as a model partitioning scheme.

[0071] 2012. From various model segmentation schemes, select the target segmentation scheme with the shortest training time.

[0072] Because different partitioning methods result in different stages for deep learning models, and each stage corresponds to different computational nodes, the subsequent training efficiency of the deep learning model also varies. Therefore, it is necessary to select the partitioning scheme with the highest training efficiency from multiple options. Training efficiency is typically reflected in training time; the longer the training time, the lower the model's training efficiency; conversely, the shorter the training time, the higher the model's training efficiency. Therefore, the target partitioning scheme selected based on training time is the one with the highest training efficiency. The training time for each partitioning scheme can be obtained by simulating the model's training process under different partitioning schemes.

[0073] 2013. Based on the target segmentation scheme, determine the attribute information of each computing node.

[0074] Once the target partitioning scheme is determined, the attribute information of each computing node under the target partitioning scheme can be determined based on the correspondence between each stage and computing node in the target partitioning scheme.

[0075] In this embodiment of the application, the deep learning model can be described using HLO (High-Level Optimization) instructions, so the computation node corresponding to each stage under the target segmentation scheme can be represented by HLOComputation.

[0076] 202. Based on the training sample set of any batch and the attribute information of each computing node, determine multiple candidate scheduling strategies.

[0077] The candidate scheduling strategy is used to indicate the training method for each computing node to train the deep learning model based on the training sample set. The candidate scheduling strategy includes a sample splitting scheme that uniformly divides the training sample set into multiple training sample subsets and the corresponding maximum K value. The maximum K value indicates the maximum number of subsets that each computing node can perform in a single forward or backward computation under the corresponding sample splitting scheme. When determining multiple candidate scheduling strategies based on any batch of training sample sets and the attribute information of each computing node, the following method can be used:

[0078] 2021. Based on the number of samples included in the training sample set, the training sample set is uniformly split to obtain multiple sample splitting schemes.

[0079] Each sample splitting scheme includes the number of subsets of the training sample set and the number of samples within each subset. Based on the number of samples in the training sample set, a factorization method is used to uniformly split the training sample set, resulting in multiple sample splitting schemes. Uniform splitting means that each subset of the training sample set contains the same number of samples. For example, if the training sample set contains 100 samples, there are several ways to uniformly split it: it can be split into 100 training sample subsets, each containing 1 sample; it can be split into 50 training sample subsets, each containing 2 samples; it can be split into 25 training sample subsets, each containing 4 samples; it can be split into 20 training sample subsets, each containing 5 samples; it can be split into 10 training sample subsets, each containing 10 samples; it can be split into 5 training sample subsets, each containing 20 samples; it can be split into 4 training sample subsets, each containing 25 samples; or it can be split into 2 training sample subsets, each containing 50 samples.

[0080] For each sample splitting scheme, the product of the number of subsets of the split training sample set and the number of samples within each subset equals the total number of samples in the training sample set. Let the size of the training sample set be denoted as `batch_size`, the number of subsets of the split training sample set be denoted as `num_micro_batches`, and the number of samples within each subset be denoted as `micro_batch_size`. Then, `micro_batch_size * num_micro_batches = batch_size`.

[0081] 2022. Using the maximum storage space of each computing node as a constraint, search for the maximum K value corresponding to each sample splitting scheme to obtain each candidate scheduling strategy.

[0082] Considering that for different sample splitting schemes, the larger the number of samples in the training sample subset, the larger the size of the input tensor, and the larger the GPU memory occupied by the computing node when computing on that training sample subset; for the same sample splitting scheme, the number of samples in the training sample subset, i.e., the micro_batch_size, is fixed. The more subsets of the training sample subset each computing node performs forward or backward computation at one time (i.e., the larger the K value), the greater the storage pressure on the computing node. Since the storage space of each computing node is limited, the maximum K value that each computing node can withstand when calling the forward stage or backward stage is different for different sample splitting schemes, and the resulting candidate scheduling strategies are also different. The maximum K value corresponding to each sample splitting scheme can be obtained by searching with the maximum storage space of each computing node as a constraint.

[0083] Specifically, the process for determining each candidate scheduling strategy is as follows: For each sample splitting scheme, based on the storage space of each computing node and the network bandwidth between different computing nodes, the processing process of each computing node in each training sample subset under each sample splitting scheme is simulated. In the process of simulating each computing node in processing the training samples in each training sample subset under each sample splitting scheme, if the maximum storage space of any computing node is not reached, the number of training sample subsets processed in parallel by each computing node is continuously increased until the maximum storage space of any computing node is reached. The number of training sample subsets processed by the computing node when the maximum storage space is reached is taken as the maximum K value corresponding to the corresponding sample splitting scheme, that is, the maximum K value corresponding to the number of samples in the training sample subset under the corresponding splitting scheme. Then, a sample splitting scheme and the corresponding maximum K value are combined to form a candidate scheduling strategy.

[0084] It should be noted that during the search for the maximum K value, for any computing node, if the computational cost in the previous round of search did not reach the maximum storage space, but the computational cost in the next round of search exceeds the maximum storage space, then the subset number of the training sample subset processed by that computing node in the previous round will be used as the maximum K value.

[0085] For example, for any sample splitting scheme, the number of training sample subsets in the scheme is 8. In the initial search, K=1 is selected, and a 1F1B scheduling strategy is used to simulate the computation process of each computing node on the training sample subset. That is, each computing node performs forward or backward computation on 1 training sample subset at a time. If the maximum storage space of any computing node is not reached, then K=2 is selected, and a 2F2B scheduling strategy is used to simulate the computation process of each computing node on the training sample subset. This can control each computing node, that is, each computing node performs forward or backward computation on 2 training sample subsets at a time. If the maximum storage space of a certain computing node is reached, then the maximum K value corresponding to this sample splitting method is determined to be 2.

[0086] The above method searches for the maximum K value for each sample splitting scheme using a fixed number of samples in a training sample subset. Alternatively, one can fix the K value and search for the maximum number of samples in the training sample subset corresponding to each K value. Since the number of samples in the training sample subset corresponding to each sample splitting scheme is fixed, once the maximum number of samples in the training sample subset corresponding to each K value is found, the maximum K value for each sample splitting scheme can be determined. The method for searching the maximum number of samples in the training sample subset corresponding to a fixed K value is similar to the method described above for searching the maximum K value for each sample splitting scheme; see the above content for details, which will not be repeated here.

[0087] The method described above for searching the maximum K value for each sample splitting scheme is also known as the Pareto front pruning method. Figure 3 The diagram illustrates the memory limit curve obtained using the Pareto front pruning method. Integer pairs on this curve represent solutions to candidate scheduling strategies when the maximum memory space of the compute node is reached. For example, point C in the diagram is on the curve, and the integer pair solution at point C represents the solution that reaches the upper bound of video memory. Integer pairs in the region below the curve represent feasible solutions to candidate scheduling strategies. For example, point A in the diagram is below the curve; although the integer pair solution at point A is feasible, the video memory is not fully utilized, and micro_batch_size or K should be increased to fully utilize the video memory. Integer pairs in the region above the curve represent infeasible solutions to candidate scheduling strategies, where the maximum video memory of the compute node is exceeded, and the compute node cannot perform the computation process.

[0088] It should be noted that the method provided in this application does not need to enumerate all feasible solutions when searching for the maximum K value corresponding to each sample splitting scheme. This reduces the pressure on subsequent generation of task images, evaluation of candidate scheduling strategies, and testing of network performance.

[0089] 203. Based on the sample splitting scheme in each candidate scheduling strategy and the correlation between each computing node, generate the task image corresponding to each candidate scheduling strategy.

[0090] Each computing node is treated as a task node on the task image. Based on the relationships between the computing nodes, a corresponding subtask image is generated for each subset of training samples under each candidate scheduling strategy. This results in multiple subtask images for each candidate scheduling strategy, with the number of subtask images being the same as the number of subsets of training samples split under each candidate scheduling strategy, and each subtask image being identical. For a training task, typically one task image is used to represent the computation process of each computing node. After obtaining multiple subtask images for each candidate scheduling strategy, these images need to be fused to obtain the task image corresponding to each candidate scheduling strategy.

[0091] 204. Based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy, determine the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies.

[0092] The training time of a model typically reflects its training efficiency; longer training times generally indicate lower efficiency, while shorter times indicate higher efficiency. When considering the training time for each candidate scheduling strategy, the target scheduling strategy with the shortest training time is selected from multiple candidate strategies; this target strategy is the one with the highest training efficiency. Specifically, when determining the target scheduling strategy with the shortest training time from multiple candidate strategies based on the current network state, the task image corresponding to each candidate scheduling strategy, and the maximum K value, the following method can be used:

[0093] 2041. Based on the task image and maximum K value corresponding to each candidate scheduling strategy, generate a candidate scheduling plan corresponding to each candidate scheduling strategy.

[0094] Each candidate scheduling strategy only indicates the direction of data flow when each computing node computes training samples within a subset of training samples. It does not specify the number of subsets or the computation order for each computing node. A training plan for each computing node to execute a specific model training task needs to be formulated based on the task image corresponding to each candidate scheduling strategy and the maximum K value. For any task image corresponding to a candidate scheduling strategy, if the maximum K value for that candidate scheduling strategy is 2, then the candidate scheduling plan instructs each computing node to compute the two training sample subsets or the two computation results from the previous forward stage in the forward computation phase, following the computation order on the task image. In the backward computation phase, it calls the forward stage twice to compute the two training sample subsets or the two computation results from the previous backward stage. In the backward computation phase, it calls the backward stage twice to compute the two forward computation results or the two computation results from the previous backward stage.

[0095] 2042. Based on the current network status, the training time for each candidate scheduling strategy is obtained by simulating the process of each computing node training the model according to the candidate scheduling plan corresponding to each candidate scheduling strategy.

[0096] Based on the maximum storage space of each computing node and the current network state, the process of model training for each computing node according to the candidate scheduling plan corresponding to each candidate scheduling strategy is simulated. By simulating the model training process, the computation time of each training sample under each candidate scheduling strategy can be estimated. Under the current network state, the communication process between each computing node and other computing nodes when executing the candidate scheduling plan corresponding to each candidate scheduling strategy is measured to obtain the communication time between each computing node and other computing nodes under each candidate scheduling strategy. Then, the overhead model is called to process the computation time of each training sample and the communication time between each computing node and other computing nodes under each candidate scheduling strategy to obtain the training time of each candidate scheduling strategy.

[0097] 2043. Select the target scheduling strategy with the shortest training time from a variety of candidate scheduling strategies.

[0098] Based on the training time of each candidate scheduling strategy, the target scheduling strategy with the shortest training time can be selected from a variety of candidate scheduling strategies.

[0099] Figure 4 The diagram shows a performance comparison between the KFKB (K=2) scheduling strategy provided in this application embodiment and the existing 1F1B scheduling strategy when performing pipelined parallel training. Figure 4The horizontal axis represents training time, the total length of the horizontal axis represents the training time (pipeline) using different scheduling strategies, the vertical axis represents different computing nodes, the number in each small box represents the micro-batch in the training sample set, and the blank space (bubble) in the figure indicates that the computing node is in an idle state.

[0100] Depend on Figure 4 It is evident that the training time using the 1F1B scheduling strategy is longer than that using the 2F2B scheduling strategy, and the number of bubbles is greater. The reason for this is as follows: In network preemption scenarios, the network communication transmission time between computing nodes cannot be ignored. Only by maximizing the operation of computations that do not depend on this communication can the communication time between computing nodes be overlapped, achieving a state where the network and computing nodes are as busy as possible simultaneously.

[0101] For the 1F1B scheduling strategy, since the network communication transmission time between computing nodes is not negligible, each micro-batch, after completing computation on the previous computing node, needs to wait for a period of time before being transmitted to the next computing node. The next computing node, having no computational tasks at this time, will be in an idle state. For example... Figure 4 After micro-batch 0 completes its computation on device 0, device 1 needs to wait for a period of time to receive the computation result for micro-batch 0 from device 0 before proceeding with further computation. For device 0, although there are no dependencies between micro-batches during the forward computation phase, it can compute uncomputed micro-batches whenever it is idle, thus overlapping the communication time between computation nodes. However, in the backward computation phase, due to the 1F1B principle, most backward stages need to wait for network transmission to finish before starting computation, resulting in a large number of buffers. This is even more pronounced for other devices, with numerous buffers forming in both the forward and backward computation phases.

[0102] For the 2F2B scheduling strategy, since two forward stages or two backward stages are grouped together, compared to the 1F1B scheduling strategy, although a backward stage cannot perform backward computation on its dependent input data because the input data is still being transmitted, the computing node can choose another data without dependencies or with ready input data for computation. In this way, all computing nodes will be fully busy, thereby overlapping the communication time between computing nodes and reducing the number of bubbles on each computing device during training.

[0103] Furthermore, after determining the target scheduling strategy with the shortest training time, this embodiment of the application also sends the target scheduling plan corresponding to the target scheduling strategy to each computing node, so that each computing node can train the model according to the target scheduling plan.

[0104] Considering the frequent changes in network conditions in public cloud scenarios, and the fact that the target scheduling plan determined in the above steps is based on the network conditions selected during the previous training round, the training efficiency of using the target scheduling plan may not be high in each subsequent training round due to changes in network conditions. Therefore, it is necessary to adjust the training plan based on the network conditions at each training round. When the number of samples in each batch of training samples is the same, since the sample splitting scheme and the attribute information of each computing node remain unchanged, it is unnecessary to repeat the process of determining the candidate scheduling plan corresponding to each candidate scheduling strategy. The selection can be made directly based on the candidate scheduling plan determined in the above steps. Specifically, when training the deep learning model based on the next batch of training sample sets, the network state of the current network is monitored. Then, based on the maximum storage space of each computing node and the current network state, the process of each computing node training the model according to the candidate scheduling plan corresponding to each candidate scheduling strategy is simulated. By simulating the model training process, the computation time of each training sample under each candidate scheduling strategy can be estimated. Under the current network state, the communication process between each computing node and other computing nodes when executing the candidate scheduling plan corresponding to each candidate scheduling strategy is measured to obtain the communication time of each computing node under each candidate scheduling strategy. Then, the overhead model is called to process the computation time of each training sample and the communication time of each computing node under each candidate scheduling strategy to obtain the training time of each candidate scheduling strategy. Finally, based on the determined training time of each candidate scheduling strategy, the target scheduling strategy with the shortest training time is reselected from multiple candidate scheduling strategies.

[0105] Of course, if the number of samples included in the training sample set in each round is different, the target candidate scheduling strategy with the shortest training time can be reselected for each round of training process according to the method shown in steps 202 to 204 above, which will not be elaborated here.

[0106] The method provided in this application can adjust the scheduling strategy in a timely manner based on the network state during each training round, rather than using a fixed scheduling strategy. Regardless of the current network state, it can achieve high model training efficiency throughout the entire training process. Especially in network preemption scenarios, when using pipelined parallel training to perform distributed training on large models, this application can maintain a good training level without requiring a supercomputing cluster, saving costs and making it possible to train large-scale models on public cloud service products.

[0107] The method provided in this application, when training a deep learning model using a pipelined parallel training approach, does not employ a fixed 1F1B scheduling strategy. Instead, during each training round, the training sample set is split according to the number of samples used in that round, resulting in multiple sample splitting schemes. Each scheme has a different number of subsets and a different number of samples within each subset. Within the maximum storage space of each computing node, for the same sample splitting scheme, the number of samples within each subset is fixed. The more training sample subsets each computing node can compute in parallel, the higher its computational efficiency. For different sample splitting schemes, the more samples within each subset, the fewer subsets each computing node can compute in parallel. Conversely, the more subsets that need parallel computation, the fewer communication sessions between computing nodes and the lower the dependence on network state. This application uses the maximum storage space of each computing node as a constraint to search for the corresponding maximum K value for each sample splitting scheme, thereby obtaining multiple candidate scheduling strategies. Then, it obtains the training time of each candidate scheduling strategy under the current network state. This training time, while comprehensively considering the network state and the computing efficiency of each computing node, reduces the impact of the network state on the training efficiency. Thus, when selecting the target scheduling strategy with the shortest training time to control the training process of each computing node, it can also achieve high model training efficiency in the scenario of network resource contention, so that the entire model training process can maintain a high training level.

[0108] Please refer to Figure 5This document illustrates a schematic diagram of a scheduling strategy determination device for pipelined parallel training, provided by an embodiment of this application. This device is used in network resource preemption scenarios and can be implemented through software, hardware, or a combination of both, becoming all or part of a computing device. The device includes:

[0109] The first determining module 501 is used to determine multiple candidate scheduling strategies based on any batch of training sample sets and the attribute information of each computing node. The computing node is deployed with training stages divided by the deep learning model. The candidate scheduling strategy is used to indicate the training method of each computing node to train the deep learning model based on the training sample set. The candidate scheduling strategy includes a sample splitting scheme that evenly splits the training sample set into multiple training sample subsets and the corresponding maximum K value. The maximum K value is used to indicate the maximum number of subsets when each computing node performs a forward or backward computation under the corresponding sample splitting scheme. The maximum K value is greater than or equal to 2.

[0110] The generation module 502 is used to generate a task image corresponding to each candidate scheduling strategy based on the sample splitting scheme in each candidate scheduling strategy and the correlation between each computing node.

[0111] The second determining module 503 is used to determine the target scheduling strategy with the shortest training time from a variety of candidate scheduling strategies based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy. The target scheduling strategy is used to control the computation process of each computing node.

[0112] In another embodiment of this application, the first determining module 501 is used to uniformly split the training sample set according to the number of samples included in the training sample set to obtain multiple sample splitting schemes. Each sample splitting scheme includes the number of subsets of the split training sample subsets and the number of samples in the training sample subsets. Using the maximum storage space of each computing node as a constraint, the maximum K value corresponding to each sample splitting scheme is searched to obtain each candidate scheduling strategy.

[0113] In another embodiment of this application, the first determining module 501 is used to continuously increase the number of training sample subsets processed in parallel by each computing node during the process of simulating each computing node processing training samples within each training sample subset under each sample splitting scheme, until the maximum storage space of any computing node is reached; the number of subsets calculated by the computing node when the maximum storage space is reached under each sample splitting scheme is taken as the maximum K value corresponding to the corresponding sample splitting scheme; and a sample splitting scheme and the corresponding maximum K value are combined to form a candidate scheduling strategy.

[0114] In another embodiment of this application, the generation module 502 is used to generate a corresponding subtask image for each training sample subset under each candidate scheduling strategy based on the association relationship between each computing node, thereby obtaining multiple subtask images corresponding to each candidate scheduling strategy. The number of subtask images is the same as the number of subsets of the training sample subset split under each candidate scheduling strategy. The multiple subtask images corresponding to each candidate scheduling strategy are then fused to obtain a task image corresponding to each candidate scheduling strategy.

[0115] In another embodiment of this application, the second determining module 503 is used to generate a candidate scheduling plan corresponding to each candidate scheduling strategy based on the task image and the maximum K value corresponding to each candidate scheduling strategy. The candidate scheduling plan is the training plan for each computing node to execute a specific model training task according to the corresponding candidate scheduling strategy. Based on the current network state, the training time of each candidate scheduling strategy is obtained by simulating the process of each computing node training the model according to the candidate scheduling plan corresponding to each candidate scheduling strategy. The target scheduling strategy with the shortest training time is selected from multiple candidate scheduling strategies.

[0116] In another embodiment of this application, the second determining module 503 is used to estimate the computation time of each training sample under each candidate scheduling strategy based on the maximum storage space of each computing node during the process of simulating model training of each computing node according to the candidate scheduling plan corresponding to each candidate scheduling strategy; under the current network state, by simulating the communication process between each computing node and other computing nodes when each computing node executes the candidate scheduling plan corresponding to each candidate scheduling strategy, the communication time of each computing node and other computing nodes under each candidate scheduling strategy is obtained; and by calling the overhead model, the computation time of each training sample and the communication time of each computing node under each candidate scheduling strategy are processed to obtain the training time of each candidate scheduling strategy.

[0117] In another embodiment of this application, the device further includes:

[0118] The sending module is used to send the target scheduling plan corresponding to the target scheduling strategy to each computing node, so that each computing node can train the model according to the target scheduling plan.

[0119] In another embodiment of this application, the device further includes:

[0120] The third determining module is used to determine multiple model segmentation schemes based on the model information of the deep learning model and the attribute information of each computing device used for model training. The model segmentation scheme is used to indicate the segmentation method of the deep learning model and the computing nodes corresponding to the training stage after segmentation. Each computing node includes at least one computing device.

[0121] The first selection module is used to select the target segmentation scheme with the shortest training time from multiple model segmentation schemes;

[0122] The fourth determination module is used to determine the attribute information of each computing node based on the target partitioning scheme.

[0123] In another embodiment of this application, the device further includes:

[0124] The monitoring module is used to monitor the current network status.

[0125] The second selection module is used to reselect the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies based on the monitored network status.

[0126] The sending module is used to send the reselected target scheduling policy to each computing node. The reselected target scheduling policy is used to control the computing process of each computing node when the deep learning model is trained on the next batch of training sample sets.

[0127] It should be noted that, Figure 1 and Figure 5 The scheduling strategy determination devices shown are both capable of selecting a suitable scheduling strategy for the current network in pipelined parallel training scenarios, thereby achieving higher model training efficiency in network preemption scenarios. Both have the same function, of course... Figure 1 The various devices included Figure 5 There must be certain relationships between the various modules included. Specifically, Figure 1 Adaptive grouper and Figure 5 The first determining module corresponds to the adaptive grouper, which may include the first determining module or may be the first determining module; Figure 1 The task image generator and Figure 5 The corresponding generation module in the text indicates that the task image generator may include or may be a generation module; Figure 1 The adaptive group scheduler in Figure 5 The second determining module and the sending module correspond to each other in the adaptive packet scheduler, which may include the second determining module and the sending module. Figure 1 Model splitter and Figure 5 The third determining module, the first selecting module, and the fourth determining module correspond to each other. The model segmenter may include the third determining module, the first selecting module, and the fourth determining module.

[0128] The apparatus provided in this application does not employ a fixed 1F1B scheduling strategy when training a deep learning model using a pipelined parallel training method. Instead, during each training round, it splits the training sample set according to the number of samples used in that round, resulting in multiple sample splitting schemes. Each scheme has a different number of subsets and a different number of samples within each subset. Within the maximum storage space of each computing node, for the same sample splitting scheme, the number of samples within each subset is fixed. The more training sample subsets each computing node can compute in parallel, the higher its computational efficiency. For different sample splitting schemes, the more samples within each subset, the fewer subsets each computing node can compute in parallel. Conversely, the more subsets that need parallel computation, the fewer communication sessions between computing nodes and the lower the dependence on network state. This application uses the maximum storage space of each computing node as a constraint to search for the corresponding maximum K value for each sample splitting scheme, thereby obtaining multiple candidate scheduling strategies. Then, it obtains the training time of each candidate scheduling strategy under the current network state. This training time, while comprehensively considering the network state and the computing efficiency of each computing node, reduces the impact of the network state on the training efficiency. Thus, when selecting the target scheduling strategy with the shortest training time to control the training process of each computing node, it can also achieve high model training efficiency in the scenario of network resource contention, so that the entire model training process can maintain a high training level.

[0129] This application provides a pipelined parallel training system, see [link to relevant documentation]. Figure 6 The system includes a scheduling strategy determination device 601 for pipelined parallel training and multiple computing nodes 602;

[0130] The scheduling strategy determines device 601 as follows: Figure 5 The apparatus shown is used to execute the above-described scheduling strategy determination method for pipelined parallel training.

[0131] Each computing node 602 is used to receive the target scheduling policy sent by the scheduling policy determination device 601 and to train the model according to the target scheduling policy.

[0132] Figure 7 This diagram illustrates a structural block diagram of a computing device 700 provided in an exemplary embodiment of this application. Typically, the computing device 700 includes a processor 701 and a memory 702.

[0133] The processor 701 can be implemented in at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 701 may also include a main processor and a coprocessor; the main processor is a processor for processing data in the wake-up state, and the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 701 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 701 may also include an artificial intelligence processor for handling computational operations related to machine learning.

[0134] The memory 702 may include one or more computer-readable storage media, which may be non-transitory computer-readable storage media, such as CD-ROM (Compact Disc Read-Only Memory), ROM, RAM (Random Access Memory), magnetic tape, floppy disk, and optical data storage devices. The computer-readable storage medium stores at least one computer program, which, when executed, enables a scheduling strategy determination method for pipelined parallel training.

[0135] Of course, the aforementioned computing device may also include other components, such as input / output interfaces and communication components. Input / output interfaces provide an interface between the processor and peripheral interface modules, which can be output devices, input devices, etc. Communication components are configured to facilitate wired or wireless communication between the computing device and other devices.

[0136] Those skilled in the art will understand that Figure 7 The structure shown does not constitute a limitation on the computing device 700, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0137] This application provides a computer-readable storage medium storing at least one computer program, which, when executed by a processor, enables a scheduling strategy determination method for pipelined parallel training.

[0138] This application provides a computer program product, which includes a computer program that, when executed by a processor, enables a scheduling strategy determination method for pipelined parallel training.

[0139] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0140] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for determining scheduling strategies for pipelined parallel training, characterized in that, The method is applied to network resource preemption scenarios, and the method includes: Based on any batch of training sample set and the attribute information of each computing node, multiple candidate scheduling strategies are determined. The computing nodes are deployed with training stages divided by the deep learning model. The candidate scheduling strategies are used to indicate the training method of each computing node to train the deep learning model based on the training sample set. The candidate scheduling strategies include a sample splitting scheme that evenly splits the training sample set into multiple training sample subsets and the corresponding maximum K value. The maximum K value is used to indicate the maximum number of subsets when each computing node performs a forward or backward computation under the corresponding sample splitting scheme. The maximum K value is greater than or equal to 2. Based on the sample splitting scheme and the relationship between each computing node in each candidate scheduling strategy, a task image corresponding to each candidate scheduling strategy is generated. Based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy, the target scheduling strategy with the shortest training time is determined from multiple candidate scheduling strategies. The target scheduling strategy is used to control the computation process of each computing node.

2. The method according to claim 1, characterized in that, The process involves determining multiple candidate scheduling strategies based on any batch of training sample sets and the attribute information of each computing node, including: Based on the number of samples included in the training sample set, the training sample set is uniformly split to obtain multiple sample splitting schemes. Each sample splitting scheme includes the number of subsets of the split training sample set and the number of samples within each training sample subset. Using the maximum storage space of each computing node as a constraint, we search for the maximum K value corresponding to each sample splitting scheme to obtain each candidate scheduling strategy.

3. The method according to claim 2, characterized in that, The process of searching for the maximum K value corresponding to each sample splitting scheme, using the maximum storage space of each computing node as a constraint, to obtain each candidate scheduling strategy includes: In the process of simulating each computing node processing training samples within each training sample subset under each sample splitting scheme, the number of training sample subsets processed in parallel by each computing node is continuously increased until the maximum storage space of any computing node is reached. The number of subsets computed by the computing node when the maximum storage space is reached under each sample splitting scheme is taken as the maximum K value corresponding to each sample splitting scheme; A candidate scheduling strategy is formed by combining a sample splitting scheme and the corresponding maximum K value.

4. The method according to claim 2, characterized in that, The process of generating a task image corresponding to each candidate scheduling strategy based on the sample splitting scheme and the correlation between each computing node in each candidate scheduling strategy includes: Based on the correlation between each computing node, a corresponding subtask image is generated for each training sample subset under each candidate scheduling strategy, resulting in multiple subtask images corresponding to each candidate scheduling strategy. The number of subtask images is the same as the number of subsets of the training sample subset split under each candidate scheduling strategy. Multiple subtask images corresponding to each candidate scheduling strategy are fused to obtain the task image corresponding to each candidate scheduling strategy.

5. The method according to claim 1, characterized in that, The step of determining the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy includes: Based on the task image and maximum K value corresponding to each candidate scheduling strategy, a candidate scheduling plan is generated for each candidate scheduling strategy. The candidate scheduling plan is the training plan for each computing node when executing the model training task according to the corresponding candidate scheduling strategy. Based on the current network state, the training time for each candidate scheduling strategy is obtained by simulating the process of each computing node training the model according to the candidate scheduling plan corresponding to each candidate scheduling strategy. From a variety of candidate scheduling strategies, the target scheduling strategy with the shortest training time is selected.

6. The method according to claim 5, characterized in that, The step of obtaining the training time for each candidate scheduling strategy by simulating the model training process of each computing node according to the candidate scheduling plan corresponding to each candidate scheduling strategy based on the current network state includes: During the simulation of model training for each computing node according to the candidate scheduling plan corresponding to each candidate scheduling strategy, the computation time of each training sample under each candidate scheduling strategy is estimated based on the maximum storage space of each computing node. In the current network state, by simulating the communication process between each computing node and other computing nodes when each computing node executes the candidate scheduling plan corresponding to each candidate scheduling strategy, the communication duration between each computing node and other computing nodes under each candidate scheduling strategy is obtained. The overhead model is invoked to process the computation time of each training sample and the communication time between each computing node and other computing nodes under each candidate scheduling strategy, so as to obtain the training time of each candidate scheduling strategy.

7. The method according to claim 5, characterized in that, After determining the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy, the process further includes: The target scheduling plan corresponding to the target scheduling strategy is sent to each computing node so that each computing node can train the model according to the target scheduling plan.

8. The method according to any one of claims 1 to 7, characterized in that, Before determining multiple candidate scheduling strategies based on any batch of training sample sets and the attribute information of each computing node, the process further includes: Based on the model information of the deep learning model and the attribute information of each computing device used for model training, multiple model segmentation schemes are determined. The model segmentation scheme is used to indicate the segmentation method of the deep learning model and the computing nodes corresponding to the training stage after segmentation. Each computing node includes at least one computing device. From various model segmentation schemes, the target segmentation scheme with the shortest training time is selected; Based on the target segmentation scheme, the attribute information of each computing node is determined.

9. The method according to any one of claims 1 to 7, characterized in that, After determining the target scheduling strategy with the shortest training time from multiple candidate scheduling strategies based on the current network state and the task image and maximum K value corresponding to each candidate scheduling strategy, the process further includes: Monitor the current network status; Based on the monitored network status, the target scheduling strategy with the shortest training time is selected from a variety of candidate scheduling strategies. The reselected target scheduling policy is sent to each computing node. The reselected target scheduling policy is used to control the computing process of each computing node when training the deep learning model with the next batch of training sample sets.

10. A pipelined parallel training system, characterized in that, The system includes a scheduling strategy determination device for pipelined parallel training and multiple computing nodes; The scheduling strategy determination device is used to execute the scheduling strategy determination method for pipelined parallel training as described in any one of claims 1 to 9; Each computing node is used to receive the target scheduling policy sent by the scheduling policy determination device and to train the model according to the target scheduling policy.

11. A computing device, characterized in that, It includes a processor and a memory; the memory stores at least one piece of program code; the at least one piece of program code is called and executed by the processor to implement the scheduling strategy determination method for pipelined parallel training as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which, when executed by a processor, is capable of implementing the scheduling strategy determination method for pipelined parallel training as described in any one of claims 1 to 9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, can implement the scheduling strategy determination method for pipelined parallel training as described in any one of claims 1 to 9.

Images