Online training-oriented computing power resource elastic distribution system

Abstract

The invention provides an online training-oriented computing power resource elastic allocation system. Comprising a user side, a cluster resource manager, an elastic scheduler and a working node using a to-be-allocated GPU, and the elastic scheduler comprises three components: a working load predictor, a working node estimator and an online learning automatic expansion device; the working node estimator adopts a resource throughput model to determine the most appropriate number of working nodes in distributed training on the basis of future working load prediction; and the online learning automatic expansion device expands or reduces the working nodes, and distributes more GPUs for the working nodes or recycles the GPUs into a resource pool after the subsequent approval of the cluster resource manager. Therefore, the performance of the online training model can be better ensured, and the computing power resource can be better saved.

Description

Technical Field

[0001] The present disclosure relates to the field of computers, and more specifically, to a computing resource elastic allocation system for online training. Background Art

[0002] Deep learning (DL) training is a process consisting of a large number of iterations, each of which reads and processes a certain number of data samples, or mini-batches. Unlike offline deep learning training, which obtains data from offline datasets, online deep learning training pipelines consume streaming data provided by streaming platforms such as Kafka. Online training is a dynamic program that incrementally learns from data streams in real time and adapts evolving algorithms to newly arrived data. To support large-scale model training using massive data, DL jobs are submitted to GPU clusters and distributed model training is performed over GPUs using different parallel techniques.

[0003] In large-scale production environments, the production rate of streaming samples for online model training usually shows periodic fluctuations. User behaviors (such as user clicks, application usage, transaction volume, etc.) have obvious peak and non-peak characteristics. A typical traffic fluctuation pattern on weekdays is as follows, with data throughput reaching the lowest value in the early morning and the highest value in the evening. It is not difficult to understand that clusters can usually maintain a high data production rate for a long time after the evening and drop sharply at night. In contrast, the average value of data traffic is higher on weekends or holidays-user data starts to rise and accumulate from the early morning and remains at an extremely high level for the rest of the day. There is a difference of 7.65 times between the maximum and minimum values ​​of traffic, and we have observed from production clusters that the gap ranges from 5 times to 30 times. This shows that dynamic automatic expansion on GPU clusters has great potential in reducing resource waste and improving model training throughput.

[0004] The purpose of this paper is to design and implement a flexible GPU scheduling system for large-scale online model training in the real world, with the following research and engineering requirements:

[0005] i) Active auto-scaling based on prediction. Existing auto-scaling strategies are reactive strategies that respond to changes in system load, but the time taken for auto-scaling is too long for online training. Once the auto-scaling is triggered, a series of subsequent steps including model saving, container startup to prepare new job nodes, image pulling, model loading, data flow reconstruction, etc. are required. When all these steps are completed, the auto-scaling may be outdated because the system load changes rapidly over time and a large number of fresh data samples are not used in a timely manner. In addition, reactive strategies are prone to frequent resource allocation, which inevitably leads to high preemption overhead. As mentioned earlier, there are significant temporal (e.g., obvious peaks and troughs) behavior patterns in terms of user clicks, video viewing time, and frequency. This observation reveals the potential of adopting active prediction methods to predict traffic and guide active auto-scaling.

[0006] ii) Timely and accurate resource estimation. The right number of GPUs must be calculated accurately and timely to accommodate the time-varying size of data samples. While some existing works use analytical performance-resource models to determine the optimal amount of resources to optimize cluster-wide training throughput, they either focus only on the amount of CPU / memory or are only effective on general DNN models in offline training models.

[0007] To meet these requirements, in this paper, we propose KALE, a new elastic GPU scheduling system for improving the performance of online deep learning model training. To achieve this goal, we propose an effective auto-scaling mechanism mainly by leveraging data traffic prediction and resource estimation based on resource-performance model. The key design concept is to use historical data traffic to proactively predict the number of data samples that a training job needs to consume and determine the most appropriate parallelism, i.e., the number of GPUs, so that the job has enough resources to process and train a good enough model using the data. Once the GPU resources are determined, auto-scaling is performed to adjust the allocated GPUs, assigning each GPU to a worker node to adapt to the incoming data samples in a timely and most appropriate manner. Summary of the invention

[0008] The purpose of the embodiments of the present disclosure is to provide a computing resource elastic allocation system for online training. The main research goal of the present invention is to realize the automatic generation of code summary documents with project granularity through the organic combination of hierarchical-based models and program analysis technology. The specific object of the study is English documents, and the programming language targeted is Java. The summary document is mainly reflected in a short natural language summary, which does not involve complex information such as directory structure, instructions for use, etc. Based on this model, an open source code repository summary automatic generation tool was designed and implemented on the Eclipse IDE in the form of an Eclipse plug-in.

[0009] In a general aspect, a computing resource elastic allocation system for online training is provided, including a user end, a cluster resource manager, an elastic scheduler, and a working node using a GPU to be allocated. The working process of the system is as follows: the user submits detailed configuration and preference settings of the online learning job to the cluster resource manager, and the cluster resource manager analyzes and approves and returns the submitted configuration to the user; thereafter, the configuration is submitted to the elastic scheduler to manage data traffic and resource allocation throughout the life cycle of the online training job, and the elastic scheduler includes three components: a workload predictor, a working node estimator, and an online learning automatic scaler;

[0010] The workload predictor receives the submitted configuration, trains a time series prediction model using historical data to predict the upcoming data traffic that the online training job may face in the upcoming period, adopts a Transformer-based time series prediction model, uses a time series decomposition method to decompose the time data into trend, seasonality, periodicity and noise components, converts the input time series data into an attention vector through an encoder, combines position encoding to capture time series information, and finally uses a decoder to predict future workload requirements;

[0011] The worker node estimator uses a resource throughput model to determine the most appropriate number of worker nodes for distributed training based on future workload predictions;

[0012] The online learning autoscaler expands or shrinks the worker nodes and, after subsequent approval by the cluster resource manager, allocates more GPUs to the worker nodes or recycles the GPUs back to the resource pool.

[0013] The detailed configuration and preference settings include model algorithms and parameters, data sources and access methods, and model export addresses.

[0014] The worker node estimator is targeted at a parameter server architecture, which consists of two types of nodes: parameter servers and worker nodes;

[0015] The parameter server is responsible for storing and maintaining global model parameters. One or more parameter server nodes are deployed according to the system scale and requirements to distribute and store model parameters. Multiple parameter server nodes are usually synchronized to ensure that the parameters they hold are consistent. The worker node is responsible for the actual model training task. Each worker node independently processes a part of the training data and accelerates the entire training process through parallel computing. The worker node obtains the latest model parameters from the parameter server, performs calculations locally, and sends the calculated gradients back to the parameter server.

[0016] The workflow of model training under the parameter server architecture is as follows:

[0017] Step 1: Parameter initialization. When training starts, the parameter server initializes the global model parameters. The parameters can be randomly generated or obtained by loading a pre-trained model.

[0018] Step 2: Parameter pulling. Before each calculation step, the worker node pulls the latest model parameters from the parameter server. This process ensures that each worker node uses the latest version of the model parameters for training.

[0019] Step 3: Local training. After pulling the parameters, the working node uses a batch of training samples to perform forward propagation and back propagation calculations locally. In the forward propagation stage, the working node calculates the predicted value and loss function based on the current model parameters; in the back propagation stage, the working node calculates the gradient based on the loss function;

[0020] Step 4: Gradient push: After the working node calculates the gradient locally, it sends these gradients back to the parameter server;

[0021] Step 5: Parameter update: After receiving the gradients from multiple working nodes, the parameter server updates the global model parameters according to these gradients.

[0022] Step 6: Iterate, the above steps will be repeated until the model converges or reaches the predetermined number of training rounds;

[0023] The time required for each working node to train one step in the parameter server architecture is: T = T D +T F +T B +T U +T O +T C ; where T D Indicates the time to pull parameters, T F represents the forward propagation time, T B represents the time of back propagation, T U Indicates the time of pushing parameters, T O Indicates the time when the optimizer updates the parameters, T C Indicates the remaining additional time overhead.

[0024] For the dense parameters and sparse parameters in the training process, let the number of parameter servers be p, the number of working nodes be w, the batch size of each working node training be m, the size of dense parameters be D, and the size of sparse parameters be m×S; assuming that the parameters are evenly distributed on the parameter servers, the number of dense parameters sent by the working nodes to the parameter servers is D / p, and the number of sparse parameters is m×S / p; define the forward propagation time for training a small batch as Tforward , the back propagation time is T backward , T F =m×T forward , T B =T backward .

[0025] T D =(m×S) / (B×p), T U =(m×S+D) / (B×p); let the time to update a small batch of parameters be T update , then T O =m×S×T update / p; other additional overhead is assumed to be linearly related to p and w, T C =λ×p+λ′×w, then the time required for each working node to train one step is:

[0026]

[0027] In synchronous training, all working nodes access the same parameter server at the same time. Assume that the bandwidth of the parameter server and the working node is B0. When a working node establishes connections with p parameter servers at the same time, the maximum bandwidth of each connection is B0 / p. When a parameter server establishes connections with w working nodes at the same time, the maximum bandwidth of each connection is B0 / w. The bandwidth B between the parameter server and the working node should be the smaller value of B0 / w and B0 / p:

[0028]

[0029] The throughput is equal to the total batch size divided by the time of one step of training. Assuming the total batch size is M, the throughput under synchronous training is:

[0030]

[0031] After linear approximation of min, it is simplified to, where θ is a non-negative coefficient:

[0032]

[0033] If there is an optimal performance ratio of w / p in a production environment, it can be further simplified to:

[0034]

[0035] In asynchronous training, each worker node can be considered to access the parameter server in a staggered manner. The bandwidth between the worker node and the parameter server is B0 / p. The total throughput is w times the throughput of each node. The throughput under asynchronous training is:

[0036]

[0037] Similarly, w and p are used as independent variables, and other variables are used as fitting parameters to merge similar items according to independent variables. Finally, the parameters are renamed as θ, where θ is a non-negative coefficient:

[0038]

[0039] If there is an optimal performance ratio of w / p in a production environment, the equation can be simplified further:

[0040]

[0041] The online learning automatic scaler is implemented by expanding or reducing the online training job based on the estimate of the required working nodes, and formulating the requirement for the number of online learning working nodes that needs to be met at each time t:

[0042] min{w(t)∈Z+|w(t)}

[0043] stf(w(t))>L(t)

[0044] Where w(t) represents the number of worker nodes allocated at time t, f(w) represents the throughput of w worker nodes using the resource throughput model fitted in the previous section, and L(t) represents the workload at time t obtained by prediction;

[0045] Keep the batch size constant during model training and keep the ratio of w / p constant during expansion.

[0046] The innovation of the embodiment of the present invention lies in:

[0047] (1) Better guarantee of the performance of online training models. Due to active prediction, KALE’s default rate and cumulative delay are significantly reduced compared to reactive scaling methods like Autopilot and HPA. Compared to HPA, KALE significantly reduces cumulative delay and downtime by 69.2% and 33.1%, respectively, while reducing the SLO default rate from 19.57% to only 2.6%. As Figure 3 As shown in the upper middle part of the figure, during the workload peak, the GPU allocation of the reactive scaling method has not yet reached the highest point. The GPU allocation curve of Autopilot is slightly misaligned with the workload peak, which may cause samples to accumulate and lead to delays. This problem is more serious in HPA, where the number of worker nodes peaks 20 minutes after the workload peak. Scaling at the peak means that more samples are accumulated for the same downtime, making it more difficult to eliminate delays, resulting in HPA having the highest default rate and accumulated delays.

[0048] (2) Save more computing resources. Thanks to accurate resource throughput modeling, KALE significantly reduces GPU hours by up to 46.8% compared to Madu. Although Madu predicts future load, its linear fitting algorithm for GPU throughput leads to over-allocation of resources. Figure 3 As shown, KALE's curve is like a pair of tweezers, tightly around the peak, while Madu shows excessive and continuous allocation at the peak, reflecting its high number of GPU hours. In addition, KALE and Madu are comparable in terms of default rate and cumulative latency, with KALE showing a default rate of 2.60% and a cumulative latency of 2203.69 minutes, while Madu shows a default rate of 2.57% and a cumulative latency of 2302.85 minutes. However, KALE achieved this performance using only 242 GPU hours, while Madu required 455 GPU hours. KALE's accuracy in resource throughput modeling and expansion timing enables it to achieve performance comparable to Madu's high resource usage with significantly lower resource consumption. Due to sawtooth detection, KALE significantly reduces downtime, up to 59.5% (from 269 minutes for HPA to 109 minutes for KALE). Reactive methods are more prone to allocation fluctuations due to their lack of foresight. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The above and other objects and features of the present disclosure will become more apparent from the following description in conjunction with the accompanying drawings.

[0050] Figure 1 The data samples in the prior art have periodic time series characteristics;

[0051] Figure 2 It is a schematic diagram of an architecture diagram showing the overall architecture of a computing resource elastic allocation system KALE for online training according to an embodiment of the present disclosure;

[0052] Figure 3 is a diagram showing a comparison of the number of GPUs in a system end-to-end experiment according to an embodiment of the present disclosure;

[0053] Figure 4 is a schematic diagram showing the GPU utilization of a production cluster according to an embodiment of the present disclosure, wherein the left figure is without using KALE, and the right figure is with using KALE. DETAILED DESCRIPTION

[0054] The following specific embodiments are provided to help the reader obtain a comprehensive understanding of the methods, devices and / or systems described herein. However, after understanding the disclosure of the present application, various changes, modifications and equivalents of the methods, devices and / or systems described herein will be clear. For example, the order of operations described herein is only an example and is not limited to those orders set forth herein, but can be changed as will be clear after understanding the disclosure of the present application, except for operations that must occur in a specific order. In addition, for greater clarity and simplicity, the description of features known in the art may be omitted.

[0055] The features described herein can be implemented in different forms and should not be construed as being limited to the examples described herein. Rather, the examples described herein have been provided to illustrate only some of the many possible ways to implement the methods, devices, and / or systems described herein, which will be clear after understanding the disclosure of the present application.

[0056] As used herein, the term "and / or" includes any one of the associated listed items and any combination of any two or more.

[0057] Although terms such as "first", "second", and "third" may be used herein to describe various members, components, regions, layers, or portions, these members, components, regions, layers, or portions should not be limited by these terms. Instead, these terms are only used to distinguish one member, component, region, layer, or portion from another member, component, region, layer, or portion. Therefore, without departing from the teachings of the examples described herein, the first member, first component, first region, first layer, or first portion referred to in the examples may also be referred to as the second member, second component, second region, second layer, or second portion.

[0058] In the specification, when an element (such as a layer, a region or a substrate) is described as being “on”, “connected to” or “coupled to” another element, the element may be directly “on”, “connected to” or “coupled to” another element, or one or more other elements may be present therebetween. Conversely, when an element is described as being “directly on”, “directly connected to” or “directly coupled to” another element, there may be no other elements present therebetween.

[0059] The terms used herein are only used to describe various examples and are not intended to limit the disclosure. Unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. The terms "comprise", "include" and "have" indicate the presence of the described features, quantities, operations, components, elements and / or combinations thereof, but do not exclude the presence or addition of one or more other features, quantities, operations, components, elements and / or combinations thereof.

[0060] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by a person of ordinary skill in the art to which the present disclosure belongs after understanding the present disclosure. Unless explicitly defined as such herein, terms (such as those defined in a general dictionary) should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and the present disclosure, and should not be interpreted in an idealized or overly formal manner.

[0061] Furthermore, in the description of examples, when it is considered that a detailed description of a well-known related structure or function would cause vague interpretation of the present disclosure, such a detailed description will be omitted.

[0062] Figure 2 It is a schematic diagram showing a computing resource elastic allocation system for online training according to an embodiment of the present disclosure.

[0063] Overall architecture:

[0064] The basic workflow and interaction between KALE components are as follows: Users can submit detailed configuration and preferences for online learning jobs to the cluster resource manager (such as Kubernetes). This information usually includes model algorithms and parameters, data sources and access methods, and model export addresses. The resource manager then analyzes and approves the submitted configuration (steps ① and ②).

[0065] After that, the elastic scheduler will be responsible for managing data traffic and resource allocation throughout the life cycle of the online training job (step ③). The core of KALE is to predict the number of upcoming data samples before dynamically allocating the resources that are most suitable for the online training job. The elastic scheduler includes three key components - workload predictor, worker node estimator, and online learning auto-scaler.

[0066] Workload Predictor. Inside the predictor, a time series prediction model is trained using historical data to predict the upcoming data traffic that the online training job may face in the upcoming period (step ④). Technically, in order to address the periodic temporal characteristics of online learning workloads, we adopted a Transformer-based time series prediction model. Specifically, we use a time series decomposition method to decompose the temporal data into components such as trend, seasonality, periodicity, and noise. An encoder converts the input time series data into an attention vector, combined with position encoding to capture the time series information. Finally, a decoder is used to predict future workload requirements. Looking ahead, our design philosophy adopts a modular architecture, which not only facilitates the current implementation of the Transformer-based model, but also paves the way for the seamless integration of more advanced prediction models in the future.

[0067] Worker Node Quantity Estimator. Based on the traffic prediction, we adopt a resource throughput model to determine the most appropriate number of worker nodes for distributed training so that the upcoming data traffic can be fully utilized to obtain a competitive online model (step ⑤). The technical details will be discussed in Section 3.2.

[0068] Online learning autoscaler. The decision result, i.e. the number of worker nodes, will be used by the online learning autoscaler, which is responsible for scaling up or down the worker nodes, allocating more GPUs to the worker nodes or recycling GPUs back to the resource pool (step ⑥).

[0069] Subsequent approval by the resource manager (steps ⑦ and ⑧). The autoscaler also optimizes the scaling strategy of online learning jobs to avoid too frequent resource adjustments and actively collects system metrics. We will explain the working principle of the autoscaler in detail in Section 3.3.

[0070] Worker Node Estimator:

[0071] The mainstream training framework for online training usually adopts the parameter server architecture. The parameter server architecture is a distributed computing architecture designed for large-scale machine learning tasks, especially for the training of deep learning models. This architecture can effectively solve the computing and communication bottlenecks in large-scale data training and significantly improve training efficiency and performance. The parameter server architecture mainly consists of two types of nodes: parameter server (PS) and worker node (WorkerNode).

[0072] Parameter Server: The parameter server is responsible for storing and maintaining global model parameters. Depending on the system scale and requirements, one or more parameter server nodes can be deployed to distribute and store model parameters. Multiple parameter server nodes are usually synchronized to ensure that the parameters they hold are consistent.

[0073] Worker Node: Worker nodes are responsible for the actual model training tasks. Each worker node processes a portion of the training data independently, and accelerates the entire training process through parallel computing. The worker node obtains the latest model parameters from the parameter server, calculates them locally, and sends the calculated gradients back to the parameter server.

[0074] The workflow of model training under the parameter server architecture is as follows:

[0075] 1) Parameter initialization: When training starts, the parameter server will initialize the global model parameters. These parameters can be randomly generated or obtained by loading a pre-trained model.

[0076] 2) Parameter Pull: Before each calculation step, the worker node pulls the latest model parameters from the parameter server. This process ensures that each worker node uses the latest version of the model parameters for training.

[0077] 3) Local training: After pulling parameters, the working node will use a batch of training samples locally for forward propagation and backward propagation calculations. In the forward propagation stage, the working node calculates the predicted value and loss function based on the current model parameters; in the backward propagation stage, the working node calculates the gradient based on the loss function.

[0078] 4) Gradient Push: After the working node calculates the gradient locally, it sends these gradients back to the parameter server. This process is called gradient push.

[0079] 5) Parameter update: After receiving the gradients from multiple worker nodes, the parameter server will update the global model parameters based on these gradients. Usually, some optimization algorithms such as stochastic gradient descent (SGD) or its variants (such as Adam) are used to update the parameters.

[0080] 6) Iteration: The above steps are repeated until the model converges or the predetermined number of training rounds is reached. The time required for each working node in the parameter server architecture to train one step is shown below. D Indicates the time to pull parameters, T F represents the forward propagation time, T B represents the time of back propagation, T U Indicates the time of pushing parameters, T O Indicates the time when the optimizer updates the parameters, T C Indicates the remaining additional time overhead.

[0081] T=T D +T F +T B +T U +T O +T C

[0082] In the online businesses of Internet companies (such as search, recommendation, and advertising), data is usually sparse. This means that most features are zero or missing most of the time. In order to process this sparse data and improve the performance and efficiency of the model, the parameters in online learning are usually divided into dense parameters and sparse parameters, which are stored and updated separately.

[0083] Dense parameters: Dense parameters are those feature parameters that appear in almost all samples. These parameters are fully involved in the calculation during model training, and all gradient information is used every time they are updated.

[0084] Sparse parameters: Sparse parameters are those feature parameters that are missing or zero in most samples. Since these parameters appear less frequently in each batch, their updates are mainly based on the feature key values ​​that actually appear.

[0085] In order to better handle these two types of parameters, the online training system adopts different storage and update strategies. Dense parameters and sparse parameters are stored in different databases or memory structures respectively. Dense parameters are usually stored in memory for fast access and update; sparse parameters may be stored in a distributed database and indexed and searched according to feature key values. Dense parameter update: In each training iteration, dense parameters are fully updated according to the calculated global gradient. This means that each dense parameter uses all sample information in the current batch to adjust its value to ensure the global consistency of the model. Sparse parameter update: The update of sparse parameters depends on specific key-value pairs. In each training iteration, only those sparse feature parameters that actually appear will be updated according to the corresponding gradients. This method reduces unnecessary computational overhead and improves update efficiency.

[0086] Generally speaking, the number of sparse parameter updates in each round of training is positively correlated with the batch size. Batch size refers to the number of samples used in each training iteration. When the batch size increases, more sparse feature parameters will appear in this batch, resulting in more sparse parameters that need to be updated. Conversely, if the batch size is small, the number of sparse parameters that need to be updated in each round of training will also decrease accordingly.

[0087] Assume that the number of parameter servers is p, the number of worker nodes is w, the batch size of each worker node is m, the size of dense parameters is D, and the size of sparse parameters is m × S. Assuming that the parameters are evenly distributed on the parameter servers, the number of dense parameters sent by the worker nodes to the parameter servers is D / p, and the number of sparse parameters is m × S / p.

[0088] In one-step training, the time of forward propagation is linearly related to the batch size, while the time of backward propagation is independent of the batch size. Let the forward propagation time for training a small batch be T forward , the back propagation time is T backward Therefore, T F =m×T forward , T B =T backward .

[0089] Sparse parameters and dense parameters are transmitted separately, so they can overlap during transmission to save time. Typically, sparse scenarios for online training result in longer transmission times for sparse parameters. In this case, the pull time of dense parameters can overlap with the update time of sparse parameters, allowing the sparse parameters to monopolize the bandwidth when pulling, increasing the speed. Therefore, the time for a worker node to pull parameters is (m×S) / (B×p), and the time to push gradients is (m×S+D) / (B×p), rather than a symmetric process. Therefore, T D =(m×S) / (B×p), T U =(m×S+D) / (B×p). Let T be the time to update a small batch of parameters. update , then T O =m×S×T update / p. Other additional overhead (such as control commands, etc.) is assumed to be linearly related to p and w. Therefore, T C =λ×p+λ′×w. Then the following formula can be derived:

[0090]

[0091] In synchronous training, all worker nodes access the same parameter server at the same time. Assume that the bandwidth of both the parameter server and the worker node is B0. When a worker node establishes connections with p parameter servers at the same time, the maximum bandwidth of each connection is B0 / p, and when a parameter server establishes connections with w worker nodes at the same time, the maximum bandwidth of each connection is B0 / w. Therefore, the bandwidth B between the parameter server and the worker node should be the smaller value of B0 / w and B0 / p.

[0092]

[0093] The throughput is equal to the total batch size divided by the time of one step of training. Let the total batch size be M, then the throughput formula under synchronous training can be derived.

[0094]

[0095] After linear approximation of min, it can be modeled as follows, where θ is a non-negative coefficient.

[0096]

[0097] If there is an optimal performance ratio of w / p in a production environment, the equation can be simplified further.

[0098]

[0099] In asynchronous training, each worker node can be considered to access the parameter server in a staggered manner. Therefore, the bandwidth between the worker node and the parameter server is B0 / p. In addition, in asynchronous training, each worker node trains independently through small batches and does not need to wait for other nodes, so the total throughput is w times the throughput of each node. The throughput formula under asynchronous training can be derived as follows:

[0100]

[0101] Similarly, w and p are used as independent variables, and other variables are used as fitting parameters to merge similar items according to independent variables. Finally, the parameters are renamed as θ, where θ is a non-negative coefficient:

[0102]

[0103] If there is an optimal performance ratio of w / p in a production environment, the equation can be simplified further:

[0104]

[0105] Application example. We use an example to show how the theoretical analysis and the proposed resource throughput model work. Assume that we have a synchronous online training job with a batch size of M = 16384. The parameter fitting results are as follows: θ0 = 0.00035, θ1 = 2.5726, θ2 = 0.9824, θ3 = 0.02786. If the workload predictor predicts the foreseeable traffic to be 30,000, then we can check the throughput generated for a given number of worker nodes. In this case, for example, assuming f(10) = 30007.326, the optimal number of worker nodes that is just enough to process the upcoming data samples would be ten. As will be explained in detail in this paper, the autoscaler can then adjust the allocated GPUs to meet the needs of elastic training.

[0106] Auto-Scaler:

[0107] Based on the estimate of required worker nodes, the online training job can be scaled up or down to ensure that its throughput meets the upstream data traffic. Specifically, we formulate the requirement for the number of online learning worker nodes that needs to be met at each time t:

[0108] min{w(t)∈Z+|w(t)}

[0109] stf(w(t))>L(t)(10)

[0110] Where w(t) represents the number of worker nodes allocated at time t, f(w) represents the throughput of w worker nodes using the resource throughput model fitted in the previous section, and L(t) represents the workload at time t obtained by prediction.

[0111] During model training, we keep the batch size constant to ensure the stability and effectiveness of model training. We follow a simple but effective pause and restart strategy to automatically scale the running worker nodes, which is currently adopted by mainstream deep learning frameworks. In order to reduce the complexity of fitting the proposed resource throughput model, the ratio of w / p is kept constant during the scaling process.

[0112] Optimization Scheme. Although the above general auto-scaling is able to calculate the appropriate number of worker nodes for the upcoming data samples, it may experience frequent auto-scaling operations, which in turn leads to unstable resource allocation. This is especially serious when the data traffic has bursts and sawtooth cycles. To solve this problem, we actually propose a sawtooth calibration mechanism so that the sawtooth cycles are detected in time and smoothed with line segments.

[0113] The optimized workflow is as follows: first, each time resources are allocated, calculate whether there will be a "sawtooth" phenomenon in resource allocation. If there is no "sawtooth" phenomenon, allocate resources normally. Otherwise, perform smoothing and calculate the number of resources that should be allocated after optimization. The core of this process is to find out the situation where the maintenance time is too short after the number of resources changes like a sawtooth, that is, the "sawtooth" phenomenon. This is because if the maintenance time after the expansion is too short, the optimal allocation time of the resources will end before the expansion is completed. In this case, there is no need to perform this expansion. The change_threshold parameter represents the threshold for detecting resource changes and is usually set to 1. The hold_threshold parameter represents the maintenance time threshold after the change occurs. When the maintenance time is less than this threshold, it will be marked as a sawtooth. The number of optimized resources after smoothing is equal to the larger value of the amount of resources on both sides of the sawtooth to ensure that the resource allocation is sufficient. Different hold_threshold selections will lead to different resource allocation results, such as Figure 3 To demonstrate the effectiveness of sawtooth smoothing, we collected workload information for a day and calculated the number of resources allocated at each moment, which is given by Figure 3 The blue line in the figure shows that the sawtooth phenomenon is clearly seen near the horizontal coordinates 300, 500, and 1300. After the sawtooth smoothing process, the thresholds set to 10 and 30 minutes cleared the visible sawtooth. The other expansion times are also slightly different. In general, the larger the threshold, the fewer expansion times and the smaller the overhead, but the lower the accuracy of resource allocation, and vice versa.

[0114] Experimental verification:

[0115] In this section, we conduct a comprehensive end-to-end evaluation of our online training computing resource elastic allocation system to demonstrate its effectiveness. We compare our online training computing resource elastic allocation system with several other automatic scaling strategies based on key performance indicators.

[0116] Evaluation Metrics:

[0117] To effectively evaluate the performance of the system, we used the following key metrics:

[0118] SLO breach rate. In the previous section, we described the impact of latency on model performance. Based on industry experience, an AUC drop of more than 0.02% should be taken seriously, so based on this system goal and the impact of latency on AUC, we can define the SLO breach range. For the model used in the experiment, a delay of more than 20 minutes is considered a default. The smaller this indicator, the better.

[0119] Cumulative delay. The cumulative delay metric is used to describe the overall situation of delay, that is, the sum of delays per minute, and measures the degree of accumulation of streaming data samples during training, as shown in the following formula. The smaller the metric, the better.

[0120]

[0121] Downtime. Downtime describes the time overhead of scaling up and down capacity, and the smaller the indicator, the better. Since the scaling up and down of online learning involves workflows such as saving models, starting new worker node containers, pulling images, loading models, and rebuilding consumer groups, the overhead of scaling up and down cannot be ignored. Fewer but more precise adjustments are what we pursue.

[0122] GPU hours. GPU hours describes the total GPU time used for online learning. The calculation formula is as follows. The smaller the indicator, the better.

[0123] Compare to baseline:

[0124] To evaluate the performance of our system, we compared it with four different scheduling schemes:

[0125] Adequate Resources: Adequate resources are always allocated and no expansion is performed.

[0126] Kubernetes default horizontal pod autoscaling (HPA): A widely used responsive autoscaling method that automatically adjusts the number of pods by monitoring resource usage and following preset rules.

[0127] Autopilot: An automatic scaling solution proposed by Google, similar to HPA. Autopilot uses a moving window to collect recent resource usage statistics and scales resources based on these statistics (e.g., average CPU and memory utilization), making it more accurate than HPA's resource prediction.

[0128] Madu: An active scaling solution combined with workload prediction, which scales containers based on the workload prediction of individual microservices.

[0129] Experimental results:

[0130] Table 1: System end-to-end experimental evaluation results

[0131]

[0132] As shown in the table, our method M outperforms HPA, Autopilot, Madu, and other auto-scaling schemes in terms of default rate, cumulative latency, downtime, and GPU hours. In terms of default rate and cumulative latency, our method M is comparable to Madu, with a default rate of 0.0260 and a cumulative latency of 1203.69 minutes for our method M, while Madu has a default rate of 0.0257 and a cumulative latency of 1702.85 minutes. However, our method M only uses 242 GPU hours, while Madu requires 455 GPU hours. The precise resource throughput modeling and scaling timing of our method M enable it to achieve performance comparable to Madu's high resource usage with significantly lower resource consumption.

[0133] Due to active prediction, the default rate and cumulative delay of our method M are significantly reduced compared to reactive expansion methods such as Autopilot and HPA, with the default rate reduced by up to 17% (from 0.1957 in HPA to 0.0260 in our method M) and the cumulative delay reduced by up to 83.2% (from 7150.61 minutes in HPA to 1203.69 minutes in our method M). In terms of downtime, our method M prevents excessive resource allocation frequency through sawtooth detection, significantly reducing the time cost, and the downtime is reduced by up to 59.5% (from 269 minutes in HPA to 109 minutes in our method M). In addition, compared with sufficient resources, our method M reduces the number of GPU hours by up to 64% (from 672 GPU hours to 242 GPU hours).

[0134] Figure 3 The GPU allocation changes of different methods over a continuous 24-hour period are shown. The differences in allocation strategies and quantities are obvious. For reactive methods, allocation timing is usually delayed, which is the main reason for sample accumulation and SLO violations. In addition, reactive methods lack foresight and are prone to allocation fluctuations.

[0135] Although Madu predicts future load, its linear fitting algorithm for GPU throughput leads to over-allocation of resources. The zoomed-in portion of the figure shows the allocation of each method at the peak. The curve of the proposed method (our method M) resembles a pair of tweezers, tightly clamping the peak, while Madu shows excessive and continuous allocation at the peak, reflecting its high GPU hours.

[0136] The lag of reactive methods such as Autopilot and HPA is also clearly visible. During the peak workload, the GPU allocation of these methods has not yet reached the peak. Autopilot's GPU curve is slightly misaligned with the workload peak, resulting in sample accumulation. HPA has a more serious problem, with its GPU count reaching its peak only 20 minutes after the workload peak. Scaling at the peak means that more samples are accumulated for the same downtime, making it more difficult to eliminate delays, so HPA has the highest default rate and accumulated delay.

[0137] The last key point is that, thanks to sawtooth detection, our approach M avoids allocation fluctuations, resulting in the least number of resource adjustments and the shortest downtime.

[0138] Production environment deployment example:

[0139] KALE has been deployed on production clusters, serving tens of thousands of online recommendation model training jobs. In order to verify the design and implementation of KALE in a real-world production environment while ensuring that the cluster works normally without interference, we collected trace data for some jobs and analyzed relevant statistics. This information comes from a GPU / CPU hybrid cluster with more than 8,000 GPUs and more than 500,000 CPU cores. Given that 90% of the recommendation models in actual production environments are trained online and updated in real time, the main performance indicators at the cluster level are GPU utilization and peak throughput. These indicators are critical to ensuring that online training can effectively utilize GPU resources while meeting the service level agreements (SLAs) for model updates.

[0140] To evaluate the impact of the key techniques proposed in KALE on production systems, we collected tracking data and metrics for a week from June 8 to June 115, 2024, after deploying KALE, and compared it with the tracking data of the same cluster from April 8 to April 15, 2024. The average GPU utilization at the cluster level can be increased by 40%, and, thanks to the elastic scheduling mechanism with accurate estimation of the required number of GPUs, the system throughput can be continuously maintained at a high level without obvious utilization drops and peak fluctuations. At the same time, this shows that more training jobs can be allowed and executed in the cluster. The cluster is now used to host the company's largest ranking models, which have model sizes of tens of TB. The proposed elastic scheduling strategy can achieve minute-level elasticity while meeting peak throughput requirements.

[0141] Although some embodiments of the present disclosure have been shown and described, it will be appreciated by those skilled in the art that modifications may be made to the embodiments without departing from the principles and spirit of the present disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. A computing resource elastic allocation system for online training, characterized in that: The system includes a user terminal, a cluster resource manager, an elastic scheduler, and a working node using a GPU to be allocated. The working process of the system is as follows: the user submits the detailed configuration and preference settings of the online learning job to the cluster resource manager, and the cluster resource manager analyzes and approves and returns the submitted configuration to the user; After that, the configuration is submitted to the elastic scheduler to manage data traffic and resource allocation throughout the life cycle of the online training job. The elastic scheduler consists of three components: workload predictor, worker node estimator, and online learning autoscaler. The workload predictor receives the submitted configuration, trains a time series prediction model using historical data to predict the upcoming data traffic that the online training job may face in the upcoming period, adopts a Transformer-based time series prediction model, uses a time series decomposition method to decompose the time data into trend, seasonality, periodicity and noise components, converts the input time series data into an attention vector through an encoder, combines position encoding to capture time series information, and finally uses a decoder to predict future workload requirements; The worker node estimator uses a resource throughput model to determine the most appropriate number of worker nodes for distributed training based on future workload predictions; The online learning autoscaler expands or shrinks the worker nodes and, after subsequent approval by the cluster resource manager, allocates more GPUs to the worker nodes or recycles the GPUs back to the resource pool.

2. The computing resource elastic allocation system for online training according to claim 1, characterized in that: The detailed configuration and preference settings include model algorithms and parameters, data sources and access methods, and model export addresses.

3. The computing resource elastic allocation system for online training according to claim 2, characterized in that: The worker node estimator is targeted at a parameter server architecture, which consists of two types of nodes: parameter servers and worker nodes; The parameter server is responsible for storing and maintaining global model parameters. One or more parameter server nodes are deployed according to the system scale and requirements to distribute and store model parameters. Multiple parameter server nodes are usually synchronized to ensure that the parameters they hold are consistent. The worker node is responsible for the actual model training task. Each worker node independently processes a part of the training data and accelerates the entire training process through parallel computing. The worker node obtains the latest model parameters from the parameter server, performs calculations locally, and sends the calculated gradients back to the parameter server. The workflow of model training under the parameter server architecture is as follows: Step 1: Parameter initialization. When training starts, the parameter server initializes the global model parameters. The parameters can be randomly generated or obtained by loading a pre-trained model. Step 2: Parameter pulling. Before each calculation step, the worker node pulls the latest model parameters from the parameter server. This process ensures that each worker node uses the latest version of the model parameters for training. Step 3: Local training. After pulling the parameters, the working node uses a batch of training samples to perform forward propagation and back propagation calculations locally. In the forward propagation stage, the working node calculates the predicted value and loss function based on the current model parameters; in the back propagation stage, the working node calculates the gradient based on the loss function. Step 4: Gradient push: After the working node calculates the gradient locally, it sends these gradients back to the parameter server; Step 5: Parameter update: After receiving the gradients from multiple working nodes, the parameter server updates the global model parameters according to these gradients. Step 6: Iterate, the above steps will be repeated until the model converges or reaches the predetermined number of training rounds; The time required for each working node to train one step in the parameter server architecture is: T = T D +T F +T B +T U +T O +T C ; where T D Indicates the time to pull parameters, T F represents the forward propagation time, T B represents the time of back propagation, T U Indicates the time of pushing parameters, T O Indicates the time when the optimizer updates the parameters, T C Indicates the remaining additional time overhead.

4. The computing resource elastic allocation system for online training according to claim 3, characterized in that: For the dense parameters and sparse parameters in the training process, let the number of parameter servers be p, the number of working nodes be w, the batch size of each working node training be m, the size of the dense parameters be D, and the size of the sparse parameters be m×S; Assuming that the parameters are evenly distributed on the parameter server, the number of dense parameters sent by the working node to the parameter server is D / p, and the number of sparse parameters is m×S / p; the forward propagation time for training a small batch is defined as T forward , the back propagation time is T backward , T F =m×T forward , T B =T backward ; T D =(m×S) / (B×p), T U =(m×S+D) / (B×p); let the time to update a small batch of parameters be T update , then T O =m×S×T update / p; other additional overhead is assumed to be linearly related to p and w, T C =λ×p+λ'×w, then the time required for each working node to train one step is: In synchronous training, all working nodes access the same parameter server at the same time. Assume that the bandwidth of the parameter server and the working node is B0. When a working node establishes connections with p parameter servers at the same time, the maximum bandwidth of each connection is B0 / p. When a parameter server establishes connections with w working nodes at the same time, the maximum bandwidth of each connection is B0 / w. The bandwidth B between the parameter server and the working node should be the smaller value of B0 / w and B0 / p: The throughput is equal to the total batch size divided by the time of one step of training. Assuming the total batch size is M, the throughput under synchronous training is: After linear approximation of min, it is simplified to, where θ is a non-negative coefficient: If there is an optimal performance ratio of w / p in a production environment, it can be further simplified to: In asynchronous training, each worker node can be considered to access the parameter server in a staggered manner. The bandwidth between the worker node and the parameter server is B0 / p. The total throughput is w times the throughput of each node. The throughput under asynchronous training is: Take w and p as independent variables, and other variables as fitting parameters to merge similar items according to independent variables, and finally rename the parameters to θ, where θ is a non-negative coefficient: Based on the existence of an optimal performance ratio of w / p in a production environment, the equation can be further simplified:

5. The computing resource elastic allocation system for online training according to claim 4, characterized in that: The online learning automatic scaler is implemented by expanding or reducing the online training job based on the estimate of the required working nodes, and formulating the requirement for the number of online learning working nodes that needs to be met at each time t: min{w(t)∈Z+|w(t)} stf(w(t))>L(t) Where w(t) represents the number of worker nodes allocated at time t, f(w) represents the throughput of w worker nodes using the resource throughput model fitted in the previous section, and L(t) represents the workload at time t obtained by prediction; Keep the batch size constant during model training and keep the ratio of w / p constant during expansion.

Images