An online training-oriented computing power resource elastic allocation system

Abstract

The disclosure provides an online training-oriented computing resource elastic allocation system. The system comprises a user end, a cluster resource manager, an elastic scheduler and a work node using a GPU to be allocated. The elastic scheduler comprises three components: a work load predictor, a work node estimator and an online learning automatic scaler. The work node estimator determines the most suitable number of work nodes for distributed training by using a resource throughput model on the basis of the prediction of future work load. The online learning automatic scaler expands or reduces the work nodes and allocates more GPUs to the work nodes or recycles the GPUs to the resource pool after subsequent approval by the cluster resource manager. Thus, the disclosure realizes the effects of better guaranteeing the performance of online training models and saving computing resources.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of computers, and more specifically, to an online training oriented elastic allocation system of computing power resources. BACKGROUND

[0002] Deep learning (DL) training is a process composed of a large number of iterations, each of which reads and processes a certain number of data samples, also known as mini-batches. Unlike offline deep learning training that obtains data from an offline dataset, an online deep learning training pipeline consumes streaming data provided by streaming platforms such as Kafka. Online training is a dynamic program that can incrementally learn from data streams in real time and adapt the evolving algorithm to newly arrived data. In order to support large-scale model training using massive data, DL jobs are submitted to GPU clusters and use different parallel techniques for distributed model training through GPUs.

[0003] In a large-scale production environment, the streaming sample production rate for online model training often presents periodic fluctuations. User behavior such as user clicks, application usage, transaction volume, etc. has obvious peak and non-peak characteristics. A typical traffic fluctuation pattern on weekdays is as follows: data throughput reaches a minimum in the early morning and a maximum in the evening. It is not difficult to understand that the cluster can usually maintain a high data production rate for a long time after the evening, and sharply decrease at night. In contrast, the average value of data traffic is higher on weekends or holidays--user data starts to rise and accumulate from the early morning and remains at a very high level for the rest of the day. The difference between the maximum and minimum values of traffic is 7.65 times, and we have observed from production clusters that the difference ranges from 5 times to 30 times. This indicates that dynamic auto-scaling on GPU clusters has great potential in reducing resource waste and improving model training throughput.

[0004] The purpose of the present disclosure is to design and implement an elastic GPU scheduling system for real-world large-scale online model training, with the following research and engineering requirements:

[0005] i) Proactive auto-scaling based on prediction. Existing auto-scaling strategies are reactive strategies that respond to system load changes, but the time-consuming auto-scaling is too long for online training. Once auto-scaling is triggered, a series of subsequent steps including model saving, container startup to prepare new job nodes, image pulling, model loading, data stream reconstruction, etc. When all these steps are completed, auto-scaling can be outdated because the system load changes rapidly over time and a large number of fresh data samples are not used in time. In addition, the reactive strategy is prone to frequent resource allocation, which inevitably leads to high preemption overhead. As mentioned earlier, there are significant timing (e.g., obvious peaks and troughs) behavior patterns in user clicks, video viewing time, and frequency. This observation reveals the potential of adopting a proactive prediction method to predict traffic and guide proactive auto-scaling.

[0006] ii) Timely and accurate resource estimation. The right number of GPUs must be calculated in a timely and accurate manner to adapt to the time-varying size of data samples. While some existing works use analytical performance-resource models to determine the optimal amount of resources to optimize cluster-wide training throughput, they either only focus on CPU / memory amount or only work on general DNN models in offline training.

[0007] To meet these needs, in this paper, we propose KALE, a new elastic GPU scheduling system to improve the performance of online deep learning model training. To achieve this goal, we mainly propose an effective auto-scaling mechanism by using data traffic prediction and resource estimation based on resource-performance models. The key design idea is to use historical data traffic to proactively predict the number of data samples that the training job needs to consume and determine the most appropriate parallelism, i.e., the number of GPUs, so that the job has enough resources to process and train a good enough model using data. Once the GPU resources are determined, auto-scaling is performed to adjust the allocated GPUs, with each GPU allocated to a worker node to adapt to incoming data samples in a timely and most appropriate manner. SUMMARY

[0008] The purpose of the embodiments of the present disclosure is to provide an online training-oriented computing resource elastic allocation system. The main research goal of the present invention is to automatically generate code summary documents at the project level by combining hierarchical model and program analysis techniques. The research object is English documents, and the programming language is Java. The summary document is mainly a short natural language summary and does not involve complex information such as directory structure, usage instructions, etc. According to this model, an open source code repository summary automatic generation tool is designed and implemented in the form of an Eclipse plug-in on the Eclipse IDE.

[0009] In one general aspect, there is provided an online training oriented elastic allocation system of computing resources, including a user end, a cluster resource manager, an elastic scheduler, and a worker node using GPUs to be allocated, the working process of the system being: the user submits detailed configurations and preference settings of an online learning job to the cluster resource manager, the cluster resource manager analyzes and approves the submitted configurations and returns them to the user; then, the configurations are submitted to the elastic scheduler to manage data flow and resource allocation throughout the life cycle of the online training job, the elastic scheduler including three components: a workload predictor, a worker node estimator, and an online learning automatic scaler;

[0010] The workload predictor receives the submitted configurations, trains a time series prediction model using historical data to predict the upcoming data flow that the online training job may face in the upcoming period, adopts a Transformer-based time series prediction model, uses a time series decomposition method to decompose time data into trend, seasonality, periodicity, and noise components, converts the input time series data into attention vectors through an encoder, combines position encoding to capture time series information, and finally uses a decoder to predict future workload demand;

[0011] The worker node estimator determines the optimal number of worker nodes for distributed training based on the prediction of future workload;

[0012] The online learning automatic scaler expands or reduces the worker nodes and allocates more GPUs to the worker nodes or recycles the GPUs into the resource pool after subsequent approval by the cluster resource manager.

[0013] The detailed configurations and preference settings include model algorithms and parameters, data sources and access methods, and model export addresses.

[0014] The worker node estimator is for a parameter server architecture, which consists of two types of nodes: parameter servers and worker nodes;

[0015] The parameter server is responsible for storing and maintaining global model parameters, and one or more parameter server nodes are deployed to distribute the storage of model parameters according to the system size and requirements, and multiple parameter server nodes usually synchronize with each other to ensure that the parameters they hold are consistent; the worker node is responsible for the actual model training task, each worker node independently processes a part of the training data, and parallel computing is used to accelerate the entire training process, the worker node obtains the latest model parameters from the parameter server, performs local calculation, and sends the calculated gradient back to the parameter server;

[0016] The workflow of model training under the parameter server architecture is as follows:

[0017] Step 1: parameter initialization, at the beginning of training, the parameter server initializes the global model parameters, which can be randomly generated or obtained by loading a pre-trained model;

[0018] Step 2: parameter pulling, before each calculation step, the worker node pulls the latest model parameters from the parameter server, which ensures that each worker node uses the latest version of the model parameters for training;

[0019] Step 3: local training, after pulling the parameters, the worker node uses a batch of training samples for forward propagation and back propagation calculation locally; in the forward propagation stage, the worker node calculates the predicted value and loss function according to the current model parameters; in the back propagation stage, the worker node calculates the gradient according to the loss function;

[0020] Step 4: gradient pushing, after calculating the gradient locally, the worker node sends the gradient back to the parameter server;

[0021] Step 5: parameter updating, after receiving the gradient from multiple worker nodes, the parameter server updates the global model parameters according to the gradient;

[0022] Step 6: iteration, the above steps are repeated until the model converges or reaches the predetermined number of training rounds;

[0023] The time required for each worker node to train one step in the parameter server architecture is: T = T D + T F + T B + T U + T O + T C ; where T D represents the time of pulling parameters, T F represents the time of forward propagation, T B represents the time of back propagation, T U represents the time of pushing parameters, T O represents the time of optimizer updating parameters, and T C represents the remaining additional time overhead.

[0024] For the dense parameters and sparse parameters in the training process, assume that the number of parameter servers is p, the number of worker nodes is w, the batch size of each worker node is m, the size of the dense parameters is D, and the size of the sparse parameters is m x S; assuming that the parameters are uniformly distributed on the parameter server, the number of dense parameters sent by the worker node to the parameter server is D / p, and the number of sparse parameters is m x S / p; define the forward propagation time of training a small batch as Tforward , the reverse propagation time is T backward , T F =m x T forward , T B =T backward .

[0025] T D =(m x S) / (B x p), T U =(m x S+D) / (B x p); let the time for updating a mini-batch parameter be T update , then T O =m x S x T update / p; other additional overheads are assumed to be linearly related to p and w, T C =λ x p+λ' x w, so the time required for each worker node to train one step is:

[0026]

[0027] In synchronous training, all worker nodes access the same parameter server at the same time; let the bandwidths of the parameter server and the worker nodes be B0, when a worker node simultaneously establishes a connection with p parameter servers, the bandwidth of each connection is at most B0 / p, and when a parameter server simultaneously establishes a connection with w worker nodes, the bandwidth of each connection is at most B0 / w, the bandwidth B between the parameter server and the worker nodes should be the smaller of B0 / w and B0 / p:

[0028]

[0029] The throughput is equal to the total batch size divided by the time for one step of training, let the total batch size be M, then the throughput under synchronous training is:

[0030]

[0031] After linear approximation of min, it is simplified to where θ is a non-negative coefficient:

[0032]

[0033] If there is an optimal performance ratio of w / p in the production environment, it is further simplified to:

[0034]

[0035] In asynchronous training, each worker node can be considered to access the parameter server staggered, the bandwidth between the worker node and the parameter server is B0 / p, the total throughput is w times the throughput of each node, and the throughput under asynchronous training is:

[0036]

[0037] Similarly, let w and p be the independent variables, and other variables be the fitting parameters, merge the same items according to the independent variables, and finally rename the parameters as θ, which is a non-negative coefficient:

[0038]

[0039] If there is an optimal performance ratio of w / p in the production environment, the equation can be further simplified:

[0040]

[0041] The implementation of the online learning automatic stretcher is to expand or reduce the online training job according to the estimation of the required worker nodes, and formulate the requirement of the number of online learning worker nodes required at each time t:

[0042] min{w(t)∈Z+|w(t)}

[0043] s.t.f(w(t))>L(t)

[0044] Where w(t) represents the number of worker nodes allocated at time t, f(w) represents the throughput of w worker nodes using the resource throughput model fitted in the previous section, and L(t) represents the workload at time t predicted by the prediction;

[0045] Keep the batch size unchanged during model training, and keep the ratio of w / p constant during expansion.

[0046] The innovation of the embodiment of the present application is:

[0047] (1) Better guarantee the performance of online training model. Due to active prediction, the default rate and cumulative delay of KALE are greatly reduced compared with reactive expansion methods such as Autopilot and HPA. Compared with HPA, KALE significantly reduces the cumulative delay and downtime, reduces by 69.2% and 33.1% respectively, and reduces the SLO default rate from 19.57% to only 2.6%. As shown in the middle part of the upper of Figure 3 As shown in the middle part of the upper of, during the peak of the workload, the GPU allocation of the reactive expansion method has not reached the highest point. The GPU allocation curve of Autopilot is slightly out of position with the peak of the workload, which may cause sample accumulation and cause delay. This problem is more serious in HPA, and the number of worker nodes reaches the peak 20 minutes after the peak of the workload. Expanding at the highest peak means that the same downtime accumulates more samples, making it more difficult to eliminate delay, resulting in HPA having the highest default rate and cumulative delay.

[0048] (2) Greater efficiency in conserving computing resources. Thanks to accurate resource throughput modeling, KALE significantly reduces GPU hours by up to 46.8% compared to Madu. While Madu predicts future load, its linear fitting algorithm for GPU throughput leads to over-allocation of resources. Figure 3 As shown, KALE's curve resembles a pair of tweezers, tightly clustering around the peak, while Madu exhibits excessive and persistent allocation at the peak, reflecting its high GPU hours. Furthermore, KALE and Madu are comparable in terms of default rate and cumulative latency, with KALE showing a 2.60% default rate and 2203.69 minutes of cumulative latency, compared to Madu's 2.57% default rate and 2302.85 minutes of cumulative latency. However, KALE achieved this performance in just 242 GPU hours, while Madu required 455 GPU hours. KALE's accuracy in resource throughput modeling and scaling timing allows it to achieve performance comparable to Madu's high resource usage with significantly lower resource consumption. Due to its sawtooth detection, KALE significantly reduces downtime, by up to 59.5% (from 269 minutes in HPA to 109 minutes in KALE). Reactive methods are more prone to allocation fluctuations due to a lack of foresight. Attached Figure Description

[0049] The above and other objects and features of this disclosure will become clearer from the following description taken in conjunction with the accompanying drawings.

[0050] Figure 1 The data samples in the existing technology exhibit periodic temporal characteristics;

[0051] Figure 2 This is a schematic diagram illustrating the overall architecture of KALE, a computing resource elastic allocation system for online training according to an embodiment of the present disclosure.

[0052] Figure 3 This is a graph showing a comparison of the number of GPUs in an end-to-end experiment of a system according to an embodiment of the present disclosure;

[0053] Figure 4 This is a schematic diagram illustrating the GPU utilization of a production cluster according to an embodiment of the present disclosure, wherein the left diagram shows the cluster without KALE and the right diagram shows the cluster with KALE. Detailed Implementation

[0054] The following detailed description is presented to aid the reader in gaining a comprehensive understanding of the methods, apparatuses, and / or systems described herein. However, various changes, modifications, and equivalents can be used, and thus particular embodiments described herein are not intended as being exhaustive of the ways in which the methods, apparatuses, and / or systems described herein can be practiced. For instance, the order in which operations are described is not intended to be limiting, except insofar as a particular order can be required for practical implementations described herein, and

[0055] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including" and "has" are to be construed to be inclusive (i.e., to include both instances of items in the

[0056] As used herein, the term "and / or" includes any one of the associated listed items, as well as any combination of any two or more of the associated listed items.

[0057] Although the terms "first," "second," and "third" can be used herein to describe various elements, components, regions, layers, or sections, these elements, components, regions, layers, or sections should not be limited by these terms. Rather, these terms are only used to distinguish one element, component, region, layer, or section from another element, component, region, layer, or section. Thus, a first element, a first component, a first region, a first layer, or a first section described in the examples herein could also be termed a second element, a second component, a second region, a second layer, or a second section.

[0058] In the description, when an element (such as a layer, a region, or a substrate) is referred to as being "on" another element, "connected to" or "coupled to" another element, it can be directly on, directly connected to, or directly coupled to the other element, or one or more other elements can be interposed therebetween. In contrast, when an element is referred to as being "directly on," "directly connected to," or "directly coupled to" another element, there are no other elements interposed therebetween.

[0059] The terminology used herein is for the purpose of describing various examples only and is not intended to be limiting. As used herein, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises," "comprising," "includes," "including" and "has" are to be construed to be inclusive (i.e., to include both instances of items in the

[0060] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains upon understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalistic manner.

[0061] Furthermore, in the description of the examples, detailed descriptions of well-known related structures or functions will be omitted when it is believed that such detailed descriptions would lead to a vague interpretation of this disclosure.

[0062] Figure 2 This is a schematic diagram illustrating an elastic allocation system of computing resources for online training according to an embodiment of the present disclosure.

[0063] Overall architecture:

[0064] The basic workflow and interactions between KALE components are detailed below: Users can submit detailed configurations and preferences for online learning jobs to the cluster resource manager (e.g., Kubernetes). This information typically includes model algorithms and parameters, data sources and access methods, and model export addresses. The resource manager then analyzes and approves the submitted configurations (steps ① and ②).

[0065] Next, the elastic scheduler will be responsible for managing data traffic and resource allocation throughout the entire lifecycle of the online training job (step ③). The core of KALE is predicting the number of upcoming data samples before dynamically allocating the resources best suited for the online training job. The elastic scheduler comprises three key components—a workload predictor, a worker node estimator, and an online learning autoscalor.

[0066] Workload Predictor. Internally, a time-series prediction model is trained using historical data to predict the upcoming data traffic that online training jobs may face in the coming period (step ④). Technically, to address the periodic temporal characteristics of online learning workloads, we employ a Transformer-based time-series prediction model. Specifically, we use a time-series decomposition method to break down the time data into components such as trend, seasonality, periodicity, and noise. An encoder transforms the input time-series data into an attention vector, combined with positional encoding to capture time-series information. Finally, a decoder is used to predict future workload demands. Looking ahead, our design philosophy adopts a modular architecture, which not only facilitates the current implementation of the Transformer-based model but also paves the way for seamless integration of more advanced prediction models in the future.

[0067] Work node number estimator. Based on the traffic prediction, we employ a resource throughput model to determine the optimal number of work nodes for distributed training, so that the upcoming data traffic can be fully utilized to obtain a competitive online model (step ⑤). The technical details will be discussed in Section 3.2.

[0068] Online learning autoscaler. The decision result, i.e., the number of work nodes, will be used by the online learning autoscaler, which is responsible for scaling up or down the work nodes, allocating more GPUs to the work nodes or recycling GPUs back to the resource pool (step ⑥).

[0069] Subsequent approval of resource manager (steps ⑦ and ⑧). The autoscaler also optimizes the scaling strategy of online learning jobs to avoid too frequent resource adjustments and actively collects system metrics. We will elaborate on the working principle of the autoscaler in Section 3.3.

[0070] Work node number estimator:

[0071] The mainstream training framework for online training usually adopts the parameter server architecture. The parameter server architecture is a distributed computing architecture designed specifically for large-scale machine learning tasks, particularly suitable for training deep learning models. This architecture can effectively solve the computational and communication bottlenecks in large-scale data training, significantly improving training efficiency and performance. The parameter server architecture mainly consists of two types of nodes: parameter servers (PS) and worker nodes (Worker Node).

[0072] Parameter Server (PS): The parameter server is responsible for storing and maintaining global model parameters. Depending on the system size and requirements, one or more parameter server nodes can be deployed to distribute the storage of model parameters. Multiple parameter server nodes usually synchronize with each other to ensure that the parameters they hold are consistent.

[0073] Worker Node (Worker Node): The worker node is responsible for the actual model training task. Each worker node independently processes a portion of the training data, accelerating the entire training process through parallel computing. The worker node obtains the latest model parameters from the parameter server, performs calculations locally, and sends the calculated gradients back to the parameter server.

[0074] The workflow of model training under the parameter server architecture is as follows:

[0075] 1) Parameter initialization: At the beginning of training, the parameter server initializes the global model parameters. These parameters can be randomly generated or obtained by loading a pre-trained model.

[0076] 2) Parameter Pull: Before each computation step, the worker node pulls the latest model parameters from the parameter server. This process ensures that each worker node uses the latest version of model parameters for training.

[0077] 3) Local Training: After pulling the parameters, the worker node performs forward and backward propagation calculations using a batch of training samples locally. During the forward propagation phase, the worker node computes the predicted values and loss function based on the current model parameters. During the backward propagation phase, the worker node computes the gradients based on the loss function.

[0078] 4) Gradient Push: After computing the gradients locally, the worker node sends these gradients back to the parameter server. This process is called gradient push.

[0079] 5) Parameter Update: After receiving gradients from multiple worker nodes, the parameter server updates the global model parameters based on these gradients. Typically, some optimization algorithms such as Stochastic Gradient Descent (SGD) or its variants (e.g., Adam) are used for parameter update.

[0080] 6) Iteration: The above steps are repeated until the model converges or reaches the predetermined number of training rounds. The time required for each worker node to train one step in the parameter server architecture is shown below. D T F represents the time for pulling parameters, T B represents the time for forward propagation, T U represents the time for backward propagation, T O represents the time for pushing parameters, T C represents the time for optimizer updating parameters, T D represents the remaining additional time overhead.

[0081] T = T F + T B + T U + T O + T C

[0082] In online businesses of internet companies (such as search, recommendation, and advertising), data is usually sparse. This means that most features are zero or missing most of the time. To handle these sparse data and improve the performance and efficiency of the model, parameters in online learning are usually divided into dense parameters and sparse parameters, and are stored and updated separately.

[0083] Dense parameters: Dense parameters refer to those feature parameters that almost always appear in all samples. These parameters participate in the calculation comprehensively during the model training process, and the full gradient information is used for each update.

[0084] Sparse parameters: Sparse parameters refer to those feature parameters that are missing or zero in most samples. Since the frequency of these parameters appearing in each batch is low, their update is mainly based on the actual appearing feature key values.

[0085] To better handle these two types of parameters, the online training system adopts different storage and update strategies. Dense parameters and sparse parameters are stored in different databases or memory structures. Dense parameters are usually saved in memory for fast access and update; while sparse parameters may be stored in a distributed database, indexed and searched by feature key values. Dense parameter update: In each training iteration, dense parameters are updated comprehensively according to the calculated global gradient. This means that each dense parameter uses all sample information in the current batch to adjust its value, ensuring the global consistency of the model. Sparse parameter update: The update of sparse parameters depends on specific key-value pairs. In each training iteration, only those actually appearing sparse feature parameters are updated according to the corresponding gradient. This way reduces unnecessary computational overhead and improves update efficiency.

[0086] Generally speaking, the number of sparse parameter updates in each round of training is positively related to the batch size. Batch size (BatchSize) refers to the number of samples used in each training iteration. When the batch size increases, more sparse feature parameters will appear in this batch, resulting in more sparse parameters needing to be updated. Conversely, if the batch size is small, the number of sparse parameters that need to be updated in each round of training will also decrease.

[0087] Let the number of parameter servers be p, the number of worker nodes be w, the batch size of each worker node be m, the size of dense parameters be D, and the size of sparse parameters be m×S. Assuming that the parameters are evenly distributed on the parameter servers, the number of dense parameters sent by the worker nodes to the parameter servers is D / p, and the number of sparse parameters is m×S / p.

[0088] In one-step training, the forward propagation time is linearly related to the batch size, while the backward propagation time is independent of the batch size. Let the forward propagation time for training a small batch be T forward , and the backward propagation time be T backward . Therefore, T F = m × T forward , T B = T backward .

[0089] Sparse and dense parameters are transmitted separately, so they can overlap in transmission to save time. Generally, the online training of sparse scenarios results in longer transmission time of sparse parameters. In this case, the pull time of dense parameters can overlap with the update time of sparse parameters, so that sparse parameters monopolize the bandwidth when pulling, improving the speed. Therefore, the time for the worker node to pull parameters is (m x S) / (B x p), and the time for pushing gradients is (m x S + D) / (B x p), not the symmetric process. Therefore, T D = (m x S) / (B x p), T U = (m x S + D) / (B x p). Let the time to update one small batch of parameters be T update , then T O = m x S x T update / p. Other additional overheads (such as control commands, etc.) are assumed to be linearly related to p and w. Therefore, T C = λ x p + λ' x w. Then the following formula can be derived:

[0090]

[0091] In synchronous training, all worker nodes access the same parameter server at the same time. Let the bandwidth of the parameter server and the worker node be B0. When a worker node simultaneously establishes a connection with p parameter servers, the bandwidth of each connection is at most B0 / p, and when a parameter server simultaneously establishes a connection with w worker nodes, the bandwidth of each connection is at most B0 / w. Therefore, the bandwidth B between the parameter server and the worker node should be the smaller of B0 / w and B0 / p.

[0092]

[0093] The throughput is equal to the total batch size divided by the time of one-step training. Let the total batch size be M, then the formula of the throughput under synchronous training can be derived.

[0094]

[0095] After linear approximation of min, it can be modeled as follows, where θ is a non-negative coefficient.

[0096]

[0097] If there is an optimal performance ratio of w / p in the production environment, the equation can be further simplified.

[0098]

[0099] In asynchronous training, each worker can be considered to access the parameter server staggered. Therefore, the bandwidth between the worker and the parameter server is B0 / p. Moreover, in asynchronous training, each worker trains by mini-batch independently without waiting for other nodes, so the total throughput is w times of the throughput of each node. The throughput formula under asynchronous training can be derived as follows:

[0100]

[0101] Similarly, take w and p as independent variables, and merge like terms according to independent variables with other variables as fitting parameters. Finally, rename the parameters as θ, which is a non-negative coefficient:

[0102]

[0103] If there is an optimal performance ratio of w / p in the production environment, the equation can be further simplified:

[0104]

[0105] Application example. We use an example to show how the theoretical analysis and the proposed resource throughput model work. Suppose we have a synchronous online training job with a batch size of M = 16384. The parameter fitting results are as follows: θ0= 0.00035, θ1= 2.5726, θ2= 0.9824, θ3= 0.02786. If the workload predictor predicts that the foreseeable traffic is 30,000, then we can check the throughput produced under a given number of workers. In this case, for example, assume f(10) = 30007.326, then the most suitable number of workers to handle the upcoming data samples will be ten. As will be explained in detail in the present specification, the autoscaler can then adjust the allocated GPUs to meet the needs of elastic training.

[0106] Autoscaler:

[0107] According to the estimate of the required number of workers, the online training job can be expanded or reduced to ensure that its throughput meets the upstream data traffic. Specifically, we formulate the requirement for the number of online learning workers that need to be met at each time t:

[0108] min{w(t)∈Z+|w(t)}

[0109] s.t.f(w(t))>L(t)(10)

[0110] where w(t) denotes the number of workers allocated at time t, f(w) denotes the throughput of w workers using the resource throughput model fitted in the previous section, and L(t) denotes the workload at time t predicted by the workload predictor.

[0111] During model training, we keep the batch size constant to ensure the stability and effectiveness of model training. We follow a simple but effective pause-and-restart strategy to automatically scale the number of workers in running, which is currently adopted by mainstream deep learning frameworks. To reduce the complexity of fitting the proposed resource throughput model, we keep the ratio of w / p constant during scaling.

[0112] Optimization. Although the above general auto-scaling can calculate the appropriate number of workers for upcoming data samples, it can experience frequent auto-scaling operations, which in turn can lead to unstable resource allocation. This is particularly serious when there are bursts and jagged periods of data traffic. To solve this problem, we actually propose a jagged calibration mechanism to timely detect jagged periods and smooth them with line segments.

[0113] The optimized workflow is as follows: first, calculate whether there will be a "jagged" phenomenon in resource allocation after this resource allocation, if there is no "jagged" phenomenon, then normally allocate resources, otherwise, perform smoothing processing, and calculate the optimized resource quantity that should be allocated. The core of this process is to find out the case where the resource quantity changes like a jagged phenomenon after a short maintenance time, i.e., the "jagged" phenomenon. This is because if the maintenance time after scaling is too short, the optimal allocation time of resources will end before the scaling is completed. In this case, there is no need to perform this scaling. The change_threshold parameter represents the threshold for detecting resource changes, which is usually set to 1. The hold_threshold parameter represents the threshold for the maintenance time after the change, which will be marked as a jagged phenomenon when the maintenance time is less than this threshold. The optimized resource quantity after smoothing is equal to the larger value of the resource quantity on both sides of the jagged phenomenon to ensure sufficient resource allocation. Different choices of hold_threshold will lead to different resource allocation results, as shown in Figure 3 To demonstrate the effectiveness of jagged smoothing, we collect the workload information for a day and calculate the resource quantity allocated at each time, which is represented by the blue line in Figure 3 It can be clearly seen that the jagged phenomenon occurs near the horizontal coordinates 300, 500, and 1300. After jagged smoothing, the thresholds set to 10 and 30 minutes eliminate the visible jagged phenomenon. For other scaling times, there are slight differences. In general, the larger the threshold, the fewer the scaling times, the smaller the overhead, but the lower the accuracy of resource allocation, and vice versa.

[0114] Experimental validation:

[0115] In this section, we conduct a comprehensive end-to-end evaluation of the online training resource elasticity allocation system to demonstrate its effectiveness. We compare the online training resource elasticity allocation system with several other automatic scaling strategies based on key performance indicators.

[0116] Evaluation metrics:

[0117] To effectively evaluate the performance of the system, we use the following key indicators:

[0118] SLO violation rate. In the previous section, we described the impact of delay on model performance. Based on industry experience, an AUC drop of more than two thousandths should be taken seriously, so according to this system target and the impact of delay on AUC, we can draw the SLO violation range. For the model used in the experiment, a delay of more than 20 minutes is considered a default violation. The smaller this indicator is, the better.

[0119] Cumulative delay. The cumulative delay indicator is used to describe the overall situation of the delay, i.e. the sum of the delay per minute, and measures the degree of accumulation of streaming data samples during training, as shown in the following formula. The smaller this indicator is, the better.

[0120]

[0121] Downtime. Downtime describes the time overhead of scaling up and down, and the smaller the indicator is, the better. Since online learning involves workflows such as saving models, starting new worker node containers, pulling images, loading models, and rebuilding consumer groups, the overhead of scaling up and down cannot be ignored. Fewer but more accurate adjustments are the direction we pursue.

[0122] GPU hours. GPU hours describe the total GPU time used by online learning, calculated as shown in the following formula. The smaller this indicator is, the better.

[0123] Comparison with baseline:

[0124] To evaluate the performance of the system, we compare it with four different scheduling schemes:

[0125] Adequate Resources: Always allocate adequate resources, do not scale.

[0126] Kubernetes default horizontal pod auto-scaling (HPA): a widely used responsive auto-scaling method that automatically adjusts the number of pods by monitoring resource usage and following pre-set rules.

[0127] Autopilot: An auto-scaling solution proposed by Google, similar to HPA. Autopilot uses a moving window to collect recent resource usage statistics and scales resources based on these statistics (e.g., average CPU and memory utilization), making it more accurate than HPA's resource prediction.

[0128] Madu: An active scaling solution that combines workload prediction, scaling containers based on workload prediction of individual microservices.

[0129] Experimental results:

[0130] Table 1: End-to-end experimental evaluation results of the system

[0131]

[0132] As shown in the table, our method M outperforms HPA, Autopilot, Madu, and other auto-scaling solutions in terms of violation rate, cumulative latency, downtime, and GPU hours. In terms of violation rate and cumulative latency, our method M is comparable to Madu, with a violation rate of 0.0260 and a cumulative latency of 1203.69 minutes, while Madu has a violation rate of 0.0257 and a cumulative latency of 1702.85 minutes. However, our method M only uses 242 GPU hours, while Madu requires 455 GPU hours. The precise resource throughput modeling and scaling timing of our method M enable it to achieve comparable performance to Madu with significantly lower resource consumption.

[0133] Due to active prediction, the violation rate and cumulative latency of our method M are significantly reduced compared to reactive scaling methods such as Autopilot and HPA, with a reduction of up to 17% in violation rate (from 0.1957 for HPA to 0.0260 for our method M) and up to 83.2% in cumulative latency (from 7150.61 minutes for HPA to 1203.69 minutes for our method M). In terms of downtime, our method M prevents excessive resource allocation frequency through sawtooth detection, significantly reducing time costs, with a maximum reduction of 59.5% in downtime (from 269 minutes for HPA to 109 minutes for our method M). Additionally, compared to sufficient resources, our method M reduces GPU hours by up to 64% (from 672 GPU hours to 242 GPU hours).

[0134] Figure 3 The figure shows the changes in GPU allocation over a 24-hour period for different methods. The differences in allocation strategy and quantity are apparent. For reactive methods, the allocation timing is usually lagging, which is the main reason for sample accumulation and SLO violation. In addition, reactive methods lack foresight and are prone to allocation fluctuations.

[0135] While Madu predicts future load, its linear fitting algorithm of GPU throughput leads to over-provisioning of resources. The zoomed-in portion of the figure shows the allocation at the peak of each method. The curve of the proposed method (our method M) resembles a pair of tweezers, tightly pinching the peak, while Madu shows excessive and persistent allocation at the peak, reflecting its high GPU hours.

[0136] The hysteresis of reactive methods like Autopilot and HPA is also clearly visible. At the peak of the workload, the GPU allocation of these methods has not yet reached the peak. The GPU curve of Autopilot is slightly misaligned with the peak of the workload, leading to sample accumulation. The problem of HPA is more serious, its GPU number reaches the peak only 20 minutes after the peak of the workload. Expanding at the highest peak means that the same downtime accumulates more samples, and it is more difficult to eliminate the delay, so HPA has the highest violation rate and accumulated delay.

[0137] Finally, the last key point is that our method M avoids allocation fluctuations due to sawtooth detection, resulting in the least number of resource adjustments and the shortest downtime.

[0138] Production environment deployment embodiments:

[0139] KALE has been deployed on a production cluster, serving tens of thousands of online recommendation model training jobs. To verify the design and implementation of KALE in real-world production environments while ensuring that the cluster works normally without interruption, we collected tracking data for some jobs and analyzed the relevant statistical information. This information comes from a GPU / CPU hybrid cluster that contains over 8000 GPUs and over 500,000 CPU cores. Given that 90% of the recommendation models in the actual production environment are trained online and updated in real time, the main performance indicators at the cluster level are GPU utilization and peak throughput. These indicators are crucial for ensuring that online training can effectively utilize GPU resources while meeting the service level agreement (SLA) of model updates.

[0140] To evaluate the impact of the key techniques proposed in KALE on production systems, we collected tracing data and metrics for a week from June 8 to June 115, 2024, after deploying KALE, and compared them with the tracing data of the same cluster from April 8 to April 15, 2024. The average GPU utilization at the cluster level can be improved by 40%, and, due to the elastic scheduling mechanism with accurate estimation of the number of GPUs required, the system throughput can be consistently maintained at a high level without significant utilization drops and peak fluctuations. At the same time, this indicates that more training jobs can be allowed and executed in the cluster. The cluster is now used to host the company's largest ranking models, which have model sizes of tens of TB. The proposed elastic scheduling strategy can achieve minute-level elasticity while also meeting peak throughput demands.

[0141] While some embodiments of the disclosure have been shown and described, it will be understood by those skilled in the art that modifications can be made to these embodiments without departing from the principles and spirit of the disclosure, the scope of which is defined by the claims and their equivalents.

Claims

1. An online training-oriented computing power resource elastic allocation system, characterized in that, The system includes a user terminal, a cluster resource manager, an elastic scheduler, and a work node using a GPU to be allocated. The working process of the system is as follows: the user submits detailed configurations and preference settings of an online learning job to the cluster resource manager, the cluster resource manager analyzes and approves the submitted configurations and returns them to the user; Then, the configurations are submitted to the elastic scheduler to manage data traffic and resource allocation throughout the life cycle of the online training job. The elastic scheduler includes three components: a workload predictor, a work node estimator, and an online learning automatic scaler; The workload predictor receives the submitted configurations, trains a time series prediction model using historical data to predict the upcoming data traffic that the online training job may face in the upcoming period, adopts a Transformer-based time series prediction model, decomposes time data into trend, seasonality, periodicity, and noise components using a time series decomposition method, converts the input time series data into an attention vector through an encoder, combines position encoding to capture time series information, and finally uses a decoder to predict future workload demand; The work node estimator determines the optimal number of work nodes for distributed training based on the prediction of future workloads using a resource throughput model; The online learning automatic scaler expands or reduces the work nodes and allocates more GPUs to the work nodes or recycles the GPUs into the resource pool after subsequent approval by the cluster resource manager.

2. The online training-oriented computing resource elastic allocation system of claim 1, wherein, The detailed configurations and preference settings include model algorithms and parameters, data sources and access methods, and model export addresses.

3. The online training-oriented computing resource elastic allocation system of claim 2, wherein, The work node estimator is for a parameter server architecture composed of two types of nodes: parameter servers and work nodes; The parameter server is responsible for storing and maintaining global model parameters. One or more parameter server nodes are deployed to distribute the storage of model parameters according to the system size and requirements. Multiple parameter server nodes usually synchronize with each other to ensure that the parameters they hold are consistent. The work node is responsible for the actual model training task. Each work node independently processes a portion of the training data, accelerates the entire training process through parallel computing, and obtains the latest model parameters from the parameter server. After local calculation, the calculated gradient is sent back to the parameter server; The workflow of model training under the parameter server architecture is as follows: Step 1: parameter initialization. At the beginning of training, the parameter server initializes the global model parameters. The parameters can be randomly generated or obtained by loading a pre-trained model; Step 2: parameter pulling. Before each calculation step, the work node pulls the latest model parameters from the parameter server. This process ensures that each work node uses the latest version of the model parameters for training. Step 3: Local training, after pulling the parameters, the worker node uses a batch of training samples to perform forward propagation and back propagation calculations locally. In the forward propagation stage, the worker node calculates the predicted value and loss function according to the current model parameters; in the back propagation stage, the worker node calculates the gradient according to the loss function; Step 4: Gradient pushing, after the worker node calculates the gradient locally, it sends the gradient back to the parameter server; Step 5: Parameter updating, after the parameter server receives the gradient from multiple worker nodes, it updates the global model parameters according to the gradient; Step 6: Iteration, the above steps are repeated until the model converges or reaches the predetermined number of training rounds; The time required for each worker node to train one step in the parameter server architecture is: T = T D +T F +T B +T U +T O +T C ; where T D T represents the time it takes to fetch parameters. F T represents the forward propagation time. B T represents the time of backpropagation. U T represents the time of the push parameter. O T represents the time it takes for the optimizer to update parameters. C This indicates the remaining additional time cost.

4. The computing resource elastic allocation system for online training of claim 3, wherein, For the dense parameters and sparse parameters in the training process, let the number of parameter servers be p, the number of worker nodes be w, the batch size of each worker node be m, the size of the dense parameters be D, and the size of the sparse parameters be m×S; Assuming that the parameters are uniformly distributed on the parameter server, the number of dense parameters sent by the worker to the parameter server is D / p, and the number of sparse parameters is m×S / p; define the forward propagation time of training a small batch as T forward , the backward propagation time as T backward , T F =m×T forward , T B =T backward ; T D = (m x S) / (B x p), T U = (m x S + D) / (B x p); let the time to update one mini-batch parameter be T update , then T O = m x S x T update / p; other additional overheads are assumed to be linear in p and w, T C = lambda x p + lambda' x w, then the time required for one step of training by each worker node is: In synchronous training, all worker nodes access the same parameter server at the same time. Let the bandwidth of the parameter server and the worker node be B0. When a worker node simultaneously establishes a connection with p parameter servers, the bandwidth of each connection is at most B0 / p. When a parameter server simultaneously establishes a connection with w worker nodes, the bandwidth of each connection is at most B0 / w. The bandwidth B between the parameter server and the worker node should be the smaller of B0 / w and B0 / p: The throughput is equal to the total batch size divided by the time of one-step training. Let the total batch size be M. The throughput under synchronous training is: After linear approximation of min, it is simplified to where θ is a non-negative coefficient: If there is an optimal performance ratio of w / p in the production environment, it is further simplified to: In asynchronous training, each worker node can be considered to access the parameter server staggered. The bandwidth between the worker node and the parameter server is B0 / p. The total throughput is w times the throughput of each node. The throughput under asynchronous training is: Taking w and p as independent variables and other variables as fitting parameters, combine like terms according to the independent variables, and finally rename the parameters to θ, which is a non-negative coefficient: Based on the existence of the optimal performance ratio of w / p in the production environment, the equation can be further simplified:

5. The online training-oriented computing resource elastic allocation system of claim 4, wherein, The implementation of the online learning autoscaler is to expand or reduce the online training job according to the estimate of the required worker nodes, and to formulate the requirement of the number of online learning worker nodes that need to be met at each time t: min{w(t)∈Z+∣w(t)} s.t.f(w(t))>L(t) Where w(t) represents the number of worker nodes allocated at time t, f(w) represents the throughput of w worker nodes using the resource throughput model fitted in the previous section, and L(t) represents the workload at time t predicted by prediction; Keep the batch size unchanged during model training, and keep the ratio of w / p constant during expansion.

Images