GPU demand prediction method and Spot GPU inventory prediction method

Abstract

The invention provides a GPU demand prediction method and a Spot GPU inventory prediction method, and the method comprises the steps: S1, collecting a historical sequence of GPU usage, and decomposing the historical sequence of GPU usage into a component sequence; s2, extracting service features and time features based on input service information and time information; s3, predicting a periodic term sequence and a trend term sequence based on the component sequence, the business features and the time features, and adding the periodic term sequence and the trend term sequence to obtain a mean value required by the GPU; s4, predicting the standard deviation of the GPU demands, and obtaining the distribution of the future GPU demands according to the mean value of the GPU demands. According to the method, a concept drift detection module and a bottom-out algorithm module are introduced, so that a concept drift phenomenon possibly occurring in a GPU usage prediction scene in a large-scale cluster by a single time sequence prediction algorithm can be timely and effectively prevented.

Description

Technical Field

[0001] The present invention belongs to the field of computer application technology, and in particular, relates to a GPU demand prediction method and a Spot GPU inventory prediction method. Background Art

[0002] In recent years, with the explosive development of emerging technologies such as big data, cloud computing, and artificial intelligence, especially generative artificial intelligence (AIGC), modern cloud service providers generally use large-scale data centers to manage computing and storage resources such as GPU, CPU, and memory.

[0003] Compared with CPU and other computing resources, GPU, or Graphics Processing Unit, is more expensive and has stronger computing power. Especially in the era of large models, it is becoming the main force of AI model computing and plays an irreplaceable and important role. The number of GPU tasks deployed in large-scale data centers is increasing, and the demand for GPU services is far greater than the supply of resources.

[0004] To alleviate the shortage of GPU resources in large-scale cloud computing clusters, when the resource pool sales rate is 100%, there are often peaks and troughs in business usage; specifically, GPU training jobs are JOB-type, and they will stop after running for a period of time, resulting in overall usage, such as Figure 3 As shown. As a cluster manager, you need to solve the problem of how to utilize the resources that have been sold but are not used by the business in this period. If the quota is oversold, once the business peak is squeezed, there is no way to guarantee the SLO of capacity expansion within the quota. We sell this part of idle resources by providing Spot instances, that is, defining differentiated SLOs, and predicting the inventory volume in the future through technical means; ensuring a certain operating SLO, that is, trying not to be preempted by online tasks during the operation period, thereby improving the reuse efficiency of GPU resources. Specifically, the minimum number of idle cards in the cluster or resource pool in the future is defined as Spot inventory.

[0005] The core of GPU Spot inventory prediction is to make accurate hourly predictions of the time series data of the actual GPU usage of online tasks for each cluster and card model. The time series prediction problem can be defined as follows: In a time series containing N variables, given historical data Predict the time series values ​​in the future T time steps Where L refers to the look-back window length, that is, the length of the recent historical time series data needed to make each prediction.

[0006] In recent years, the academic community has proposed many general time series prediction algorithms and achieved certain results on public data sets in the fields of finance, meteorology, and residential electricity consumption. Currently, the commonly used time series prediction algorithms are divided into the following categories: the first category is traditional statistical algorithms, such as ARIMA, Prophet, etc.; the second category is prediction algorithms based on machine learning, such as generalized linear models, such as LTSF-Linear, GBRT, Xgboost, etc.; the third category is prediction algorithms based on deep learning, such as LSTM and DARNN based on recurrent neural networks, i.e., Recurrent Neural Network, RNN, TCN and LSTNet based on convolutional neural networks, i.e., Convolutional Neural Network, CNN, Informer, Autoformer, FEDformer, etc. based on Transformer.

[0007] The only drawback is that the above prediction algorithms are all single-value predictions, which cannot adapt well to the need to predict the distribution probability of future online task GPU demand in the scenario of large-scale cluster SpotGPU inventory prediction.

[0008] At present, time series prediction algorithms with the ability to predict probability distribution include Deepar, Timegrad, TLAE, etc. There are few such algorithms, with high computational overhead, poor versatility and generalization ability, and because they do not consider the business characteristics of Spot GPU inventory prediction scenarios, the accuracy of single value prediction or probability distribution prediction cannot meet the needs of business scenarios. In a production environment, the organizational structure that uses resources, divides and manages costs within a group or enterprise is called a cost organization. Online tasks within a cluster are submitted by multiple different cost organizations. The GPU usage mode of each cost organization has its own characteristics. Each cost organization uses different spatial ranges, such as clusters and resource pools. The GPU resources also have complex corresponding relationships. Existing time series prediction algorithms are difficult to support the modeling and mining of these business information.

[0009] This problem needs to be solved urgently. Summary of the invention

[0010] In view of the defects in the prior art, the object of the present invention is to provide a GPU demand prediction method and a SpotGPU inventory prediction method.

[0011] A GPU demand prediction method provided by the present invention includes:

[0012] Step S1: Collect the historical sequence of GPU usage and decompose the historical sequence of GPU usage into component sequences;

[0013] Step S2: extracting business features and time features based on the input business information and time information;

[0014] Step S3: Based on the component sequence, business characteristics and time characteristics, predict the period item sequence and the trend item sequence, and then add the period item sequence and the trend item sequence to obtain the mean value of the GPU demand;

[0015] Step S4: predicting the standard deviation of the GPU demand, and deriving the distribution of future GPU demand based on the mean of the GPU demand.

[0016] Preferably, in step S1, the component sequence includes a period item sequence and a trend item sequence;

[0017] In step S2, the mathematical expression of the service characteristic is:

[0018] c i =Attention(concat(Embedding(v i1 ),Embedding(v i2 ),…,Embedding(v ij )))(1)

[0019] Among them, c i represents the business characteristics of cost organization i; v i1 ,v i2 ,…,v ij represents the business feature input contained in the cost organization i; Embedding represents the embedding layer, concat represents the concatenation function, and Attention represents the self-attention mechanism;

[0020] The mathematical expression of the self-attention mechanism is:

[0021]

[0022] Among them, Q, K and V represent Query, key and Value respectively; Query, key and Value are obtained by transforming the input matrix through the linear layer; d represents the dimension size of K; Softmax represents the Softmax activation function; the superscript T represents the transposition of the key matrix;

[0023] The time characteristic is expressed mathematically as follows:

[0024] c t =concat(Embedding(t.hour),Embedding(t.weeikday),Embedding(t.holiday)) (3)

[0026] Among them, c t Indicates time characteristics; t.hour indicates hour; t.weekday indicates day of the week; t.holiday indicates holidays.

[0027] Preferably, in step S3, the periodic item sequence and the trend item sequence of future GPU demand are predicted by a Linear layer;

[0028] The mathematical expression of the periodic term sequence is:

[0029]

[0030] in, represents a periodic term sequence; Linear represents a linear layer; χ i,s represents the periodic component;

[0031] The mathematical expression of the trend item sequence is:

[0032]

[0033] in, represents the trend term sequence; i,t represents the trend term component;

[0034] In step S3, the mathematical expression of the mean value of the GPU requirement is:

[0035]

[0036] Among them, μ i Indicates the mean value of GPU demand.

[0037] Preferably, in step S4, the mathematical expression of the standard deviation of the GPU requirement is:

[0038] σ i =Softplus(h i ) (8)

[0039] Among them, σ i represents the standard deviation of GPU requirements; Softplus represents the Softplus activation function; h i represents the sequence of fluctuation terms;

[0040] The mathematical expression of the fluctuation term sequence is:

[0041] h i =Linear(concat(χ i ,c i ,c t )) (7)

[0042] Among them, χ i represents the input sequence;

[0043] In step S4, the number of GPU cards required in the future obeys Gaussian distribution, and the mathematical expression of its loss function is:

[0044]

[0045] Among them, loss represents the Loss loss function; p(y i |μ i ,σ i ) represents the probability density function of Gaussian distribution, μ i represents the mean value of GPU demand, σ i Indicates the standard deviation of the mean of GPU requirements; y i Represents the real future GPU usage sequence; Represents the probability density function p(y i |μ i ,σ i ) to perform a continuous multiplication operation.

[0046] According to a Spot GPU inventory prediction method provided by the present invention, the GPU demand prediction method is adopted, including:

[0047] Step M1: collecting business information, and predicting the probability distribution of the number of online GPU demand cards of each cost organization based on the business information and historical sequences; the prediction adopts the GPU demand prediction method;

[0048] Step M2: Calculate the upper limit of the number of online GPU cards required based on the probability distribution of the number of online GPU cards required and the user's Spot guarantee requirement;

[0049] Step M3: Based on the upper limit of the number of online GPU cards required, determine whether concept drift occurs; if the result is yes, use the bottom-up algorithm to predict the result; if the result is no, use the upper limit of the number of online GPU cards required in step M2 as the prediction result;

[0050] Step M4: spatiotemporally aggregate the prediction results and output the GPU Spot predicted inventory card count.

[0051] Preferably, in the step M1, the business information includes workload, resource trace and cost organization.

[0052] Preferably, the mathematical expression for predicting the upper limit of the number of online GPU cards required is:

[0053]

[0054] in, Indicates the upper limit of the number of online GPU cards required; norm.ppf indicates the quantile function of the Gaussian distribution; p indicates the Spot guarantee rate.

[0055] Preferably, in step M3, it is determined whether the ratio of online GPU usage to the predicted upper limit of online GPU card number demand is greater than a preset threshold; if the result is yes, a bottom-up algorithm is used to predict the result; if the result is no, the upper limit of online GPU card number demand in step M2 is used as the prediction result; the preset threshold is 0.1; the bottom-up algorithm is to take the peak value of GPU usage in the past as the predicted value of the upper limit of online GPU card number demand of the cost organization in the future;

[0056] The prediction result, i.e. the predicted value of the upper limit of the future demand for online GPU cards, is expressed as follows:

[0057]

[0058] in, Indicates the predicted upper limit of the future demand for online GPU cards; max indicates the maximum value; χ i [-step:] represents the actual GPU usage sequence of cost organization i in the past step hours;

[0059] In step M4, the GPU Spot predicts the number of inventory cards, and the mathematical expression is:

[0060]

[0061] Among them, M represents the number of GPU Spot predicted inventory cards; C represents resource capacity; Indicates the upper limit of the total number of online GPU cards within the specified space range;

[0062] The upper limit of the total number of online GPU cards within the specified space range is expressed as:

[0063]

[0064] in, Indicates the predicted upper limit of the number of online GPU cards required in the future.

[0065] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method are implemented.

[0066] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the SpotGPU inventory prediction method are implemented.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The present invention is applicable to the problem of predicting the number of GPU Spot inventory cards in a highly dynamic large-scale production cluster, and overcomes the problems of low accuracy, high overhead, and difficulty in accurately predicting probability distribution in the existing general time series prediction algorithms in GPU demand prediction or inventory card number prediction scenarios; specifically, compared with the existing Transformer algorithm and DLinear algorithm, the accuracy of the OrgLinear algorithm proposed in the present invention is improved by 18-33%; compared with the Deepar algorithm for predicting the probability distribution of time series data, the accuracy of single value prediction of the OrgLinear algorithm is improved by 19-50%, and the accuracy of distribution prediction is improved by 45.61%, and the time overhead is only equivalent to 1.63% of Deepar.

[0069] 2. The present invention decomposes the problem of predicting the number of GPU Spot inventory cards in a large-scale production cluster into a probabilistic prediction of the GPU usage distribution of each cost organization using the cluster, and proposes an OrgLinear algorithm that perceives business information for the GPU demand prediction problem at the cost organization granularity.

[0070] 3. The present invention extracts business features from business information of a given spatial scope and cost organization, and extracts time features from the current time, which are used as inputs for the OrgLinear time series prediction algorithm, i.e., the OrgLinear algorithm mines favorable feature pattern inputs; specifically, after adding the feature extraction module, the accuracy of OrgLinear is improved by 16.38%.

[0071] 4. The present invention uses Linear network learning to input the periodicity, trend and volatility characteristics contained in the online GPU usage time series data to predict and describe the probability distribution of future online GPU usage.

[0072] 5. The present invention introduces concept drift detection and backup algorithm modules, which can timely and effectively prevent the concept drift phenomenon that may occur in the GPU usage prediction scenario of a single time series prediction algorithm in a large-scale cluster.

[0073] 6. The present invention can perform spatiotemporal aggregation on the prediction results of various cost organizations and time lengths, calculate the prediction results of the number of GPU Spot inventory cards with various spatial ranges and guaranteed time lengths on demand, and achieve a one-time prediction to meet the functional requirements of GPU Spot products with various guaranteed time lengths such as 1, 2, and 4 hours, and can provide the cluster scheduler with detailed user portrait data at the granularity of clusters, resource pools, and cost organizations for its decision-making reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0075] Figure 1 A schematic diagram of the GPU demand prediction algorithm framework provided by the present invention;

[0076] Figure 2 A schematic diagram of the Spot GPU inventory prediction algorithm framework provided by the present invention;

[0077] Figure 3 A schematic diagram of the Spot selling space provided by the present invention;

[0078] Figure 4 A schematic diagram of the GPU business feature extraction module framework in the OrgLinear algorithm provided by the present invention;

[0079] Figure 5 This is a schematic diagram of the framework of the time feature extraction module in the OrgLinear algorithm provided by the present invention. DETAILED DESCRIPTION

[0080] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0081] The present invention uses cost organization as the minimum granularity of Spot inventory prediction, and decomposes the actual GPU usage of online tasks in a specific spatial range, namely cluster-card model-resource pool, into separate predictions of GPU usage for each cost organization. The usage time series data of each cost organization is further decomposed into periodic items and trend items, and business features and time features are added as multi-dimensional feature inputs to improve the accuracy of the GPU organization granularity GPU usage time series prediction algorithm. The business features, such as clusters, resource pools, cost organizations, etc., use Self-Attention, i.e., the self-attention mechanism to learn the correlation between them.

[0082] Considering that the Spot guarantee in actual production applications requires an eviction rate of less than 5%, the present invention adopts a probability distribution prediction of the actual usage of the online task GPU to replace the conventional single-value time series prediction, which can more conveniently obtain the maximum number of GPU cards required for online tasks with a 95% guarantee probability, that is, the number of GPU cards required for online tasks in the future period of time has a theoretical probability of 95% not exceeding this maximum number of cards required, and the Spot quota issued according to this demand prediction also has a 95% probability of not conflicting with the demand of online tasks. On the basis of accurately predicting the GPU demand for online tasks in the future period of time, aggregation is performed according to different guarantee durations and spatial ranges, and the maximum number of inventory GPU cards that can be used by Spot-level tasks under different guarantee durations and spatial ranges is calculated. The spatial range includes: cluster, card model and resource pool. The guarantee duration includes: 1 hour, 2 hours and 4 hours.

[0083] The present invention discloses a GPU demand prediction algorithm for perceiving business information of large-scale clusters, namely, the OrgLinear algorithm, comprising:

[0084] Step S1: Decompose the GPU usage history sequence of the input cost organization into two component sequences: a periodic item and a trend item;

[0085] Step S2: extracting business features from business information organized by a given spatial scope and cost, and extracting time features from the current time;

[0086] Step S3: According to the two component sequences of the periodic item and the trend item decomposed in step S1 and the business features and time features extracted in step S2, the future periodic item sequence and the trend item sequence are predicted through the Linear layer, and the mean value of the predicted future GPU demand is obtained by adding them up;

[0087] Step S4: predict the standard deviation of the future GPU demand through the Linear layer and the Softplus activation function, and profile the distribution of the future GPU demand of the cost organization with the mean and standard deviation of the future GPU demand.

[0088] The present invention discloses a Spot GPU inventory prediction algorithm for large-scale clusters, including:

[0089] Step M1: Based on the collected workload, resource trajectory and business information related to the cost organization, the probability distribution of the number of online GPU cards required by each cost organization in the future is predicted using the above-mentioned business information-aware GPU demand prediction algorithm, namely, the OrgLinear algorithm, specifically, the Gaussian distribution;

[0090] Step M2: Calculate the upper limit of the number of online GPU cards required based on the predicted probability distribution of the number of online GPU cards required by each cost organization in the future and the user's Spot guarantee demand, and write it to the OrgLinear usage forecast database;

[0091] Step M3: Check whether concept drift occurs based on the historical OrgLinear algorithm usage prediction output and actual usage. If the error is too large, use the fallback algorithm to calibrate the usage prediction result. The fallback algorithm refers to another set of timing prediction algorithms that runs in parallel with the main algorithm to deal with the worst case scenario of serious errors in the main algorithm. The default fallback algorithm in this invention is to take the past GPU usage peak as the predicted value of the upper limit of the future online GPU card demand of the cost organization.

[0092] Step M4: Perform spatiotemporal aggregation on the prediction results and output the number of GPU Spot predicted inventory cards within the specified spatial range and guaranteed duration.

[0093] Embodiment 1:

[0094] Technical solution of the present invention: The present invention aims to provide a Spot GPU inventory prediction algorithm suitable for large-scale clusters, which can realize online GPU demand card number prediction and Spot GPU inventory card number prediction for any cluster, card model, resource pool and cost organization dimension, any guarantee duration and any target guarantee rate.

[0095] In view of the limitations of current mainstream time series prediction algorithms such as Transformer, the present invention has made two major optimizations. First, for the problem of Spot GPU inventory prediction in large-scale production clusters, the present invention proposes a prediction method based on cost organization decomposition, predicts the probability distribution of the number of online GPU demand cards of each cost organization, and calculates the number of Spot GPU inventory cards with various spatial granularities, guarantee durations, and guarantee rates.

[0096] Secondly, for the online GPU usage timing prediction of each GPU business organization, compared with the existing general timing prediction algorithm, the present invention can select a suitable feature extraction model to effectively extract and learn information from the input business features and time features, and can achieve learning and more accurate distribution prediction of the three dimensions of periodicity, trend and volatility of the online GPU usage sequence at a lower cost. The accuracy of both single value prediction and distribution prediction is better than that of the existing mainstream timing prediction algorithm.

[0097] The present invention proposes the OrgLinear algorithm, which is a GPU demand prediction algorithm that perceives business information. OrgLinear aims to use lightweight feature extraction networks and prediction networks to fully explore the features contained in input time series data, business information and time patterns, and accurately predict the value and fluctuation of the number of GPU demand cards at the cost organization granularity in the future. On this basis, OrgLinear helps cluster managers accurately predict the probability distribution of the number of online GPU demand cards in the cluster and each cost organization in the future, and provides Spot GPU inventory cards and Spot GPU products with high guarantee rates and SLOs, so as to better improve the efficiency of cluster resource utilization and service quality and obtain greater economic benefits.

[0098] According to the GPU demand prediction method for perceiving service information of large-scale clusters provided by the present invention, namely, the OrgLinear algorithm, as shown in FIG. Figure 1 As shown, including:

[0099] Step S1: Decompose the GPU usage history sequence of the input cost organization into two component sequences: a periodic item and a trend item;

[0100] Step S2: extracting business features from business information organized by a given spatial scope and cost, and extracting time features from the current time;

[0101] Specifically, in step S2:

[0102] The business feature and time feature extraction methods include:

[0103] Features are extracted from the business and forecast time as the input of the prediction neural network. For each set of cost organization i related business information, unique hot encoding is used, and each is first transformed into a continuous feature vector of the same dimension using the Embedding layer, and then the Self-Attention mechanism is used to learn the correlation between them, and finally the business feature c is obtained. i The relevant business information includes: cluster, card model, resource pool and cost organization.

[0104] GPU business feature extraction module, such as Figure 4 As shown, the formal expression is:

[0105] c i =Attention(concat(Embedding(v i1 ),Embedding(v i2 ),…,Embedding(v ij )))(1)

[0106] Among them, the business characteristic input contained in cost organization i is v i1 ,v i2 ,…,v ij , Embedding represents the embedding layer, concat represents the concatenation function, and Attention represents the self-attention mechanism

[0107] The Self-Attention mechanism is expressed as follows:

[0108]

[0109] Among them, Q, K and V represent Query, key and Value respectively, which are obtained by transforming the input matrix through the linear layer; d represents the dimension of K; Softmax represents the Softmax activation function; the superscript T represents the transposition of the key matrix;

[0110] Since GPU usage time series data has certain periodic characteristics, and the patterns on weekdays and holidays may be different, the input of the time feature extraction module is the time point of the current prediction. First, the three dimensions of hour t.hour, day of the week t.weekday and whether it is a holiday t.holiday are parsed from the time t, and then the feature vectors of the same dimension are obtained through the embedding layer respectively, and the time feature c is obtained by splicing. t Output. The temporal feature extraction module is as follows Figure 5 As shown, the formal expression is as follows:

[0111] c t =concat9Embedding(t.hour),Embedding(t.weekday),Embedding(t.holiday)) (3)

[0113] Step S3: According to the two component sequences of the period item and the trend item decomposed in step S1 and the business features and time features extracted in step S2, the future period item sequence and the trend item sequence are predicted through the Linear layer, and the mean value of the predicted future GPU demand is obtained by adding them up;

[0114] Specifically, in step S3:

[0115] Methods for predicting the mean future GPU demand include:

[0116] Two different Linear layers are used to predict the periodic term sequence and trend term sequence of future GPU demand.

[0117] For the periodic term prediction of GPU usage series, Linear represents the Linear layer, and the input is the historical periodic term component χ i,s 、Business characteristicsc i and time feature c t The output is the predicted periodic term sequence, that is, Specifically, the length is the number of hours to predict into the future, and defaults to 4.

[0118]

[0119] The predicted trend item sequence, i.e.

[0120]

[0121] Finally, the predicted periodic term sequence and trend item sequence Add together to get the mean μ of the predicted future GPU demand i :

[0122]

[0123] Step S4: predict the standard deviation of the future GPU demand through the Linear layer and the Softplus activation function, and profile the distribution of the future GPU demand of the cost organization with the mean and standard deviation of the future GPU demand.

[0124] Specifically, in step S4:

[0125] Methods for predicting the standard deviation of future GPU demand include:

[0126] According to the input sequence χ i 、Business characteristicsc i and time feature c t Use Linear layers to predict the volatility sequence h of future GPU demand i , it is also necessary to obtain a non-negative prediction standard deviation σ through the Softplus activation function Softplus i :

[0127] h i =Linear(concat(χ i ,c i ,c t )) (7)

[0128] σ i =Softplus(h i ) (8)

[0129] Based on the predicted mean future GPU usage μi and standard deviation σ i We can use Gaussian distribution to predict the distribution of future GPU usage sequences. The entire OrgLinear network uses the probability density function of Gaussian distribution as the Loss function for training, namely:

[0130]

[0131] Among them, p(y i |μ i ,σ i ) represents the probability density function of Gaussian distribution, μ i represents the mean value of GPU demand, σ i Indicates the standard deviation of the mean of GPU requirements; y i Represents the real future GPU usage sequence; Represents the probability density function p(y i |μ i ,σ i ) to perform a continuous multiplication operation.

[0132] Embodiment 2:

[0133] Embodiment 2 is a preferred example of Embodiment 1, and is used to illustrate the present invention in more detail.

[0134] The present invention also provides a Spot GPU inventory prediction algorithm for large-scale clusters. The Spot GPU inventory prediction algorithm for large-scale clusters can be implemented by executing the process steps of the GPU demand prediction algorithm for perceiving business information for large-scale clusters, that is, those skilled in the art can understand the GPU demand prediction algorithm for perceiving business information for large-scale clusters as one of the preferred implementation methods of the Spot GPU inventory prediction algorithm for large-scale clusters.

[0135] According to the Spot GPU inventory prediction algorithm for large-scale clusters provided by the present invention, Figure 2 As shown, including:

[0136] Step M1: Based on the collected workload, resource trajectory and business information related to the cost organization, the probability distribution of the number of online GPU cards required by each cost organization in the future is predicted using the above-mentioned GPU demand prediction algorithm that perceives business information, namely, the OrgLinear algorithm; specifically, the Gaussian distribution;

[0137] Step M2: Calculate the upper limit of the number of online GPU cards required based on the predicted probability distribution of the number of online GPU cards required by each cost organization in the future and the user's Spot guarantee demand, and write it to the OrgLinear usage forecast database;

[0138] Specifically, in step M2:

[0139] Methods for predicting the upper limit of online GPU card demand include:

[0140] Assume that the historical GPU usage sequence of cost organization i obtained from the resource trace data of the cluster is x i , using the OrgLinear algorithm to predict the future online GPU usage of cost organization i, the mean and standard deviation are μ i and σ i , the Spot guarantee rate required by the business side is p, for example, 95%. Assuming that the future online GPU usage follows a Gaussian distribution, norm.ppf represents the quantile function of the Gaussian distribution, that is, the inverse of CDF, then the probability of the Spot guarantee rate p not conflicting with the upper limit of the number of online GPU cards required

[0141]

[0142] Step M3: Check whether concept drift occurs based on the historical OrgLinear algorithm usage prediction output and actual usage. If the error is too large, use the fallback algorithm to calibrate the usage prediction result.

[0143] Specifically, in step M3:

[0144] The methods for calibrating the bottom-line algorithm include:

[0145] When the ratio of the actual online GPU usage in the past period to the upper limit of the online GPU card demand predicted by the OrgLinear algorithm is greater than a certain threshold, such as 0.1, it is considered that concept drift has occurred. At this time, the bottom-up algorithm is used to take the peak GPU usage in the past period of time, which defaults to the guaranteed duration of step hours, as the predicted value of the upper limit of the online GPU card demand of the cost organization in the future. where χ i [-step:] represents the actual GPU usage sequence of cost organization i in the past step hour:

[0146]

[0147] Step M4: Perform spatiotemporal aggregation on the prediction results and output the number of GPU Spot predicted inventory cards within the specified spatial range and guaranteed duration.

[0148] Specifically, in step M4:

[0149] Methods for spatiotemporal aggregation include:

[0150] Spatial range aggregation only requires summing up the predicted upper limits of the number of online GPU cards required for all cost organizations that have historically used GPUs within the spatial range to obtain the upper limit of the total number of online GPU cards used within the spatial range.

[0151]

[0152] Time granularity aggregation takes the predicted peak GPU usage in the next step hour and combines it with the resource capacity C within the space to calculate the number of inventory GPU cards M available for the future Spot product:

[0153]

[0154] Embodiment 3:

[0155] Embodiment 3 is a preferred example of Embodiment 2, and is used to illustrate the present invention in more detail.

[0156] The present invention discloses a GPU demand prediction algorithm and a SpotGPU inventory prediction algorithm for perceiving business information for large-scale clusters. The present invention proposes OrgLinear, a lightweight GPU demand distribution prediction algorithm based on business and time feature modeling. First, we parse the GPU usage time series data at the cost organization granularity from the production cluster Trace data, and decompose it into periodic item components and trend item components. Secondly, the present invention models and mines features from business information and forecast time information and the correlation between them. On this basis, the present invention uses a Linear network to learn the periodicity, trend and volatility characteristics of GPU usage time series data, and realizes the distribution prediction of the future GPU demand card number of each cost organization. The OrgLinear algorithm proposed in the present invention can reduce the error by 18% to 33%, and provide a more accurate and less expensive distribution prediction capability.

[0157] The present invention proposes a lightweight algorithm that can online and rollingly predict the number of GPU cards required in the future, and based on the algorithm, further proposes a method that can effectively predict the sales space and guarantee amount of Spot GPUs, and gives the design details and specific implementation mechanism of the method.

[0158] 1. Framework structure

[0159] Figure 1 , Figure 2The figure shows the framework of the GPU demand prediction algorithm for perceiving business information for large-scale clusters proposed by the present invention, namely the OrgLinear algorithm and the Spot GPU inventory prediction algorithm. Among them, the SpotGPU inventory prediction algorithm proposed by the present invention includes a GPU demand prediction algorithm, namely the OrgLinear algorithm module and a Spot guaranteed inventory calculation module. The OrgLinear algorithm module aims to predict the probability distribution of future GPU demand for each cost organization, including a GPU usage time series decomposition module, a business and time feature extraction module, a usage mean prediction module and a usage volatility prediction module, namely the quantity volatility standard deviation prediction module. The basic idea of ​​the algorithm is to propose a lightweight resource management algorithm logic that is independent of the cluster scheduler, which can predict the number of online demand cards and the number of available Spot inventory cards in the future period of time based on the business characteristics, time characteristics and real-time changing GPU resource usage in actual production, determine the supply quota and sales space of Spot resources, and improve the utilization efficiency of precious GPU resources.

[0160] 2. Module working process

[0161] (1) Cost organization granularity GPU demand card number prediction module, namely OrgLinear algorithm;

[0162] 1) GPU usage timing decomposition module

[0163] GPU Spot inventory prediction is essentially a prediction of the GPU usage of online tasks in the future. The present invention obtains the historical sequence of online task GPU usage submitted by each cost organization from the real production cluster Trace data. i Inspired by Autoformer and FEDformer, combined with the periodic characteristics of GPU usage sequence, we use the historical sequence of GPU usage χ i The periodic component χ is decomposed into i,s and the trend term χ i,t :

[0164] χ i,t =Avgpool(Padding(χ i )) (14)

[0165] χ i,s =v i -x i,t (15)

[0166] The Axgpool function represents the sliding average of the input sequence, and the sliding average kernel size is 1 day, that is, the GPU usage sequence is decomposed from the perspective of daily periodicity.

[0167] 2) Business and time feature extraction module

[0168] In addition to the input time series itself, the present invention also extracts features from the business and prediction time as the input of the prediction neural network. For each set of cost organization i related business information, including: cluster, card model, resource pool, cost organization, unique hot encoding is used, and each is first transformed into a continuous feature vector of the same dimension using the Embedding layer, and then Self-Attention, that is, the self-attention mechanism, is used to learn the correlation between them, and finally the business feature c is obtained. i .

[0169] GPU business feature extraction module, such as Figure 4 As shown, the formal expression is as follows, where the business characteristics implied by cost organization i are input as v i1 ,v i2 ,…,v ij , Embedding represents the embedding layer, concat represents the concatenation function, and Attention represents the self-attention mechanism:

[0170] c i =Attention(concat(Embedding(v i1 ),Embedding(v i2 ),…,Embedding(v ij )))(1)

[0171] The Self-Attention mechanism is expressed as follows:

[0172]

[0173] Among them, Q, K, V represent Query, key and Value respectively, which are obtained by transforming the input matrix through the linear layer; d represents the dimension size of K; Softmax() represents the Softmax activation function.

[0174] Since GPU usage time series data has certain periodic characteristics, and the patterns on weekdays and holidays may be different, the input of the time feature extraction module is the time point of the current prediction. First, the three dimensions of hour t.hour, day of the week t.weekday and whether it is a holiday t.holiday are parsed from the time t, and then the feature vectors of the same dimension are obtained through the embedding layer respectively, and the time feature c is obtained by splicing. t Output. The temporal feature extraction module is as follows Figure 5 As shown, the formal expression is as follows:

[0175] c t=concat(Embedding(t.hour),Embedding(t.weekday),Embedding(t.holiday)) (3)

[0177] 3) Usage mean prediction module

[0178] After completing the GPU usage time series decomposition and business time feature extraction, the present invention uses two different Linear layers to predict the periodic item sequence and trend item sequence of future GPU demand. This is because inspired by DLinear, compared with the complex Transformer-type time series prediction model, the simple linear model is sufficient to extract the periodic item and trend item information contained in the time series, which may achieve a 20% to 50% advantage over Transformers.

[0179] For the periodic term prediction of GPU usage series, Linear() represents the Linear layer, and the input is the historical periodic term component χ i,s 、Business characteristicsc i and time feature c t The output is the predicted period item sequence (length is the number of hours to predict in the future, default is 4):

[0180]

[0181] Predicted trend item sequence

[0182]

[0183] Finally, the predicted periodic term sequence and trend item sequence Add together to get the mean μ of the predicted future GPU demand i :

[0184]

[0185] 4) Usage volatility standard deviation prediction module

[0186] Since the OrgLinear algorithm module needs to predict the probability distribution of the number of GPU cards required by each cost organization in the future, according to the input sequence χ i 、Business characteristicsc i and time feature c t Use Linear layers to predict the volatility sequence h of future GPU demand i , it is also necessary to obtain a non-negative prediction standard deviation σ through the Softplus activation function Softplus() i:

[0187] h i =Linear(concat(x i ,c i ,c t )) (7)

[0188] σ i =Softplus(h i ) (8)

[0189] Based on the predicted mean future GPU usage μ i and standard deviation σ i We can use Gaussian distribution to predict the distribution of future GPU usage sequences. The entire OrgLinear network uses the probability density function of Gaussian distribution as the Loss function for training. The mathematical expression is:

[0190]

[0191] Among them, loss represents the Loss function; y i represents the real future GPU usage sequence, p(y i |μ i ,σ i ) represents the probability density function of the Gaussian distribution.

[0192] (2) Spot Guaranteed Inventory Calculation Module

[0193] The OrgLinear algorithm is used to predict the mean value of future GPU usage μ i and standard deviation σ i After that, the upper limit of the number of online GPU cards required with a probability of p and no conflict can be calculated based on the Spot guarantee rate p required by the business party, for example, 95%. In actual production, sometimes due to some sudden business or operation and maintenance reasons, such as large-scale machine migration, the GPU usage pattern of the cluster or cost organization may change dramatically, which is the so-called "concept drift", i.e., ConceptDrift. The present invention dynamically detects the concept drift phenomenon. When it is detected that the MAPE prediction error of a period of time in the past, such as 1 hour, is greater than a certain threshold, such as 0.1, the bottom-up algorithm is used instead - the peak GPU usage of the same length of time in the past is taken as the predicted value of the upper limit of the future online GPU card demand. The concept drift detection and bottom-up algorithm of the present invention can quickly and timely detect concept drift, and effectively correct and control the prediction error of the number of online GPU demand cards.

[0194] After obtaining the maximum number of online GPU cards required for each cost organization, the upper limit of the online GPU card demand for each cost organization can be aggregated according to the guaranteed duration and predicted spatial range requirements to obtain the guaranteed number of Spot inventory cards under the spatial range. The fine-grained cost organization dimension prediction results can also provide more information for the subsequent Spot scheduling algorithm than directly predicting the number of online Spot inventory cards.

[0195] Taking the cluster-card model space granularity as an example, the Spot guaranteed inventory calculation module is shown in Algorithm 1.

[0196]

[0197] Based on Algorithm 1, the present invention can flexibly provide Spot GPU inventory card supply with multiple guarantee durations (such as 1 hour, 2 hours, 4 hours, etc.), spatial granularity and guarantee rate. The spatial granularity includes: cluster, card model, resource pool.

[0198] 3. Algorithm Implementation

[0199] The Spot GPU inventory prediction algorithm for large-scale clusters proposed in the present invention can be implemented using the flask framework + celery distributed task execution framework + celery message middleware, and lightly deployed on a distributed cloud service container. Distributed tasks such as synchronizing Trace data, updating model parameters, and real-time inventory prediction can be run through Web url calls at regular intervals. The Spot GPU inventory prediction results are written into the cloud database, and users call the api to read the latest SpotGPU inventory prediction data.

[0200] The present invention can be used as a bypass algorithm for the current cluster scheduler without intrusion, and is applicable to the cluster scheduling framework of mainstream cloud service providers such as PAI. It can synchronize the production trace data of large-scale data centers online, update model parameters and perform real-time optimization. Specifically, the present invention works in conjunction with the data center resource monitoring system, quota operation system and scheduling system.

[0201] For the resource monitoring system, the present invention relies on the monitoring platform of the large-scale production cluster to read the GPU usage related data and the business information data using the GPU. For the quota operation system, it regularly calls the query API to update the total inventory number of Spot GPU cards in the resource pool, and distributes the total inventory number of Spot GPU cards to each cost organization that uses the total inventory number of Spot GPU cards. When users deploy Spot containers on a certain resource pool host, they must meet the corresponding quota constraints. For the scheduling system, the cluster scheduler will ensure that online instances do not preempt Spot instances as much as possible, and the resource requirements of the two will not conflict as much as possible.

[0202] The operation process of the Spot GPU inventory prediction algorithm proposed in this invention includes 6 steps:

[0203] Step 1: For any cost organization i, at any time t, based on the collected trace data, the GPU usage history sequence χ i The periodic component χ is decomposed into i,s and the trend term χ i,t , as shown in formula (14) and formula (15); extract business characteristics c according to cost organization business information i , as shown in formula (1); extract the time feature c according to time t t , as shown in formula (3).

[0204] Step 2: According to the periodic component χ i,s 、Business characteristicsc i , time feature c t , predict the periodic term sequence of future GPU demand cards As shown in formula (4); according to the trend term component χ i,t 、Business characteristicsc i , time feature c t , predict the trend item sequence of future GPU demand card number As shown in formula (5), the mean value μ of the number of GPU cards required in the future is obtained by synthesis i , as shown in formula (6).

[0205] Step 3: Based on the GPU usage history series χ i 、Business characteristicsc i , time feature c t , predict the standard deviation of the number of GPU cards required in the future σ i , as shown in formula (7) and formula (8).

[0206] Step 4: Organize the future online GPU usage average μ according to the predicted cost i and standard deviation σ i and the Spot guarantee rate p required by the business side, and calculate the upper limit of the number of online GPU cards required by cost organization i As shown in formula (10).

[0207] Step 5: Forecast the upper limit of the number of online GPU cards required for each cost organization i Perform concept drift detection. When concept drift is detected, use the fallback algorithm as shown in formula (11).

[0208] Step 6: Forecast the upper limit of the number of online GPU cards required for each cost organization i Perform spatiotemporal aggregation to obtain the number M of inventory GPU cards available for Spot products in the specified spatial range in the future, as shown in formulas (12) and (13).

[0209] 4. Evaluation of the Effect of the Invention

[0210] In order to verify the effect of the OrgLinear algorithm proposed in the present invention in the online GPU usage prediction scenario. Six months of GPU Trace data were selected from a large-scale production cluster of an Internet company mainly engaged in e-commerce business, covering a total of 6 card types and 14 GPU business organizations. The data from the first 5 months were used as the training set, and the data from the last month were used as the test set. The model parameters are updated every day, and the peak online GPU usage in the next 4 hours is predicted every 5 minutes. The single-value prediction error is evaluated using four indicators: mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and mean absolute percentage error (MAPE). The smaller the error, the more accurate the prediction algorithm:

[0211]

[0212] Among them, y i Indicates the actual value of GPU usage of cost organization i, represents the predicted GPU usage of cost organization i, and n represents the number of cost organizations in the test set.

[0213] The comparison results of the OrgLinear algorithm and the existing general time series prediction algorithms are shown in the following table. Compared with the existing mainstream Transformer-based time series prediction algorithms and the DLinear algorithm, the error of the OrgLinear algorithm proposed in the present invention is reduced by 18-33%.

[0214] Table 1 Comparison of the results of the OrgLinear algorithm and the general timing prediction algorithm in the online GPU usage prediction scenario

[0215] MAE MSE RMSE MAPE FEDformer 55.47 10804.65 103.95 0.199 Autoformer 58.62 12828.20 113.26 0.210 Informer 54.56 16043.29 126.66 0.201 Transformer 41.25 8249.51 90.83 0.180 DLinear 32.65 6848.49 82.76 0.131 OrgLinear 24.39 4570.68 67.61 0.097 Improved results 25.30％ 33.26％ 18.31％ 25.95％

[0216] Since the OrgLinear algorithm is an algorithm for predicting the probability distribution of online GPU usage, the OrgLinear algorithm is compared with the classic algorithm Deepar for predicting the probability distribution of time series data, and a new evaluation index for the p-quantile prediction error, p-MAQE, is defined, which is the Mean Absolute Quantile Error. It represents the absolute value of the error between the actual quantile of the p-quantile prediction value and p. For example, only 94% of the time series prediction results are less than the 0.95-quantile, and the 0.95-MAQE error is 0.01. In formula (20), represents the p-quantile value sequence of the probability distribution of the predicted time series, Z represents the actual value sequence, Indicates the frequency of occurrence that the actual value is less than the p-quantile value, and n represents the number of cost organizations in the test set. In the present invention, since the business party requires the Spot GPU guarantee rate to reach 95%, p is set to 0.95.

[0217]

[0218] Among them, p-MAQE represents the evaluation indicator.

[0219] The comparison results are shown in the following table. The OrgLinear algorithm is not only significantly better than the Deepar time series prediction algorithm in terms of the accuracy index of the above-mentioned single value prediction, but also supports the ability to predict distribution probability. When predicting the 0.95-percentile online GPU usage time series data, the 0.95-MAQE error index is significantly improved by 45.61% compared with the Deepar algorithm. In addition, since Deepar is a time series prediction algorithm based on LSTM, and the OrgLinear algorithm proposed in the present invention is mainly based on the Linear network, its time overhead is only equivalent to 1.63% of the Deepar algorithm.

[0220] Table 2 Comparison of the results of the OrgLinear algorithm and the distributed timing prediction algorithm in the online GPU usage prediction scenario

[0221] MAE MSE RMSE MAPE 0.95-MAQE Time(s) Deepar 32.22 9181.70 95.82 0.120 0.057 28610.11 OrgLinear 24.39 4570.68 67.61 0.097 0.026 464.94 Improved results 24.30％ 50.22％ 29.44％ 19.17％ 45.61％ 98.37％

[0222] According to a computer-readable storage medium storing a computer program provided by the present invention, when the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method are implemented.

[0223] An electronic device provided according to the present invention includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the SpotGPU inventory prediction method are implemented.

[0224] Those skilled in the art know that, in addition to realizing the system and its various devices, modules, and units provided by the present invention in a purely computer-readable program code, it is entirely possible to realize the same functions in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered as a hardware component, and the devices, modules, and units included therein for realizing various functions can also be regarded as structures within the hardware component; the devices, modules, and units for realizing various functions can also be regarded as both software modules for realizing the method and structures within the hardware component.

[0225] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A GPU demand prediction method, characterized in that: include: Step S1: Collect the historical sequence of GPU usage and decompose the historical sequence of GPU usage into component sequences; Step S2: extracting business features and time features based on the input business information and time information; Step S3: Based on the component sequence, business characteristics and time characteristics, predict the period item sequence and the trend item sequence, and then add the period item sequence and the trend item sequence to obtain the mean value of the GPU demand; Step S4: predicting the standard deviation of the GPU demand, and deriving the distribution of future GPU demand based on the mean of the GPU demand.

2. The GPU demand prediction method according to claim 1, characterized in that: In the step S1, the component sequence includes a period item sequence and a trend item sequence; In step S2, the mathematical expression of the service characteristic is: c i =Attention(concat(Embedding(v i1 ),Embedding(v i2 ),…,Embedding(v ij ))) (1) Among them, c i represents the business characteristics of cost organization i; v i1 ,v i2 ,…,v ij represents the business feature input contained in the cost organization i; Embedding represents the embedding layer, concat represents the concatenation function, and Attention represents the self-attention mechanism; The mathematical expression of the self-attention mechanism is: Among them, Q, K and V represent Query, key and Value respectively; Query, key and Value are obtained by transforming the input matrix through the linear layer; d represents the dimension size of K; Softmax represents the Softmax activation function; the superscript T represents the transposition of the key matrix; The time characteristic is expressed mathematically as follows: c t =concat(Embedding(t.hout),Embedding(t.weekday),Embedding(t.holiday))(3) Among them, c t Indicates time characteristics; t.hour indicates hour; t.weekday indicates day of the week; t.holiday indicates holidays.

3. The GPU demand prediction method according to claim 2, characterized in that: In the step S3, the periodic item sequence and the trend item sequence of future GPU demand are predicted through the Linear layer; The mathematical expression of the periodic term sequence is: in, represents a periodic term sequence; Linear represents a linear layer; χ i,s represents the periodic component; The mathematical expression of the trend item sequence is: in, represents the trend term sequence; i,t represents the trend term component; In step S3, the mathematical expression of the mean value of the GPU requirement is: Among them, μ i Indicates the mean value of GPU demand.

4. The GPU demand prediction method according to claim 3, characterized in that: In step S4, the mathematical expression of the standard deviation of the GPU requirement is: σ i =Softplus(h i ) (8) Among them, σ i represents the standard deviation of GPU requirements; Softplus represents the Softplus activation function; h i represents the sequence of fluctuation terms; The mathematical expression of the fluctuation term sequence is: h i =Linear(concat(χ i ,c i ,c t )) (7) Among them, χ i represents the input sequence; In step S4, the number of GPU cards required in the future obeys Gaussian distribution, and the mathematical expression of its loss function is: Among them, loss represents the Loss loss function; p(y i |μ i ,σ i ) represents the probability density function of Gaussian distribution, μ i represents the mean value of GPU demand, σ i Indicates the standard deviation of the mean of GPU requirements; y i Represents the real future GPU usage sequence; Represents the probability density function p(y i |μ i ,σ i ) to perform a continuous multiplication operation.

5. A Spot GPU inventory prediction method, using the GPU demand prediction method according to claim 1, characterized in that: include: Step M1: collecting business information, and predicting the probability distribution of the number of online GPU demand cards of each cost organization based on the business information and historical sequences; the prediction adopts the GPU demand prediction method; Step M2: Calculate the upper limit of the number of online GPU cards required based on the probability distribution of the number of online GPU cards required and the user's Spot guarantee requirement; Step M3: Based on the upper limit of the number of online GPU cards required, determine whether concept drift occurs; if the result is yes, use the bottom-up algorithm to predict the result; if the result is no, use the upper limit of the number of online GPU cards required in step M2 as the prediction result.

6. The Spot GPU inventory prediction method according to claim 5, characterized in that: In the step M1, the business information includes workload, resource trace and cost organization.

7. The Spot GPU inventory prediction method according to claim 6, characterized in that: The mathematical expression for predicting the upper limit of the number of online GPU cards required is: in, Indicates the upper limit of the number of online GPU cards required; norm.ppf indicates the quantile function of the Gaussian distribution; p indicates the Spot guarantee rate.

8. The Spot GPU inventory prediction method according to claim 7, characterized in that: In the step M3, it is determined whether the ratio of online GPU usage to the predicted upper limit of online GPU card number demand is greater than a preset threshold; if the result is yes, the bottom-up algorithm is used to predict the result; if the result is no, the upper limit of online GPU card number demand described in step M2 is used as the prediction result; the preset threshold is 0.1; the bottom-up algorithm is to take the past peak value of GPU usage as the predicted value of the upper limit of online GPU card number demand of the cost organization in the future; The prediction result, i.e. the predicted value of the upper limit of the future demand for online GPU cards, is expressed as follows: in, Indicates the predicted upper limit of the future demand for online GPU cards; max indicates the maximum value; χ i [-step:] represents the actual GPU usage sequence of cost organization i in the past step hour; In step M4, the GPU Spot predicts the number of inventory cards, and the mathematical expression is: Among them, M represents the number of GPU Spot predicted inventory cards; C represents resource capacity; Indicates the upper limit of the total number of online GPU cards within the specified space range; The upper limit of the total number of online GPU cards within the specified space range is expressed as: in, Indicates the predicted upper limit of the number of online GPU cards required in the future.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method according to any one of claims 5 to 8 are implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method according to any one of claims 5 to 8 are implemented.

Images