Gpu demand prediction method and spot gpu inventory prediction method

Abstract

The application provides a GPU demand prediction method and a Spot GPU inventory prediction method, comprising the following steps: S1, collecting a historical sequence of GPU usage and decomposing the historical sequence of GPU usage into component sequences; S2, extracting business features and time features based on input business information and time information; S3, predicting periodic term sequences and trend term sequences based on the component sequences, the business features and the time features, and then adding the periodic term sequences and the trend term sequences to obtain a mean value of GPU demand; and S4, predicting a standard deviation of the GPU demand, and obtaining a distribution of future GPU demand according to the mean value of the GPU demand. The application introduces a concept drift detection and bottom-up algorithm module, and can effectively prevent the concept drift phenomenon that may occur in the GPU usage prediction scene of a large-scale cluster by using a single time series prediction algorithm.

Description

Technical Field

[0001] The present invention belongs to the field of computer application technology, and in particular relates to a GPU demand forecasting method and a Spot GPU inventory forecasting method. Background Art

[0002] In recent years, with the explosive development of emerging technologies such as big data, cloud computing, and artificial intelligence, especially generative artificial intelligence (AIGC), modern cloud service providers generally use large-scale data centers to manage computing and storage resources such as GPUs, CPUs, and memories.

[0003] Compared to CPUs and other computing resources, GPUs (Graphics Processing Units) are more expensive and offer greater computing power. They are increasingly becoming the primary force in AI model computing, playing an irreplaceable and important role, especially in the era of large models. The number of GPU tasks deployed in large-scale data centers is increasing, and demand for GPUs far outstrips the supply of resources.

[0004] To alleviate the shortage of GPU resources in large-scale cloud computing clusters, when the resource pool sales rate is 100%, there are often peaks and troughs in business usage; specifically, GPU training jobs are JOB-type and will stop after running for a period of time, resulting in overall usage, such as Figure 3 As shown in the figure, cluster managers need to figure out how to utilize resources whose quotas have been sold but are not currently being used by businesses. If the quota is oversold, the SLO for capacity expansion within the quota cannot be met during peak business demand. We offer Spot Instances, which define differentiated SLOs to sell these idle resources. We use technology to predict inventory levels over the next period of time and ensure a certain operating SLO, which minimizes preemption by online tasks over a period of time, thereby improving GPU resource reuse efficiency. Specifically, the minimum number of idle GPUs in a cluster or resource pool over the next period of time is defined as Spot inventory.

[0005] The core of GPU Spot inventory prediction is to make accurate hourly predictions based on the time series data of actual GPU usage for online tasks of various clusters and card models. The time series prediction problem can be defined as follows: In a set of time series containing N variables, given historical data Predict the time series value in the next T time steps Where L refers to the look-back window size, that is, the length of the recent historical time series data required to make each prediction.

[0006] In recent years, the academic community has proposed many general time series forecasting algorithms, achieving certain results on public datasets in fields such as finance, meteorology, and residential electricity consumption. Currently, commonly used time series forecasting algorithms can be divided into the following categories: First, traditional statistical algorithms, such as ARIMA and Prophet; second, machine learning-based forecasting algorithms, such as generalized linear models (LTSF-Linear, GBRT, and Xgboost); and third, deep learning-based forecasting algorithms, such as LSTM and DARNN based on recurrent neural networks (RNNs), TCN and LSTNet based on convolutional neural networks (CNNs), and Informer, Autoformer, and FEDformer based on Transformers.

[0007] The only drawback is that the above prediction algorithms are all single-value predictions, which cannot well adapt to the need to predict the distribution probability of future online task GPU demand in the scenario of large-scale cluster SpotGPU inventory prediction.

[0008] Currently, time series forecasting algorithms capable of predicting probability distributions include Deepar, Timegrad, and TLAE. However, these algorithms are relatively few in number, computationally expensive, and lack versatility and generalization. Furthermore, because they fail to consider the business characteristics of Spot GPU inventory forecasting scenarios, the accuracy of both single-value predictions and probability distribution predictions falls short of meeting the needs of these scenarios. In a production environment, the organizational structure that uses resources, divides costs, and manages costs within a group or enterprise is called a cost organization. Online tasks within a cluster are submitted by multiple cost organizations, each with its own unique GPU usage patterns. Each cost organization utilizes different spatial scopes, and for example, GPU resources within clusters and resource pools have complex correspondences. Existing time series forecasting algorithms struggle to support the modeling and mining of this business information.

[0009] This problem needs to be solved urgently. Summary of the Invention

[0010] In view of the defects in the prior art, the purpose of the present invention is to provide a GPU demand forecasting method and a SpotGPU inventory forecasting method.

[0011] A GPU demand prediction method provided by the present invention includes:

[0012] Step S1: Collect the historical sequence of GPU usage and decompose the historical sequence of GPU usage into component sequences;

[0013] Step S2: extracting business features and time features based on the input business information and time information;

[0014] Step S3: Based on the component sequence, business characteristics and time characteristics, predict the period item sequence and trend item sequence, and then add the period item sequence and trend item sequence to obtain the average value of GPU demand;

[0015] Step S4: predicting the standard deviation of the GPU demand, and deriving the distribution of future GPU demand based on the mean of the GPU demand.

[0016] Preferably, in step S1, the component sequence includes a period item sequence and a trend item sequence;

[0017] In step S2, the mathematical expression of the service characteristic is:

[0018] c i =Attention(concat(Embedding(v i1 ),Embedding(v i2 ),…,Embedding(v ij )))(1)

[0019] Among them, c i represents the business characteristics of cost organization i; v i1 ,v i2 ,…,v ij Represents the business feature input contained in cost organization i; Embedding represents the embedding layer, concat represents the concatenation function, and Attention represents the self-attention mechanism;

[0020] The mathematical expression of the self-attention mechanism is:

[0021]

[0022] Where Q, K, and V represent Query, key, and Value, respectively. Query, key, and Value are obtained by transforming the input matrix through a linear layer. d represents the dimension of K. Softmax represents the Softmax activation function. The superscript T indicates the transposition of the key matrix.

[0023] The mathematical expression of the time characteristic is:

[0024] c t =concat(Embedding(t.hour),Embedding(t.weeikday),Embedding(t.holiday)) (3)

[0026] Among them, c t Indicates time characteristics; t.hour indicates hour; t.weekday indicates day of the week; t.holiday indicates holiday.

[0027] Preferably, in step S3, the periodic item sequence and trend item sequence of future GPU demand are predicted by a Linear layer;

[0028] The mathematical expression of the periodic term sequence is:

[0029]

[0030] in, Represents a periodic term sequence; Linear represents a linear layer; χ i,s represents the periodic component;

[0031] The mathematical expression of the trend item sequence is:

[0032]

[0033] in, represents the trend term sequence; χ i,t represents the trend term component;

[0034] In step S3, the mathematical expression of the mean value of the GPU requirements is:

[0035]

[0036] Among them, μ i Indicates the average GPU demand.

[0037] Preferably, in step S4, the mathematical expression of the standard deviation of the GPU requirement is:

[0038] σ i =Softplus(h i ) (8)

[0039] Among them, σ i represents the standard deviation of GPU requirements; Softplus represents the Softplus activation function; h i represents the sequence of fluctuation terms;

[0040] The mathematical expression of the fluctuation term sequence is:

[0041] h i =Linear(concat(χ i ,c i ,c t )) (7)

[0042] Among them, χ i represents the input sequence;

[0043] In step S4, the number of GPU cards required in the future follows a Gaussian distribution, and the mathematical expression of its loss function is:

[0044]

[0045] Among them, loss represents the Loss loss function; p(y i |μ i ,σ i ) represents the probability density function of Gaussian distribution, μ i represents the mean value of GPU demand, σ i Indicates the standard deviation of the mean of GPU requirements; y i Represents the real future GPU usage sequence; Represents the probability density function p(y i |μ i ,σ i ) to perform continuous multiplication operations.

[0046] According to a Spot GPU inventory forecasting method provided by the present invention, the GPU demand forecasting method is adopted, including:

[0047] Step M1: collecting business information, and predicting the probability distribution of the number of online GPU cards required by each cost organization based on the business information and historical sequences; the prediction adopts the GPU demand prediction method;

[0048] Step M2: Calculate the upper limit of the number of online GPU cards required based on the probability distribution of the number of online GPU cards required and the user's Spot guarantee requirement;

[0049] Step M3: Based on the upper limit of the number of online GPU cards required, determine whether concept drift occurs; if the result is yes, use the bottom-up algorithm to predict the result; if the result is no, use the upper limit of the number of online GPU cards required in step M2 as the prediction result;

[0050] Step M4: spatiotemporally aggregate the prediction results and output the GPU Spot predicted inventory number.

[0051] Preferably, in the step M1, the business information includes workload, resource trace and cost organization.

[0052] Preferably, the mathematical expression for predicting the upper limit of the number of online GPU cards required is:

[0053]

[0054] in, Indicates the upper limit of the number of online GPU cards required; norm.ppf represents the quantile function of the Gaussian distribution; and p represents the spot guarantee rate.

[0055] Preferably, in step M3, it is determined whether the ratio of online GPU usage exceeding the predicted upper limit of online GPU card demand is greater than a preset threshold; if the result is yes, a fallback algorithm is used to predict the result; if the result is no, the upper limit of online GPU card demand described in step M2 is used as the prediction result; the preset threshold is 0.1; the fallback algorithm uses the past peak value of GPU usage as the predicted value of the upper limit of online GPU card demand of the cost organization in the future;

[0056] The prediction result, i.e. the predicted upper limit of the future demand for online GPU cards, is expressed as follows:

[0057]

[0058] in, Indicates the predicted upper limit of the future demand for online GPU cards; max indicates the maximum value; χ i [-step:] represents the actual GPU usage sequence of cost organization i in the past step hour;

[0059] In step M4, the GPU Spot predicts the number of inventory cards, and the mathematical expression is:

[0060]

[0061] Where M represents the number of GPU Spot predicted inventory cards; C represents resource capacity; Indicates the upper limit of the total number of online GPU cards within the specified space range;

[0062] The upper limit of the total number of online GPU cards within the specified space is expressed as follows:

[0063]

[0064] in, Indicates the predicted upper limit of the future demand for online GPU cards.

[0065] According to the present invention, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method are implemented.

[0066] According to the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the SpotGPU inventory prediction method are implemented.

[0067] Compared with the prior art, the present invention has the following beneficial effects:

[0068] 1. The present invention is applicable to the problem of predicting the number of GPU Spot inventory cards in highly dynamic large-scale production clusters. It overcomes the problems of low accuracy, high overhead, and difficulty in accurately predicting probability distribution in existing general time series prediction algorithms in GPU demand prediction or inventory card number prediction scenarios. Specifically, compared with the existing Transformer algorithm and DLinear algorithm, the accuracy of the OrgLinear algorithm proposed in the present invention is improved by 18-33%. Compared with the Deepar algorithm for predicting the probability distribution of time series data, the accuracy of the OrgLinear algorithm for single value prediction is improved by 19-50%, and the accuracy of distribution prediction is improved by 45.61%, with a time overhead equivalent to only 1.63% of Deepar.

[0069] 2. This paper decomposes the problem of predicting the number of GPU Spot inventory cards in a large-scale production cluster into a probabilistic prediction of the GPU usage distribution of each cost organization using the cluster, and proposes the OrgLinear algorithm that perceives business information for the GPU demand prediction problem at the cost organization granularity.

[0070] 3. The present invention extracts business features from business information organized by a given spatial scope and cost, and extracts time features from the current time, which are used as inputs for the OrgLinear time series prediction algorithm, i.e., the OrgLinear algorithm mines favorable feature patterns; specifically, after adding the feature extraction module, the accuracy of OrgLinear is improved by 16.38%.

[0071] 4. The present invention uses Linear network learning to input the periodicity, trend and volatility characteristics contained in the online GPU usage time series data to predict and describe the probability distribution of future online GPU usage.

[0072] 5. The present invention introduces concept drift detection and fallback algorithm modules, which can timely and effectively prevent the concept drift phenomenon that may occur in the GPU usage prediction scenario of a single time series prediction algorithm in a large-scale cluster.

[0073] 6. The present invention can perform spatiotemporal aggregation on the prediction results of various cost organizations and time lengths, and calculate the prediction results of the number of GPU Spot inventory cards with various spatial ranges and guaranteed time lengths on demand, so as to achieve a single prediction that meets the functional requirements of GPU Spot products with various guaranteed time lengths such as 1, 2, and 4 hours, and can provide the cluster scheduler with detailed user portrait data at the cluster, resource pool, and cost organization granularity for its decision-making reference. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Other features, objects and advantages of the present invention will become more apparent upon reading the detailed description of non-limiting embodiments with reference to the following drawings:

[0075] Figure 1 A schematic diagram of the GPU demand prediction algorithm framework provided by the present invention;

[0076] Figure 2 This is a schematic diagram of the Spot GPU inventory prediction algorithm framework provided by the present invention;

[0077] Figure 3 A schematic diagram of the Spot vending space provided by the present invention;

[0078] Figure 4 This is a schematic diagram of the GPU service feature extraction module framework in the OrgLinear algorithm provided by the present invention;

[0079] Figure 5 This is a schematic diagram of the framework of the time feature extraction module in the OrgLinear algorithm provided by the present invention. DETAILED DESCRIPTION

[0080] The present invention will be described in detail below with reference to specific embodiments. The following examples will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several changes and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0081] This paper uses cost organization as the minimum granularity for Spot inventory forecasting. It decomposes the actual GPU usage of online tasks within a specific spatial scope (cluster, card model, and resource pool) into separate GPU usage forecasts for each cost organization. The usage time series data for each cost organization is further decomposed into periodic and trend terms, and business and time features are added as multi-dimensional feature inputs to improve the accuracy of the GPU usage time series forecasting algorithm at the GPU organization granularity. The correlations between these business features, such as cluster, resource pool, and cost organization, are learned using a self-attention mechanism.

[0082] Taking into account the fact that the Spot guarantee requirement in actual production applications is an eviction rate of less than 5%, the present invention uses a probability distribution prediction of the actual GPU usage of online tasks to replace the conventional single-value time series prediction, which can more conveniently obtain the maximum number of GPU cards required for online tasks with a 95% guarantee probability. That is, there is a 95% probability that the number of GPU cards required for online tasks in the future will not exceed this maximum number of cards required, and the Spot quota issued according to this demand prediction also has a 95% probability of not conflicting with the demand of online tasks. On the basis of accurately predicting the GPU demand of online tasks in the future, aggregation is performed according to different guarantee durations and spatial ranges to calculate the maximum number of inventory GPU cards that can be used by Spot-level tasks under different guarantee durations and spatial ranges. The spatial range includes: cluster, card model and resource pool. The guarantee duration includes: 1 hour, 2 hours and 4 hours.

[0083] The present invention discloses a GPU demand prediction algorithm for perceiving business information of large-scale clusters, namely the OrgLinear algorithm, comprising:

[0084] Step S1: Decompose the GPU usage history series of the input cost organization into two component series: the periodic item and the trend item;

[0085] Step S2: extracting business features from business information organized by a given spatial range and cost, and extracting time features from the current time;

[0086] Step S3: Based on the two component sequences of the periodic item and the trend item decomposed in step S1 and the business features and time features extracted in step S2, the linear layer is used to predict the future periodic item sequence and the trend item sequence, and the sum is added to obtain the mean value of the predicted future GPU demand;

[0087] Step S4: Predict the standard deviation of the future GPU demand through the Linear layer and the Softplus activation function, and use the mean and standard deviation of the future GPU demand to profile the distribution of the future GPU demand of the cost organization.

[0088] The present invention discloses a Spot GPU inventory prediction algorithm for large-scale clusters, including:

[0089] Step M1: Based on the collected workload, resource trajectory, and business information related to the cost organization, use the aforementioned business information-aware GPU demand prediction algorithm, namely the OrgLinear algorithm, to predict the probability distribution of the future online GPU demand for each cost organization, specifically, the Gaussian distribution.

[0090] Step M2: Based on the predicted probability distribution of the number of online GPU cards required by each cost organization in the future and the user's Spot guarantee requirements, calculate the upper limit of the online GPU card demand and write it to the OrgLinear usage forecast database;

[0091] Step M3: Check for concept drift based on the historical OrgLinear algorithm usage forecast output and actual usage. If the error is too large, use a fallback algorithm to calibrate the usage forecast results. The fallback algorithm is another time series prediction algorithm that runs in parallel with the main algorithm to address the worst-case scenario of severe errors in the main algorithm. The default fallback algorithm in this invention uses the past peak GPU usage as the predicted upper limit for the cost organization's future online GPU card demand.

[0092] Step M4: Perform spatiotemporal aggregation on the prediction results and output the GPU Spot predicted inventory number for the specified spatial range and guaranteed duration.

[0093] Example 1:

[0094] The technical solution of the present invention: The present invention aims to provide a Spot GPU inventory prediction algorithm suitable for large-scale clusters. The algorithm can realize online prediction of the number of GPU demand cards and the number of Spot GPU inventory cards for any cluster, card model, resource pool and cost organization dimension, any guarantee duration and any target guarantee rate.

[0095] To address the limitations of current mainstream time series prediction algorithms, such as the Transformer, this paper implements two key optimizations. First, for the problem of Spot GPU inventory prediction in large-scale production clusters, this paper proposes a prediction method based on cost organization decomposition. This method predicts the probability distribution of online GPU demand for each cost organization and, based on this, calculates Spot GPU inventory counts for various spatial granularities, guarantee durations, and guarantee rates.

[0096] Secondly, for the online GPU usage time series prediction of each GPU business organization, compared with existing general time series prediction algorithms, the present invention can select a suitable feature extraction model to effectively extract and learn information from the input business features and time features. It can achieve learning and more accurate distribution prediction of the three dimensions of periodicity, trend and volatility of online GPU usage series at a low cost. The accuracy of both single value prediction and distribution prediction is superior to existing mainstream time series prediction algorithms.

[0097] This paper proposes the OrgLinear algorithm, a business-information-aware GPU demand prediction algorithm. OrgLinear aims to leverage lightweight feature extraction and prediction networks to fully exploit the features inherent in input time series data, business information, and temporal patterns, and accurately predict the numerical value and degree of fluctuation of GPU demand at the cost organization level over the coming period. On this basis, OrgLinear helps cluster managers accurately predict the probability distribution of future online GPU demand for the cluster and each cost organization, providing Spot GPU inventory and Spot GPU products with high guarantee rates and SLOs, thereby improving cluster resource utilization efficiency and service quality, and achieving greater economic benefits.

[0098] According to the GPU demand prediction method for perceiving business information of large-scale clusters provided by the present invention, namely the OrgLinear algorithm, as shown in FIG. Figure 1 Shown, including:

[0099] Step S1: Decompose the GPU usage history series of the input cost organization into two component series: the periodic item and the trend item;

[0100] Step S2: extracting business features from business information organized by a given spatial range and cost, and extracting time features from the current time;

[0101] Specifically, in step S2:

[0102] Business feature and time feature extraction methods include:

[0103] Extract features from business and forecast time as input to the prediction neural network. For each set of cost organization i related business information, use one-hot encoding, first use the Embedding layer to transform each into a continuous feature vector of the same dimension, then use Self-Attention, that is, the self-attention mechanism to learn the correlation between them, and finally obtain the business feature c i The relevant business information includes: cluster, card model, resource pool and cost organization.

[0104] GPU business feature extraction module, such as Figure 4 As shown, the formal expression is:

[0105] c i =Attention(concat(Embedding(v i1 ),Embedding(v i2 ),…,Embedding(v ij )))(1)

[0106] Among them, the business characteristic input contained in cost organization i is v i1 ,v i2 ,…,v ij , Embedding represents the embedding layer, concat represents the concatenation function, and Attention represents the self-attention mechanism

[0107] The Self-Attention mechanism is expressed as follows:

[0108]

[0109] Where Q, K, and V represent Query, key, and Value, respectively, which are obtained by transforming the input matrix through a linear layer; d represents the dimension of K; Softmax represents the Softmax activation function; the superscript T represents the transposition of the key matrix;

[0110] Since GPU usage time series data has certain periodic characteristics and its patterns on weekdays and holidays may be different, the input of the time feature extraction module is the time point of the current prediction. First, the three dimensions of hour t.hour, day of the week t.weekday and whether it is a holiday t.holiday are parsed from the time t. Then, the feature vectors of the same dimension are obtained through the embedding layer respectively, and the time feature c is obtained by splicing. t Output. The temporal feature extraction module is as follows Figure 5 As shown, the formal expression is as follows:

[0111] c t =concat9Embedding(t.hour),Embedding(t.weekday),Embedding(t.holiday)) (3)

[0113] Step S3: Based on the two component sequences of the periodic item and the trend item decomposed in step S1 and the business features and time features extracted in step S2, the linear layer is used to predict the future periodic item sequence and the trend item sequence, and the sum is added to obtain the mean value of the predicted future GPU demand;

[0114] Specifically, in step S3:

[0115] Methods for predicting the mean future GPU demand include:

[0116] Two different Linear layers are used to predict the periodic term sequence and trend term sequence of future GPU demand.

[0117] For the periodic term prediction of GPU usage series, Linear represents the Linear layer, and the input is the historical periodic term component χ i,s 、Business characteristics c i and time characteristics c t The splicing of is output as the predicted periodic item sequence, that is, Specifically, the length is the number of hours to predict in the future, and defaults to 4.

[0118]

[0119] The predicted trend item sequence, i.e.

[0120]

[0121] Finally, the predicted periodic term sequence and trend item sequence Add up to get the mean μ of the predicted future GPU demand i :

[0122]

[0123] Step S4: Predict the standard deviation of the future GPU demand through the Linear layer and the Softplus activation function, and use the mean and standard deviation of the future GPU demand to profile the distribution of the future GPU demand of the cost organization.

[0124] Specifically, in step S4:

[0125] Methods for predicting the standard deviation of future GPU demand include:

[0126] According to the input sequence χ i 、Business characteristics c i and time characteristics c t Use the Linear layer to predict the fluctuation sequence h of future GPU demand i , it is also necessary to obtain a non-negative prediction standard deviation σ through the Softplus activation function Softplus i :

[0127] h i =Linear(concat(χ i ,c i ,c t )) (7)

[0128] σ i =Softplus(h i ) (8)

[0129] According to the average μ of the predicted future GPU usagei and standard deviation σ i We can use Gaussian distribution to approximate the distribution of future GPU usage sequences. The entire OrgLinear network uses the probability density function of Gaussian distribution as the loss function for training, namely:

[0130]

[0131] Among them, p(y i |μ i ,σ i ) represents the probability density function of Gaussian distribution, μ i represents the mean value of GPU demand, σ i Indicates the standard deviation of the mean of GPU requirements; y i Represents the real future GPU usage sequence; Represents the probability density function p(y i |μ i ,σ i ) to perform continuous multiplication operations.

[0132] Example 2:

[0133] Example 2 is a preferred example of Example 1 and is used to illustrate the present invention in more detail.

[0134] The present invention also provides a Spot GPU inventory prediction algorithm for large-scale clusters. The Spot GPU inventory prediction algorithm for large-scale clusters can be implemented by executing the process steps of the GPU demand prediction algorithm for perceiving business information for large-scale clusters. That is, those skilled in the art can understand the GPU demand prediction algorithm for perceiving business information for large-scale clusters as one of the preferred implementation methods of the Spot GPU inventory prediction algorithm for large-scale clusters.

[0135] According to the Spot GPU inventory prediction algorithm for large-scale clusters provided by the present invention, Figure 2 Shown, including:

[0136] Step M1: Based on the collected workload, resource trajectory, and business information related to the cost organization, use the aforementioned business information-aware GPU demand prediction algorithm, namely the OrgLinear algorithm, to predict the probability distribution of the future online GPU demand for each cost organization; specifically, the Gaussian distribution.

[0137] Step M2: Based on the predicted probability distribution of the number of online GPU cards required by each cost organization in the future and the user's Spot guarantee requirements, calculate the upper limit of the online GPU card demand and write it to the OrgLinear usage forecast database;

[0138] Specifically, in step M2:

[0139] Methods for predicting the upper limit of online GPU card demand include:

[0140] Assume that the historical GPU usage sequence of cost organization i obtained from the cluster resource trace data is χ i , using the OrgLinear algorithm to predict the future online GPU usage of cost organization i, the mean and standard deviation are μ i and σ i , the Spot guarantee rate required by the business side is p, for example 95%. Assuming that the future online GPU usage follows a Gaussian distribution, norm.ppf represents the quantile function of the Gaussian distribution, that is, the inverse of the CDF, then the probability of the Spot guarantee rate p not conflicting with the upper limit of the number of online GPU cards required

[0141]

[0142] Step M3: Check whether concept drift occurs based on the historical OrgLinear algorithm usage prediction output and actual usage. If the error is too large, use the fallback algorithm to calibrate the usage prediction result;

[0143] Specifically, in step M3:

[0144] The methods for calibrating the backup algorithm include:

[0145] When the ratio of the actual online GPU usage in the past period to the upper limit of the online GPU card demand predicted by the OrgLinear algorithm is greater than a certain threshold, such as 0.1, it is considered that concept drift has occurred. In this case, the fallback algorithm is used instead - taking the peak GPU usage in the past period, with the default guarantee time step hour, as the predicted value of the upper limit of the online GPU card demand of the cost organization in the future where χ i [-step:] represents the actual GPU usage sequence of cost organization i in the past step hour:

[0146]

[0147] Step M4: Perform spatiotemporal aggregation on the prediction results and output the GPU Spot predicted inventory number for the specified spatial range and guaranteed duration.

[0148] Specifically, in step M4:

[0149] Methods for spatiotemporal aggregation include:

[0150] Spatial range aggregation only requires summing up the predicted upper limit of online GPU card demand for all cost organizations that have historically used GPUs within the spatial range to obtain the upper limit of the total online GPU card usage within the spatial range.

[0151]

[0152] Time-granularity aggregation takes the predicted peak GPU usage in the next step hour and combines it with the resource capacity C within the spatial range to calculate the number of inventory GPU cards M available for the future Spot product:

[0153]

[0154] Example 3:

[0155] Example 3 is a preferred example of Example 2 and is used to illustrate the present invention in more detail.

[0156] The present invention discloses a GPU demand prediction algorithm and a SpotGPU inventory prediction algorithm for perceiving business information for large-scale clusters. The present invention proposes OrgLinear, a lightweight GPU demand distribution prediction algorithm based on business and time feature modeling. First, we parse the GPU usage time series data at the cost organization granularity from the production cluster Trace data and decompose it into periodic item components and trend item components. Secondly, the present invention models and mines features from business information and forecast time information as well as the correlation between them. On this basis, the present invention uses a Linear network to learn the periodicity, trend and volatility characteristics of GPU usage time series data to achieve a distribution prediction of the future number of GPU cards required for each cost organization. The OrgLinear algorithm proposed in the present invention can reduce the error by 18% to 33% and provide a more accurate and less expensive distribution prediction capability.

[0157] This paper proposes a lightweight algorithm that can perform online rolling predictions on the future number of GPU cards required. Based on this algorithm, it further proposes a method that can effectively predict the sales space and guarantee amount of Spot GPUs, and provides the design details and specific implementation mechanism of this method.

[0158] 1. Framework structure

[0159] Figure 1 、 Figure 2The figure shows the framework of the GPU demand prediction algorithm for perceiving business information for large-scale clusters proposed by the present invention, namely the OrgLinear algorithm and the Spot GPU inventory prediction algorithm. Among them, the SpotGPU inventory prediction algorithm proposed by the present invention includes a GPU demand prediction algorithm, namely the OrgLinear algorithm module and a Spot guaranteed inventory calculation module. The OrgLinear algorithm module is designed to predict the probability distribution of future GPU demand for each cost organization, including a GPU usage time series decomposition module, a business and time feature extraction module, a usage mean prediction module and a usage volatility prediction module, namely the usage volatility standard deviation prediction module. The basic idea of ​​the algorithm is to propose a lightweight resource management algorithm logic that is independent of the cluster scheduler, which can predict the number of online demand cards and the number of available Spot inventory cards in the future period based on the business characteristics, time characteristics and real-time changes in GPU resource usage in actual production, determine the supply quota and sales space of Spot resources, and improve the utilization efficiency of precious GPU resources.

[0160] 2. Module working process

[0161] (1) Cost organization granularity GPU demand card number prediction module, namely OrgLinear algorithm;

[0162] 1) GPU usage timing decomposition module

[0163] GPU Spot inventory prediction is essentially a prediction of the GPU usage of online tasks in the future. The present invention obtains the historical sequence of online task GPU usage submitted by each cost organization from the real production cluster Trace data. i Inspired by Autoformer and FEDformer, combined with the periodic characteristics of GPU usage sequence, we use the historical sequence of GPU usage χ i Decompose the periodic component χ i,s and the trend component χ i,t :

[0164] χ i,t =Avgpool(Padding(χ i )) (14)

[0165] χ i,s =v i -χ i,t (15)

[0166] The Axgpool function represents the sliding average of the input sequence, and the sliding average kernel size is 1 day, that is, the GPU usage sequence is decomposed from the perspective of daily periodicity.

[0167] 2) Business and time feature extraction module

[0168] In addition to the input time series itself, the present invention also extracts features from the business and forecast time as input to the prediction neural network. For each set of cost organization i related business information, including: cluster, card model, resource pool, cost organization, one-hot encoding is used, and each is first transformed into a continuous feature vector of the same dimension using the Embedding layer, and then the self-attention mechanism is used to learn the correlation between them, and finally the business feature c is obtained. i .

[0169] GPU business feature extraction module, such as Figure 4 As shown, the formal expression is as follows, where the business characteristics of cost organization i are input as v i1 ,v i2 ,…,v ij , Embedding represents the embedding layer, concat represents the concatenation function, and Attention represents the self-attention mechanism:

[0170] c i =Attention(concat(Embedding(v i1 ),Embedding(v i2 ),…,Embedding(v ij )))(1)

[0171] The Self-Attention mechanism is expressed as follows:

[0172]

[0173] Among them, Q, K, and V represent Query, key, and Value, respectively, which are obtained by transforming the input matrix through the linear layer; d represents the dimension of K; Softmax() represents the Softmax activation function.

[0174] Since GPU usage time series data has certain periodic characteristics and its patterns on weekdays and holidays may be different, the input of the time feature extraction module is the time point of the current prediction. First, the three dimensions of hour t.hour, day of the week t.weekday and whether it is a holiday t.holiday are parsed from the time t. Then, the feature vectors of the same dimension are obtained through the embedding layer respectively, and the time feature c is obtained by splicing. t Output. The temporal feature extraction module is as follows Figure 5 As shown, the formal expression is as follows:

[0175] c t=concat(Embedding(t.hour),Embedding(t.weekday),Embedding(t.holiday)) (3)

[0177] 3) Usage mean prediction module

[0178] After decomposing the GPU usage time series and extracting business time features, the present invention uses two different Linear layers to predict the periodic and trend terms of future GPU demand. This is inspired by DLinear, and compared to complex Transformer-like time series forecasting models, simple linear models are sufficient to extract the periodic and trend terms contained in the time series, potentially achieving a 20% to 50% advantage over Transformers.

[0179] For the periodic term prediction of GPU usage series, Linear() represents the Linear linear layer, and the input is the historical periodic term component χ i,s 、Business characteristics c i and time characteristics c t The output is the prediction period item sequence (The length is the number of hours to predict in the future, the default is 4):

[0180]

[0181] Forecasted trend item sequence

[0182]

[0183] Finally, the predicted periodic term sequence and trend item sequence Add up to get the mean μ of the predicted future GPU demand i :

[0184]

[0185] 4) Usage volatility standard deviation prediction module

[0186] Since the OrgLinear algorithm module needs to predict the probability distribution of the number of GPU cards required by each cost organization in the future, according to the input sequence χ i 、Business characteristics c i and time characteristics c t Use the Linear layer to predict the fluctuation sequence h of future GPU demand i , it is also necessary to obtain a non-negative prediction standard deviation σ through the Softplus activation function Softplus() i:

[0187] h i = Linear(concat(x i , c i , c t )) (7)

[0188] σ i = Softplus(h i ) (8)

[0189] According to the mean μ i and the standard deviation σ i of the predicted future GPU usage, we can approximately use the Gaussian Distribution to predict the distribution of the future GPU usage sequence. The entire OrgLinear network uses the probability density function of the Gaussian Distribution as the Loss function for training, and the mathematical expression is:

[0190]

[0191] Where, loss represents the Loss function; y i represents the real future GPU usage sequence, and p(y i | μ i , σ i ) represents the probability density function of the Gaussian Distribution.

[0192] (2) Spot safeguard inventory calculation module

[0193] After predicting the mean μ i and the standard deviation σ i of the future GPU usage through the OrgLinear algorithm, the upper limit of the online GPU card number demand with a probability of p of not occurring conflict can be calculated according to the business side's required Spot guarantee rate p, for example, 95%. In actual production, sometimes due to some sudden business or operation and maintenance reasons, such as large-scale machine migration in and out, the GPU usage mode of the cluster or cost organization will change dramatically, which is called Concept Drift. The present application dynamically detects the concept drift phenomenon, and when it is detected that the MAPE prediction error of a period of time, for example, 1 hour, is greater than a certain threshold, for example, 0.1, a bottom algorithm is used, that is, the peak GPU usage of the same length of time in the past is taken as the prediction value of the upper limit of the future online GPU card number demand. The concept drift detection and bottom algorithm of the present application can quickly and timely detect the concept drift, effectively correct and control the prediction error of the online GPU demand card number.

[0194] After determining the maximum online GPU demand for each cost organization, we can aggregate the upper limits of each cost organization's online GPU demand based on the guaranteed duration and predicted spatial range requirements to determine the guaranteed Spot inventory within that spatial range. Fine-grained cost organization-level predictions also provide more information for subsequent Spot scheduling algorithms than directly predicting the online Spot inventory.

[0195] Taking the cluster-card model space granularity as an example, the Spot guaranteed inventory calculation module is shown in Algorithm 1.

[0196]

[0197] Based on Algorithm 1, the present invention can flexibly provide Spot GPU inventory card supply with various guarantee durations (e.g., 1 hour, 2 hours, 4 hours, etc.), spatial granularity, and guarantee rate. The spatial granularity includes: cluster, card model, and resource pool.

[0198] 3. Algorithm Implementation

[0199] The Spot GPU inventory prediction algorithm for large-scale clusters proposed in this invention can be implemented using the Flask framework + Celery distributed task execution framework + Celery message middleware, and lightweight deployed on a distributed cloud service container. Distributed tasks such as synchronizing trace data, updating model parameters, and real-time inventory prediction can be run regularly through Web URL calls. The Spot GPU inventory prediction results are written to the cloud database, and users call the API to read the latest Spot GPU inventory prediction data.

[0200] This invention can be used non-invasively as a bypass algorithm for current cluster schedulers and is applicable to cluster scheduling frameworks of mainstream cloud service providers such as PAI. It can synchronize production trace data from large-scale data centers online, update model parameters, and perform real-time analysis. Specifically, this invention works in conjunction with data center resource monitoring systems, quota operation systems, and scheduling systems.

[0201] For the resource monitoring system, the present invention relies on the monitoring platform of the large-scale production cluster to read the GPU usage-related data and the business information data of the GPU. For the quota operation system, it regularly calls the query API to update the total inventory number of Spot GPU cards in the resource pool, and distributes the total inventory number of Spot GPU cards to each cost organization that uses the total inventory number of Spot GPU cards. When users deploy Spot containers on a certain resource pool host, they must meet the corresponding quota constraints. For the scheduling system, the cluster scheduler will ensure that online instances do not preempt Spot instances as much as possible, and the resource requirements of the two will not conflict as much as possible.

[0202] The operation process of the Spot GPU inventory prediction algorithm proposed in this invention includes 6 steps:

[0203] Step 1: For any cost organization i, at any time t, based on the collected trace data, the GPU usage history sequence χ i Decompose the periodic component χ i,s and the trend component χ i,t , as shown in formula (14) and formula (15); extract business characteristics c based on cost organization business information i , as shown in formula (1); extract the time feature c according to time t t , as shown in formula (3).

[0204] Step 2: According to the periodic component χ i,s 、Business characteristics c i , time characteristics c t , predict the periodic item sequence of the future GPU demand card number As shown in formula (4); according to the trend term component χ i,t 、Business characteristics c i , time characteristics c t , predict the trend item sequence of future GPU demand card number As shown in formula (5), the mean value μ of the number of GPU cards required in the future is obtained by synthesis i , as shown in formula (6).

[0205] Step 3: Based on the GPU usage history series χ i 、Business characteristics c i , time characteristics c t , predict the standard deviation of the number of GPU cards required in the future σ i , as shown in formula (7) and formula (8).

[0206] Step 4: Organize the future online GPU usage average μ based on the predicted cost i and standard deviation σ i Based on the Spot guarantee rate p required by the business side, calculate the upper limit of the number of online GPU cards required by cost organization i As shown in formula (10).

[0207] Step 5: Forecast the upper limit of the number of online GPU cards required for each cost organization i based on historical data Perform concept drift detection. When concept drift is detected, use the fallback algorithm as shown in formula (11).

[0208] Step 6: Forecast the upper limit of online GPU card requirements for each cost organization i based on historical data Perform spatiotemporal aggregation to obtain the number M of inventory GPU cards available for Spot products in the specified spatial range in the future, as shown in formulas (12) and (13).

[0209] 4. Evaluation of Invention Effect

[0210] In order to verify the effect of the OrgLinear algorithm proposed in this invention in the online GPU usage prediction scenario. Six months of GPU Trace data were selected from a large-scale production cluster of an Internet company mainly engaged in e-commerce business, covering a total of 6 card types and 14 GPU business organizations. The data of the first 5 months were used as the training set, and the data of the last month were used as the test set. The model parameters are updated every day, and the online GPU usage peak for the next 4 hours is predicted every 5 minutes. The single-value prediction error is evaluated using four indicators: mean absolute error (MAE), mean square error (MSE), root mean square error (RMSE), and mean absolute percentage error (MAPE). The smaller the error, the more accurate the prediction algorithm:

[0211]

[0212] Among them, y i Indicates the actual value of GPU usage of cost organization i, represents the predicted GPU usage of cost organization i, and n represents the number of cost organizations in the test set.

[0213] The comparison results of the OrgLinear algorithm and existing general time series prediction algorithms are shown in the following table. Compared with the existing mainstream Transformer-based time series prediction algorithms and the DLinear algorithm, the OrgLinear algorithm proposed in this paper reduces the error by 18-33%.

[0214] Table 1 Comparison of the results of the OrgLinear algorithm and the general timing prediction algorithm in the online GPU usage prediction scenario

[0215] MAE MSE RMSE MAPE FEDformer 55.47 10804.65 103.95 0.199 Autoformer 58.62 12828.20 113.26 0.210 Informer 54.56 16043.29 126.66 0.201 Transformer 41.25 8249.51 90.83 0.180 DLinear 32.65 6848.49 82.76 0.131 OrgLinear 24.39 4570.68 67.61 0.097 Improved results 25.30％ 33.26％ 18.31％ 25.95％

[0216] Since the OrgLinear algorithm is an algorithm for predicting the probability distribution of online GPU usage, the OrgLinear algorithm is compared with the classic algorithm Deepar for predicting the probability distribution of time series data, and a new evaluation indicator for the p-quantile prediction error is defined, p-MAQE, namely Mean Absolute Quantile Error, which represents the absolute value of the error between the actual quantile of the p-quantile prediction value and p. For example, if only 94% of the time series prediction results are less than the 0.95-quantile, then the 0.95-MAQE error is 0.01. In formula (20), represents the p-quantile value sequence of the probability distribution of the predicted time series, Z represents the actual value sequence, Indicates the frequency of occurrence of actual values ​​being less than the p-percentile value, and n represents the number of cost organizations in the test set. In this invention, since the business side requires a 95% guarantee rate for Spot GPUs, p is set to 0.95.

[0217]

[0218] Among them, p-MAQE represents the evaluation indicator.

[0219] The results are compared in the table below. The OrgLinear algorithm not only significantly outperforms the Deepar time series prediction algorithm in terms of the accuracy of the aforementioned single-value predictions, but also supports the ability to predict distribution probabilities. When predicting 0.95-percentile online GPU usage time series data, the 0.95-MAQE error metric shows a significant improvement of 45.61% compared to the Deepar algorithm. Furthermore, because Deepar is an LSTM-based time series prediction algorithm, while the OrgLinear algorithm proposed in this paper is primarily based on a Linear network, its time overhead is only 1.63% of that of the Deepar algorithm.

[0220] Table 2 Comparison of the results of the OrgLinear algorithm and the distributed time series prediction algorithm in the online GPU usage prediction scenario

[0221] MAE MSE RMSE MAPE 0.95-MAQE Time(s) Deepar 32.22 9181.70 95.82 0.120 0.057 28610.11 OrgLinear 24.39 4570.68 67.61 0.097 0.026 464.94 Improved results 24.30％ 50.22％ 29.44％ 19.17％ 45.61％ 98.37％

[0222] According to the present invention, a computer-readable storage medium storing a computer program is provided. When the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method are implemented.

[0223] According to the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the SpotGPU inventory prediction method are implemented.

[0224] Those skilled in the art will appreciate that, in addition to implementing the system and its various devices, modules, and units provided by the present invention in purely computer-readable program code, it is entirely possible to implement the same functions of the system and its various devices, modules, and units provided by the present invention in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system and its various devices, modules, and units provided by the present invention can be considered a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; the devices, modules, and units for implementing various functions can also be considered as both software modules implementing the method and structures within the hardware component.

[0225] The above describes specific embodiments of the present invention. It should be understood that the present invention is not limited to the specific embodiments described above, and that those skilled in the art may make various changes or modifications within the scope of the claims, without affecting the essence of the present invention. Unless there is a conflict, the embodiments of this application and the features in the embodiments may be combined with each other in any manner.

Claims

1. A GPU demand forecasting method, characterized in that: include: Step S1: collecting a historical sequence of GPU usage and decomposing the historical sequence of GPU usage into component sequences; in step S1, the component sequence includes a periodic item sequence and a trend item sequence; Step S2: extracting business features and time features based on the input business information and time information; The mathematical expression of the service characteristics is: in, Indicates cost organization business characteristics; Indicates cost organization The business characteristics input contained; represents the embedding layer, represents the concatenation function, Represents the self-attention mechanism; The mathematical expression of the time characteristic is: in, Represents time characteristics; Indicates hours; Indicates the day of the week; Indicates holidays; Step S3: Based on the component sequence, business characteristics, and time characteristics, the future period item sequence and trend item sequence are predicted through the Linear layer, and then the period item sequence and trend item sequence are added to obtain the mean value of the GPU demand; Step S4: predicting the standard deviation of GPU demand, and deriving the distribution of future GPU demand based on the mean of the GPU demand; In step S4, the mathematical expression of the standard deviation of the GPU requirement is: （8） in, Indicates the standard deviation of GPU demand; Represents the Softplus activation function; represents the sequence of fluctuation terms; The mathematical expression of the fluctuation term sequence is: （7） in, represents the input sequence; In step S4, the number of GPU cards required in the future follows a Gaussian distribution, and the mathematical expression of its loss function is: （9） in, Represents the Loss loss function represents the probability density function of the Gaussian distribution, represents the mean value of GPU demand, Indicates the standard deviation of the mean of GPU requirements; Represents the real future GPU usage sequence; Represents the probability density function Perform continuous multiplication operations.

2. The GPU demand forecasting method according to claim 1, wherein: In step S2, the self-attention mechanism is expressed as follows: （2） in, Represents Query, key and Value respectively; Query, key and Value are obtained by transforming the input matrix through the linear layer; express The dimension size of represents the Softmax activation function; the superscript Indicates transposing the key matrix.

3. The GPU demand forecasting method according to claim 2, wherein: In step S3, the periodic item sequence （4） in, represents a sequence of periodic terms; Represents the Linear layer; represents the periodic component; The mathematical expression of the trend item sequence is: （5） in, represents a sequence of trend items; represents the trend term component; In step S3, the mathematical expression of the mean value of the GPU requirements is: （6） in, Indicates the average GPU demand.

4. A Spot GPU inventory forecasting method, using the GPU demand forecasting method according to claim 1, characterized in that: include: Step M1: collecting business information, and predicting the probability distribution of the number of online GPU cards required by each cost organization based on the business information and historical sequences; the prediction adopts the GPU demand prediction method; Step M2: Calculate the upper limit of the number of online GPU cards required based on the probability distribution of the number of online GPU cards required and the user's Spot guarantee requirement; Step M3: Based on the upper limit of the number of online GPU cards required, determine whether concept drift occurs; if the result is yes, use the bottom-up algorithm to predict the result; if the result is no, use the upper limit of the number of online GPU cards required in step M2 as the prediction result; Step M4: spatiotemporally aggregate the prediction results and output the GPU Spot predicted inventory number.

5. The Spot GPU inventory prediction method according to claim 4, wherein: In the step M1, the business information includes workload, resource trace and cost organization.

6. The Spot GPU inventory prediction method according to claim 5, characterized in that: The mathematical expression for predicting the upper limit of the number of online GPU cards required is: （10） in, Indicates the upper limit of the number of online GPU cards required; represents the quantile function of the Gaussian distribution; Indicates the Spot guarantee rate.

7. The Spot GPU inventory prediction method according to claim 6, characterized in that: In step M3, it is determined whether the ratio of online GPU usage to the predicted upper limit of online GPU card demand is greater than a preset threshold. If the result is yes, a fallback algorithm is used to predict the result. If the result is no, the upper limit of online GPU card demand described in step M2 is used as the prediction result. The preset threshold is 0.

1. The fallback algorithm uses the past peak value of GPU usage as the predicted value of the upper limit of online GPU card demand of the cost organization in the future. The prediction result, i.e. the predicted upper limit of the future demand for online GPU cards, is expressed as follows: （11） in, Indicates the predicted upper limit of the future demand for online GPU cards; Indicates taking the maximum value; Indicates the past Hourly Cost Organization Real GPU usage sequence; In step M4, the GPU Spot predicts the number of inventory cards, and the mathematical expression is: （13） in, Indicates the number of GPU Spot predicted inventory cards; Indicates resource capacity; Indicates the upper limit of the total number of online GPU cards within the specified space range; The upper limit of the total number of online GPU cards within the specified space is expressed as follows: （12） in, Indicates the predicted upper limit of the future demand for online GPU cards.

8. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method according to any one of claims 4 to 7 are implemented.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the computer program is executed by a processor, the steps of the Spot GPU inventory prediction method according to any one of claims 4 to 7 are implemented.

Images