Resource index prediction method and device based on Transform framework and medium

Abstract

The invention provides a resource index prediction method and device based on a Transform framework and a medium, and the method comprises the steps: carrying out the pre-training of a PatchTST model through employing the historical data of resource indexes, such as a CPU, a memory, and the like, improving the future values of the prediction indexes of a conventional PatchTST model into the future distribution of the prediction indexes, obtaining a prediction mean value and a prediction variance, and carrying out the prediction of the prediction mean value and the prediction variance. And further converting into prediction value output, prediction interval upper bound output and prediction interval lower bound output. Compared with a traditional time sequence prediction method based on a Transform architecture, the method has the advantages that the prediction accuracy is high, the training time is short, the upper and lower bounds of a prediction interval can be output, and the predicted upper and lower bounds can be used for representing a confidence interval of a predicted value. According to the invention, richer and more accurate guidance can be provided for intelligent operation and maintenance scenes such as dynamic capacity expansion and contraction, capacity management, capacity planning and the like.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a resource indicator prediction method, device and medium based on a Transformer framework. Background Art

[0002] Existing technical solutions mainly include four types of algorithms: 1) resource indicator time series prediction based on expert experience; 2) resource indicator time series prediction based on statistical methods; 3) resource indicator time series prediction based on machine learning algorithms; 4) resource indicator time series prediction based on deep learning algorithms.

[0003] Time series forecasting of resource indicators based on expert experience: This type of method is mainly based on the professional knowledge and professional analysis capabilities accumulated by experts in various fields. Experts usually make judgments based on the changes in time series type indicators and static variable indicators that may have an impact on the future trend of the forecast indicators, and give long-term and short-term forecast conclusions. The main disadvantage of this type of method is that it relies too much on the level of expert experience, and the forecast effects of different experts are uneven and the confidence level is low.

[0004] Resource indicator time series prediction based on statistical methods: Common statistical prediction methods include sliding average prediction, Arima model, linear fitting, etc.; the prediction based on sliding average is overly dependent on the most recent historical data and cannot capture the paradigm of data changes; although Arima, linear fitting and multivariate high-order fitting can output explicit representations of historical data and future data, they have two obvious defects: 1. The fitting expression is fixed, resulting in the final model being too simple and only suitable for simple trend prediction. 2. Only when the data trend is close to the formula to be fitted can a good fitting accuracy be obtained, and it is easy to cause overfitting problems, resulting in poor extrapolation performance of the obtained model.

[0005] Resource indicator time series prediction based on machine learning methods: With the rise of machine learning methods, MLOps has gradually been applied in the operation and maintenance systems of major companies and banking systems. Traditional machine learning models are mainly used in classification and regression tasks. The time series prediction of resource indicators is essentially the use of machine learning methods to complete autoregression. In recent years, many advanced machine learning methods: LightGBM, XGBoost, random forest, etc. have been proven to have high prediction accuracy and prediction efficiency in regression tasks, especially the capture of correlations and regularities between nonlinear data. However, this type of method also has disadvantages: 1. It can only support single output, that is, it can only rely on historical data to predict the resource indicator value at a future time point each time. 2. The historical dependency is fixed, and the long historical dependency will lead to data feature redundancy. The decisive historical dependency may lead to insufficient information and affect the prediction accuracy. Although some Python-based methods such as MultiRegressor can now realize multi-output regression prediction of machine learning, this type of method only establishes multiple models for time series prediction, ignoring time information. Based on the above two defects, machine learning methods cannot output resource trends for a period of time in the future at one time, and operation and maintenance scenarios such as dynamic expansion and contraction require a period of prediction data to make more accurate operations.

[0006] Resource indicator time series prediction based on deep learning methods: RNN is the first proposed sequence-based neural network, which is widely used in the field of time series prediction. LSTM solves the problem of catastrophic forgetting when RNN has long historical dependencies by designing a long-term and short-term gating unit to selectively control the correlation between long-term and short-term historical data and future data. In addition, there are deep learning models such as BiLSTM and GRU. These models have well solved the problem that machine learning models cannot output resource trends for a period of time in the future at one time. However, there is a prediction error that will increase exponentially with the increase of prediction length. It is impossible to predict the trend of resource indicators in the long term in the future with high accuracy.

[0007] In order to solve the defects of the existing technical solutions, the present application proposes a resource indicator prediction method, device and medium based on the Transformer framework. Transformer is widely used in sequence problems and is good at processing long sequence problems. More specifically: The present application uses the PatchTST model. The PatchTST model is a Transformer model based on Patch token input. The main problems solved by the present application are as follows: 1. It is good at processing longer historical time series dependencies. Compared with the traditional Transformer model, it can process N2 historical dependencies under the same computing time requirements and memory resources (Transformer can process when the historical dependency is N). 2. It can pay more attention to local information and can deal with the problem of insufficient local information extraction of general deep learning models. 3. Baseband prediction can be performed based on the Transformer architecture. The present application uses the maximum likelihood function as the loss function, and optimizes the operation of predicting the true value of the traditional PatchTST model to predicting the distribution of the true value. The output baseband is more instructive for resource regulation in the production environment. Summary of the invention

[0008] The embodiment of the present application provides a resource indicator prediction method, device and medium based on the Transformer framework to solve at least one problem existing in the related technology. The technical solution is as follows:

[0009] In a first aspect, an embodiment of the present application provides a resource indicator prediction method based on a Transformer framework, comprising: Step 1: Computing power configuration: basic CPU configuration, consumer GPU selection configuration, and a set of servers with appropriate cost and capable of supporting daily data volume for model training; Step 2: Data preprocessing: data cleaning, removing missing values, aligning timestamps, aggregating data according to application requirements, and performing missing data filling and feature engineering as appropriate; Step 3, time series decomposition: decompose the data after data preprocessing into periodic terms and trend terms, and input the periodic terms and trend terms into the PatchTST model respectively; Step 4: Model training: Use the self-developed PatchTST model code that can achieve upper and lower baseline outputs, and use the corresponding computing power configuration to perform model training; Step 5: Model prediction: Deploy the trained model on the Jiawei Blue Whale computing and AI platform for model prediction, and link the output results with the capacity management system.

[0010] In one embodiment, the computing power configuration includes: Step 1.1: The computing power configuration requires a consumer-level graphics card of Tesla V100 with 16GB video memory, RTX 3090Ti with 24GB video memory, or RTX4090 with 24GB video memory. The CPU configuration requires 32 cores or more. Step 1.2: Configure NIVIDA CUDA drivers on the server: GPU Driver, CUDA Toolkit.

[0011] In one embodiment, the data preprocessing includes: Step 2.1: Data cleaning, removing invalid values ​​and null values; Step 2.2: Sort the data according to the time sequence and use the sliding average method to fill the invalid values ​​and null values ​​removed in step 2.1; Step 2.3: Determine the input and output data length parameters according to the application requirements, and aggregate the original data according to the target data granularity to obtain the time series data of the target granularity.

[0012] In one embodiment, the model training includes: Step 4.1: The periodic term and trend term calculated in step 3 are respectively input into the PatchTST model; the PatchTST model is a time series prediction model based on the Transformer framework; Step 4.2: Divide the input time series data into N patches, N=\left [ {\frac {LP} {S}} \right ]+2 , L is the length of the input time series L, which is divided into N patches of length P, S is the number of points in the non-overlapping area, and the brackets represent rounding down; the patch is a subsequence of the specified length of the time series; Step 4.3: Patch data is input into the Embedding layer; the Embedding layer maps the original data into higher-dimensional data as the input of the model through a linear neural network; Step 4.4: Input the Transformer layer; the Transformer layer consists of a multi-head attention mechanism, a forward propagation layer, and a residual link; Step 4.5: The output of the Transformer layer is a single high-dimensional matrix, which is copied into two matrices; the first matrix is ​​input to the linear layer and the Droupout layer and output as the predicted mean, and the second matrix is ​​input to the linear layer and the Droupout layer and normalized by Softplus as the predicted variance; the Droupout layer is a method to avoid overfitting of the neural network, and the Softplus is an activation function of the neural network; Step 4.6: Use the predicted mean and predicted variance as the predicted distribution parameters, input the maximum likelihood function to calculate the deviation between the predicted value distribution and the true value distribution as the loss function, and the model updates the neural network weight parameters through back propagation of the loss function calculation value; Step 4.7: Iterate until the loss function error converges.

[0013] In one embodiment, the model prediction includes: Step 5.1: Save the trained model file as a pkl file; Step 5.2: Access the real-time data stream of Kafka resource indicator data; Step 5.3: Build the Dataflow canvas model flow chart, including data real-time computing nodes, model nodes, and storage nodes; Step 5.4: Start the model flow chart constructed in step 5.3; Step 5.5: Query the prediction results stored in the database.

[0014] In one embodiment, the data source is the historical monitoring data of resource indicators of the Jiawei Blue Whale platform.

[0015] In a second aspect, an embodiment of the present application provides an electronic device, including: a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned embodiments.

[0016] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, the method in any one of the above-mentioned embodiments is implemented.

[0017] The beneficial effects of the above technical solution include at least: Predict future trends for observable resource indicators in all operation and maintenance fields, and guide the implementation of intelligent scenarios for automatic capacity expansion in capacity management; Output resource indicator prediction upper and lower baselines to guide the implementation of intelligent operation and maintenance scenarios such as abnormal alarms; Deployed on Jiawei's Blue Whale computing AI platform, the model configuration is highly flexible and the algorithm is easy to use.

[0018] The above summary is for illustrative purposes only and is not intended to be limiting in any way. In addition to the illustrative aspects, embodiments and features described above, further aspects, embodiments and features of the present application will be readily apparent by reference to the accompanying drawings and the following detailed description. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the accompanying drawings required for use in the embodiments will be briefly introduced below. In the accompanying drawings, unless otherwise specified, the same reference numerals throughout multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that the following drawings only illustrate certain embodiments of the present application and are therefore not to be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can also be obtained based on these drawings without paying creative work.

[0020] Figure 1 It is the resource indicator time series prediction process based on Transformer architecture; Figure 2 This is the flow chart of the resource indicator timing prediction framework based on PatchTST; Figure 3 It is the Softplus function graph; Figure 4 This is the algorithm configuration diagram of Jiawei's Blue Whale computing AI platform; Figure 5 The output effect of the PatchTST resource indicator prediction model on the public dataset; Figure 6 This is the output effect of the PatchTST resource indicator prediction model on the Blue Whale dataset. DETAILED DESCRIPTION

[0021] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the spirit or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary and non-restrictive in nature.

[0022] The resource indicator time series prediction framework based on PatchTST proposed in this application can be found in Figure 1 ,It is mainly divided into two parts: offline training and online prediction. ,In the offline training phase, the historical data of resource indicators are ,obtained from the Blue Whale database and the training data set is constructed through ,data cleaning. The training data set is input into the PatchTST model for ,finetuning. Figure 1 It is the flow chart of resource indicator time series prediction model. Figure 2 It is the training flow chart and model architecture diagram of the PatchTST resource indicator time series prediction model.

[0023] PatchTST is a model based on the Transformer architecture. Based on the Transformer architecture (Encoder-Decoder), it adopts only the Encoder architecture. After the data enters the model, it first performs instance normalization (Instance Norm) and patching operations. The patching operation will split the input time series of length L into N patches of length P. These patches can overlap or not overlap. For S non-overlapping area points, the brackets represent rounding down. The number of patches N split out is: N=\left [ {\frac {LP} {S}} \right ]+2

[0024] The Patch sequence is input into the Embedding model for Embedding operation and then into the Transformer. The Transformer here only uses the Encoder part of the traditional Transformer architecture and uses a multi-head attention mechanism to calculate the attention mechanism:

[0025] The output of the Transformer is Flattened and then input into the Linear layer for transformation to obtain μ and σ respectively:

[0026] μ is the predicted mean, and σ is the predicted variance. In order to ensure the positive property of the variance, the Softplus activation function is used here. The Softplus activation function is evolved from the Relu activation function. Figure 3 is the image of the softplus function, which has good non-negative continuity.

[0027] The predicted μ and σ are the predicted mean and variance, which are essentially the predicted data distribution. The predicted data distribution and the true value data distribution are estimated by maximum likelihood to obtain the loss function:

[0028] The loss function is used for back propagation and continuous training to obtain the final model.

[0029] The training steps of the PatchTST resource indicator time series prediction model are summarized as follows: Obtain resource indicator time series data, perform data cleaning, filter invalid values, and fill in empty values; Decompose the historical resource indicator data into time series, into periodic items and trend items; The periodic term and trend term are respectively input into the PatchTST model for training; Output predictions μ and σ; The negative of the maximum likelihood estimate calculated with the true value is used as the loss for a single training run; The loss is back-propagated within the model, and the gradient is back-propagated to update the network parameters; Repeat 2 to 6 until the model loss converges.

[0030] The production environment prediction steps of PatchTST are as follows: Input the historical dependency of the sequence to be predicted; Determine whether the historical dependency is met and whether there are any gaps; If step 2 is not satisfied, return to unpredictable; if step 2 is satisfied, proceed to step 4; Use the trained model to predict the output μ and σ; Returns the upper and lower bounds of the baseband according to the set baseband width coefficient a , ; Drawing display; Repeat steps 1 to 6 after new historical data is entered.

[0031] In one implementation, the present application can be applied to many data platforms such as computing AI platforms, monitoring platforms, capacity management platforms, and resource scheduling platforms of operation and maintenance products, and can perform time series predictions on all operation and maintenance indicators to assist capacity management platforms, monitoring platforms, and resource scheduling platforms in executing corresponding strategies.

[0032] First, we build a large data set through the historical data of resource indicators in the Pod dimension collected by the computer platform. After a certain amount of cleaning work on the data set, we get the training data set. We use the PatchTST model to perform model pre-training and parameter adjustment optimization, and finally configure it on the Blue Whale platform. Figure 4 .

[0033] In a possible implementation, the above steps are implemented using a public dataset and a Blue Whale historical resource indicator dataset respectively: Figure 5The display shows the prediction effect of PatchTST output on the public data set. It can be seen that the real-time prediction accuracy of PatchTST is very high, almost coinciding with the real curve, and the predicted upper and lower bounds of the baseband are output. The upper and lower basebands can be combined to control the resource scheduling risk when the predicted resource value is obtained. For example, when a more robust adjustment strategy is needed based on the prediction results, resource scheduling can be performed based on the fluctuation of the upper bound; when a more aggressive adjustment strategy is needed based on the prediction results, it can be adjusted based on the lower bound. At the same time, the baseband width coefficient here is an adjustable value, and a wider or narrower baseband can be obtained by adjusting the baseband width coefficient. The baseband prediction results can also be used for abnormal point detection.

[0034] Figure 6 What is displayed is the internal data set of Blue Whale. It can be seen that the data set has a high regularity. The prediction output of PatchTST also shows that the real-time prediction accuracy of PatchTST is very high, almost coinciding with the real curve, and the upper and lower bounds of the predicted baseband also have a high regularity. The strong periodicity of the original data indicates that the distribution of the original data is stable. The obtained mean variance is ideally also a set of stable values, and the baseband indicates the distribution stability of the data. It can be seen that whether it is mean prediction or distribution prediction, the model has a good grasp of the nonlinear law of the original data.

[0035] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the application release orchestration method provided in the above embodiment.

[0036] In addition, an embodiment of the present application provides an electronic device, including: a processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement a method in any one of the above-mentioned embodiments.

[0037] Furthermore, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed, the method in any one of the above-mentioned embodiments is implemented.

[0038] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A resource index prediction method based on Transformer framework, characterized in that: include: Step 1: Computing power configuration: basic CPU configuration, consumer GPU selection configuration, and a set of servers with appropriate cost and capable of supporting daily data volume for model training; Step 2: Data preprocessing: data cleaning, removing missing values, aligning timestamps, aggregating data according to application requirements, and performing missing data filling and feature engineering as appropriate; Step 3, time series decomposition: decompose the data after data preprocessing into periodic terms and trend terms, and input the periodic terms and trend terms into the PatchTST model respectively; Step 4: Model training: Use the self-developed PatchTST model code that can achieve upper and lower baseline outputs, and use the corresponding computing power configuration to perform model training; Step 5: Model prediction: Deploy the trained model on the Jiawei Blue Whale computing and AI platform for model prediction, and link the output results with the capacity management system.

2. The method according to claim 1, characterized in that The computing power configuration includes: Step 1.1: The computing power configuration requires a consumer-level graphics card of Tesla V100 with 16GB video memory, RTX 3090Ti with 24GB video memory, or RTX4090 with 24GB video memory. The CPU configuration requires 32 cores or more. Step 1.2: Configure NIVIDA CUDA drivers on the server: GPU Driver, CUDA Toolkit.

3. The method according to claim 1, characterized in that The data preprocessing includes: Step 2.1: Data cleaning, removing invalid values ​​and null values; Step 2.2: Sort the data according to the time sequence and use the sliding average method to fill the invalid values ​​and null values ​​removed in step 2.1; Step 2.3: Determine the input and output data length parameters according to the application requirements, and aggregate the original data according to the target data granularity to obtain the time series data of the target granularity.

4. The method according to claim 1, characterized in that: The time series decomposition includes: The time-series data without missing parts is decomposed using the sliding average method. The trend item is obtained after the sliding average calculation. The period item is obtained by subtracting the trend item from the original data. The window parameters of the sliding average can be customized.

5. The method according to claim 1, characterized in that The model training includes: Step 4.1: Input the periodic term and trend term calculated in step 3 into the PatchTST model respectively; the PatchTST model is a time series prediction model based on the Transformer framework; Step 4.2: Divide the input time series data into N patches, N=\left [ {\frac {LP} {S}} \right ]+2 , L is the length of the input time series, which is divided into N patches of length P, S is the number of points in the non-overlapping area, and the brackets represent rounding down; the patch is a subsequence of the specified length of the time series; Step 4.3: Patch data is input into the Embedding layer; the Embedding layer maps the original data into higher-dimensional data as the input of the model through a linear neural network; Step 4.4: Input the Transformer layer; the Transformer layer consists of a multi-head attention mechanism, a forward propagation layer, and a residual link; Step 4.5: The output of the Transformer layer is a single high-dimensional matrix, which is copied into two matrices; the first matrix is ​​input to the linear layer and the Droupout layer and output as the predicted mean, and the second matrix is ​​input to the linear layer and the Droupout layer and normalized by Softplus as the predicted variance; the Droupout layer is a method to avoid overfitting of the neural network, and the Softplus is an activation function of the neural network; Step 4.6: Use the predicted mean and predicted variance as the predicted distribution parameters, input the maximum likelihood function to calculate the deviation between the predicted value distribution and the true value distribution as the loss function, and the model updates the neural network weight parameters through back propagation of the loss function calculation value; Step 4.7: Iterate until the loss function error converges.

6. The method according to claim 1, characterized in that The model predictions include: Step 5.1: Save the trained model file as a pkl file; Step 5.2: Access the real-time data stream of Kafka resource indicator data; Step 5.3: Build the Dataflow canvas model flow chart, including data real-time computing nodes, model nodes, and storage nodes; Step 5.4: Start the model flow chart constructed in step 5.3; Step 5.5: Query the prediction results stored in the database.

7. The method according to claim 3, characterized in that The data source is the historical monitoring data of resource indicators of Jiawei Blue Whale platform.

8. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores instructions, and the instructions are loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is executed.

Images