Data missing oriented random workflow load prediction method

Abstract

This invention discloses a method for predicting the load of stochastic workflows with missing data. The method includes: a workflow preprocessing stage, where the workflow is transformed into an embedded vector form and clustered based on historical data similarity; a workflow missing value completion and load prediction stage, where a kernel density missing data filling method based on common features is used to fill in missing data, and a workflow load analysis and prediction method is proposed based on the characteristics of the computing network-cloud-edge-device structure and historical information on workflow computation, storage, and communication; a cluster modeling and feature engineering stage; and a cluster load prediction stage, where this invention combines kernel methods and Kalman filtering techniques to predict missing values ​​and workflow load, optimizing the robustness and generalization of the prediction. This method has broad application value and promising prospects in the fields of load prediction and workflow scheduling.

Description

Technical Field

[0001] This invention relates to a method for predicting the load of random workflows with missing data, belonging to the technical fields of data analysis, deep learning, load prediction, and workflow. Background Technology

[0002] In stochastic workflow scheduling, missing data hinders effective workload prediction and reduces workflow scheduling performance. To address the missing data problem, many imputation prediction algorithms have been proposed to handle missing values, mainly based on supervised learning, i.e., imputing missing values ​​by building a prediction model using the remaining complete data. Ma et al. [1] used a deep denoising autoencoder to study the correlation between missing and non-missing values. Yoon et al. [2] developed a generative adversarial imputation network (GAIN), in which the discriminator aims to distinguish whether the value is real or imputed. Fortuin et al. [3] used a deep variational autoencoder and Gaussian process to transform incomplete time series data into a low-dimensional space for better prediction. In addition, Spinelli et al. [4] used a graph imputation neural network to build graph similarity between complete and incomplete datasets. Pan et al. [5] proposed an adaptive learning median imputation deep autoencoder (AM-DAE) to imput missing values ​​in industrial time series data in an unsupervised manner. It continuously replaces missing values ​​with the median of the input data and its reconstruction, thus allowing imputation information to be transmitted along with the training process. These methods demonstrate the excellent expressive power of deep learning-based methods.

[0003] Cluster load forecasting is a key technology in cloud computing and data center management. Accurately predicting the usage of cluster resources has become the key to resource scheduling and optimization. Domestic and foreign scholars have proposed a variety of forecasting methods for this problem, from traditional statistical methods to deep learning models, to continuously improve the accuracy and efficiency of forecasting. Early studies mainly used statistical methods and traditional machine learning models for load forecasting. Martinez et al. [6] used Kalman filtering to predict node CPU utilization in a serverless environment. Deepika et al. [7] used a multilayer perceptron (MLP) regression model to predict the power consumption of virtual machines. Iqbal et al. [8] proposed an adaptive model selector, which dynamically selects the most suitable forecasting model from linear regression (LR), support vector machine (SVM), gradient boosting decision tree (GBDT) and Gaussian process (GP) through random forest (RF) to estimate the consumption of data center CPU resources, especially time series and bursty behavior. These traditional methods have achieved load forecasting to a certain extent, but they are not good when facing complex nonlinear patterns.

[0004] [1] Q.Ma, W.Lee, T.Fu, Y.Gu and G.Yu, "MIDIA: Exploring denoisingautoencoders for missing data imputation", Data Min.Knowl.Disc., vol.34, no.6, pp.1859-1897, 2020.

[0005] [2] J. Yoon, J. Jordon and M. van der Schaar, "GAIN: Missing data imputation using generative adversarial nets", arxiv:1806.02920, 2018.

[0006] [3] V. Fortuin, D. Baranchuk, G. and S. Mandt, "GP-VAE: Deepprobabilistic time series imputation", arxiv:1907.04155, 2019.

[0007] [4]I.Spinelli,S.Scardapane,M.Scarpiniti and A.Uncini,"Efficient dataaugmentation using graph imputation neural networks"in Progresses inArtificial Intelligence and Neural Systems,Singapore:Springer,pp.57-66,2021.

[0008] [5]Z.Pan,Y.Wang,K.Wang,H.Chen,C.Yang and W.Gui,"Imputation of MissingValues in Time Series Using an Adaptive-Learned Median-Filled DeepAutoencoder,"in IEEE Transactions on Cybernetics,vol.53,no.2,pp.695-706,Feb.2023,doi:10.1109 / TCYB.2022.3167995.

[0009] [6]Martinez,M.M.,&Pandey,S.R.(2022,March).Predictive functionplacement for distributed serverless environments.In 2022 25th Conference onInnovation in Clouds,Internet and Networks(ICIN)(pp.86-90).IEEE.

[0010] [7]Deepika,T.,&Prakash,P.(2020).Power consumption prediction in clouddata center using machine learning.Int.J.Electr.Comput.Eng.(IJECE),10(2),1524-1532.

[0011] [8] Iqbal, W., Berral, JL, Erradi, A., & Carrera, D. (2019). Adaptiveprediction models for data center resources utilization estimation. IEEE Transactions on Network and Service Management, 16(4), 1681-1693. Summary of the Invention

[0012] Technical Problem: In the face of the complexity and diversity of workflow tasks, in order to further improve the robustness and generalization of random workflow load prediction, this invention considers the case of missing values ​​in random workflows. It utilizes historical records to propose a kernel density missing data imputation method based on common features. Based on this, workflow load prediction is performed. Furthermore, in a shared GPU cluster environment, a cluster load prediction method for random workflows with missing data is provided to address the dynamic nature of the cluster environment and resource contention issues.

[0013] Technical Solution: To achieve the above objectives, the technical solution adopted in this invention is: a method for predicting the load of random workflows with missing data, comprising the following stages:

[0014] A. Workflow Preprocessing Stage: The workflow is transformed into embedded vectors. Based on historical random workflow data from the computing power network, time-series data information is memorized and learned. The time-series features of the requests are extracted, and spectral clustering is performed based on common features using three convolutional neural network models responsible for the head node, tail node, and relationship, respectively.

[0015] B. Workflow Missing Value Completion and Load Prediction Phase: For each request class obtained from clustering, a kernel density model is constructed. Combined with Kalman filtering, the weights of the kernel density estimate are dynamically updated. Simultaneously, the recursive properties of the filter are calculated using maximum likelihood estimation to optimize the kernel bandwidth and other dynamic control parameters. Finally, the random workflow load is predicted based on methods such as calculating the expectation of the probability density function.

[0016] C Cluster Modeling and Feature Engineering Phase: Assigning cluster load x t Transform it into a high-dimensional feature vector x containing multiple time series features. t In order to make more accurate predictions and analyses,

[0017] In the D cluster load prediction stage: the N-BEATS model is used to predict the cluster load. The N-BEATS model normalizes the input load data X, and then uses STL decomposition technology to decompose the time series into a linear combination of basis functions, including trend, seasonality and residual. Each basis function is learned and represented by a neural network block. The prediction result can be formed by modeling the above components respectively.

[0018] The specific steps of the workflow preprocessing stage are as follows:

[0019] A1. A directed acyclic graph G(V,E) represents a workflow, where V represents the nodes in the directed acyclic graph and E represents the adjacency matrix of the graph. The graph information is then transformed into an embedding vector W.

[0020] A2. Based on historical data T of random workflow from computing power networks W The neural network model is built to memorize the information of neurons at the current moment and learn the dependencies of time-series data;

[0021] A3. Extract the temporal features t of each request arrival. W ;

[0022] A4. Workflow requirement similarity measurement based on information entropy d W =d(t) W ,T W Based on one-dimensional time series features t W Perform spectral clustering to find tasks Y=G with high similarity. p (t W ,T W ).

[0023] The specific steps of the workflow missing value completion and load prediction phase are as follows:

[0024] B1. Construct three convolutional neural network models, each responsible for completing the head node, tail node, and dependencies.

[0025]

[0026] The structure of a convolutional neural network is as follows:

[0027] B1.1 will input two embedding vectors Y i and Y j Link them into a two-dimensional or three-dimensional array, and use a fusion algorithm to fully merge the two embedding vectors so that they can interact with each other under the action of the convolution kernel;

[0028] B1.2 outputs an edge embedding vector W after passing through a fully connected layer. E Or node embedding vector W GThis vector is transformed into a workflow task node or inter-task dependency relationship to complete the missing data information of the workflow;

[0029] In B1.3, when training the model, a workflow is processed simultaneously using three convolutional neural network models, and all three models use the same loss function and are trained synchronously.

[0030] B2. Construct a kernel density model K for each request class obtained by clustering, and use the original calculation sequence of the workflow in each class as a data record in the kernel density prediction model of that class;

[0031] B3. By using techniques such as linear filtering, exponential weighted filtering, and Kalman filtering, the weights are automatically updated at each time point through the covariance matrix C. To estimate the time-varying density, Kalman filtering and kernel density estimation are combined, and the weights of the dynamic kernel density estimation are updated using the Kalman filtering method.

[0032]

[0033] B4. Maximum likelihood estimation is applied as follows:

[0034] B4.1 Obtain estimates of kernel bandwidth MaxL(h) and other control parameters based on maximizing the likelihood function;

[0035] B4.2 Calculate the estimates of the probability density function, cumulative distribution function, and quantiles using these estimates;

[0036] B5 predicts the random workflow load E(w) based on methods such as calculating the expectation of the probability density function. t ).

[0037] The specific steps of the cluster modeling and feature engineering phase are as follows:

[0038] C1. Reduce cluster load x t Defined as a sequence that changes over time, where t represents a timestamp:

[0039] X = {x1, x2, x3, ..., x} t ,};

[0040] C2. Extract features from the timestamp t, such as month, day, hour, minute, weekday, etc. These features can be represented as: T t ={month(t),day(t),hour(t),minute(t),weekday(t)};

[0041] C3. Use the sliding window technique to generate hysteresis features. For example, for a window size of k, the hysteresis feature can be represented as: L t ={x t-1,x t-2 ,...,x t-k};

[0042] C4. Calculate the statistic within the sliding window, considering the mean μ. t , median m t and standard deviation σ t :

[0043]

[0044] C5. Reduce the original cluster load by x t With the extracted time features T t Lag characteristic L t and statistical characteristics {μ t ,m t ,σ t} combined, forming a high-dimensional feature vector X t :

[0045] X t ={x t ,T t ,L t ,μ t ,m t ,σ t}

[0046] X = {X1, X2, ..., X} t}

[0047] In this way, the cluster load x t Transform it into a high-dimensional feature vector X containing multiple time series features. t The advantages are as follows:

[0048] 1. Comprehensively capture system load dynamics, avoiding the limitations of relying on a single indicator, and providing more accurate predictions and analysis for load forecasting;

[0049] 2. Learn more potential patterns and the impact of seasonal, periodic, or sudden events in order to capture more complex nonlinear relationships, improve the model's predictive ability, and enhance the system's intelligence.

[0050] 3. By combining and selecting high-dimensional features in different ways, the model can focus on the most important load features, reduce redundant features, and improve training and inference efficiency.

[0051] The cluster load prediction phase specifically includes the following steps:

[0052] D1. Normalize the input load data X: X = Normalization(X);

[0053] D2. Using a feedforward neural network to extract features through a block structure, and employing STL decomposition techniques to decompose the time series into a linear combination of basis functions, including three parts: trend, seasonality, and residuals.

[0054] trend,seasonal,residual=STL Decomposition(X);

[0055] D3. Model the above components using specific activation functions and models:

[0056] For the prediction of the trend section pred Represented as:

[0057] trend pred =W t2 (ELU(W t1 x+b t1 ))+b t2 ,

[0058] Where ELU is the exponential linear unit, W t1 ,b t1 For the weights and biases of ELU, W t2 ,b t2 Weights and biases for trend prediction.

[0059] For seasonal forecasts pred Represented as:

[0060] seasonal pred =W s2 (GLU(W s1 x+b s1 ))+b s2 ,

[0061] GLU stands for gated linear unit, W s1 ,b s1 W represents the weights and biases of the GLU. s2 ,b s2 Weights and biases for seasonal forecasts.

[0062] For the prediction of the residual part pred Represented as:

[0063] residual pred =W r2 (ELU(W r1 x+b r1 ))+b r2 ,

[0064] GLU(x)=(σ(W g1 x+bg1 ))⊙(W g2 x+b g2 ),

[0065] Where σ is the Sigmoid activation function, ⊙ represents the Hadamard product, and W r1 ,b r1 W represents the weights and biases of the sigmoid activation function. r2 ,b r2 The weights and biases for the residual prediction.

[0066] D4. Predicting Time Series Represented as the sum of the three:

[0067]

[0068] Decomposing time series forecasts into trend and seasonal residuals, processing them separately, and then integrating them has several significant advantages:

[0069] 1. Improve model interpretability: The trend component reflects long-term changes in the data, the seasonal component reveals periodic fluctuations, and the residual component represents random noise. Through this decomposition, the model can more clearly represent the dynamic characteristics of the time series.

[0070] 2. Enhance the predictive power of the model: The trend component typically represents changes over a longer time scale, while seasonality reflects short-term cyclical fluctuations. Modeling and optimizing these two components separately can enable the model to capture these characteristics more accurately.

[0071] 3. Improve the robustness of the model: After decomposing the data, each component uses different modeling techniques, so that each part can be optimized according to its characteristics, thereby improving the robustness and accuracy of the overall model.

[0072] Beneficial Effects: The random workflow load prediction method for data missing issues provided by this invention has the following beneficial effects compared with the prior art:

[0073] (1) This invention uses a time series prediction model to predict the workflow itself and the cluster load in the future time period. It not only considers the task uncertainty in the cluster workflow scheduling process, but also reasonably refers to historical load information to predict the load. The prediction results can serve as an important basis for determining task priority and scheduling strategy.

[0074] (2) This invention combines a kernel density missing data imputation method based on common features with Kalman filtering technology to provide a powerful tool for estimating and predicting missing values ​​in time series, providing accurate assistance for workflow load prediction.

[0075] (3) The N-BEATS model used in this invention can adaptively capture various fluctuations and periodicities in the cluster time series, and adopts a highly parallel design, which can improve computational efficiency. Attached Figure Description

[0076] Figure 1 This is a flowchart of the method of the present invention.

[0077] Figure 2 This is a schematic diagram of the overall process of the present invention.

[0078] Figure 3 This is a schematic diagram of the missing value imputation network in the workflow of the method of the present invention.

[0079] Figure 4 This is a schematic diagram of the cluster load prediction stage of the method of the present invention.

[0080] Figure 5 This is a diagram illustrating the dataset used in the load forecasting problem. Detailed Implementation

[0081] The present invention will be further illustrated below with reference to the accompanying drawings and specific implementation examples. It should be understood that these examples are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0082] Example:

[0083] like Figure 1 The diagram shows a flowchart of a random workflow load prediction method for data-missing scenarios. Figure 2 This is a schematic diagram of the overall process of the present invention. When performing cluster load prediction, the N-BEATS model is used to predict the load for future time periods, thereby enhancing the accuracy and efficiency of the prediction.

[0084] A. Workflow preprocessing stage:

[0085] Information in the workflow is converted into embedded vectors W, and then, based on the workflow's requirements, the request logs in the cloud center are divided into historical data and predicted data. Historical data T from random workflows within the computing power network is used as the basis. W A neural network model is constructed to record the information of neurons at the current moment and learn temporal dependencies. The temporal features of each request are extracted, and the one-dimensional temporal feature t of each request sequence is calculated. W Based on similarity d W =d(t) W ,T W Clustering is performed to obtain tasks Y=G that have high similarity in the temporal feature space. p (t W ,TW ).

[0086] B. Workflow missing value completion and load forecasting phase:

[0087] B1. Figure 3 The diagram shows a workflow missing value imputation network. Three convolutional neural network models are constructed to impute the head node, tail node, and dependencies, respectively. The structure of a convolutional neural network is as follows: B1.1 takes the two input embedding vectors Y... i and Y j The two embedding vectors are linked into a two-dimensional or three-dimensional array, and a fusion algorithm is used to fully integrate them so that they can interact with each other under the action of the convolutional kernel; B1.2 outputs the edge embedding vector W after passing through a fully connected layer. E Or node embedding vector W G The vector is transformed into a workflow task node or inter-task dependency to complete the missing data information of the workflow; B1.3 When training the model, a workflow is processed by three convolutional neural network models simultaneously, and the three models use the same loss function and are trained synchronously; B2. For each class of requests obtained by clustering, a kernel density model K is constructed, and the original calculation sequence of the workflow in each class is used as a data record in the kernel density prediction model of that class; B3. By using techniques such as linear filtering, exponential weighted filtering, and Kalman filtering, the weights are automatically updated at each time point through the covariance matrix C. To estimate the time-varying density, Kalman filtering and kernel density estimation are combined, and the weights of the dynamic kernel density estimation are updated using the Kalman filtering method:

[0088] B4. Maximum likelihood estimation is applied as follows: B4.1 Obtain estimates of kernel bandwidth MaxL(h) and other control parameters based on the maximum likelihood function; B4.2 Calculate estimates of probability density function, cumulative distribution function, and quantiles using these estimates; B5 Predict stochastic workload E(w) based on methods such as calculating the expectation of the probability density function. t ).

[0089] C. Cluster modeling and feature engineering stage:

[0090] Assume the cluster load at time t, i.e., the proportion of busy GPUs to the total number of GPUs, is x. t The value range is [0,1]. Let x... t Defined as a sequence that changes over time, where t represents a timestamp: X = {x1, x2, x3, ..., x...} t At the same time, relevant time features, such as month, day, hour, minute, and weekday, are extracted from the timestamp t, and represented as: T t={month(t),day(t),hour(t),minute(t),weekday(t)}. In obtaining x... t Next, a sliding window technique is needed to generate hysteresis features. For a window size of k, the hysteresis feature is represented as: L t ={x t-1 ,x t-2 ,...,x t-k Consider the average value. Medianm t =median(x t-1 ,x t-2 ,...,x t-k ) and standard deviation Reduce the original cluster load x t With the extracted time features T t Lag characteristic L t and statistical characteristics {μ t ,m t ,σ t} combined, forming a high-dimensional feature vector X t ={x t ,T t ,L t ,μ t ,m t ,σ t Then the high-dimensional feature vector X at all times can be represented as X = {X1, X2, ..., X}. t}

[0091] D. Cluster load prediction phase:

[0092] Figure 4 The diagram illustrates the cluster load forecasting stage, with the following steps: D1. Normalize the input load data X: X = Normalization(X); D2. Extract features using a feedforward neural network through a block structure, and decompose the time series into a linear combination of basis functions using STL decomposition technology, including three parts: trend, seasonal, and residual: trend, seasonal, residual = STL Decomposition(X); D3. Model the above components using specific activation functions and models: trend pred =W t2 (ELU(W t1 x+b t1 ))+b t2 ,seasonl pred =W s2 (GLU(W s1 x+b s1 ))+b s2,residual pred =W r2 (ELU(W r1 x+b r1 ))+b r2 GLU(x)=(σ(W) g1 x+b g1 ))⊙(W g2 x+b g2 GLU stands for Gated Linear Unit, ELU for Exponential Linear Unit, σ for Sigmoid activation function, and ⊙ for Hadamard product; D4. Predicting Time Series Represented as the sum of the three:

[0093] Regarding the data in this example, as follows: Figure 5 As shown, this study focuses on the dynamic scheduling problem of deep learning tasks in large enterprises or research institutions, using the workflow data and cluster tracking data they generate.

[0094] It should be noted that the above embodiments are not intended to limit the scope of protection of the present invention. Equivalent transformations or substitutions made based on the above technical solutions all fall within the scope of protection of the claims of the present invention.

Claims

1. A method for predicting workload in stochastic workflows with missing data, characterized in that: Includes the following stages: A. Workflow Preprocessing Stage: The workflow is transformed into embedded vectors. Based on historical random workflow data from the computing power network, time-series data information is memorized and learned. The time-series features of the requests are extracted, and spectral clustering is performed based on common features using three convolutional neural network models responsible for the head node, tail node, and relationship, respectively. B. Workflow Missing Value Completion and Load Prediction Phase: For each request class obtained from clustering, a kernel density model is constructed. Combined with Kalman filtering, the weights of the kernel density estimate are dynamically updated. Simultaneously, kernel bandwidth and other dynamic control parameters are calculated using maximum likelihood estimation. Finally, the random workflow load is predicted based on the expectation method of the probability density function. C. Cluster Modeling and Feature Engineering Phase: This phase involves configuring the cluster load... Transform into a high-dimensional feature vector containing multiple time series features In order to make more accurate predictions and analyses, D. Cluster Load Forecasting Stage: The N-BEATS model is used to forecast the cluster load. The N-BEATS model normalizes the input load data X and then uses STL decomposition technology to decompose the time series into a linear combination of basis functions, including trend, seasonality and residual components. Each basis function is learned and represented by a neural network block. The forecast result can be formed by modeling the above components respectively.

2. The method for predicting load in a random workflow with missing data according to claim 1, characterized in that: The specific steps of the workflow preprocessing stage are as follows: A1. Workflow information is converted into embedded vector form; A2. Based on historical data of random workflow in a computing power network, construct a neural network model to memorize the information of neurons at the current moment and learn the dependencies of time-series data; A3. Extract the timing characteristics of each request arrival; A4. Workflow requirement similarity measurement based on information entropy, using spectral clustering based on one-dimensional temporal features.

3. The method for predicting load in random workflows with missing data according to claim 2, characterized in that: The specific steps of the workflow missing value completion and load prediction phase are as follows: B1. Construct three convolutional neural network models, each responsible for completing the head node, tail node, and relationships, respectively; B2. Construct a kernel density model for each request class obtained by clustering, and use the original calculation sequence of the workflow in each class as a data record in the kernel density model of that class; B3. By using linear filtering, exponential weighted filtering, and Kalman filtering techniques, the weights are automatically updated at each time point through the covariance matrix; B4. Calculate kernel bandwidth and other dynamic control parameters using maximum likelihood estimation; B5. Predicting random workflow load based on the expectation method of calculating probability density function.

4. The method for predicting load in random workflows with missing data according to claim 3, characterized in that: The specific steps of the cluster modeling and feature engineering phase are as follows: C1. Define a time series. C2. Extract time features. C3. Create hysteresis features. C4. Calculate statistical characteristics. C5. Construct high-dimensional features.

5. The method for predicting load in random workflows with missing data according to claim 4, characterized in that: The specific steps of the cluster load prediction phase are as follows: D1. Normalize the input load data X; D2. Use a feedforward neural network to extract features through a block structure, and use STL decomposition technology to decompose the time series into a linear combination of basis functions, including three parts: trend, seasonality, and residual. D3. Model the above components using specific activation functions and models; D4. Predicting Time Series It is represented as the sum of the three.

Images