A task failure prediction method based on Transformer in cloud data center environment

Abstract

The present invention discloses a method for predicting task failures in a cloud data center environment based on Transformer. The method first organizes information such as CPU usage and memory usage during task execution into a time series, and models the task failure prediction problem into a binary classification problem based on the time series. Then, a one-dimensional convolutional neural network is combined with the Transformer to extract the local dependencies of the time series through the one-dimensional convolutional neural network, and through the multi-head attention mechanism of the Transformer core, weights are adaptively assigned to each time point to extract global dependencies in different subspaces. The present invention combines the 1DCNN and Transformer models to complementarily extract the features of the time series, avoiding the shortcoming of traditional time series neural networks that are prone to information forgetting, improving the quality of feature extraction, and more accurately predicting whether a task will fail.

Description

Technical Field

[0001] The present invention belongs to the field of cloud computing and mainly relates to a task failure prediction method in a cloud data center environment based on Transformer. Background Art

[0002] In the cloud computing model, users can connect to cloud data centers over the internet and utilize the computing resources and services provided by cloud data centers to complete various computing tasks. However, the task execution process in cloud data centers is relatively complex. Specifically, as cloud data centers expand in scale and the number of devices increases, the interdependencies and dependencies between various devices and systems also increase. This interaction and collaboration fosters closer connections between devices, but it also means that a failure in any one device can significantly impact the normal operation of other devices, leading to the risk of failure in tasks executed in the cloud data center. Furthermore, cloud service providers typically assign customer tasks to different virtual machines or containers for execution. The allocation and scheduling of these virtual machines or containers is dynamically adjusted based on load and resource utilization. This dynamism leads to task instability, increasing the risk of task failure. Fault prediction can promptly identify potential failures during task execution and take appropriate measures to mitigate their impact, thereby preventing any impact on service quality. In short, the complexity and dynamic nature of cloud environments make tasks executed in cloud data centers prone to failure. Accurate fault prediction can help data center managers quickly respond and take necessary measures to ensure smooth task operation.

[0003] Traditional fault prediction methods are primarily based on statistical analysis of historical data and empirical judgment. These methods cannot fully utilize the vast amount of real-time data in cloud data centers, nor can they adapt to the increasingly complex and changing conditions in cloud data center environments. Consequently, in recent years, a growing number of researchers have begun exploring fault prediction methods based on artificial intelligence technologies such as machine learning and deep learning. These technologies hold great potential for fault prediction. They can leverage the vast amount of historical data in cloud data centers to rapidly detect faults and anomalies, thereby enabling fault prediction and prevention. Furthermore, these technologies can automatically learn and optimize prediction models, continuously improving the accuracy and reliability of predictions.

[0004] Traditional machine learning typically relies on manually designed features for learning and prediction. These features often require preprocessing, such as feature selection and data cleaning, to remove noise and redundant information from the data and extract useful features. This is because traditional machine learning models have high data requirements, requiring meaningful and interpretable features for learning and prediction. Deep learning models, on the other hand, can automatically learn and extract feature representations from raw data, reducing reliance on feature engineering. Furthermore, traditional machine learning models struggle to process complex time series. While many deep learning models, such as RNNs and LSTMs, have been widely used for time series modeling and prediction, these models, based on traditional recurrent neural networks, rely to a certain extent on the features output at the previous time point. This makes information forgetting very common when processing long sequences. Therefore, how to effectively utilize deep learning models to solve data center failure prediction problems remains a highly research-oriented issue. Summary of the Invention

[0005] In order to solve the above problems, the present invention proposes a task failure prediction method in a cloud data center environment based on Transformer.

[0006] The present invention comprises the following steps:

[0007] S1. Model task failure prediction in a cloud data center environment as a binary classification problem based on time series:

[0008] Collect task resource usage information within a certain period of time and form a time series, and predict whether the task will fail based on the time series.

[0009] A job consists of one or more tasks, each of which is accompanied by a set of resource requirements. For job-level fault prediction, a job is predicted to be fault-free only when all tasks in the job are predicted to be fault-free.

[0010] S2. Identify characteristics associated with task failures, including resource usage, task scheduling priority, number of task resubmissions, and scheduling delay.

[0011] S3. Combine a one-dimensional convolutional neural network and a Transformer model to extract features from the input time series and classify them based on the extracted features to predict whether tasks and jobs in the cloud data center will fail. The specific process is as follows:

[0012] S31. Input layer:

[0013] At each time point, observe the task's CPU usage, memory usage, cache usage, disk I / O time, average disk usage, task scheduling priority, number of task resubmissions, and scheduling delay; set these features into a vector as the observation results of the task at each time point. These vectors will form a time series as the input of the prediction model.

[0014] S32. One-dimensional convolution operation layer:

[0015] After obtaining the input time series of the task, it is passed through a one-dimensional convolution operation layer to extract the local features of the time series.

[0016] S33.Transformer layer:

[0017] After obtaining the results of the one-dimensional convolution operation layer, the global features of the time series are extracted through the Transformer layer.

[0018] S34. Mean pooling and fully connected layers:

[0019] After obtaining the output of the Transformer layer, it passes through a layer of mean pooling and a fully connected layer to learn the interaction between different features, and then passes through an activation function to output the classification probability.

[0020] The output is compared with the threshold to predict whether the corresponding task will fail. A job consists of one or more tasks, and whether it fails is determined by the failure prediction of all tasks in the job.

[0021] Beneficial effects of the present invention: By using the fault prediction method proposed in the present invention, it is possible to accurately predict whether a task failure will occur in a cloud data center environment, thereby facilitating cloud data center managers to take corresponding measures in a timely manner to avoid serious consequences.

[0022] The 1DCNN-Transformer algorithm proposed in this paper combines the 1DCNN and Transformer models to process the information of local features and global dependencies respectively, making full use of the respective advantages of the two models, and can better model time series data, thereby improving the accuracy of fault prediction. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 This is a framework diagram of the Transformer-based task fault prediction method proposed in this invention;

[0024] Figure 2 This is a diagram illustrating the input processing of the 1DCNN-Transformer algorithm proposed in this invention;

[0025] Figure 3This is a diagram showing the classification effects of the present invention and other methods on samples at different thresholds. DETAILED DESCRIPTION

[0026] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings.

[0027] like Figure 1 As shown, the present invention includes the following steps:

[0028] S1. Model task failure prediction in a cloud data center environment as a binary classification problem based on time series:

[0029] In order to more accurately predict task failures, it is considered to collect information such as resource usage in the past and form a time series, and analyze and predict task failures based on the time series. Therefore, this embodiment models task failure prediction in a cloud data center environment as a binary classification problem based on time series. A job in a cloud data center environment consists of one or more tasks, each of which is accompanied by a set of resource requirements. A job consisting of I tasks Y = {X1,…,X i ,…,X I The input of} can be specifically expressed by formula (1):

[0030]

[0031]

[0032] where X i represents the time series of observation results of the i-th task in job Y at T time points, where T represents the size of the time window, and x it Represents task X i The observation result at the t-th time point is specifically expressed by formula (2), where A represents the feature dimension. The output of this embodiment is the fault prediction result for each task and each job. For job Y, its output can be expressed by formula (3):

[0033]

[0034]

[0035] in Represents task X i The prediction result of , the result of 0 means that the task is predicted to have no failure, and the result of 1 means that the task is predicted to have failure. represents the fault prediction result of job Y, and the sum of all task prediction results of job Y is the fault prediction result of job I. When , job Y will fail. This means that as long as any task in a job is predicted to fail, the job is predicted to fail. Correspondingly, the job is predicted to run smoothly without failure only when every task in the job is predicted to not fail.

[0036] The main purpose of this embodiment is to build and evaluate a fault prediction model to accurately predict whether jobs and tasks will fail during execution in a cloud data center environment.

[0037] S2. Feature analysis and selection:

[0038] Understanding the resource usage and characteristics of tasks during execution is key to developing an accurate task failure prediction method. This example conducted an in-depth study of the cluster workload analysis dataset provided by Google and, combined with previous research, identified the following characteristics associated with task failures:

[0039] Resource Usage: The study found significant differences in resource usage between failed and successfully completed tasks, including CPU usage, memory usage, cache usage, disk I / O time, and average disk usage. For example, the average CPU usage and memory consumption of failed tasks were at least twice that of successfully completed tasks, demonstrating a strong correlation between task resource usage and failure.

[0040] Task Scheduling Priority: Each task has a scheduling priority. Tasks with higher priority numbers generally receive resources before tasks with lower priority numbers. Studies have shown that tasks with the highest and lowest scheduling priorities have higher failure rates than tasks with medium scheduling priorities. When a task priority is higher than 3, the likelihood of a task failing increases significantly. Therefore, there is a positive correlation between the occurrence of task failures and task priority.

[0041] Task resubmissions: Task resubmissions refer to the resubmission of tasks to the cluster after they have been evicted, aborted, or failed. The number of task resubmissions for failed jobs is higher than that for successfully completed jobs. For failed and successfully completed jobs, the percentage of tasks that were executed multiple times was 35.8% and 0.9%, respectively.

[0042] Scheduling latency: Scheduling latency refers to the waiting time between a task being submitted to the cloud data center and being scheduled for execution on a virtual machine. Failed tasks have higher scheduling latency than successfully completed tasks.

[0043] S3. Transformer-based task fault prediction method: The Transformer-based task fault prediction algorithm 1DCNN-Transformer combines a one-dimensional convolutional neural network and a Transformer model to extract features from the input time series, and classifies them according to the extracted features to predict whether tasks and jobs in the cloud data center will fail. In this embodiment, the batch number of the 1DCNN-Transformer algorithm is set to 50, the total number of training epochs is set to 1000, and the learning rate is set to 0.001.

[0044] The specific process is as follows:

[0045] S31. Input layer:

[0046] This embodiment observes eight features of the task at each time point, including CPU usage, memory usage, cache usage, disk I / O time, average disk usage, task scheduling priority, number of task resubmissions, and scheduling delay. This embodiment sets these features as a vector as the observation results of the task at each time point. These vectors will form a time series as the input of the input layer. For each task, this embodiment uses the observation results of its most recent w time points as a time series as the input of the model for prediction. For the part of the task that runs for less than w time points, the insufficient part is padded with zeros. In this way, a time series with a certain time window size can be used to train the model and predict whether a task or job will fail.

[0047] like Figure 2 As shown, assuming that the time window size w is set to 6, then task X i The total length of the time series is greater than the size of the time window w, so the vector of the last six time points is taken. j For example, since the total length of the time series is insufficient, an automatic padding operation is required. In a preferred embodiment, an all-zero vector is added at the end of the sequence until the sequence length reaches the time window size w, as shown in the dark red part in the figure.

[0048] Then, the padded sequence is normalized and used as the input of the 1DCNN-Transformer model.

[0049] S32. One-dimensional convolution operation layer:

[0050] The main function of the one-dimensional convolution operation layer is to extract the local dependency of the input time series. The one-dimensional convolution operation layer will undergo one-dimensional convolution, pooling and other operations. The calculation formula of one-dimensional convolution is shown in formula (5):

[0051]

[0052] where f 1,i,k,m For task X i For example, the input data is the output of the mth position after the kth convolution kernel operation, the number of input channels represented by A is the dimension of the input data, and b k,a It represents the bias value of channel a of the kth convolution kernel, S represents the convolution kernel size of one-dimensional convolution, ω k,a,s represents the data at position s on channel a of the kth convolution kernel. The second summation symbol indicates traversing the convolution kernel and multiplying and summing the corresponding input data. The output of the one-dimensional convolution layer is obtained by calculating each output position using Equation (5).

[0053] After the one-dimensional convolution layer, the activation function layer is required. Its purpose is to introduce nonlinear transformations to allow the neural network to learn more complex representations. In a preferred example, the ReLU activation function can be selected. It has a fast calculation speed and a low risk of gradient vanishing. It can be calculated by formula (6):

[0054]

[0055] Next, the feature map is compressed through the maximum pooling layer, which can reduce the parameters that need to be trained and filter out some redundant information, thereby improving the robustness of the model and enhancing local features, thereby improving the expressiveness of the model. It can be calculated by formula (7):

[0056]

[0057] Where S′ represents the size of the pooling window. In this embodiment, three layers of one-dimensional convolution operation are stacked to better capture the complex patterns in the input data.

[0058] S33.Transformer layer:

[0059] Since the Transformer model cannot capture the order of the input data, it is necessary to use position encoding to attach it to the input of the Transformer encoder so that the model can learn the position information in the input sequence. It can be calculated by formula (8):

[0060]

[0061] Use C i Represents the output of the one-dimensional convolution operation layer, E represents the position embedding, and D i represents the input of the Transformer encoder layer. Then in formula (8) For task X i For example, the t-th output of the one-dimensional convolution operation layer, et represents the position embedding information at the t-th moment, Represents the t-th vector input to the Transformer encoder.

[0062] Since the features of different time points in a task contribute differently to the final prediction results, it is necessary to explore the weights of different time points and assign higher weights to time points with high correlation. For example, for some tasks, the features at a certain time point may have a greater impact on the prediction results than at other time points. This may be because the features at that time point are strongly correlated with certain key features of the task. To discover this correlation, the multi-head self-attention mechanism in the Transformer can be used. Through this mechanism, the model can mine the weights of different time points in the time series from different subspaces to more accurately predict whether the task will fail. In addition, residual connections can alleviate the occurrence of problems such as gradient vanishing or exploding to a certain extent, helping the neural network learn better feature representations. The output of multi-head self-attention and residual can be calculated by formula (9-12):

[0063] Z i =D i +MultiHead(D i )#(9)

[0064] MultiHead(X)=Concat(head1,head2,...,head h )·W0#(10)

[0065]

[0066]

[0067] Where W0 is the matrix to be trained, Concat represents the connection operation, head i is the single-head attention in the i-th subspace, Wi i Q , is the transformation matrix of the i-th attention head, d k Represents the dimension of the vector in matrix k, and Softmax represents column-wise normalization.

[0068] Layer normalization can effectively improve the training speed and performance of neural networks and improve generalization ability. The result obtained by residual connection is normalized by layer, which is expressed by formula (13-15):

[0069]

[0070]

[0071]

[0072] where μ ip For task X i For example, the mean of the p-th row of input data in the layer normalization layer after multi-head attention and residual, Represents the variance of the pth row of input data, m represents the feature dimension of the input data. In order to avoid the denominator being zero, a small constant ε is usually added to the denominator, while α and β are parameters that need to be trained.

[0073] Next, the features of each time point need to be further extracted through the feedforward neural network layer. First, the input dimension is increased through a fully connected layer to obtain a richer information representation. The activation function is ReLU. Finally, another fully connected layer is used to reduce the dimension to obtain a vector of the same length as the input vector. Specifically, it can be calculated by formula (16):

[0074] H i =FFN(Z i )=max(0,Z i W1+b1)W2+b2#(16)

[0075] The max function represents the ReLU activation function operation, and W1, b1, W2, and b2 are all matrices to be trained. After passing through the feedforward neural network layer, it also needs to go through residual connection and layer normalization, that is, the output of the Transformer encoder layer is obtained, as shown in Equations (17) and (18):

[0076] H i =H i +Z i #(17)

[0077] H i =LayerNorm(H i )#(18)

[0078] This embodiment stacks two Transformer encoder layers to better explore the complex global dependencies in the input data.

[0079] S34. Mean pooling and fully connected layers:

[0080] After obtaining the output of the Transformer encoder, it needs to go through a layer of mean pooling, that is, taking the mean of the feature representation output at each time point, which can be specifically expressed by formula (19):

[0081]

[0082] Where averagePool represents the average pooling function, k represents the sequence length output by the Transformer encoder, and h ij For task X i For example, the Transformer encoder outputs the j-th feature vector in the sequence.

[0083] CNN often needs to go through a fully connected layer for classification after the convolution layer and pooling layer. Similarly, the model proposed in this invention needs to perform classification on the final output feature vector Y after 1DCNN, Transformer and mean pooling. i The interaction between different features is learned through the fully connected layer, and then the activation function is used to output the classification probability, which can be specifically expressed by formula (20):

[0084] y i =sigmoid(MLP(Y i ))=sigmoid(Y i W+b)#(20)

[0085] Where MLP represents the fully connected layer operation, and W and b are matrices to be trained.

[0086] Compare the output with the threshold δ to predict whether the corresponding task will fail, which can be specifically expressed by formula (21):

[0087]

[0088] Whether a job fails is determined by the failure prediction of all tasks in the job, which is expressed by formula (4).

[0089] This paper combines a one-dimensional convolutional neural network and a Transformer model to complementarily extract the potential dependencies of time series for task failure prediction in cloud data centers. Compared with the algorithms proposed in previous related work, it has better classification performance. Figure 3 The following table shows the ROC curves of different methods at the task level and the operation level, respectively, reflecting the classification effect of each method on samples at different thresholds. Figure 3 It can be found that the curve of the 1DCNN-Transformer method is generally above the curves of other models, and obviously has a larger area under the curve. Therefore, the 1DCNN-Transformer method proposed in this invention shows excellent performance in a binary classification problem such as fault detection.

[0090] Specifically, the results, from best to worst, are 1DCNN-Transformer > BiLSTM > LSTM > RNN > SVM. SVM has the worst performance because it can only process relatively simple data. Its performance is relatively poor in scenarios with large data volumes, such as the Google cluster dataset. RNNs can exploit correlations between time points in time series data, but they tend to forget distant time points. LSTM, by introducing gating units, addresses some of the shortcomings of RNNs, resulting in better performance than RNNs. BiLSTM introduces two layers of LSTM, partially compensating for the LSTM's drawback of different weights for near and far distances. This invention fundamentally addresses the problem of traditional recurrent neural networks, where time point weights are affected by distance and prone to information forgetting. The Transformer's core self-attention mechanism adaptively assigns each time point a corresponding weight based on its own characteristics. Simultaneously, 1DCNN extracts local features, while Transformer extracts global features, fully leveraging their respective strengths to effectively exploit dependencies between time points in a time series, thereby improving model classification performance.

[0091] In summary, the present invention uses a one-dimensional convolutional neural network to mine the local features of the time series, and uses Transformer to mine the global dependencies of the time series, combining the advantages of each to accurately predict whether the task will fail. The attention mechanism at the core of Transformer can adaptively assign weights to each time point based on the characteristics of the time point itself, making up for the problem that long-term dependent information in traditional recurrent neural networks needs to be transmitted layer by layer, which leads to the problem of not paying enough attention to distant time points. At the same time, the global dependencies in different subspaces are extracted through the multi-head attention mechanism to improve the expressive power of the model.

Claims

1. A task failure prediction method based on Transformer in a cloud data center environment, characterized by The method comprises the following steps: S1. Model task failure prediction in a cloud data center environment as a binary classification problem based on time series: Collect task resource usage information over a certain period of time and form a time series. Based on this time series, predict whether a task will fail. A job consists of one or more tasks, each of which is accompanied by a set of resource requirements. For job-level fault prediction, a job is predicted to be fault-free only when all tasks in the job are predicted to be fault-free. S2. Identify characteristics associated with task failures, including resource usage, task scheduling priority, number of task resubmissions, and scheduling delays; S3. Combine a one-dimensional convolutional neural network and a Transformer model to extract features from the input time series. Classify the extracted features to predict whether cloud data center tasks and jobs will fail. The specific process is as follows: S31. Input layer: At each time point, observe the task's CPU usage, memory usage, cache usage, disk I / O time, average disk usage, task scheduling priority, number of task resubmissions, and scheduling delay; set these features as a vector as the observation results of the task at each time point; these vectors will form a time series as the input of the prediction model; S32. One-dimensional convolution operation layer: After obtaining the input time series of the task, it is passed through a one-dimensional convolution operation layer to extract the local features of the time series; the one-dimensional convolution in the one-dimensional convolution operation layer is calculated as follows: where f 1,i,k,m For task X i For example, the input data is the output of the mth position after the kth convolution kernel operation, A represents the number of input channels, that is, the dimension of the input data, b k,a Represents the bias value of channel a of the kth convolution kernel, S represents the convolution kernel size of one-dimensional convolution, ω k,a,s Represents the data at position s on channel a of the k-th convolution kernel; The output of the one-dimensional convolutional layer can be obtained by performing the above calculations on each output position; After the one-dimensional convolution layer, it passes through the activation function layer, introducing nonlinear transformations to allow the neural network to learn more complex representations; Compress the feature map through the maximum pooling layer; S33.Transformer layer: After obtaining the results of the one-dimensional convolution operation layer, the global features of the time series are extracted through the Transformer layer; S34. Mean pooling and fully connected layers: After obtaining the output of the Transformer layer, it passes through a layer of mean pooling and a fully connected layer to learn the interaction between different features, and then passes through an activation function to output the classification probability; The output is compared with the threshold to predict whether the corresponding task will fail. A job consists of one or more tasks, and whether it fails is determined by the failure prediction of all tasks in the job.

2. The method for predicting task failures in a cloud data center environment according to claim 1, characterized in that: The time window size corresponding to the time series described in S31 is w; for the part of the task that runs for less than w time points, the insufficient part is filled with zeros.

3. The method for predicting task failures in a cloud data center environment according to claim 1, wherein: The convolution kernel size of each channel is 3, the number of convolution kernels is 16, and the stride is 1.

4. The method for predicting task failures in a cloud data center environment according to claim 1, wherein: Transformer layer described in S33: First, encode the position: in Represents task X i After the t-th output of the one-dimensional convolution operation layer, e t represents the position embedding information at the t-th moment, represents the t-th vector of the input encoder; Secondly, the multi-head self-attention mechanism in Transformer is used to mine the weights of different time points in the time series from different subspaces; the results obtained by residual connection are normalized by layer; Then, the features of each time point are further extracted through the feedforward neural network layer; Finally, after passing through the feedforward neural network layer, it goes through residual connection and layer normalization again to obtain the output of the Transformer encoder layer.

Images