A clustering method for data center batch jobs

Abstract

The application discloses a clustering method for data center batch processing jobs, which is divided into five steps: initialization, batch processing job original resource use feature modeling, batch processing job hidden layer feature extraction model construction, batch processing job clustering method construction, and batch processing job classification. The application extracts the dependency structure of jobs and the resource use feature attribute set of tasks in the data center log for batch processing jobs. Based on the extracted features and dependency relationships, a batch processing job clustering method is constructed by using a graph convolutional neural network-based autoencoder. Through the clustering method provided by the application, batch processing jobs can be classified more accurately, thereby improving the accuracy of capacity planning and load generation results.

Description

Technical Field

[0001] The present invention belongs to the field of big data computing, and in particular relates to a clustering method for batch processing jobs in a data center cluster. Background Art

[0002] Data centers serve as the information infrastructure for the internet and related industries. Multiple applications are deployed within a data center, and workloads are becoming increasingly diverse. Batch jobs are a type of workload within data center clusters. They are not latency-sensitive, run offline, and are often used to operate on large, static datasets. Typically, a batch job consists of multiple tasks, each performing distinct functional logic and often subject to data dependencies. A task begins executing only after one or more dependent tasks have completed. Each task implements parallel computing by executing multiple instances. An instance is the smallest unit of resource consumption for a batch job.

[0003] To perform data center capacity planning and load generation, it's first necessary to model the resource usage characteristics of the load. The complexity and diversity of current loads make it difficult to extract an accurate load model. To reduce load complexity, clustering algorithms can be used to segment diverse loads into multiple similar subsets. Within each subset, a representative load is selected to represent all loads within the subset. Clustering reduces load modeling complexity and improves accuracy.

[0004] Existing data center batch job clustering has the following flaws: 1) It cannot handle the characteristics of inter-task dependencies. The dependencies between tasks in a job are usually represented using a directed acyclic graph (DAG). The resource usage characteristics of a batch job are affected by task dependencies. The same set of tasks under different dependencies often produces different resource usage fluctuations and concurrency, forming different resource usage patterns. Existing clustering methods for batch jobs can only classify jobs with different resource consumption patterns based on statistical characteristics of job resource usage, such as average and peak values. 2) Most clustering methods target the task level, and there is a lack of job-level clustering methods. Task-level clustering methods cannot perceive the dynamic resource requirements during the execution of batch jobs. The few methods that cluster job-level loads use task statistics, but lack fine-grained task information. The use of task statistics results in the loss of specific task information and cannot distinguish between the direct differences between some jobs.

[0005] The present invention aims to classify batch processing jobs in data centers with dependency structures based on their task dependency characteristics and resource usage characteristics. The resulting job classification results can be used for capacity planning, load generation, and so on. Summary of the Invention

[0006] To address these challenges, this paper proposes a data center batch job clustering method. This method extracts the dependency structure of jobs and the resource usage attribute sets of tasks from data center logs. Based on these extracted features and dependencies, a batch job clustering method is constructed using an autoencoder based on a graph convolutional neural network. The model constructed using this method can more accurately classify batch jobs, thereby improving the accuracy of capacity planning and load generation results.

[0007] The proposed data center batch job clustering method consists of five steps: initialization, modeling of batch job raw resource usage features, construction of batch job hidden layer feature extraction model, construction of batch job clustering method, and batch job classification. In this method, the following important parameters are: the number of columns c of the output matrix of the 1st to 3rd layer graph convolutional neural network (1) 、c (2) 、c (3) , the number of rows P of the output matrix of the pooling layer, the number of clusters k of K-means clustering. c (1) 、c (2) 、c (3) It can be obtained based on the commonly recommended parameters of graph convolution, P can be obtained based on the average number of tasks in each job in the log, and k can be determined based on the elbow rule. (1) Take 32, c (2) Take 32, c (3) Take 8, P takes 15, and k takes 8.

[0008] Before executing this method, read in the required log data.

[0009] (1) Initialization

[0010] Use the batch job information in the log to initialize the data, and let the full set of attributes contained in the log be A = {a1, a1...a F}, select attributes related to task resource usage characteristics, including attributes related to CPU, memory, execution time, and network bandwidth, and define the attribute subset as S is the number of attributes in the attribute subset T. The set of jobs included in the log is W = {w1, w2...w Z}, where Z is the total number of jobs in the log. The i-th job w in W i It consists of multiple tasks and can be expressed as w i ={wt i1 , wt i2 ...wt iH}, 1≤i≤Z, where H is the job w i The total number of tasks in , each task consists of multiple parallel instances with the same execution logic, and the jth task can be expressed as wtij = {wi ij1 , wi ij2 ,..., wi ijV ,}, 1≤i≤Z, 1≤J≤H, where V is the total number of instances in the task wtij. The start time of instance wi ijq may be defined as time_wi ijq , 1≤i≤Z, 1≤J≤H, 1≤q≤V. The start time of a task is the start time of the earliest instance that starts executing in that task, task wt ij may be represented as time ij , time ij = min(time_wi ij1 , time_wi ij2 ,..., time_wi ijV ).

[0011] (2) Modeling of the original resource usage features of batch jobs

[0012] 2.1) For any job w i , 1≤i≤Z, Z is the total number of jobs in the log, the original resource usage features of the batch job include the dependency structure and the task resource attribute features. First, the dependency structure of the job is defined by an adjacency matrix, which is in the following form

[0013]

[0014] The number of rows and columns of the matrix is H, and H means the number of tasks included in the job wi. Each row of the matrix represents a task, and each column represents a task. Any element m ua , 1≤u≤H, 1≤a≤H, of the matrix represents whether the task wt ia depends on the task wt iu in the batch job. If there is a dependency, m ua = 1; otherwise, m ua = 0. The definition of dependency is described in the background art, and whether there is a dependency between tasks can be determined according to whether wt ia can only start after the task wt iu ends.

[0015] 2.2) For any job w i , 1≤i≤Z, the resource usage features of each task are defined using a task feature matrix, which is in the following form:

[0016]

[0017] The number of rows in the matrix is ​​H and the number of columns is S. Where S is the total number of attributes related to the task resource usage characteristics selected from the log, and any element f in the matrix j，q , 1≤j≤H, 1≤q≤S, represents the job w i In the task wt ij In the attribute t q The attribute value on .

[0018] 2.3) For each job w i , according to its dependency structure, initialize the adjacency matrix M i , initialize the feature matrix F according to its task resource usage characteristics i .

[0019] (3) Construction of batch processing hidden layer feature extraction model

[0020] 3.1) An autoencoder neural network is used as the hidden layer feature extraction model for batch jobs. The autoencoder neural network consists of an encoder and a decoder. The encoder consists of three graph convolutional layers, a convolutional layer, a pooling layer, and a fully connected layer. The decoder consists of five fully connected layers, a depooling layer, a deconvolution layer, and three deconvolutional layers. The autoencoder neural network is trained by reducing the difference between the encoder input and the decoder output. After training, the encoder output is used to partially output the hidden layer feature vectors of the job.

[0021] The model inputs are an H×H adjacency matrix M and an H×S feature matrix F. The output is the hidden feature vector of the batched job, which captures the job's dependency structure and resource usage. A learning rate of 0.01 or 0.001 is generally recommended. The learning rate for the autoencoder neural network is set to 0.01, and the training batch size is the size of the entire log sample dataset.

[0022] The calculation method of the graph convolution layer in the encoder is shown in formula (1) to formula (3). First, calculate the augmented adjacency matrix of the task w The calculation formula is shown in formula (1), where M is the adjacency matrix of job w, and I is the identity matrix with the same number of rows as M. Then calculate the augmented diagonal matrix of job w The calculation formula is shown in formula (2), where Representation matrix and the value of row i and column j, Representation matrix The value of the i-th row and q-th column in the matrix H is The number of rows is also the total number of tasks in job w. Formula (3) is the iterative calculation formula of the graph convolution layer, where G (m) is the input matrix of the m-th layer graph convolution, G (0) is the feature matrix F of job w. θ(m) It is the parameter matrix of the graph convolution operation. The specific value needs to be obtained through training. (m) The mth layer has c (m) dimensional feature matrix G (m) Mapped to c (m+1) dimension, θ (m) The number of rows is c (m) , the number of columns is c (m+1) ξ(·) is the tanh activation function. After three graph convolutions, we get the matrix G (3) .

[0023] The convolution layer is followed by the graph convolution layer. Before the convolution starts, the start time of each task in the job is calculated based on the time. ij Re-convolutional layer output matrix G (3) Sort, matrix G (3) Each row represents a task, after reordering, the start time is time ij The rows corresponding to smaller tasks will be located at the top of the matrix, and the rows corresponding to tasks with later start times will be located at the bottom of the matrix. The number of layers l of each task in the DAG of the job is added to the right side of the corresponding row in the matrix. The number of layers l is the shortest path from the root task (not dependent on other tasks) to the task. At this time, the matrix G (3) The number of columns becomes c (3) +1. The convolution kernel size of the convolution layer is 2×(c (3) +1), the stride is 1, that is, the number of columns of the convolution kernel is equal to the number of columns of the input matrix. The pooling layer consists of adaptive average pooling and weighted adaptive average pooling. The parameter of adaptive average pooling is the size of the output matrix output_size, output_size = (P, 1). The parameter stride and kernel number kernel_size of the pooling are calculated by output_size and the size of the input matrix input_size. The calculation is shown in formula (4) and formula (5), where the floor() function is rounded down. The calculation formula of the pooling layer is shown in formula (6), P is the number of rows of the output matrix, H is the number of rows of the input matrix, and adaptiveAvgPool() is adaptive average pooling. The size of the output matrix of the pooling layer is P×2. The feature vector V of the output operation of the fully connected layer w , the number of neurons is 8.

[0024]

[0025]

[0026]

[0027] stride = floor(input_size ÷ output_size) # (4)

[0028] kernel_size = input_size - (output_size - 1) x stride # (5)

[0029]

[0030] The number of neurons in the full connection layer in the decoder is 2P. The anti-pooling layer calculation is shown in equation (7), and the avgUnpool() function is the anti-average pooling, with the same parameters as the adaptive pooling in the encoder. The parameters of the anti-convolution layer are the same as those of the convolution layer in the encoder. Then, according to the order in the encoder, each row in the output matrix of the convolution layer is restored to the original position. The calculation formula of the anti-graph convolution layer is the same as equation (1), the parameters of the first layer of anti-convolution layer are the same as those of the third layer of convolution layer, the parameters of the second layer of anti-convolution layer are the same as those of the second layer of convolution layer, and so on.

[0031]

[0032] The loss value during training is calculated according to the difference between the input matrix F and the decoder output , and the specific formula is shown in equation (8). Where F is the feature matrix of the encoder input, is the output matrix of the decoder, ||·||F F is the Frobenius norm, H is the number of tasks in the job w, N is the number of job samples with H tasks in the log, and Z is the total number of job samples in the log.

[0033]

[0034] 3.2) Train the auto-encoding neural network model AE using the batch job set W.

[0035] 3.2.1) Train the constructed auto-encoding neural network AE, and use all jobs w i in the job set W i as sample data, w i represents the i-th job in the job set. The adjacency matrix M i and the feature matrix F i are used as input values of the model, and the model parameters are updated by the forward propagation algorithm and the Adam optimizer according to the loss value, and the input is repeated until all jobs are trained

[0036] 3.2.2) Repeat the process of 3.2.1), and update the parameters of the model for multiple rounds until the difference in loss value between two iterations is less than 2%, and stop iteration. After the parameter update is completed, the job training is completed.

[0037] (4) Construction of batch job clustering model

[0038] 4.1) Select the hidden layer feature vector X of the batch job as the feature value required for clustering, use the K-means clustering algorithm for clustering, define the number of K-means clustering clusters k = 8, and the batch job clusters CL after clustering = {cl1, cl2, ..., cl k Initialize the center points of k randomly selected clusters, the center point set is CT = {c1, c2, ...c k}, where c i Represents the center point sample c of the i-th cluster i →cl i .

[0039] 4.2) Traverse the job set W, for any w i Calculate the distance between it and the k center points according to formula (9). i For homework i The eigenvector of X j is the feature vector of the center point of the jth cluster, ||·|| F is the Frobenius norm. With the task w i The cluster with the smallest distance between the centers is the task w i The belonging of is shown in formula (10), where the cluster() function returns the cluster to which it belongs, and w i Join the set cluster(w i ). Traverse the job set W, any job w in W i After the belonging cluster exists, recalculate the center point of each cluster and update the center point of each cluster according to formula (11), c i For cluster cl i The center point of cluster cl i The number of elements in X j The corresponding w i ∈cl i .

[0040]

[0041]

[0042]

[0043] 4.3) Repeat step 4.2) until the center point of the cluster no longer changes. This completes the clustering of the job set W and obtains the clustering model.

[0044] 4.4) Based on the clustering results, the batch job dataset is divided into clusters. The jobs in each cluster have similar DAG structures and resource consumption patterns.

[0045] (5) Batch job classification

[0046] 5.1) For any task w, input its adjacency matrix M and feature matrix F into the autoencoder neural network in step 3). Use the autoencoder neural network to extract hidden layer features. The encoder part of the autoencoder neural network outputs the feature vector X of the batch job.

[0047] 5.2) Repeat step 5.1) until feature extraction of all batch jobs is completed.

[0048] 5.3) For any batch job w, calculate the distance between the feature vector X of job w and each center point in the batch clustering model. The calculation formula is shown in formula (9). The cluster to which the center point with the smallest distance from job w belongs is the cluster to which job w belongs. This completes the classification of job w. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A clustering platform for batch job clustering methods in data centers.

[0050] Figure 2 This is a diagram of the architecture of the present invention.

[0051] Figure 3 Flowchart of the present invention.

[0052] Figure 4 Flowchart for modeling raw resource usage characteristics for batch jobs.

[0053] Figure 5 Flowchart for building a hidden layer feature extraction model for a batch job.

[0054] Figure 6 A flowchart constructed for the batch job clustering method. DETAILED DESCRIPTION

[0055] The present invention will be described below with reference to the accompanying drawings and specific embodiments.

[0056] The clustering method for batch processing jobs in data centers proposed in this invention is built on multiple connected servers and is implemented by writing corresponding functions. Figure 1It is a deployment diagram of the platform built by this method. The platform is composed of multiple computer servers (platform nodes), which are connected through a network to store data and execute tasks in a distributed manner. The platform nodes are divided into two categories: a management node and multiple computing nodes. The platform built by the method of the present invention contains three types of core software modules: a data management module, a data receiving module, and a data processing module. Among them, the data management module is responsible for allocating the required log data to the data receiving module and collecting the management data results, and is only deployed on the management node; the data receiving module is responsible for pulling the required log data and needs to be deployed on each computing node; the data processing module is responsible for running the corresponding algorithm and returning the results to the resource management module, which is deployed on the computing node. The above three types of software modules are all deployed and run when the platform is started.

[0057] Figure 2 The following is an architectural diagram of the method of the present invention. The present invention uses the non-intrusive log of the data center as input and first generates a corresponding adjacency matrix and feature matrix for each job. Based on the generated adjacency matrix and feature matrix set, an autoencoder neural network model is constructed for the batch job. By training the model, a batch job hidden layer feature extraction model is obtained. The adjacency matrix and feature matrix of any job sample are input into the feature extraction model to obtain the hidden layer feature vector of the job. A clustering model is constructed for the job set, and after training, the center points of different classifications are obtained. The hidden layer feature vector of any job is input into the clustering model to obtain the classification result of the job.

[0058] The following combination Figure 3 The overall process of the invention describes the specific implementation method of this method. In this implementation method, the basic parameters are set as follows: the number of columns c of the output matrix of the first layer of the graph convolutional neural network (1) =32, the number of columns of the output matrix of the second graph convolution layer c (2) =32, the number of columns of the output matrix of the third graph convolution layer c (3) =8, the number of rows in the output matrix of the pooling layer P = 15, and the number of clusters k in the K-means clustering is k = 8.

[0059] The specific implementation method can be divided into the following steps:

[0060] 1. Initialization

[0061] The log used in the present invention has a total of 49 log attributes, including the full set of attributes A = {a1, a1...a 49}, there are 23 attributes related to the task, from which the required attribute subset T = {t1, t2...t4} is manually selected, which are execution time duration, number of instances, CPU utilization cpu, and memory utilization mem. There are 4201014 jobs in the log, and the job set is W = {w1, w2...w 4201014}; Take job w1 as an example, job w1 consists of 12 tasks, w1 = {wt 1，1 , wt 1，2 ...wt 1，12}, the start time of each task is time ij , for example, task wt in job w1 1，1 The start time is 362010.

[0062] 2. Modeling the raw resource usage characteristics of batch jobs

[0063] 2.1) For each job w i According to the actual execution of the job and its tasks, the adjacency matrix and feature matrix are constructed. Taking job w5 as an example, there are 6 tasks to which w5 belongs, w5={wt 1，1 , wt 1，2 ,…,wt 1，6}, attribute subset T = {t1, t2...t4}, so according to the method of steps 2.1) and 2.2) in the invention summary, its adjacency matrix M5 and feature matrix F5 are defined as follows:

[0064]

[0065]

[0066] 2.2) For each job w i , based on the model constructed in 2.1) i and F i Initialization and data filling.

[0067] 2.2.1) Select a task w that does not generate an adjacency matrix and a feature matrix i Taking task w5 as an example, the adjacency matrix and feature matrix of task w5 are established by the method of step 2.3) in the invention content. 1，2 , wt 1，3 , wt 1，4 Depends on wt 1，1 , wt 1，5 Depends on wt 1，2 , wt 1，3 , wt 1，6 Depends on wt1，4 , wt 1，5 , so the adjacency matrix M5 is constructed as follows:

[0068]

[0069] Continue to build the feature matrix F5, filling the matrix according to the duration, instance, cpu, and mem eigenvalues ​​of each task. The feature matrix is ​​as follows:

[0070]

[0071] 2.2.2) Repeat steps 2.2) to 2.2.1) until the adjacency matrix and feature matrix are established for all jobs.

[0072] 3. Batch job feature extraction model construction

[0073] 3.1) The basic structure of the autoencoder neural network is constructed based on the basic parameters. The number of columns of the output matrix of the first layer of the graph convolutional neural network is c (1) =32, the number of columns of the output matrix of the second graph convolution layer c (2) =32, the number of columns of the output matrix of the third graph convolution layer c (3) =8, the number of rows of the output matrix of the pooling layer P = 15, the neural network learning rate is set to 0.01, and the training batch size is the total number of samples.

[0074] 3.2) Train the autoencoder neural network.

[0075] 3.2.1) Using the job set W as an example, the sample input is the adjacency matrix and feature matrix of each job in the job set. The adjacency matrix size varies for different samples. Training is performed using the method in step 3.2) of the Summary of the Invention until all jobs in the job set W are trained.

[0076] 3.3) Repeat the process in 3.2.1) for multiple rounds of parameter updates, stopping the iteration when the difference in loss between two rounds is less than 2%. After the parameter updates are complete, the training is complete, and the autoencoder neural network (AE) is constructed.

[0077] 4. Batch job clustering model construction

[0078] 4.1) Build a K-means clustering model and set the number of clusters to 8. Initialize the model and randomly select 8 samples from the job set W as the center points of each cluster.

[0079] 4.2) Train the batch job clustering model.

[0080] 4.2.1) Update the center point of each cluster in the model according to the method in step 4.2) in the summary of the invention.

[0081] 4.2.2) Repeat step 4.2.2) until the center point in the model no longer changes, completing the training of the clustering model.

[0082] 5. Batch job classification

[0083] 5.1) Take task w5 in task set W as an example. Its adjacency matrix M5 and feature matrix F5 are input into the feature extraction model. The encoder of the autoencoder neural network outputs the feature vector X5 of task w5, which is as follows:

[0084] X5={0.1, 1.2, -3.1,…, 2.1}

[0085] 5.2) Repeat step 5.1) until feature extraction of all batch jobs is completed.

[0086] 5.3) For any batch job w i Classify according to the method of step 5.3) in the invention content to obtain the job w i The type of the batch job is now classified.

[0087] The inventors conducted performance tests based on the proposed method for clustering batch jobs in data centers. The test results demonstrate that the method is applicable to Alibaba's massive data center logs. This method can accurately classify jobs in data center logs.

[0088] The performance test compares this method with existing clustering methods to demonstrate the advantages of the proposed method in terms of interference degree prediction accuracy. The comparison method is as follows:

[0089] (1) K-Means clustering method based on resource usage data

[0090] This method uses the resource usage of batch jobs as multiple feature dimensions of samples and uses the basic K-means algorithm as the clustering method.

[0091] (2) Clustering method for clustering and recombining single feature dimensions

[0092] This method uses the resource usage of batch jobs as features, clustering each feature dimension separately. The types are then combined to obtain the final classification of the samples.

[0093] The performance test was run on a computer with the following hardware configuration: Intel E5-2660@2.20GHz CPU and 64GB DDR4 RAM.

[0094] The Silhouette Coefficient and the Calinski-Harabaz Index are often used to evaluate the classification effect of the clustering model. The calculation method of the Silhouette Coefficient is shown in formula (12), wherein a is the average distance from the sample to other samples in the same class, and b is the average distance from the sample to the sample in the nearest different class. The calculation method of the CH index is shown in formula (13), wherein m is the number of training set samples, k is the number of classes, B k is the covariance matrix between classes, W k is the covariance matrix of the data inside the class, and tr is the trace of the matrix.

[0095]

[0096]

[0097] The performance test selects the batch processing job in the Alibaba log, and the evaluation results for the features of the graph structure (including the number of nodes, the critical path length, the maximum parallelism, and the edge density) are shown in Table 1. The evaluation results for the resource usage features (execution time, CPU, and memory usage) are shown in Table 2.

[0098] Table 1: Comparison results of clustering effects for graph structure features

[0099]

[0100] Table 2: Comparison results of clustering effects for resource usage features

[0101]

[0102] From the data in Table 1 and Table 2, it can be concluded that, compared with the comparison method, the index of the method of the present application is improved higher on the graph structure features, the Silhouette Coefficient is improved by 0.33, and the CH index is improved by 52 times. The method of the present application is not much different from the comparison method on the resource usage features. In summary, the method of the present application has a great improvement on the graph structure features while ensuring the clustering effect of the resource usage features.

[0103] Finally, it should be noted that: the above examples are only used to illustrate the present application and do not limit the described technology of the present application, and all technical solutions and improvements thereof that do not deviate from the spirit and scope of the present application should be covered in the scope of the claims of the present application.

Claims

1. A clustering method for batch processing jobs in data centers, characterized by: It consists of five steps: initialization, modeling of batch job original resource usage characteristics, construction of batch job hidden layer feature extraction model, construction of batch job clustering method, and batch job classification: The following parameters are available: the number of columns c of the output matrix of the 1st to 3rd layer graph convolutional neural network (1) 、c (2) 、c (3) , the number of rows P of the output matrix of the pooling layer, the number of clusters k of K-means clustering; c (1) 、c (2) 、c (3) The recommended parameters are obtained based on graph convolution. P is obtained based on the average number of tasks in each job in the log, and k is determined based on the elbow rule. (1) Take 32, c (2) Take 32, c (3) Take 8, P takes 15, k takes 8; (1) Initialization Use the batch job information in the log to initialize the data, and let the full set of attributes contained in the log be A={a1,…a F }, select attributes related to the resource usage characteristics of the task, including attributes related to CPU, memory, execution time and network bandwidth, and define the attribute subset as T = {t1, t2…t S }, S is the number of attributes in the attribute subset T; the set of jobs included in the log is W = {w1,w2…w Z }, where Z is the total number of jobs in the log; the i-th job w in W i It consists of multiple tasks, represented by w i ={wt i1 , wt i2 …wt iH }, 1≤i≤Z, where H is the job w i The total number of tasks in , each task consists of multiple parallel instances with the same execution logic, and the jth task is represented by wt ij ={wi ij1 ,wi ij2 …wi ijV },1≤i≤Z,1≤j≤H, where V is the task wt ij The total number of instances in ijq The start time is defined as time_wi ijq ,1≤i≤Z,1≤j≤H,1≤q≤V; the start time of a task is the start time of the earliest instance of the task, task wt ij The start time is expressed as time ij , time ij =min(time_wi ij1 ,time_wi ij ,…,time_wi ijV ); (2) Modeling the raw resource usage characteristics of batch jobs 2.1) For any job w i ,1≤i≤Z, Z is the total number of jobs in the log, and the original resource usage characteristics of the batch job include the dependency structure and task resource attribute characteristics. First, the dependency structure of the job is defined by the adjacency matrix. The adjacency matrix is ​​as follows The number of rows and columns of the matrix is ​​H; each row of the matrix represents a task, and each column represents a task; any element m of the matrix ua ,1≤u≤H,1≤a≤H, represents the task wt in the batch job ia Whether it depends on task wt iu ; If there is a dependency, then m ua =1; otherwise m ua =0; whether there is dependency between tasks is determined by wt ia Is it only in the task wt iu Only after it is over can we start to judge, if so, there is dependence; 2.2) For any job w i ,1≤i≤Z, the resource usage characteristics of each task are defined using the task feature matrix, which is in the following form: The number of rows of the matrix is ​​H, the number of columns is S; any element f of the matrix j,q ,1≤j≤H,1≤q≤S, represents the job w i In the task wt ij In the attribute t q The attribute value on ; 2.3) For each job w i , according to its dependency structure, initialize the adjacency matrix M i , initialize the feature matrix F according to its task resource usage characteristics i ; (3) Construction of batch processing hidden layer feature extraction model 3.1) Use an autoencoder neural network as the hidden layer feature extraction model for batch processing jobs. The autoencoder neural network consists of an encoder and a decoder. The encoder consists of three graph convolutional layers, a convolutional layer, a pooling layer, and a fully connected layer. The decoder consists of a fully connected layer, a depooling layer, a deconvolution layer, and three deconvolutional layers. The autoencoder neural network is trained by reducing the difference between the encoder input and the decoder output. After training, the encoder partially outputs the hidden layer feature vector of the job. The model input is an H×H adjacency matrix M and an H×S feature matrix F. The output is the hidden feature vector of the batch job, which aggregates the characteristics of the job in terms of dependency structure and resource usage. The learning rate is 0.01 or 0.001, and the training batch size is set to the size of the entire log sample dataset; The calculation method of the graph convolution layer in the encoder is shown in formulas (1) to (3); first calculate the augmented adjacency matrix of the task w The calculation formula is shown in formula (1), where M is the adjacency matrix of job w, and I is the identity matrix with the same number of rows as M; then the augmented diagonal matrix of job w is calculated The calculation formula is shown in formula (2), where Representation matrix The value of row i and column j in Representation matrix The value of the i-th row and q-th column in the matrix H is The number of rows is also the total number of tasks in job w; Formula (3) is the iterative calculation formula of the graph convolution layer, where G (m) is the input matrix of the m-th layer graph convolution, G (0) is the feature matrix F of job w; θ (m) It is the parameter matrix of the graph convolution operation. The specific value needs to be obtained through training. (m) The mth layer has c (m) dimensional feature matrix G (m) Mapped to c (m+1) dimension,θ (m) The number of rows is c (m) , the number of columns is c (m+1) A real matrix; ξ(·) is the tanh activation function; after three graph convolutions, the matrix G is obtained (3) ; The convolution layer is followed by the graph convolution layer. Before the convolution starts, the start time of each task in the job is calculated based on the time. ij Re-convolutional layer output matrix G (3) Sort, matrix G (3) Each row represents a task, after reordering, the start time is time ij The rows corresponding to smaller tasks will be located at the top of the matrix, and the rows corresponding to tasks with later start times will be located at the bottom of the matrix. The number of layers l of each task in the DAG of the job is added to the right side of the corresponding row in the matrix. The number of layers l is the shortest path from the root task to the task. At this time, the matrix G (3) The number of columns becomes c (3) +1; the convolution kernel size of the convolution layer is 2×(c (3) +1), the stride is 1, that is, the number of columns of the convolution kernel is equal to the number of columns of the input matrix; the pooling layer is composed of adaptive average pooling and weighted adaptive average pooling; the parameter of adaptive average pooling is the size of the output matrix output_size, output_size=(P,1); the parameter stride and kernel number kernel_size of the pooling are calculated by output_size and the size of the input matrix input_size, and the calculation is shown in formula (4) and formula (5), where the floor() function is rounded down; the calculation formula of the pooling layer is shown in formula (6), P is the number of rows of the output matrix, H is the number of rows of the input matrix, and adaptiveAvgPool() is adaptive average pooling; the size of the output matrix of the pooling layer is P×2; the feature vector V of the output operation of the fully connected layer w , the number of neurons is 8; stride=floor(input_size÷output_size) (4) kernel_size=input_size-(output_size-1)×stride (5) The number of neurons in the fully connected layer in the decoder is 2P; The calculation of the unpooling layer is shown in formula (7). The avgUnpool() function is the unaverage pooling, and its parameters are the same as those of the adaptive pooling in the encoder. The parameters of the deconvolution layer are the same as those of the convolution layer in the encoder. Then, according to the order in the encoder, each row in the output matrix of the convolution layer is restored to its original position. The calculation formula of the inverse convolution layer is the same as formula (1). The parameters of the first deconvolution layer are the same as those of the third convolution layer, the parameters of the second deconvolution layer are the same as those of the second convolution layer, and so on. During training, the loss value is calculated based on the input matrix F and the decoder output. The specific formula is shown in formula (8); where F is the feature matrix of the encoder input, is the output matrix of the decoder, ‖·‖ F is the Frobenius norm, H is the number of tasks in job w, N is the number of job samples with H tasks in the log, and Z is the total number of job samples in the log; 3.2) Using the batch job set W to train the autoencoder neural network model AE; 3.2.1) Train the constructed autoencoder neural network AE and convert all the jobs w in the job set W into i As sample data, w i represents the i-th job in the job set; the adjacency matrix M i With the feature matrix F i As the input value of the model, the model parameters are updated through the forward propagation algorithm and Adam optimizer according to the loss value for training. Repeat the input until all jobs are trained; 3.2.2) Repeat the process in 3.2.1) for multiple rounds of model parameter updates until the difference in loss between two iterations is less than 2%. After the parameter updates are complete, the training job is complete. (4) Construction of batch job clustering model 4.1) Select the hidden layer feature vector X of the batch job as the feature value required for clustering, use the K-means clustering algorithm for clustering, define the number of K-means clustering clusters k = 8, and the batch job clusters CL after clustering = {cl1, cl2, ..., cl k }; Initialize and randomly select k cluster center points, the center point set is CT = {c1, c2, ... c k }, where c i Represents the center point sample c of the i-th cluster i →cl i ; 4.2) Traverse the job set W, for any w i Calculate the distance between it and k center points according to formula (9); where X i For homework i The eigenvector of X j is the feature vector of the center point of the j-th cluster, ‖·‖ F is the Frobenius norm; and task w i The cluster with the smallest distance between the centers is the task w i The belonging of is shown in formula (10), where the cluster() function returns the cluster to which it belongs, and w i Join the set cluster(w i ) in the set of jobs W; traverse the set of jobs W, any job w in W i After the belonging cluster exists, recalculate the center point of each cluster and update the center point of each cluster according to formula (11), c i For cluster cl i The center point of cluster cl i The number of elements in X j The corresponding w i ∈cl i ; 4.3) Repeat step 4.2) until the center of the cluster no longer changes. This completes the clustering of the job set W and obtains the clustering model. 4.4) Based on the clustering results, the batch job dataset is divided into clusters. The jobs in each cluster have similar DAG structures and resource consumption patterns. (5) Batch job classification 5.1) For any job w, input its adjacency matrix M and feature matrix F into the autoencoder neural network in step 3). Use the autoencoder neural network to extract hidden layer features. The encoder part of the autoencoder neural network outputs the feature vector X of the batch job; 5.2) Repeat step 5.1) until feature extraction of all batch jobs is completed; 5.3) For any batch job w, calculate the distance between the feature vector X of job w and each center point in the batch clustering model. The calculation formula is shown in formula (9). The cluster to which the center point with the smallest distance from job w belongs is the cluster to which job w belongs. This completes the classification of job w.

Images