A cluster-aware method for HPC jobs based on hybrid clusters

Abstract

The present invention relates to a method for perceiving a HPC job cluster based on a hybrid cluster, including: A. obtaining job operation log record information of a traditional virtualized cloud cluster environment and a container cloud cluster environment; B. initially constructing and processing job operation data sets in a traditional virtualized cloud cluster environment and a container cloud cluster environment; C. training a prediction model based on an RFR random forest regression algorithm in two environments, a traditional virtualized cloud cluster environment and a container cloud cluster environment; D. according to the job demand information submitted by the user, the prediction model gives a prediction of the running time, CPU usage rate, and memory usage rate; E. processing the submitted job with a cluster perception model using the job operation result information, and finally forming a method for perceiving a node in an HPC job cluster. The present invention establishes prediction models in two environments, and performs cluster node perception based on the prediction results, thereby achieving the effect of automatically perceiving the best cluster node according to the job demand.

Description

Technical Field

[0001] The present invention relates to the fields of cloud computing and high performance computing, and in particular to a hybrid cluster-based HPC job cluster node perception method. Background Art

[0002] As an important means of scientific and technological innovation, high-performance computing is applied to more fields, including nuclear explosion simulation, weather forecasting, seismic research for oil and gas exploration, high-resolution solid and fluid mechanics, social media analysis and molecular dynamics, etc. In recent years, it has been used more deeply in physics research, nuclear weapon design, aerospace vehicle design, energy exploration, satellite image processing, national economic forecasting and decision-making, intelligence analysis, Internet services, industrial simulation and other fields. The development of high-performance computing has become one of the important signs of scientific and technological progress, and it is also a concentrated reflection of a country's comprehensive scientific and technological strength. High-performance computing usually executes computing tasks through supercomputers or large-scale high-performance computing clusters, and is completed through the high-speed collaboration of each node in the cluster. Supercomputers or high-performance computing clusters have become important infrastructures of modern supercomputing centers. HPC (High Performance Computing, high-performance computer cluster) users can usually access local high-performance computing clusters, but based on resources and time levels, the operating conditions provided by the local high-performance computing cluster may not be optimal for the jobs and requirements submitted by users. For example, when resources are insufficient to meet the new jobs submitted by users, the jobs may wait in the queue for a long time and have to run. In this context, due to the advantages of cloud computing in terms of elastic supply, lightweight virtualization, and resource control, more and more academic and commercial HPC users have begun to try to use the cloud as an economic alternative to supercomputers or high-performance computing clusters. Existing studies have shown that due to the certain overhead of cloud computing virtualization in network and storage performance, HPC applications cannot obtain performance comparable to supercomputer clusters on the cloud, but cloud computing can be used as a supplement to supercomputers or high-performance computing. First, from the perspective of HPC users, there are some users who do not have high performance requirements and pay more attention to cost-effectiveness, and are willing to migrate HPC applications to the cloud to obtain shorter queuing time and lower machine time rental fees. Second, from the perspective of platform operators, transferring appropriate HPC jobs to the cloud can reduce resource fragmentation of HPC clusters, improve the load balancing and utilization of the overall platform resources, and effectively shorten user response time. At the same time, the visualization and data management functions, flexible software licensing models, and cloud value-added services in the cloud platform make the cloud more and more user-friendly for the development of HPC applications. At present, although most supercomputing centers have the capabilities of heterogeneous computing clusters (such as high-performance computing clusters and cloud computing clusters), each cluster still manages and loads its own applications separately, and has not achieved application awareness capabilities based on hybrid clusters. It cannot timely and accurately determine for users which cluster node the submitted job is suitable for running on, that is, it cannot help users better determine whether the submitted job runs on a local cluster or a cloud-based cluster. Summary of the invention

[0003] In order to overcome the shortcomings of the above-mentioned technical problems, the present invention provides a method for perceiving nodes in an HPC job cluster based on a hybrid cluster, and provides a selection function to support decision-making. It includes: predicting which cluster node the HPC job is suitable for running on based on the submitted HPC job demand information, so as to perform scheduling perception on the HPC job, make appropriate and better choices for HPC users, and obtain the running results of the job. Among them, for existing different categories of HPC applications, the running results of the HPC job in two different environments based on the traditional virtualized cloud cluster environment and the container cloud cluster environment are obtained through the data center, and the characteristic values ​​of the operation are obtained; by obtaining the characteristic values, a data set and a perception model are established to predict and perceive the clusters suitable for HPC jobs with different characteristics, so as to achieve the most suitable effect for HPC job perception, and give users feedback on the job results running under the appropriate work allocation solution.

[0004] Terminology explanation:

[0005] A data center is a global collaborative network of specific devices used to transmit, accelerate, display, calculate, and store data information on the Internet network infrastructure.

[0006] StandardScalarTransform(), an algorithm in the Scikitlearn Python package, is used to normalize the data set that needs sample training by transferring it to its respective mean and scaling it by its respective standard deviation so that the data range is within the acceptable range of the model.

[0007] The technical solution of the present invention is:

[0008] A method for HPC job cluster awareness based on a hybrid cluster includes the following steps:

[0009] A. Obtain job running log information in traditional virtualized cloud cluster environments and container cloud cluster environments;

[0010] B. Based on the job running log record information of the traditional virtualized cloud cluster environment and the container cloud cluster environment obtained in step A, initially construct and process the job running data set in the traditional virtualized cloud cluster environment and the container cloud cluster environment, including: obtaining feature values, filtering and processing feature value information, standardizing the feature information, splitting the feature value data set into a training set and a test set, and performing data normalization on the data set;

[0011] C. The feature data sets in the two cluster environments obtained in step B are trained on the RFR random forest regression algorithm respectively to obtain prediction models based on the RFR random forest regression algorithm in the traditional virtualized cloud cluster environment and the container cloud cluster environment;

[0012] D. Based on the job requirement information submitted by the user, the job requirement information includes: job application type name, job running node requirement number, and job running CPU core number;

[0013] The prediction model trained in step C gives predictions of running time, CPU usage, and memory usage;

[0014] The prediction results obtained by the prediction model are described as follows:

[0015] Running time refers to the time required for the job to complete, as predicted by the prediction model based on the job demand information submitted by the user; CPU usage refers to the CPU usage ratio of the job to the current CPU environment in the cluster, as predicted by the prediction model based on the job demand information submitted by the user; memory usage refers to the memory usage ratio of the job to the current memory environment in the cluster, as predicted by the prediction model based on the job demand information submitted by the user;

[0016] E. Use the job running result information predicted in step D to process the submitted job using the cluster awareness model, and finally form an HPC job cluster node awareness method.

[0017] Preferably, in step A, obtaining the job running log record information of the traditional virtualized cloud cluster environment and the container cloud cluster environment includes:

[0018] Obtain log records of jobs submitted by HPC users from the data center to run in traditional virtualized cloud cluster environments and container cloud cluster environments. The log records include: application category of the running job, number of nodes and cores required to run the job, status of the job running result, running time of the job, CPU usage during the job running, and memory usage during the job running.

[0019] Preferably, in step B, initially constructing and processing a job running data set in a traditional virtualized cloud cluster environment and a container cloud cluster environment includes:

[0020] Extract feature values ​​from the obtained log records running in the traditional virtualized cloud cluster environment and the container cloud cluster environment. The feature values ​​include job name JobName, job completion status State, number of job running nodes AllocNodes, number of job running CPU cores AllocCPUS, job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage.

[0021] Filter and process characteristic value information: including: filtering out job completion status State non-COMPLETED log records; filtering out job running information entries with CPU running time CPUTimeRAW being 0;

[0022] Standardize the feature information: including: converting non-digital features into digital codes and converting the units of data information into a unified format;

[0023] Split the eigenvalue data set into training set and test set: divide the standardized eigenvalue data into training set and test set according to a certain ratio.

[0024] Further preferably, the ratio of the training set to the test set in the traditional virtualized cloud cluster environment and the container cloud cluster environment is 4:1.

[0025] The data of the dataset is normalized, including: using StandardScalarTransform() in the Scikitlearn Python package to normalize the dataset features obtained in the traditional virtualized cloud cluster and container cloud cluster environments, namely, the job completion status State, the number of job running nodes AllocNodes, the number of job running CPU cores AllocCPUS, the job running time Elapsed, the CPU running time CPUTimeRAW, the CPU usage CPUusage, and the memory usage in the two environments, by transferring them to their respective average values ​​and scaling them according to their respective standard deviations.

[0026] Preferably, according to the present invention, the feature data sets in the two cluster environments obtained in step B are trained on the RFR random forest regression algorithm respectively to obtain prediction models based on the RFR random forest regression algorithm in the two environments of the traditional virtualized cloud cluster environment and the container cloud cluster environment; including:

[0027] (a) Randomly extract m sample points from the training sample set S to obtain a new sub-training set {S1…Sn};

[0028] (b) Use the sub-training set to train a regression tree. During the training process, the splitting rule for each node is: first randomly select k features from all features, then select the optimal split point from these k features to divide the left and right subtrees;

[0029] (c) Generate many regression trees through step (b);

[0030] (d) The final prediction result of each regression tree is the mean of the leaf nodes reached by the sample point;

[0031] (e) The final prediction result of the random forest is the mean of the prediction results of all regression trees.

[0032] Preferably, according to the present invention, the job running result information predicted in step D is used to process the submitted job using a cluster-aware model;

[0033] For job i submitted by the user, the job requirement information is i JobName 、i AllocNodes 、i AllocCPUS ,i JobName The job application type name of job i submitted by the user; i AllocNodes The number of job running nodes required for job i submitted by the user; i AllocCPUs The number of CPU cores for job i submitted by the user;

[0034] The running time Elapsed predicted by the prediction model in the traditional virtualized cloud cluster environment Publii CPU usage Public Memory usage Publii , named the set predicted Public , where the set information is: predicted Public ={Elapsed Publii , CPU usage Public , Memory usage Public};

[0035] For the container cloud cluster environment, the predicted running time Elapsed Container CPU usage Container Memory usage Container , named set predicted Container , where the collection information is:

[0036] predicted Container ={Elapsed Container , CPU usage Container , Memory usage Container};

[0037] The set predicted Public and the set predicted Container The included features serve as the basis for the cluster awareness model to perceive the HPC cluster nodes; including:

[0038] Step 1: Input matrix X; matrix X includes matrix X1 and matrix X2; each job running record in the feature data set in the traditional virtualized cloud cluster environment includes the job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage, which is recorded as matrix X1; each job running record in the feature data set in the container cloud cluster environment includes the job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPUusage, and memory usage Memory usage, which is recorded as matrix X2;

[0039] The sample objects include 1 to m;

[0040] Each sample object has n features, n=4, including job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage;

[0041] Step 2: Initialize m=1; n=1;

[0042] Step 3: Perform dimensionless processing on each data, i.e., perform inverse processing: After the features of each job log record are processed by data standardization, m is increased by 1, and the iterative process is performed until m reaches the last job log record in the data set in each environment;

[0043] Step 4: Perform standard deviation processing on each feature: S(ij) to obtain the index variability, n increases by 1, and iterate until n reaches 4;

[0044] Step 5: Process the information volume of each feature: C(j), obtain the index conflict, n increases by 1, and iterate until n reaches 4;

[0045] Step 6: Calculate the objective weight of each feature to obtain the weight value of each feature, n increases by 1, and iterate until n reaches 4;

[0046] Elapsed Public 、CPU usage Public 、Memory usage Public After processing from step 1 to step 6, the weight values ​​obtained are W pe , W pc , W pm ; Elapsed Container 、CPU usage Container 、Memory usage Container After processing from step 1 to step 6, the weight values ​​obtained are Wce , W cc , W cm .

[0047] Further preferably, the inversion treatment formula is as shown in formula (I):

[0048]

[0049] In formula (II), x ij represents the jth feature value in the i-th job log record; Represents the maximum value of the eigenvalue of the jth column in the input data set matrix X; Represents the minimum value of the eigenvalues ​​in the jth column of the input data set matrix X; max (j) Indicates the maximum value of the j-th feature value in the i-th job log record.

[0050] Further preferably, in step 4, the standard deviation S of the reaction index variability is obtained for each feature. j , as shown in formula (III):

[0051]

[0052] in, is the mean value of each indicator data.

[0053] Further preferably, in step 5, information processing is performed on each feature, including:

[0054] The conflict R is obtained as shown in formula (III):

[0055]

[0056] Calculate the conflict coefficient A j , as shown in formula (IV):

[0057]

[0058] In formula (IV), r ij It represents the correlation coefficient between the lowest i index and the lowest j index;

[0059] Calculate the information carrying capacity C j , as shown in formula (V):

[0060] C j =S j ×A j (V).

[0061] Further preferably, in step 6, an objective weight W is assigned to each feature. j Calculate, as shown in formula (VI):

[0062]

[0063] Preferably, according to the present invention, a method for forming a node perception of an HPC job cluster includes:

[0064] 1) Obtain the cluster node perception function value based on the traditional virtualized cloud cluster environment and the container cloud cluster environment:

[0065] The perception function value in the traditional virtualized cloud cluster environment is:

[0066] G Public =W pe *Elapsed Public +W pc *CPU Usage Public +W pm *Memory Usage Public ;

[0067] The perception function value in the container cloud cluster environment is:

[0068] G Container =W ce *Elapsed Container +W cc *CPU Usage Container +W cm *Memory Usage Container ;

[0069] 2) If G Public >G Container , then job i is sensed to run in the node of the container cloud cluster environment; if G Public <G Container , then job i will be perceived as running in the node of the traditional virtualized cloud cluster environment.

[0070] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of a hybrid cluster-based HPC job cluster node perception method when executing the computer program.

[0071] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a hybrid cluster-based HPC job cluster node perception method.

[0072] The beneficial effects of the present invention are:

[0073] 1. A dataset of HPC job operation characteristics in different environments was formed. Based on the log records of job operation submitted by HPC users in the traditional virtualized cloud cluster environment and the container cloud cluster environment, the data center was processed to form a dataset of HPC job operation characteristics in the traditional virtualized cloud cluster environment and a dataset of HPC job operation characteristics in the container cloud cluster environment.

[0074] 2. A method for predicting cluster nodes for high-performance jobs based on hybrid clusters has been developed. Based on the characteristic data sets of HPC jobs running in traditional virtualized cloud cluster environments and container cloud cluster environments, a runtime prediction model based on job demand information submitted by users in the two environments is established, and cluster node perception is performed based on the prediction results, achieving the effect of automatically perceiving the best cluster nodes according to job requirements;

[0075] 3. It solves the requirement of reducing the overall completion time or cost of user tasks as much as possible while meeting user needs, and promotes the development of HPC applications in hybrid clusters. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 It is a flow chart of the node perception method of HPC job cluster based on hybrid cluster of the present invention; DETAILED DESCRIPTION

[0077] The present invention will be further defined below in conjunction with the accompanying drawings and embodiments, but is not limited thereto.

[0078] Example 1

[0079] A cluster-aware method for HPC jobs based on hybrid clusters, such as Figure 1 As shown, the steps include:

[0080] A. Obtain job running log information in traditional virtualized cloud cluster environments and container cloud cluster environments;

[0081] B. Based on the job running log record information of the traditional virtualized cloud cluster environment and the container cloud cluster environment obtained in step A, initially construct and process the job running data set in the traditional virtualized cloud cluster environment and the container cloud cluster environment, including: obtaining feature values, filtering and processing feature value information, standardizing the feature information, splitting the feature value data set into a training set and a test set, and performing data normalization on the data set;

[0082] C. The feature data sets in the two cluster environments obtained in step B are trained on the RFR random forest regression algorithm respectively to obtain prediction models based on the RFR random forest regression algorithm in the traditional virtualized cloud cluster environment and the container cloud cluster environment;

[0083] D. Based on the job requirement information submitted by the user, the job requirement information includes: job application type name, job running node requirement number, and job running CPU core number;

[0084] The prediction model trained in step C gives predictions of running time, CPU usage, and memory usage;

[0085] The prediction results obtained by the prediction model are described as follows:

[0086] Running time refers to the time required for the job to complete, as predicted by the prediction model based on the job demand information submitted by the user; CPU usage refers to the CPU usage ratio of the job to the current CPU environment in the cluster, as predicted by the prediction model based on the job demand information submitted by the user; memory usage refers to the memory usage ratio of the job to the current memory environment in the cluster, as predicted by the prediction model based on the job demand information submitted by the user;

[0087] E. Use the job running result information predicted in step D to process the submitted job using the cluster awareness model, and finally form an HPC job cluster node awareness method.

[0088] Example 2

[0089] The difference between the hybrid cluster-based HPC job cluster perception method described in Example 1 is that:

[0090] In step A, the job running log record information of the traditional virtualized cloud cluster environment and the container cloud cluster environment is obtained, including:

[0091] We obtained log records of jobs submitted by HPC users from the data center to run in traditional virtualized cloud cluster environments and container cloud cluster environments. A total of 1.4 million log records were obtained for traditional virtualized cloud cluster environments, and 1.2 million log records were obtained for container cloud cluster environments. The log records include: the application category of the job, the number of nodes and cores required to run the job, the status of the job's running results, the running time of the job, the CPU usage rate of the job, and the memory usage rate of the job. These feature information will be used to build a data set based on job running in the two environments.

[0092] In step B, the job running data sets in the traditional virtualized cloud cluster environment and the container cloud cluster environment are initially constructed and processed, including:

[0093] Extract feature values ​​from the obtained log records running in the traditional virtualized cloud cluster environment and the container cloud cluster environment. The feature values ​​include job name JobName, job completion status State, number of job running nodes AllocNodes, number of job running CPU cores AllocCPUS, job running time Elapsed, CPU running time CPUTimeRAW (job running time * number of cores), CPU usage CPU usage, and memory usage Memory usage;

[0094] Filtering and processing feature value information: processing some job records in the obtained feature information that do not meet the requirements for building the prediction data set. Including: filtering out log records with a job completion status State other than COMPLETED (successful operation); specifically, using the job completion status State as a filtering condition, deleting job operation information entries with a job completion status State of FAILED (indicating job failure), CANCELLED (indicating job cancellation), TIMEOUT (job timeout), PENDING (job waiting), and NODE_FAIL (job node selection failure). At the same time, processing missing values ​​in the data, using the CPUTimeRAW value other than 0 as a filtering condition, filtering out job operation information entries with a CPU operation time CPUTimeRAW of 0; providing more accurate data information for the construction of the data set; after filtering and processing, 920,000 log records are still retained in the traditional virtualization cloud cluster environment, and 740,000 log records are still retained in the container cloud cluster environment;

[0095] Standardize the feature information: including: converting non-numeric features into numbers, and converting the units of data information into a unified format; the job name JobName cannot be used for mathematical calculations of the model, so it needs to be encoded into numbers; the job running time Elapsed is converted into a unified counting unit, which is s (seconds); that is, the unit of each feature data is processed in a unified format;

[0096] Split the eigenvalue data set into training set and test set: The eigenvalue data after standardization is divided into training set and test set according to a certain ratio. In this method, the data has a special physical meaning, so it is necessary to divide the data training set and test set in advance, and no longer divide randomly. The feature information data set after standardization is split based on model training and testing, that is, divided into training set and test set; the ratio of training set and test set in traditional virtualized cloud cluster environment and container cloud cluster environment is 4:1.

[0097] Performing data normalization on the data of the data set: including: performing data normalization on the features of the data set obtained by splitting into training sets, and processing the training set feature data in the data set into a numerical range that is recognizable and acceptable to the training model.

[0098] The selected features in the traditional virtualized cloud cluster environment and the container cloud cluster environment, namely, the job name JobName, the job completion status State, the number of job running nodes AllocNodes, the number of job running CPU cores AllocCPUS, the job running time Elapsed, the CPU running time CPUTimeRAW, the CPU usage CPU usage, and the memory usage Memory usage are further processed and analyzed. Because CPUTimeRAW is the result of Elapsed*AllocNodes, that is, the job running time*the number of cores, it has the significance of filtering and processing in the early stage of data set construction, but it has no testing significance, so it is not used as the training and testing features of the prediction model here. The StandardScalarTransform() in the Scikitlearn Python package is used to normalize the data set features obtained in the traditional virtualized cloud cluster and the container cloud cluster environment, namely, the job completion status State, the number of job running nodes AllocNodes, the number of job running CPU cores AllocCPUS, the job running time Elapsed, the CPU running time CPUTimeRAW, the CPU usage CPUusage, and the memory usage in the two environments by transferring to their respective average values ​​and scaling them according to their respective standard deviations.

[0099] The job application type submitted by the user is used as the main node to form a data set, including a model training data set and a model test data set, to prepare data for the high-performance computing job cluster node perception prediction model;

[0100] The feature data sets in the two cluster environments obtained in step B are trained on the RFR random forest regression algorithm respectively to obtain prediction models based on the RFR random forest regression algorithm in the traditional virtualized cloud cluster environment and the container cloud cluster environment; including:

[0101] The event of obtaining new prediction results by training the model according to historical data can be regarded as a regression problem. Therefore, the RFR random forest regression algorithm establishes a regression training model based on the data sets in the traditional virtualized cloud cluster environment and the container cloud cluster environment. That is, the RFR random forest regression model is used to train the data sets in two different environments to obtain the prediction models in the two environments respectively;

[0102] (a) Randomly extract m sample points from the training sample set S (one of the feature data sets in the two cluster environments obtained in step B) to obtain a new sub-training set {S1…Sn};

[0103] (b) Use the sub-training set to train a regression tree (decision tree). During the training process, the splitting rule for each node is: first randomly select k features from all features, and then select the optimal split point from these k features to divide the left and right subtrees; (the decision trees obtained here are all binary trees).

[0104] (c) Generate many regression trees through step (b);

[0105] (d) The final prediction result of each regression tree is the mean of the leaf nodes reached by the sample point;

[0106] (e) The final prediction result of the random forest is the mean of the prediction results of all regression trees.

[0107] In step D, based on the job demand information submitted by the user, the prediction model trained in step C is used to predict the running time, CPU usage, and memory usage, including the following steps:

[0108] Based on the two prediction models trained in the traditional virtualized cloud cluster environment and the container cloud cluster environment, the Elapsed, CPU usage, and Memory usage of the jobs submitted by the user are predicted in the two environments. The prediction information is shown in Table 1 below:

[0109] Table 1

[0110]

[0111]

[0112] i JobName The job application type name of job i submitted by the user; i AllocNodes The number of job running nodes required for job i submitted by the user; i AllocCPUS The number of cores required for job i submitted by user.

[0113] Process the submitted job using the cluster-aware model based on the job running result information predicted in step D;

[0114] For job i submitted by the user, the job requirement information is i JobName 、i AllocNodes 、i AllocCPUS ,i JobName The job application type name of job i submitted by the user; i AllocNodesThe number of job running nodes required for job i submitted by the user; i AllocCPUs The number of CPU cores for job i submitted by the user;

[0115] The running time Elapsed predicted by the prediction model in the traditional virtualized cloud cluster environment Publii CPU usage Public Memory usage Public , named the set predicted Public , where the set information is: predicted Public ={Elapsed Public , CPU usage Public , Memory usage Public};

[0116] For the container cloud cluster environment, the predicted running time Elapsed Container CPU usage Container Memory usage Container , named set predicted Container , where the collection information is:

[0117] predicted Container ={Elapsed Container , CPU usage Container , Memory usage Container};

[0118] The set predicted Public and the set predicted Container The included features serve as the basis for the cluster awareness model to perceive the HPC cluster nodes; including:

[0119] Step 1: Input matrix X; matrix X includes matrix X1 and matrix X2; each job running record in the feature data set in the traditional virtualized cloud cluster environment includes the job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage, which is recorded as matrix X1; each job running record in the feature data set in the container cloud cluster environment includes the job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPUusage, and memory usage Memory usage, which is recorded as matrix X2;

[0120] The sample objects include 1 to m. The sample objects here are each job log record in the feature data set in the traditional virtual cloud cluster environment and the container cloud cluster environment.

[0121] Each sample object has n features, n=4, including job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage;

[0122] Step 2: Initialize m=1; n=1;

[0123] Step 3: Perform dimensionless processing on each data, i.e., reverse processing: the main purpose is to eliminate the impact of dimensions so that all data can be measured with a unified standard. If the larger the value of a feature is, the more positive the impact on the result is, then the feature is a positive feature value and is processed positively; if the smaller the value of a feature is, the more positive the impact on the result is, then the feature is changed to a reverse feature value and is processed reversely; after the features of each job log record are processed by data standardization, m is increased by 1, and the iterative process is performed until m reaches the last job log record in the data set in each environment;

[0124] In the dimensionless processing, the forward processing formula is as follows:

[0125]

[0126] In formula (I), x ij represents the jth eigenvalue in the i-th job log record; x′ ij Represents x ij The scaled eigenvalues; Represents the minimum value of the eigenvalues ​​in the jth column of the input data set matrix X; Represents the maximum value of the eigenvalue of the jth column in the input data set matrix X; max (j) Indicates the maximum value of the j-th feature value in the i-th job log record.

[0127] The inverse processing formula is shown in formula (I):

[0128]

[0129] In formula (II), x ij represents the jth feature value in the i-th job log record; Represents the maximum value of the eigenvalue of the jth column in the input data set matrix X; Represents the minimum value of the eigenvalues ​​in the jth column of the input data set matrix X; max (j)Indicates the maximum value of the jth eigenvalue in the i-th job log record. The three characteristic attributes in the data set matrix passed into the model in the traditional virtual cloud cluster environment and the container cloud cluster environment, job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage, all have smaller eigenvalues, which have a more positive impact on the results, so these three characteristic attributes are all reversed.

[0130] Step 4: Perform standard deviation processing on each feature: S(ij) to obtain the index variability, n increases by 1, and iterates until n reaches 4; for each feature, obtain the standard deviation S of the response index variability j , as shown in formula (III):

[0131]

[0132] in, The mean value of each indicator (column) data.

[0133] Step 5: Process the information volume of each feature: C(j), obtain the index conflict, n increases by 1, and iterate until n reaches 4; process the information volume of each feature, including:

[0134] The conflict R is obtained as shown in formula (III):

[0135]

[0136] Calculate the conflict coefficient A j , as shown in formula (IV):

[0137]

[0138] In formula (IV), r ij It represents the correlation coefficient between the lowest i index and the lowest j index;

[0139] Calculate the information carrying capacity C j , as shown in formula (V):

[0140] C j =S j ×A j (V).

[0141] Step 6: Calculate the objective weight of each feature to obtain the weight value of each feature, n increases by 1, and iterate until n reaches 4; calculate the objective weight W for each feature j Calculate, as shown in formula (VI):

[0142]

[0143] Elapsed Public 、CPU usage Public 、Memory usage Public After processing from step 1 to step 6, the weight values ​​obtained are W pe , W pc , W pm ; Elapsed Container 、CPU usage Container 、Memory usage Container After processing from step 1 to step 6, the weight values ​​obtained are W ce , W cc , W cm .

[0144] Forming a node awareness method for HPC job clusters, including:

[0145] 1) Obtain the cluster node perception function value based on the traditional virtualized cloud cluster environment and the container cloud cluster environment:

[0146] The perception function value in the traditional virtualized cloud cluster environment is:

[0147] G Public =W pe *Elapsed Public +W pc *CPU Usage Public +W pm *Memory Usage Public ;

[0148] The perception function value in the container cloud cluster environment is:

[0149] G Container =W ce *Elapsed Container +W cc *CPU Usage Container +W cm *Memory Usage Container ;

[0150] 2) If G Public >G Container , then job i is sensed to run in the node of the container cloud cluster environment; if G Public <G Container , then job i will be perceived as running in the node of the traditional virtualized cloud cluster environment.

[0151] Example 3

[0152] A computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the HPC job cluster node perception method based on a hybrid cluster in embodiment 1 or 2 are implemented.

[0153] Example 4

[0154] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the hybrid cluster-based HPC job cluster node perception method of embodiment 1 or 2.

Claims

1. A method for HPC job cluster awareness based on hybrid clusters, characterized in that: The steps include: A. Obtain job running log information in traditional virtualized cloud cluster environments and container cloud cluster environments; B. Based on the job running log record information of the traditional virtualized cloud cluster environment and the container cloud cluster environment obtained in step A, initially construct and process the job running data set in the traditional virtualized cloud cluster environment and the container cloud cluster environment, including: obtaining feature values, filtering and processing feature value information, standardizing the feature information, splitting the feature value data set into a training set and a test set, and performing data normalization on the data set; C. The feature data sets in the two cluster environments obtained in step B are trained on the RFR random forest regression algorithm respectively to obtain prediction models based on the RFR random forest regression algorithm in the traditional virtualized cloud cluster environment and the container cloud cluster environment; D. Based on the job requirement information submitted by the user, the job requirement information includes: job application type name, job running node requirement number, and job running CPU core number; The prediction model trained in step C gives predictions of running time, CPU usage, and memory usage; The prediction results obtained by the prediction model are described as follows: Running time refers to the time required for the job to complete, as predicted by the prediction model based on the job demand information submitted by the user; CPU usage refers to the CPU usage ratio of the job to the current CPU environment in the cluster, as predicted by the prediction model based on the job demand information submitted by the user; memory usage refers to the memory usage ratio of the job to the current memory environment in the cluster, as predicted by the prediction model based on the job demand information submitted by the user; E. Use the job running result information predicted in step D to process the submitted job using the cluster awareness model, and finally form an HPC job cluster node awareness method.

2. The method for HPC job cluster perception based on hybrid cluster according to claim 1, characterized in that: In step A, the job running log record information of the traditional virtualized cloud cluster environment and the container cloud cluster environment is obtained, including: Obtain log records of jobs submitted by HPC users from the data center to run in traditional virtualized cloud cluster environments and container cloud cluster environments. The log records include: application category of the running job, number of nodes and cores required to run the job, status of the job running result, running time of the job, CPU usage during the job running, and memory usage during the job running.

3. The method for HPC job cluster perception based on hybrid cluster according to claim 1, characterized in that: In step B, the job running data sets in the traditional virtualized cloud cluster environment and the container cloud cluster environment are initially constructed and processed, including: Extract feature values ​​from the obtained log records running in the traditional virtualized cloud cluster environment and the container cloud cluster environment. The feature values ​​include job name JobName, job completion status State, number of job running nodes AllocNodes, number of job running CPU cores AllocCPUS, job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage. Filter and process characteristic value information: including: filtering out job completion status State non-COMPLETED log records; filtering out job running information entries with CPU running time CPUTimeRAW being 0; Standardize the feature information: including: converting non-digital features into digital codes and converting the units of data information into a unified format; Split the eigenvalue data set into training set and test set: divide the standardized eigenvalue data into training set and test set according to a certain ratio; Normalize the data in the dataset: using StandardScalarTransform() in the Scikitlearn Python package to normalize the features of the dataset obtained in the traditional virtualized cloud cluster and the container cloud cluster, i.e., the job completion status State, the number of job running nodes AllocNodes, the number of job running CPU cores AllocCPUS, the job running time Elapsed, the CPU running time CPUTimeRAW, the CPU usage CPUusage, and the memory usage in the two environments by transferring them to their respective average values ​​and scaling them according to their respective standard deviations; The ratio of training set and test set in traditional virtualized cloud cluster environment and container cloud cluster environment is 4:

1.

4. The method for HPC job cluster perception based on hybrid cluster according to claim 1, characterized in that: In step C, the feature data sets in the two cluster environments obtained in step B are trained on the RFR random forest regression algorithm respectively to obtain prediction models based on the RFR random forest regression algorithm in the two environments of the traditional virtualized cloud cluster environment and the container cloud cluster environment; including: (a) Randomly extract m sample points from the training sample set S to obtain a new sub-training set {S1…Sn}; (b) Use the sub-training set to train a regression tree. During the training process, the splitting rule for each node is: first randomly select k features from all features, then select the optimal split point from these k features to divide the left and right subtrees; (c) Generate many regression trees through step (b); (d) The final prediction result of each regression tree is the mean of the leaf nodes reached by the sample point; (e) The final prediction result of the random forest is the mean of the prediction results of all regression trees.

5. The method for HPC job cluster perception based on hybrid cluster according to claim 1, characterized in that: In step E, the submitted job is processed by the cluster-aware model using the job running result information predicted in step D; For job i submitted by the user, the job requirement information is i JobName 、i AllocNodes 、i AllocCPUS ,i JobName The job application type name of job i submitted by the user; i AllocNodes The number of job running nodes required for job i submitted by the user; i AllocCPUS The number of CPU cores for job i submitted by the user; The running time Elapsed predicted by the prediction model in the traditional virtualized cloud cluster environment Publoc CPU usage Public Memory usage Public , named the set predicted Public , where the set information is: predicted Public ={Elapsed Public , CPU usage Public , Memory usage Public }; For the container cloud cluster environment, the predicted running time Elapsed Container 、CPU usage Container Memory usage Container , named set predicted Container , where the collection information is: predicted Container ={Elapsed Container ,GPU usage Container ,Memory usage Container }; The set predicted Public and the set predicted Container The included features serve as the basis for the cluster awareness model to perceive the HPC cluster nodes; including: Step 1: Input matrix X; matrix X includes matrix X1 and matrix X2; each job running record in the feature data set in the traditional virtualized cloud cluster environment includes the job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage, which is recorded as matrix X1; each job running record in the feature data set in the container cloud cluster environment includes the job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPUusage, and memory usage Memory usage, which is recorded as matrix X2; The sample objects include 1 to m; Each sample object has n features, n=4, including job running time Elapsed, CPU running time CPUTimeRAW, CPU usage CPU usage, and memory usage Memory usage; Step 2: Initialize m=1; n=1; Step 3: Perform dimensionless processing on each data, i.e., perform inverse processing: After the features of each job log record are processed by data standardization, m is increased by 1, and the iterative process is performed until m reaches the last job log record in the data set in each environment; Step 4: Perform standard deviation processing on each feature to obtain the indicator variability, n increases by 1, and iterate until n reaches 4; Step 5: Process the information volume of each feature to obtain the index conflict, increase n by 1, and iterate until n reaches 4; Step 6: Calculate the objective weight of each feature to obtain the weight value of each feature, n increases by 1, and iterate until n reaches 4; Elapsed Public 、CPU usage Public 、Memory usage Public After processing from step 1 to step 6, the weight values ​​obtained are W pe , W pc , W pm ; Elapsed Container 、CPU usage Container 、Memory usage Container After processing from step 1 to step 6, the weight values ​​obtained are W ce , W cc , W cm .

6. The method for HPC job cluster perception based on hybrid cluster according to claim 5, characterized in that: In step 3, the inverse processing formula is shown in formula (I): In formula (II), x ij represents the jth feature value in the i-th job log record; Represents the maximum value of the eigenvalue of the jth column in the input data set matrix X; Represents the minimum value of the eigenvalues ​​in the jth column of the input data set matrix X; max (j) Indicates the maximum value of the j-th feature value in the i-th job log record.

7. The method for HPC job cluster perception based on hybrid cluster according to claim 5, characterized in that: In step 4, the standard deviation S of the response index variability is calculated for each feature. j , as shown in formula (III): in, is the mean value of each indicator data; In step 5, each feature is processed for information content, including: The conflict R is obtained as shown in formula (III): Calculate the conflict coefficient A j , as shown in formula (IV): In formula (IV), r ij It represents the correlation coefficient between the lowest i index and the lowest j index; Calculate the information carrying capacity C j , as shown in formula (V): C j =S j ×A j (V) In step 6, each feature is objectively weighted W j Calculate, as shown in formula (VI):

8. The method for HPC job cluster perception based on hybrid cluster according to claim 1, characterized in that: In step E, a method for sensing nodes in an HPC job cluster is formed, including: 1) Obtain the cluster node perception function value based on the traditional virtualized cloud cluster environment and the container cloud cluster environment: The perception function value in the traditional virtualized cloud cluster environment is: G Publig =W pe *Elapsed Public +W pc *CPU Usage Public +W pm *Memory Usage Public ; The perception function value in the container cloud cluster environment is: G Container =W ce *Elapsed Container +W cc *CPU Usage Container +W cm *Memory Usage Container ; 2) If G Public >G Container , then job i is sensed to run in the node of the container cloud cluster environment; if G Public <G Container , then job i will be perceived as running in the node of the traditional virtualized cloud cluster environment.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of any one of claims 1-8 of the HPC job cluster node perception method based on a hybrid cluster are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of any one of claims 1-8 of the HPC job cluster node perception method based on a hybrid cluster are implemented.

Images