HPC job power consumption prediction method and system based on power consumption curve and script information

Abstract

The application relates to the field of high-performance computing, and provides an HPC job power consumption prediction method and system based on power consumption curves and script information. The method comprises the following steps: obtaining a first similarity value based on historical job power consumption curve data; obtaining a second similarity value based on historical job script information data; assigning weights to the first similarity value and the second similarity value according to requirements, calculating a weighted sum, obtaining a comprehensive similarity value, and constructing a similarity adjacency matrix in this way; dividing HPC jobs into different categories according to the similarity adjacency matrix on the principle of maximizing a module degree index; training different neural network models based on historical data in different categories after the division, obtaining trained neural network models; matching a historical job category similar to a target HPC job, and using the neural network model of the historical job category to predict script information data of the target HPC job to obtain a prediction result.

Description

Technical Field

[0001] This invention belongs to the field of high-performance computing (HPC), and in particular relates to a method and system for predicting HPC job power consumption based on power consumption curves and script information. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] HPC systems typically consist of a large number of computing nodes and complex interconnected networks. To make full use of resources and improve energy efficiency, accurate prediction and management of power consumption for HPC operations are crucial.

[0004] There are various methods for predicting HPC power consumption, mainly including statistical regression models, machine learning algorithms, and time series analysis methods. These methods will be introduced one by one below.

[0005] Statistical regression models are a common method for power consumption prediction. They utilize historical job data and power consumption data to build a model, predicting future job power consumption by analyzing the relationship between job characteristics (such as job type, data size, resource requirements, etc.) and power consumption. Common statistical regression models include linear regression and multinomial regression. These models are simple, intuitive, easy to implement, and easy to interpret. However, this method ignores the complex correlations and dynamics between jobs. Furthermore, the accuracy of predictions may be affected for data with strong nonlinear relationships, and the model lacks generalization ability for new types of jobs.

[0006] Machine learning methods have also been widely used in HPC power consumption prediction. This approach utilizes machine learning algorithms, such as decision trees, support vector machines, and neural networks, to learn the complex mapping relationship between job features and power consumption from historical job data, in order to predict the power consumption of future jobs. Compared to statistical regression models, machine learning methods can better capture complex nonlinear relationships and the interrelationships between features, possessing a certain degree of generalization ability and being applicable to different types of jobs and scenarios. However, the drawbacks of this method lie in its heavy dependence on the quality and size of the dataset, and its lack of generalization ability when processing different types of data.

[0007] Time series analysis is a common technique used in power consumption forecasting. It predicts future power consumption by modeling and analyzing historical power consumption data over time. Common time series analysis methods include Autoregressive Moving Average (ARMA), Autoregressive Integral Moving Average (ARIMA), and seasonality decomposition. These methods can capture the trends, periodicity, and seasonality of power consumption data, making them suitable for data with strong long-term dependencies and time correlations. However, time series analysis methods have high requirements for data; they require power consumption data with a certain time series and continuity. Incomplete, missing, or discontinuous data may adversely affect the prediction results. Furthermore, time series analysis is usually based on patterns and trends in historical data, and its predictive ability may decrease when faced with situations that differ from historical data.

[0008] In summary, different methods have their own advantages and disadvantages in HPC power consumption prediction. Statistical regression models are simple and intuitive, but have limitations in capturing complex relationships. Machine learning methods can handle complex nonlinear relationships, but are dependent on data quality and scale. Time series analysis methods are suitable for data with time correlation, but require high data continuity and stability. Summary of the Invention

[0009] To address the technical problems mentioned above, this invention provides a method and system for predicting HPC job power consumption based on power consumption curves and script information. Guided by power consumption curves and script information, this invention captures the power consumption changes of HPC jobs under different states, and combines a neural network model to predict HPC job power consumption, thus achieving a more comprehensive and accurate prediction of HPC job power consumption.

[0010] To achieve the above objectives, the present invention adopts the following technical solution:

[0011] The first aspect of the present invention provides a method for predicting the power consumption of HPC jobs based on power consumption curves and script information.

[0012] HPC job power consumption prediction methods based on power consumption curves and script information include:

[0013] Acquire historical job power consumption curve data and historical job script information data;

[0014] Based on historical operation power consumption curve data, the first similarity algorithm is used to obtain the first similarity value;

[0015] Based on historical assignment script information data, a second similarity algorithm is used to obtain a second similarity value;

[0016] Based on the requirements, weights are assigned to the first similarity value and the second similarity value, and a weighted sum is calculated to obtain the comprehensive similarity value, thereby constructing a similarity adjacency matrix;

[0017] Based on the principle of maximizing modularity, HPC jobs are divided into different categories according to the similarity adjacency matrix;

[0018] Based on historical data in different categories after division, different neural network models are trained to obtain trained neural network models;

[0019] Obtain the script information data of the target HPC job, match it with historical job categories similar to the target HPC job, and use the trained neural network model of the historical job category to predict the script information data of the target HPC job to obtain the prediction result.

[0020] Furthermore, the process of obtaining the first similarity value based on historical operation power consumption curve data and using the first similarity algorithm includes: selecting several power consumption-related indicators based on historical operation power consumption curve data, and using the CBD algorithm based on the indicators to obtain the first similarity value.

[0021] Furthermore, the process of employing the CBD algorithm includes:

[0022] Each HPC job power consumption curve is encoded, and the CBD algorithm is used to convert continuous power consumption values ​​into discrete symbol sequences. A dictionary is constructed to record the recurring symbol sequences and replacement keys. The recurring sequences in the symbol sequences of each job power consumption curve are replaced with the keys in the dictionary to compress the job power consumption curve.

[0023] Calculate the size of each job power consumption curve after individual compression, compare the lengths of the compressed sequences, and calculate the compression ratio, where the compression ratio represents the ratio of the length of the compressed sequence to the length of the original sequence.

[0024] Based on the calculated compression ratio, the similarity between the power consumption curves of the operation is determined.

[0025] Furthermore, the process of obtaining the second similarity value based on historical task script information data and using the second similarity algorithm includes: using the KNN algorithm based on historical task script information data to calculate the similarity between historical task script information data and obtain the second similarity value.

[0026] Furthermore, the process of classifying HPC jobs into different categories includes: using a community detection algorithm based on the Leuven algorithm to divide HPC jobs into multiple communities, each community representing a category, indicating that HPC jobs within a community are similar to some extent.

[0027] Furthermore, after the communities are divided, the division results are evaluated by calculating the modularity value of the divided communities.

[0028] Furthermore, the prediction process of the trained neural network model also includes optimizing the parameters and weights of the neural network model based on real-time input data.

[0029] A second aspect of the present invention provides an HPC job power prediction system based on power consumption curves and script information.

[0030] HPC job power prediction system based on power consumption curves and script information, including:

[0031] The data acquisition module is configured to acquire historical job power consumption curve data and historical job script information data.

[0032] The first similarity calculation module is configured to: obtain a first similarity value based on historical operation power consumption curve data and using a first similarity algorithm;

[0033] The second similarity calculation module is configured to: obtain a second similarity value based on historical job script information data and using a second similarity algorithm;

[0034] The matrix construction module is configured to: assign weights to the first similarity value and the second similarity value according to the requirements, calculate the weighted sum to obtain the comprehensive similarity value, and construct the similarity adjacency matrix accordingly;

[0035] The job partitioning module is configured to divide HPC jobs into different categories based on the similarity adjacency matrix, with the principle of maximizing modularity index.

[0036] The model training module is configured to train different neural network models based on historical data in different categories after partitioning, thereby obtaining trained neural network models.

[0037] The prediction module is configured to: acquire script information data of the target HPC job, match historical job categories similar to the target HPC job, use a trained neural network model of the historical job category to predict the script information data of the target HPC job, and obtain the prediction result.

[0038] A third aspect of the present invention provides a computer-readable storage medium.

[0039] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the HPC job power prediction method based on power consumption curves and script information as described in the first aspect above.

[0040] A fourth aspect of the present invention provides a computer device.

[0041] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the steps in the HPC job power prediction method based on power consumption curves and script information as described in the first aspect above.

[0042] Compared with the prior art, the beneficial effects of the present invention are:

[0043] Traditional HPC job power consumption prediction methods often rely on simple features or models, failing to fully capture power consumption variations under different job states. This invention uses power consumption curves and script information as guidance, performing training and prediction in three stages to achieve more comprehensive and accurate HPC job power consumption prediction.

[0044] During the model training phase, this invention classifies jobs using the shape of power consumption curves and script information, and trains separately within each category. This classification training method helps to better capture the similarities and differences between different jobs, improving training efficiency and prediction accuracy.

[0045] This invention utilizes job script information to guide model selection, choosing candidate models suitable for the current job. This avoids using unsuitable models, further improving the accuracy of HPC job power consumption prediction. The candidate models are used to predict power consumption curve data, and online learning and adaptive adjustments are performed based on the curve characteristics of the target job. By dynamically adjusting model parameters, the model can better adapt to the power consumption changes of the current job, thereby improving prediction accuracy.

[0046] This invention adjusts and optimizes the model according to actual needs, which makes the model more optimized and flexible, and better suited to different types of HPC jobs, achieving more accurate power consumption prediction targets. Attached Figure Description

[0047] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0048] Figure 1 This is a diagram illustrating the HPC operation power consumption prediction framework of the present invention;

[0049] Figure 2 This is a bar chart illustrating the correlation analysis of indicators as shown in this invention;

[0050] Figure 3 This is a framework diagram of the similarity judgment clustering module shown in this invention;

[0051] Figure 4 This is a diagram illustrating the prediction effect of the method of this invention in operations with large fluctuations.

[0052] Figure 5 This is a diagram illustrating the prediction effect of the method of this invention on phased type operations;

[0053] Figure 6 This is a diagram illustrating the prediction effect of the method of this invention on periodic types of operations. Detailed Implementation

[0054] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0055] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0056] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0057] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and systems according to various embodiments of this disclosure. It should be noted that each block in a flowchart or block diagram may represent a module, segment, or portion of code, which may include one or more executable instructions for implementing the logical functions specified in the various embodiments. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than that shown in the drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, or they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, may be implemented using a dedicated hardware-based system that performs the specified functions or operations, or using a combination of dedicated hardware and computer instructions.

[0058] Example 1

[0059] This embodiment provides a method for predicting HPC job power consumption based on power consumption curves and script information. This embodiment uses the application of this method to a server as an example for illustration. It is understood that this method can also be applied to terminals, and can also be applied to systems including terminals, servers, and other components, and can be implemented through interaction between the terminal and the server. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network servers, cloud communication, middleware services, domain name services, CDN security services, and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In this embodiment, the method includes the following steps:

[0060] Step 1: First, a large amount of HPC job data is collected, including power consumption curves and script information. A compression-based dissimilarity (CBD) method is used, along with assigning different weights to script information similarity, to evaluate the similarity between each pair of jobs, thus identifying the degree of similarity between them. Second, a community detection algorithm based on the Leuven algorithm is introduced. A similarity adjacency matrix is ​​constructed based on comsim, and the modularity between power consumption sequences is calculated. This massive amount of job information is then categorized according to the similarity between jobs. This joint analysis approach allows for a more accurate understanding of the correlation between jobs and provides valuable guidance for subsequent HPC job optimization and scheduling.

[0061] Step 2: After data processing, deep learning methods are used to input these data into the prediction model. Separate training will be performed for different types of tasks to generate offline prediction models, which will then be saved to a model library to prepare for subsequent task power consumption prediction.

[0062] Step 3: For the target job to be predicted, first use the script information to perform a similarity search and match, calculate the job category, and then call the corresponding offline model to predict the power consumption of the HPC job.

[0063] Step 4: During the target job prediction process, the trained historical model is used to predict the power consumption of the target job. To further improve prediction accuracy, especially when using deep learning methods, an online incremental update module is added to enhance prediction accuracy.

[0064] The specific solution of this embodiment will now be described with reference to the accompanying drawings:

[0065] like Figure 1 As shown, this embodiment includes a model preparation stage and a model application stage.

[0066] The model preparation phase, including steps 1 and 2 above, will be explained in detail below:

[0067] Data collection aims to acquire key information about a large number of HPC jobs, providing foundational data for subsequent similarity calculations and classification. The data collection phase primarily covers two aspects of job information: job power consumption curve data and job script information data.

[0068] For the power consumption curve data, several power-related data were collected, including over twenty indicators such as "iowait," "interrupts," "load1," "ctx_switches," "rx_throughput," "read_throughput," "qos_token_latency," and "write_iops." Using statistical methods, a correlation threshold of 0.45 was set, and then the seven features most correlated with the power consumption data were selected. The correlation coefficients were obtained by plotting a bar chart, as shown below. Figure 2 As shown, the following features were ultimately selected as indicators: “cpu_power”, “perf_ddrc_rd_total”, “perf_ddrc_wr_total”, “perf_ips”, “load1”, “perf_mflops”, and “usage”.

[0069] On the other hand, HPC job script information data is collected, including key characteristics such as job ID, job name, user name, queue name, and number of CPU cores requested (AlloNodes). This script information can better reflect the nature and behavior patterns of the job, thus more accurately classifying the job type.

[0070] To ensure data integrity and diversity, the data should cover as many different types and sizes of HPC jobs as possible, encompassing various application areas and tasks. Collecting diverse data allows for more comprehensive and representative job information, thereby enhancing the reliability and accuracy of subsequent similarity calculations and classifications. This collected data will serve as a crucial foundation for power consumption prediction and optimization, providing valuable guidance for improving HPC job performance and effectively utilizing system resources.

[0071] For the task classification module, two similarity determination methods were employed. First, the CBD algorithm was used to calculate the similarity of task power consumption data curves. Second, the KNN algorithm was used to determine the similarity of task script information. The values ​​obtained from these two similarity determinations were assigned different weights and combined to form a comprehensive task similarity index. The framework is as follows: Figure 3 As shown, CBD_sim and script_sim are calculated using curve similarity and job script similarity, respectively. com_sim is the job similarity, which is calculated by assigning different weights to CBD_sim and script_sim.

[0072] Similarity assessment of job power consumption data curves using the CBD algorithm: When using the Compression-Based Dissimilarity Measurement (CBD) method to determine the similarity of job power consumption data curves, the following are the key steps in the implementation process:

[0073] The power consumption curve for each task is encoded, and a compression-based dissimilarity (CBD) method is used to convert continuous power consumption values ​​into discrete symbol sequences. A dictionary is then constructed to record recurring symbol sequences and their replacement keys. Recurring sequences in the symbol sequence of each power consumption curve are replaced with keys from the dictionary, thus achieving power consumption curve compression.

[0074] Calculate the size of each power consumption curve after individual compression, compare the lengths of the compressed sequences, and calculate the compression ratio. The compression ratio represents the ratio of the compressed sequence length to the original sequence length.

[0075] Based on the calculated compression ratio, the similarity between the operation power consumption curves is determined. A higher compression ratio indicates that there are shared patterns and trends, suggesting a higher degree of similarity between the two curves.

[0076] The KNN algorithm determines the similarity of job power consumption script information by including the following steps:

[0077] Collect script information data for HPC jobs, including key characteristics such as job ID, job name, user name, and partition name. This script information reflects the nature and behavior patterns of the job.

[0078] For the script information of the target task, the KNN algorithm is used to calculate its similarity to the script information of historical tasks. The KNN algorithm is a supervised classification algorithm that can be used to calculate the similarity between samples.

[0079] For each target task's script information, KNN similarity is calculated between it and the script information of historical tasks to obtain a similarity score. A higher similarity score indicates that the target task's script information is more similar to the historical tasks, while a lower similarity score indicates that the target task's script information is more different from the historical tasks.

[0080] Different weights are assigned and classifications are performed: two similarity scores are included, one calculated using the CBD algorithm for power consumption curve similarity, and the other calculated using the KNN algorithm for script information similarity. These two similarity scores are assigned different weights to determine whether power consumption curve similarity or script information similarity is prioritized based on actual needs. The two similarity scores are then weighted and summed to obtain the final similarity score. Based on the magnitude of the similarity score, similarity data is provided for subsequent clustering.

[0081] A community detection algorithm based on the Leuven algorithm is used to cluster jobs.

[0082] The similarity values ​​calculated using the CBD algorithm and the job script information similarity algorithm are used to construct a similarity adjacency matrix. In this matrix, each node represents a job, and the elements of the matrix are the similarity values ​​between jobs.

[0083] The Louvain-based community detection method takes a constructed similarity adjacency matrix as input. These algorithms attempt to divide the task into distinct communities to maximize the modularity metric.

[0084] The Leuven algorithm divides jobs into multiple communities based on the principle of modularity optimization. Jobs within each community are considered to be similar to some extent.

[0085] Community detection methods based on the Louvain algorithm aim to maximize the connectivity within communities in a network while minimizing connectivity between communities. The following are the detailed steps of community detection based on the Louvain algorithm:

[0086] a initialization:

[0087] Treat each node as a separate community.

[0088] Calculate the modularity of the initial network.

[0089] b. Iterative optimization:

[0090] For each node, calculate the modularity gain resulting from moving it to an adjacent community. The modularity gain represents the change in overall modularity after moving a node from one community to another.

[0091] Move the node to the community that maximizes the modularity gain, if the gain is positive.

[0092] Continue iterating until no more nodes can be moved.

[0093] c Merging communities:

[0094] After the above iteration is completed, each original community is treated as a separate node, and the network is rebuilt.

[0095] The connection weights between nodes in the new network are recalculated; these weights are typically the sum of the connection strengths between nodes within the original community.

[0096] d. Repeated iterations:

[0097] Repeat steps b and c until the modularity no longer increases or the number of iterations reaches the predetermined value.

[0098] Output of result:

[0099] Output the final community assignment, i.e., which community each node belongs to.

[0100] The modularity value of the partitioned graph is calculated to evaluate the quality of the partitioning result. Modularity measures the difference between actual and expected connections; a higher modularity value indicates a better partitioning result.

[0101] The following formula is used to calculate the modularity value of the partitioned graph:

[0102]

[0103] Where Q is the modularity value, and A ij k represents the connection weight between node i and node j. i and k j The sum of the connection weights represents the degree of nodes i and j, m is the total weight of all connections in the network, and c is the degree of nodes i and j. i and c j It is the community to which nodes i and j belong, δ(c i ,c h ) in c i equals c h The value is 1 if the node is in the same community and 0 otherwise, indicating whether node i and node j are in the same community.

[0104] Through the above steps, the power consumption data curves and script information of the jobs are compared for similarity, and weighted classification is performed based on the similarity scores. This joint analysis approach can more accurately understand the correlation between jobs, and the use of a community detection algorithm based on the Leuven algorithm for job clustering provides useful guidance for subsequent HPC job optimization and scheduling.

[0105] After the data processing phase is completed, deep learning methods are used, with the data that has undergone similarity assessment and classification serving as input to the prediction model. Separate training is performed for different types of jobs, generating offline prediction models for each job type. These models are then saved to a model library to prepare for subsequent job power consumption prediction.

[0106] The specific steps are as follows:

[0107] Data preparation: The job data that has been similarity-based and classified has been organized and processed to serve as input data for the prediction model.

[0108] Classification pre-training: First, for each job category, its corresponding data is used for pre-training. During the pre-training phase, the model will learn to capture the features and patterns of each category.

[0109] Deep learning models: Employ deep learning models suitable for time series data processing, such as Convolutional Neural Networks (CNN), Recurrent Neural Networks (RNN), or Long Short-Term Memory Networks (LSTM). These models excel at processing sequential data, effectively capturing temporal relationships and patterns within the data.

[0110] Category-specific training: Training the model on data for each job category. For each category's model, a deep learning training process is performed using data from the corresponding category.

[0111] Model saving: After training is complete, the model for each job category is saved to the model library. This is done to provide fast and efficient access for subsequent job power consumption prediction, as well as to facilitate management and maintenance.

[0112] The deep learning methods described above can generate offline prediction models for different types of jobs. These models can provide accurate predictions for subsequent job power consumption. Based on actual conditions, these models can be further optimized and updated as needed to adapt to new data and scenarios, achieving even higher accuracy in power consumption prediction. This deep learning framework can provide important support and guidance for the management and optimization of HPC systems, helping to improve the utilization efficiency of computing resources and overall performance.

[0113] Model application phase: The model application phase includes steps 3 and 4, and the specific implementation steps are as follows:

[0114] Prediction of target assignments: For the predicted target assignments, the script information is first used to perform a similarity search and matching to find historical assignment categories that are similar to the target assignments.

[0115] The specific steps are as follows:

[0116] Target job script information: First, collect the script information of the target job, including key features such as job ID, job name (JobName), username (User), queue name (Partition), and number of CPU cores requested (AlloNodes).

[0117] Similarity search matching: Next, the script information of the target job is compared with the script information of historical jobs using similarity search matching methods. This similarity search matching can be based on methods such as KNN algorithm and cosine similarity.

[0118] Job category calculation: Through similarity search matching, find the historical job category that is most similar to the target job. Classifying the target job into a similar historical job category can better predict its power consumption.

[0119] Invoking the offline model: Once the target job is classified into a specific historical job category, the corresponding offline prediction model for that category will be invoked. These prediction models were obtained during the data processing and deep learning training phases.

[0120] Power consumption prediction: After calling the corresponding offline model, the power consumption curve data of the target job is input into the prediction model to perform power consumption prediction. These prediction models have been optimized for different types of jobs in the previous training phase, so they can provide relatively accurate power consumption prediction results.

[0121] By following the steps above, similarity search and matching can be performed on the target job using script information to find its corresponding historical job category. Then, the corresponding offline model can be invoked to predict the power consumption of the HPC job. This prediction method can provide useful guidance for the management and resource scheduling of HPC systems, improving job efficiency and system resource utilization.

[0122] Model parameter updates during prediction: During the target job prediction process, pre-trained historical models are used to predict the power consumption of the target job. These historical models are obtained through data processing and deep learning methods, and are capable of accurate power consumption prediction for different types of jobs.

[0123] To further improve prediction accuracy, especially when using deep learning methods, an online incremental update module has been added. The purpose of this module is to adjust and optimize historical models in real time to adapt to new data and changing job characteristics.

[0124] The specific steps are as follows:

[0125] Real-time data updates: As new job data is continuously generated during the target job prediction process, this real-time data is input into the historical model.

[0126] Online Incremental Updates: The online incremental update module continuously optimizes the parameters and weights of the historical model based on real-time input data. Through this incremental update method, the historical model can continuously learn new job characteristics and patterns, thereby better adapting to constantly changing job behaviors.

[0127] Dynamic adjustment: As data is continuously updated and the model is dynamically adjusted, the predictive ability of the historical model will be continuously improved. This dynamic adjustment and online incremental update method can keep the predictive model closer to the actual situation, improving the accuracy and reliability of predictions.

[0128] By introducing an online incremental update module, historical models can continuously learn and adapt to new data, thereby further improving prediction accuracy. This dynamic prediction method can better cope with changes in job behavior and real-time data updates, providing more accurate and efficient prediction results for HPC system management and resource scheduling, thus optimizing system performance and resource utilization efficiency.

[0129] The prediction results of a three-stage HPC job power prediction deep learning framework jointly guided by power consumption curve shape and script information are shown in the figure for several different jobs. Figure 4 , Figure 5 , Figure 6 As shown in the figure, the predicted HPC power consumption using the method of the present invention is basically consistent with the actual value, proving the accuracy of the method of the present invention.

[0130] Example 2

[0131] This embodiment provides an HPC job power prediction system based on power consumption curves and script information.

[0132] HPC job power prediction system based on power consumption curves and script information, including:

[0133] The data acquisition module is configured to acquire historical job power consumption curve data and historical job script information data.

[0134] The first similarity calculation module is configured to: obtain a first similarity value based on historical operation power consumption curve data and using a first similarity algorithm;

[0135] The second similarity calculation module is configured to: obtain a second similarity value based on historical job script information data and using a second similarity algorithm;

[0136] The matrix construction module is configured to: assign weights to the first similarity value and the second similarity value according to the requirements, calculate the weighted sum to obtain the comprehensive similarity value, and construct the similarity adjacency matrix accordingly;

[0137] The job partitioning module is configured to divide HPC jobs into different categories based on the similarity adjacency matrix, with the principle of maximizing modularity index.

[0138] The model training module is configured to train different neural network models based on historical data in different categories after partitioning, thereby obtaining trained neural network models.

[0139] The prediction module is configured to: acquire script information data of the target HPC job, match historical job categories similar to the target HPC job, use a trained neural network model of the historical job category to predict the script information data of the target HPC job, and obtain the prediction result.

[0140] It should be noted that the data acquisition module, the first similarity calculation module, the second similarity calculation module, the matrix construction module, the job partitioning module, the model training module, and the prediction module described above are the same examples and application scenarios implemented in the steps of Embodiment 1, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0141] Example 3

[0142] This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the HPC job power consumption prediction method based on power consumption curves and script information as described in Embodiment 1 above.

[0143] Example 4

[0144] This embodiment provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the HPC job power consumption prediction method based on power consumption curves and script information as described in Embodiment 1 above.

[0145] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0146] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0147] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0148] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0149] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0150] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for predicting HPC job power consumption based on power consumption curves and script information, characterized in that, include: Acquire historical job power consumption curve data and historical job script information data; Based on historical operation power consumption curve data, the first similarity algorithm is used to obtain the first similarity value; The process of obtaining a first similarity value based on historical operation power consumption curve data and using a first similarity algorithm includes: selecting several power consumption-related indicators based on historical operation power consumption curve data, and using the CBD algorithm based on the indicators to obtain a first similarity value; wherein the CBD algorithm is a compressed dissimilarity measurement method. Based on historical assignment script information data, a second similarity algorithm is used to obtain a second similarity value; Based on the requirements, weights are assigned to the first similarity value and the second similarity value, and a weighted sum is calculated to obtain the comprehensive similarity value, thereby constructing a similarity adjacency matrix; A community detection algorithm based on the Leuven algorithm is used to cluster jobs. The similarity values ​​calculated using the CBD algorithm and the job script information similarity algorithm are used to construct a similarity adjacency matrix; in this matrix, each node represents a job, and the elements of the matrix are the similarity values ​​between jobs; A community detection method based on the Leuven algorithm is used, with the constructed similarity adjacency matrix as input; the task is divided into different communities to maximize the modularity index. A community detection algorithm based on the Leuven algorithm is used to divide HPC jobs into multiple communities, each community representing a category, indicating that HPC jobs within a community are similar to some extent. Based on the principle of maximizing modularity, HPC jobs are divided into different categories according to the similarity adjacency matrix; The community detection method based on the Leuven algorithm aims to maximize the connection strength within communities in a network while minimizing the connections between communities. Based on historical data in different categories after division, different neural network models are trained to obtain trained neural network models; Obtain the script information data of the target HPC job, match it with historical job categories similar to the target HPC job, and use the trained neural network model of the historical job category to predict the script information data of the target HPC job to obtain the prediction result.

2. The HPC job power consumption prediction method based on power consumption curves and script information according to claim 1, characterized in that, The process of using the CBD algorithm includes: Each HPC job power consumption curve is encoded, and the CBD algorithm is used to convert continuous power consumption values ​​into discrete symbol sequences. A dictionary is constructed to record the recurring symbol sequences and replacement keys. The recurring sequences in the symbol sequences of each job power consumption curve are replaced with the keys in the dictionary to compress the job power consumption curve. Calculate the size of each job power consumption curve after individual compression, compare the lengths of the compressed sequences, and calculate the compression ratio, where the compression ratio represents the ratio of the length of the compressed sequence to the length of the original sequence. Based on the calculated compression ratio, the similarity between the power consumption curves of the operation is determined.

3. The HPC job power consumption prediction method based on power consumption curves and script information according to claim 1, characterized in that, The process of obtaining a second similarity value based on historical task script information data and using a second similarity algorithm includes: using the KNN algorithm to calculate the similarity between historical task script information data and obtaining a second similarity value.

4. The HPC job power consumption prediction method based on power consumption curves and script information according to claim 1, characterized in that, After the communities are divided, the modularity value of the divided communities is calculated to evaluate the division results.

5. The HPC job power consumption prediction method based on power consumption curves and script information according to claim 1, characterized in that, The prediction process of a trained neural network model also includes optimizing the parameters and weights of the neural network model based on real-time input data.

6. An HPC job power consumption prediction system based on power consumption curves and script information, characterized in that, include: The data acquisition module is configured to acquire historical job power consumption curve data and historical job script information data. The first similarity calculation module is configured to: obtain a first similarity value based on historical operation power consumption curve data and using a first similarity algorithm; The process of obtaining a first similarity value based on historical operation power consumption curve data and using a first similarity algorithm includes: selecting several power consumption-related indicators based on historical operation power consumption curve data, and using the CBD algorithm based on the indicators to obtain a first similarity value; The second similarity calculation module is configured to: obtain a second similarity value based on historical job script information data and using a second similarity algorithm; The matrix construction module is configured to: assign weights to the first similarity value and the second similarity value according to the requirements, calculate the weighted sum to obtain the comprehensive similarity value, and construct the similarity adjacency matrix accordingly; A community detection algorithm based on the Leuven algorithm is used to cluster jobs. The similarity values ​​calculated using the CBD algorithm and the job script information similarity algorithm are used to construct a similarity adjacency matrix; in this matrix, each node represents a job, and the elements of the matrix are the similarity values ​​between jobs; A community detection method based on the Leuven algorithm is used, with the constructed similarity adjacency matrix as input; the task is divided into different communities to maximize the modularity index. A community detection algorithm based on the Leuven algorithm is used to divide HPC jobs into multiple communities, each community representing a category, indicating that HPC jobs within a community are similar to some extent. The job partitioning module is configured to divide HPC jobs into different categories based on the similarity adjacency matrix, with the principle of maximizing modularity index. The community detection method based on the Leuven algorithm aims to maximize the connection strength within communities in a network while minimizing the connections between communities. The model training module is configured to train different neural network models based on historical data in different categories after partitioning, thereby obtaining trained neural network models. The prediction module is configured to: acquire script information data of the target HPC job, match historical job categories similar to the target HPC job, use a trained neural network model of the historical job category to predict the script information data of the target HPC job, and obtain the prediction result.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the HPC job power prediction method based on power curves and script information as described in any one of claims 1-5.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the HPC job power prediction method based on power curves and script information as described in any one of claims 1-5.

Images