An I / O feature prediction method, a model training method, and related devices

Abstract

The application provides an I / O feature prediction method, a model training method and related devices. The job submission script of a job is converted into a grayscale image, and a deep learning method of image recognition is used to realize the prediction task of the I / O feature. When the I / O feature is predicted, the processing method accumulated in the field of deep learning can be reused, including various classical image recognition network structures, to process the above-mentioned grayscale image to obtain the I / O feature of the job. Compared with the existing method of using a "job name-user name-computing node number" triplet as the I / O feature and using a clustering algorithm to process a large-scale data set for prediction, the I / O feature can be predicted based on the job submission script, the time consumption of the prediction of the I / O feature is reduced, and the I / O feature of the job in the high-performance computing cluster is accurately and efficiently predicted.

Description

Technical Field

[0001] This application relates to the field of communications, and in particular to an I / O feature prediction method, a model training method, and related apparatus. Background Technology

[0002] With the rapid development of data-intensive applications, job I / O load has become a key indicator to monitor during job scheduling in high-performance computing clusters. If job I / O characteristics can be effectively predicted, resource allocation within the high-performance computing cluster can be optimized based on these characteristics, reducing job execution time and improving cluster utilization.

[0003] Known job I / O feature prediction methods often ignore or simplify job submission scripts during feature engineering. For example, they might only use the job-related triplet "job name-username-number of compute nodes" as I / O features for prediction. This leads to feature loss and limitations in the prediction method, resulting in I / O feature predictions that are often qualitative and inaccurate, making them unsuitable for complex use cases. Furthermore, existing I / O feature prediction methods typically employ clustering algorithms, which have poor applicability and are difficult to port to new computing platforms. Moreover, using clustering algorithms to process large-scale datasets during model training and inference requires calculating the distance between each sample and other samples, resulting in high computational costs and increasing the prediction time for I / O features.

[0004] Therefore, how to accurately and efficiently predict the I / O characteristics of high-performance computing cluster jobs in order to optimize resource allocation, reduce job execution time, and improve the utilization rate of high-performance computing clusters has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides an I / O feature prediction method, a model training method, and related apparatus. By converting the job submission script into a grayscale image, a deep learning method for image recognition is used to predict I / O features. This method can more accurately and efficiently predict the I / O features of jobs in high-performance computing clusters, thereby optimizing resource allocation, reducing job execution time, and improving the utilization rate of I / O resources in high-performance computing clusters.

[0006] In a first aspect, this application provides an I / O feature prediction method applied to a high-performance computing cluster, which is used to run multiple jobs. The method includes: obtaining a job submission script of a first job, wherein the first job is one of multiple jobs; converting the job submission script into at least one grayscale image; and inputting the at least one grayscale image into an I / O feature prediction model to obtain the I / O features of the first job.

[0007] In this application, the job submission script is converted into a grayscale image, and a deep learning method for image recognition is used to predict I / O features. When predicting I / O features, existing processing methods in the field of deep learning, including various classic image recognition network structures, can be reused to process the grayscale image and obtain the job's I / O features. This ability to predict I / O features based on the job submission script reduces the prediction time, achieving accurate and efficient prediction of job I / O features. This optimizes resource allocation in high-performance computing clusters, reduces job execution time, and improves the utilization of high-performance computing clusters.

[0008] In some embodiments, the job submission script includes K lines of text characters, each grayscale image corresponds to an M-line two-dimensional matrix, and converting the job submission script into at least one grayscale image includes: obtaining a line of text characters from the K lines, converting the line of text characters into a line of integer elements in the two-dimensional matrix, until the K lines of text characters are converted into at least one M-line two-dimensional matrix.

[0009] In this application, the parameters in each row of the job submission script, representing the number of nodes, cores, graphics processors, job instance, job queue, program startup parameters, job start time, and job end time, are transformed into a row of integer elements in a two-dimensional matrix. This allows the I / O feature prediction model to process the data using grayscale images, resulting in more accurate and efficient prediction of the I / O features of jobs in high-performance computing clusters.

[0010] In some embodiments, inputting the grayscale image into an I / O feature prediction model to obtain the I / O features of the first job includes: inputting the at least one two-dimensional matrix into the I / O feature prediction model, and using the I / O feature prediction model to predict the I / O features of the first job based on the grayscale values ​​of the integer elements. The I / O features include one or more of the read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate of the first job.

[0011] In this application, when predicting I / O features, the processing methods accumulated in the field of deep learning, including various classic image recognition network structures, can be reused to process the grayscale image to obtain the I / O features of the job, thereby reducing the prediction time of I / O features and realizing accurate and efficient prediction of the I / O features of jobs in high-performance computing clusters.

[0012] In some embodiments, the high-performance computing cluster includes a job scheduling system, and when the I / O feature prediction model is deployed in the job scheduling system, the method further includes: loading the job submission script of the first job or delaying the loading of the first job based on the I / O features of the first job.

[0013] In this application, by deploying an I / O feature prediction model in a job scheduling system, a global I / O view of currently running jobs and jobs to be scheduled can be obtained using the I / O feature prediction model. When making scheduling decisions, the computing and storage resources in the high-performance computing cluster can be utilized as fully as possible while meeting the constraints of throughput and metadata read / write capabilities of the high-performance computing cluster system. This optimizes resource allocation within the high-performance computing cluster and improves its utilization rate.

[0014] In some embodiments, the high-performance computing cluster includes an intelligent storage system, and when the I / O feature prediction model is deployed in the intelligent storage system, the method further includes: storing the working data of the first job in a cold storage device or a hot storage device based on the I / O features of the first job.

[0015] In this application, by deploying the I / O feature prediction model in the intelligent storage system, the complete time series of the I / O features of the first job can be obtained using the I / O feature prediction model. This provides a more accurate basis for classifying hot and cold data in the working directory, adjusting caching strategies, and data layout, so as to optimize the resource allocation of the high-performance computing cluster and improve the utilization rate of the high-performance computing cluster.

[0016] In some embodiments, the high-performance computing cluster includes a job fault diagnosis system. When the I / O feature prediction model is deployed in the job fault diagnosis system, the method further includes: determining the working status of the high-performance computing cluster based on the I / O features of multiple jobs and the performance data of multiple jobs.

[0017] In this application, by deploying the I / O feature prediction model in the job fault diagnosis system, it is possible to obtain the I / O throughput pattern sequence of multiple jobs under ideal conditions using the I / O feature prediction model, and to judge the I / O mode of the current job based on the I / O throughput pattern sequence under ideal conditions, thereby improving the utilization rate of the high-performance computing cluster.

[0018] In some embodiments, the high-performance computing cluster includes a task profiling and cluster design system. When the I / O feature prediction model is deployed in the task profiling and cluster design system, the method further includes: determining the ratio of front-end and back-end servers, solid-state drives, and hard disk drives in the high-performance computing cluster based on the I / O features of multiple jobs.

[0019] In this application, an I / O feature prediction model is deployed in a task profiling and cluster design system. The model is used to predict and collect the I / O features of user-submitted jobs. Based on the I / O features, the application distribution and job characteristics of high-performance computing cluster users are determined. The aggregated application distribution and job characteristics of system users guide the optimization of the user's HPC design and the cluster design, thereby improving the utilization rate of the high-performance computing cluster.

[0020] Secondly, embodiments of this application provide a training method for an I / O feature prediction model. The method includes: acquiring the job submission script, I / O performance data, and job start and end times of a current job; the I / O performance data includes performance information of read bandwidth, write bandwidth, read metadata, and write metadata; obtaining a first grayscale image based on the job submission script of the current job; determining a feature vector of the current job based on the I / O performance data; determining a feature dataset based on the first grayscale image, the feature vector of the current job, and the job start and end times; inputting the feature dataset into a neural network model for pre-training to obtain a first machine learning model, the first machine model being used to predict the runtime of the current job; acquiring the job submission script of a target job; converting the job submission script of the target job into a second grayscale image; inputting the second grayscale image into the first machine model, and retraining the output layer of the first machine model to obtain an I / O feature prediction model, the I / O feature prediction model being used to predict the I / O features of the target job, the I / O features including one or more of the read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate of the target job.

[0021] In this application, the job submission script is converted into a grayscale image, and a deep learning method for image recognition is used to train a neural network model. The neural network model automatically extracts I / O features related to the I / O feature vector from the current job submission script, eliminating the need for manual feature engineering. This method fully preserves the information in the current job submission script, reducing information loss and noise interference caused by human factors, and achieving more accurate feature learning. Furthermore, based on the first machine learning model, this training method retrains the output layer based on the target job's submission script to adapt to prediction tasks with different I / O features. This optimizes the generalization ability of the neural network model, improving its adaptability and flexibility, allowing the model to quickly transfer to new prediction tasks while maintaining training efficiency.

[0022] In some embodiments, the step of inputting the training dataset into a neural network model for pre-training to obtain a first machine learning model includes: inputting the training dataset into the neural network model; using the neural network model to output the prediction result of the current task; and using the prediction result of the current task and a loss function to train the neural network model to obtain a trained first machine learning model, wherein the loss function is used to indicate the deviation between the prediction result of the current task and the start and end times of the current task.

[0023] In this application, a neural network model is trained using a deep learning method for image recognition on a first grayscale image. The neural network model automatically extracts I / O features related to the I / O feature vector from the current job submission script without the need for manual feature engineering. This method can completely preserve the information in the current job submission script, reduce information loss and noise interference caused by human factors, and achieve more accurate feature learning.

[0024] Thirdly, embodiments of this application provide an I / O feature prediction apparatus, the apparatus comprising: a communication module for acquiring a job submission script of a target job input by a user; a processing module for converting the job submission script into at least one grayscale image; the processing module is further configured to input the at least one grayscale image into an I / O feature prediction model to obtain the I / O features of the job.

[0025] In some embodiments, when the job submission script includes K lines of text characters and each grayscale image corresponds to an M-line two-dimensional matrix, the processing module is specifically used to: obtain a line of text characters from the K lines, convert the line of text characters into a line of integer elements in the two-dimensional matrix, until the K lines of text characters are converted into at least one M-line two-dimensional matrix.

[0026] In some embodiments, the processing module is specifically configured to: input the at least M-row two-dimensional matrix into an I / O feature prediction model, and use the I / O feature prediction model to predict the I / O features of the first job based on the grayscale values ​​of the integer elements. The I / O features include one or more of the read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate of the first job.

[0027] Fourthly, this application provides a high-performance computing cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the high-performance computing cluster performs the aforementioned I / O feature prediction method.

[0028] Fifthly, this application provides a computer program product containing instructions that, when executed by a high-performance computing cluster, cause the high-performance computing cluster to perform the aforementioned I / O feature prediction method.

[0029] Sixthly, this application provides a computer-readable storage medium including computer program instructions, which, when executed by a high-performance computing cluster, perform the aforementioned I / O feature prediction method.

[0030] It should be understood that the beneficial effects of the technical solutions and corresponding possible implementations of the third, fourth, fifth and sixth aspects of the embodiments of this application can be referred to the above-described technical effects of the first and second aspects, and will not be repeated here. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a job scheduling scenario in a high-performance computing cluster provided in an embodiment of this application;

[0032] Figure 2 This is a schematic diagram of the assignment submission script visualized as a two-dimensional matrix;

[0033] Figure 3 This is a schematic diagram illustrating the determination of a job's I / O feature vector based on its performance data.

[0034] Figure 4 This is a schematic diagram of the feature dataset used to generate the job;

[0035] Figure 5 This is a schematic diagram of the training scenario for the ResNet-50 feature extraction network;

[0036] Figure 6 This is a schematic diagram of the first machine learning transfer scenario;

[0037] Figure 7 This is a schematic diagram illustrating the application scenario of the second machine learning model;

[0038] Figure 8 This is a flowchart illustrating the I / O feature prediction method;

[0039] Figure 9 This is a flowchart illustrating the training method for the I / O feature prediction model.

[0040] Figure 10 This is a schematic diagram of the structure of the I / O feature prediction device;

[0041] Figure 11 This is a schematic diagram of the architecture of a high-performance computing cluster. Detailed Implementation

[0042] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0043] To facilitate understanding of the technical solution of this application, the relevant terms used in this document are explained below.

[0044] High-performance computing (HPC) clusters refer to computing systems that connect multiple computing devices (usually called nodes) together to solve problems requiring large amounts of computing resources through collaborative work. They are widely used in scientific research, engineering computing, financial modeling, big data analysis, and other fields. HPC clusters typically employ a job scheduling system to manage user-submitted jobs and cluster system resources. When a job is submitted to the cluster, it is assigned a priority and queued. The job scheduling system schedules jobs based on their priority and requested resources, and monitors the entire lifecycle of the job.

[0045] The job scheduling system is a core component of high-performance computing cluster management. It is responsible for accepting user-submitted jobs, using the intelligent storage system scheduling engine to match suitable resources to the jobs, launch the jobs, and monitor task execution. During job execution, the job scheduling system can also utilize the performance monitoring system to obtain real-time job status and provide feedback to the user upon job completion.

[0046] The intelligent storage system scheduling engine is responsible for optimizing the allocation and management of storage resources in high-performance computing clusters to improve data processing efficiency and performance. It enables large-scale automated scheduling, management, monitoring, and reporting of HPC workloads. The intelligent storage system scheduling engine uses multi-dimensional strategies and advanced future modeling to optimize workload startup and runtime across various resources.

[0047] Performance monitoring systems are crucial tools for ensuring the efficient operation of high-performance computing clusters and optimizing resource allocation. They provide graphical displays of overall cluster performance and individual node performance, offering monitoring of metrics such as disk capacity, load, memory, CPU, and network I / O. Performance monitoring systems can also help job scheduling systems optimize cluster resource allocation, including task scheduling, resource management, and load balancing, to ensure efficient utilization of the high-performance computing cluster and rapid task response.

[0048] In high-performance computing clusters, cluster I / O performance is also one of the key factors affecting overall system performance. With the rapid improvement of computing power of high-performance computing clusters, the I / O performance bottleneck of storage systems has become increasingly prominent. Furthermore, with the rapid development of data-intensive applications, such as big data analytics and artificial intelligence, these applications typically rely on massive parallel processing and complex I / O operations to process and analyze massive amounts of data, placing significant demands on cluster storage I / O.

[0049] However, traditional high-performance computing cluster job scheduling systems primarily aim to optimize computing resource utilization and job throughput, lacking effective predictive and control methods for scheduling and managing job I / O resources. If a large number of high I / O demanding jobs are executed by the scheduling system solely based on the amount of hardware resources they require, it will cause I / O load conflicts in the high-performance computing cluster.

[0050] Therefore, in addition to resource quantity and job throughput, job scheduling systems should also consider the I / O load of the cluster and jobs. Effectively predicting job I / O characteristics can help optimize resource allocation in high-performance computing clusters, reduce job execution time, and improve cluster utilization. Developing a method to accurately predict job I / O characteristics is of great significance.

[0051] One known method for predicting I / O burst distribution uses the triple "job name-username-number of compute nodes" as job features. It can use an unsupervised clustering method to divide the jobs into groups, and then use the I / O burst interval and runtime of the first job in the group as the prediction result for the I / O burst interval and runtime of the group.

[0052] However, the above-mentioned methods for job I / O features often ignore or simplify the job submission script during feature engineering, such as only taking the "job name-username-number of compute nodes" triple. This results in the loss of job features and limitations of the prediction method. The I / O prediction results of this method are often in a "qualitative" form, which limits its application scenarios.

[0053] A known multi-tenant resource optimization scheduling method for different types of loads calculates the data read / write rate during the job's Map-Shuffle phase. If the data read / write rate exceeds the disk I / O bandwidth, the job is classified as I / O-intensive; otherwise, it is classified as compute-intensive. Based on this classification, the method employs two resource optimization scheduling approaches: 1) Directly schedule and execute compute-intensive jobs on the current node; 2) Perform delayed scheduling for I / O-intensive jobs. This method maintains job locality as much as possible while distributing computational tasks across nodes in the cluster, fully utilizing cluster resources to achieve distributed load balancing.

[0054] However, the algorithms used in the above methods have poor applicability and are difficult to port to new computing platforms. Because these methods lack "transfer" or "fine-tuning" design, when deploying to new systems for job I / O feature prediction, it is necessary to first accumulate millions or even tens of millions of high-quality job samples and then train the model from scratch; otherwise, it is difficult to achieve good results. Limited by platform runtime and management experience, new systems in high-performance computing clusters cannot quickly accumulate this data, and therefore cannot optimize cluster resources from a job I / O perspective.

[0055] One known I / O request prediction method calculates the average number of received requests and the average number of completed requests within a preset time period by acquiring and using the number of received I / O requests and the number of completed I / O requests within a preset period. Then, based on the preset period, the average number of received requests, and the average number of completed requests, the method predicts the average processing time for the storage node to handle the next I / O request. Based on the prediction results, this method can further distribute I / O requests reasonably according to the predicted average processing time, ultimately achieving the ideal overall performance of the distributed storage system.

[0056] However, since this method uses a clustering algorithm to process large-scale datasets, each sample has to calculate the distance to other samples during model training and inference, which is computationally expensive and increases the prediction time of I / O features.

[0057] Therefore, how to accurately and efficiently predict the I / O characteristics of high-performance computing cluster jobs in order to optimize resource allocation, reduce job execution time, and improve the utilization rate of high-performance computing clusters has become an urgent technical problem to be solved.

[0058] To more accurately and efficiently predict the I / O characteristics of high-performance computing (HPC) clusters, thereby optimizing resource allocation, reducing job execution time, and improving I / O resource utilization, this application proposes an I / O characteristic prediction method, a model training method, and related apparatus.

[0059] To facilitate understanding of the technical solution of this application, an application scenario of an embodiment of this application will be introduced first below.

[0060] For example, Figure 1 This is a schematic diagram illustrating a job scheduling scenario in a high-performance computing cluster, as provided in an embodiment of this application. Figure 1 As shown, the high-performance computing cluster includes a job scheduling system 100, an intelligent storage system 200, a computing cluster 300, and a performance monitoring system 400.

[0061] The job scheduling system 100 includes multiple jobs input by the user. These jobs are submitted through job submission scripts. Each job submission script corresponds to one job and includes the job's configuration information, such as job name, required resources (number of CPU cores, memory size, running time, etc.), the program to be executed, and input / output files.

[0062] In some possible implementations, the job scheduling system 100 is also used to obtain the user identifier of the aforementioned user.

[0063] The job scheduling system 100 schedules multiple jobs based on the configuration information in the job submission script and the current resource status of the computing cluster 300 obtained from the intelligent storage system 200. It allocates computing resources to each job and dispatches the job submission script to the computing cluster 300 to execute the job corresponding to each job submission script.

[0064] It is worth noting that, in this application, the job scheduling system used by the high-performance computing cluster can be the Slurm job scheduling system, or one or more of the following: a load-sharing facility (LSF), a portable batch system (PBS), or a high-throughput computing (HTC) job scheduling system. In other words, this application does not limit the specific type of job scheduling system used by the high-performance computing cluster.

[0065] The intelligent storage system 200 includes a scheduling engine 210, which is used to obtain the current resource status in the computing cluster 300 and send the current resource status to the job scheduling system 100. In some possible implementations, the scheduling engine 210 is also used to dynamically adjust the storage resource scheduling plan of the job based on the job's real-time data and feedback. For example, it can schedule "cold" and "hot" data of the job.

[0066] In some possible implementations, for a specific storage system such as Lustre, the scheduling engine 210 can also be used to record the storage performance of the job and output fine-grained I / O information of the output job.

[0067] The computing cluster 300 includes multiple computing devices to provide computing resources for the computing tasks of the high-performance computing cluster, in order to run user-submitted jobs.

[0068] In some possible implementations, the computing cluster 300 also includes a hot storage device and a cold storage device, the hot storage device being used to store frequently accessed "hot" data and the cold storage device being used to store infrequently accessed "cold" data.

[0069] The performance monitoring system 400 is used to monitor the nodes running jobs in the computing cluster 300 and obtain the I / O performance data corresponding to the jobs at predetermined intervals. For example, the node identifier of at least one node running the job, the I / O performance data of the aforementioned at least one node, and the read throughput performance, write throughput performance, read metadata performance, and write metadata performance of each node at the job level, obtained by sampling at a cycle of 3 minutes.

[0070] After a job completes execution, the performance monitoring system 400 also records the start and end times of that job. The I / O performance data corresponding to these jobs, along with their start and end times, are typically recorded in the job log file of the high-performance computing cluster.

[0071] In some possible implementations, for a job executed by multiple nodes, the job's I / O performance data includes the sum of the I / O performance data of the multiple nodes.

[0072] For illustrative purposes only and not as a limitation, the performance monitoring system 400 in this application can be a performance monitoring system implemented based on Ganglia or a performance monitoring system implemented based on Prometheus. In other words, this application does not limit the specific implementation method of the performance monitoring system.

[0073] To accurately and efficiently predict the I / O characteristics of a job, this application converts the text-represented job submission script into a grayscale image, enabling the use of deep learning methods for image recognition to achieve the aforementioned I / O characteristic prediction task. For example, a neural network model can be used to predict the I / O characteristics of the job.

[0074] As mentioned above, this application can obtain multiple job submission scripts submitted by users through the job scheduling system 100. The high-performance computing cluster can convert the above job submission scripts into grayscale images and determine the training dataset for the neural network model based on the grayscale images.

[0075] To improve the accuracy of I / O feature prediction, the training dataset also includes I / O feature vectors associated with each job submission script, as well as the start and end times of the job.

[0076] In some possible implementations, the training dataset described above may also include fine-grained I / O information for the jobs. For example, metadata operation counts, current disk I / O pending counts, and distribution of Remote Procedure Call (RPC) request sizes.

[0077] It is worth noting that this application can use a Residual Network (ResNet) to recognize the aforementioned grayscale images to obtain predicted I / O features, or it can use Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), or Capsule Networks to recognize the aforementioned grayscale images to obtain predicted I / O features. In other words, this application does not limit the type of neural network model used for grayscale image recognition.

[0078] Next, we will introduce the grayscale image converted from the assignment submission script.

[0079] A grayscale image is an image type where each pixel's value represents its grayscale or brightness, typically ranging from 0 (black) to 255 (white). A grayscale image can be represented as a two-dimensional matrix, where each element of the matrix corresponds to a pixel in the image, and the element's value represents that pixel's grayscale value. For example, if an element's value is 0, the corresponding pixel is black; if the element's value is 255, the corresponding pixel is white; and if the element's value is between 0 and 255, the corresponding pixel is a different shade of gray.

[0080] Take the job submission script of the Slurm job scheduling system as an example.

[0081] For example, Figure 2 This is a schematic diagram of the assignment submission script visualized as a two-dimensional matrix, as shown on the left side of the figure. The assignment submission script includes multiple lines of text strings.

[0082] The text character “#! / bin / bash” indicates that the job scheduling system is set to use “! / bin / bash” as the command interpreter.

[0083] The text string "#SBATCH-J FIRE" indicates that the job corresponding to the job submission script is identified as "FIRE".

[0084] The text character "#SBATCH-p normal" indicates that the job queue corresponding to the job submission script is "normal".

[0085] The text character "#SBATCH-N 1" indicates that the number of nodes in the job request corresponding to the job submission script is "1".

[0086] The text character "#SBATCH-n 32" indicates that the number of cores in the job request corresponding to the job submission script is "32".

[0087] The text string "#SBATCH--gres=gpu:2" indicates that the number of Graphics Processing Units (GPUs) requested by the job submission script is "2".

[0088] The text string "echo${SLURM_JOB_NODELIST}" indicates that the list of nodes occupied by the job corresponding to the job submission script is "SLURM_JOB_NODELIST".

[0089] The text string "echo start on $(date)" indicates the start time of the job corresponding to the job submission script.

[0090] The text string "echo end on $(date)" indicates the end time of the job corresponding to the job submission script.

[0091] As mentioned earlier, if the element value of the matrix is ​​0, the corresponding pixel is black; if the element value is 255, the corresponding pixel is white; if the element value is between 0 and 255, the corresponding pixel is a different shade of gray.

[0092] The ASCII codes for "1" and "32" in the text characters "#SBATCH-N 1" and "#SBATCH-n 32" are "u0031" and "u0033 u0032" respectively. Therefore, after the job submission script is converted into a grayscale image, the machine learning model can identify the size of the computing resources requested by the job based on the grayscale values ​​represented by the ASCII codes.

[0093] This application can convert each line of the job submission script into a row of a matrix in sequence, and the characters in each line of the script are converted into integer elements in the matrix in sequence, with the integer element value being the ASCII code of the character.

[0094] As an illustration, the text characters "#! / bin / bash" in the first line can be converted into a line of integer elements "a11a12 a13……a1n" in ASCII code. Each element represents an ASCII code value.

[0095] Based on the above method, a job submission script can be converted into a 256x256 integer matrix as shown on the right side of the figure.

[0096] If the size of a submitted script is less than 256x256, then a 256x256 integer matrix can be filled with 0s.

[0097] If the size of a job submission script exceeds 256x256 after conversion, the integer matrix will be truncated to 256x256, and the remaining content will be converted into another 256x256 integer matrix, until the job submission script is completely converted into at least one 256x256 integer matrix.

[0098] It is worth noting that the size of the matrix described above is intended to facilitate understanding of this application. In actual implementation, other sizes of two-dimensional integer matrices can also be used, such as 128x128 or 1024*512. In other words, this application does not limit the size of the matrix converted by the job submission script.

[0099] As mentioned earlier, this application can obtain I / O performance data for each job through the performance monitoring system 400. The high-performance computing cluster can process the I / O performance data to obtain the I / O feature vector corresponding to the job. The I / O feature vector will be introduced next.

[0100] For example, Figure 3 This is a schematic diagram illustrating the determination of a job's I / O feature vector based on its performance data. As shown in the diagram, the job's performance data includes information obtained by the performance monitoring system 400 at predetermined intervals, covering the read bandwidth performance, write bandwidth performance, read metadata performance, and write metadata performance of at least one node executing the job. It can be understood that the aforementioned job performance data may also include the node identifier corresponding to at least one node.

[0101] First, the high-performance computing cluster can clean outliers from the job record set, constructing an I / O feature dataset with a sampling interval of 3 minutes. It then removes job samples whose runtime exceeds the cluster's duration limit and those that terminated abnormally. For example, it removes job samples whose runtime exceeds 7 days.

[0102] Then, the high-performance computing cluster can use the Z-score method to detect outliers in I / O performance and calculate the Z-score for the corresponding values ​​of the same application job. For example, the threshold for the Z-score can be set to 3 to filter data points whose absolute Z-score value is greater than the set threshold.

[0103] Finally, the high-performance computing cluster can encode the I / O feature vectors in the samples into four floating-point numerical vectors based on four attributes: read throughput performance, write throughput performance, read metadata performance, and write metadata performance. For example, the length of the four floating-point numerical vectors can be set to 3360, corresponding to sample features with a sampling interval of 3 minutes and a sampling duration of 7 days.

[0104] It is worth noting that if a job takes less than 7 days to run, resulting in a feature vector length of less than 3360, the empty vector elements in the floating-point value vector can be set to "0" to make up the feature vector length of 3360.

[0105] In some possible implementations, before determining the training dataset for the machine learning model, it is also necessary to correlate the job submission scripts provided by the job scheduling system 100, the job performance information provided by the performance monitoring system 300, the job start and end times, and / or the fine-grained I / O information of the storage performance provided by the intelligent storage system in terms of time and nodes.

[0106] For example, Figure 4This is a schematic diagram of the generated job feature dataset. As shown in the figure, the job feature dataset includes an interrelated grayscale image (first grayscale image) transformed by the current job's job submission script, the current job's I / O feature vector, the current job's start and end times, and / or fine-grained I / O information about the current job's storage performance (not shown in the figure). As mentioned earlier, the first grayscale image is at least one two-dimensional matrix transformed by the current job's job submission script, and the aforementioned I / O feature vector is determined based on performance information processing of the job's read throughput, write throughput, read metadata, and write metadata.

[0107] In some possible real-time approaches, the feature dataset can be divided into a feature training set, a feature validation set, and a feature test set in an 8:1:1 ratio by randomly selecting data to ensure that each subset has a similar distribution.

[0108] By way of example and not limitation, in this application, the user identifier (UID) can be used as the association information between the job submission script, the job's performance information, the job's start and end times, and fine-grained I / O information. Alternatively, the job identity identifier in the job submission script and the node identifiers of one or more nodes running the job can be used as the association information between the job submission script, the job's performance information, the job's start and end times, and fine-grained I / O information. In other words, this application does not limit the association method between the job submission script, the job's performance information, the job's start and end times, and fine-grained I / O information.

[0109] Taking the ResNet-50 feature extraction network as an example as a neural network model for I / O feature prediction. For instance, Figure 5 This is a schematic diagram of the training scenario for the ResNet-50 feature extraction network. As shown in the figure, a high-performance computing cluster can input the feature training set into a pre-trained model for training to obtain the first machine learning model.

[0110] The basic unit of the ResNet-50 feature extraction network is the residual block.

[0111] Schematic, the ResNet-50 feature extraction network described above may include multiple residual blocks. Each residual block may include three convolutional layers, a batch normalization layer, and a ReLU activation function. For example, the ResNet-50 feature extraction network includes 16 residual blocks, each of which includes a 1×1 dimensionality-reducing convolution, a 3×3 feature extraction convolution, and a 1×1 dimensionality-increasing convolution.

[0112] The ResNet-50 feature extraction network stacks multiple bottleneck modules in a hierarchical order, with each stage extracting more abstract features by increasing the number of channels and reducing the feature map size. This ResNet-50 feature extraction network can output features of the job submission script matrix through global average pooling and fully connected layers. In this application, the first trained machine model is used to predict the runtime of the aforementioned job.

[0113] During training, the ResNet-50 feature extraction network performs forward propagation to obtain the model output; the ResNet-50 feature extraction network performs backpropagation to calculate the gradient.

[0114] In practice, the first step is to input the current job feature dataset (the first feature dataset) into the ResNet-50 feature extraction network for pre-training. As mentioned earlier, the first feature dataset includes the grayscale image converted from the current job's submission script, the current job's I / O feature vector, and the job's start and end times. The I / O feature vector is determined based on the performance data corresponding to the current job.

[0115] The ResNet-50 feature extraction network can perform forward propagation, automatically learn the gray values ​​of integer elements in a two-dimensional integer matrix based on I / O feature vectors, and output the runtime of the job.

[0116] Indicatively, the two-dimensional matrix corresponding to the first grayscale image includes an integer element "u0031" representing the number of nodes "1", an integer element "u0033 u0032" representing the number of cores "32", and an integer element "u0032" representing the number of image processors "3".

[0117] The ResNet-50 feature extraction network can learn the runtime of a job based on the gray values ​​of integer elements "u0031", "u0033u0032" and "u0032", as well as the I / O feature vectors of jobs associated with the aforementioned integer elements.

[0118] It is worth noting that the integer elements in the two-dimensional matrix corresponding to the first grayscale image, representing the number of nodes, cores, and image processors, are intended to facilitate understanding of this application. In actual implementation, the two-dimensional matrix may also include various integer elements representing the job usage instance, job queue, program startup parameters, job start time, and job end time. The ResNet-50 feature extraction network can also learn based on the above integer elements, and this application does not limit this.

[0119] The ResNet-50 feature extraction network can also perform backpropagation, using a loss function to calculate the difference between the ResNet-50 feature extraction network output value and the true label to obtain the loss value.

[0120] Then, the high-performance computing cluster can input the test set of features from the job feature dataset into the ResNet-50 feature extraction network for testing, and use the Adam optimizer to update the model's weights. Once optimization is complete, the first trained machine learning model can be obtained.

[0121] Finally, the feature validation sets can be input into the first machine learning model after training, and the start and end times of the jobs predicted by the first machine learning model can be validated based on the actual start and end times of the jobs.

[0122] Indicatively, the loss function of the ResNet-50 feature extraction network can be defined as the deviation between the predicted job run time and the actual time, and the adaptive gradient descent (Adam) optimizer can be used as the optimization algorithm for the first machine learning model to adaptively learn.

[0123] In the adaptive gradient descent optimizer, the loss function can be defined as the deviation between the predicted job execution time and the actual execution time. In some possible implementations, the learning rate of the first machine learning model can be set to 0.001 to better optimize the model.

[0124] By training the structural parameters of the ResNet-50 feature extraction network using deep learning methods and taking "job runtime" as the training objective, features of the job submission script "image" can be automatically extracted layer by layer, avoiding noise and potentially missed information introduced by manual feature engineering.

[0125] It is worth noting that, in order to avoid overfitting, the number of training steps for the first machine learning model should not be too many. For example, the number of training steps for the first machine learning model can be set to "50".

[0126] As previously described, the high-performance computing cluster includes a job scheduling system 100, an intelligent storage system 200, and a performance monitoring system 400. In some possible implementations, the high-performance computing cluster may also include a high-performance computing task profiling and cluster design system, as well as a job fault diagnosis system (not shown in the figure).

[0127] This application can retrain the output layer of the trained first machine learning model to obtain a second machine learning model, and then transfer the second machine learning model to the aforementioned system of a high-performance computing cluster.

[0128] For example, Figure 6 This is a schematic diagram of a transfer learning scenario in the first machine learning model. As shown in the figure, this application uses transfer learning technology to fine-tune the weights of the output layer of the first machine learning model, replacing the training objective from "job runtime" to "job I / O features".

[0129] The output layer is retrained so that the second machine learning model can be applied to four different I / O prediction tasks, namely the job's read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate.

[0130] To train a different task objective than the first machine learning model, the weights of the final fully connected layer of the ResNet-50 feature extraction network will be randomly initialized, but the pre-trained weights of the other layers will be retained.

[0131] Similar to the optimization method of the first machine learning model, the training of the second machine learning model also uses an adaptive gradient descent optimizer. The loss function is set as the difference between the predicted value and the actual value of any one of the task objectives: read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate.

[0132] In practice, firstly, the high-performance computing cluster can collect job submission scripts for a small number of jobs (hereinafter referred to as target jobs) on the target system.

[0133] Then, using the same training method as the first machine learning model, the target job submission script is converted into a second grayscale image, and this second grayscale image is input into the first machine learning model to retrain its output layer, thereby obtaining the I / O feature prediction model. The training process of the I / O feature prediction model will not be elaborated here.

[0134] After training the I / O feature prediction model is completed, the high-performance computing cluster can transfer the I / O feature prediction model to the target system of the high-performance computing cluster, obtain the job submission script of the target system, convert the job submission script into at least one grayscale image, and input the at least one grayscale image into the I / O feature prediction model to obtain the I / O features of the job.

[0135] It is understandable that the grayscale image described above is a two-dimensional matrix transformed from the target job's submission script. For example, Figure 6 The two-dimensional integer matrix 1, two-dimensional integer matrix 2, and two-dimensional integer matrix 3 are shown.

[0136] In some possible implementations, the learning rate of all layers in the second machine learning model, except for the final fully connected layer, needs to be reduced to 0.0001. Then, during training, the learning rate of other layers is fine-tuned based on performance feedback, and the weights of some layers in the network are updated.

[0137] Indicative, Figure 7 This is a schematic diagram illustrating the application scenario of the second machine learning model. As shown in the figure, the second machine learning model can be deployed on the job scheduling system 100. The goal of the second machine learning model is to use the I / O feature prediction results of jobs to guide the scheduling system in allocating tasks, ensuring that when scheduling tasks, it considers not only available resources such as CPU and memory, but also the constraints of storage system throughput and metadata read / write capabilities. The following sections will introduce I / O feature prediction.

[0138] First, a second machine learning model is configured on the job scheduling system 100, the polling time interval is set, and the scheduling process is started. Multiple job submission scripts are obtained from the job queue, and sample data for multiple jobs is generated based on these scripts. The sample data includes multiple grayscale images, which are two-dimensional matrices transformed from the job submission scripts.

[0139] At the beginning of each time interval, a second machine learning model is used to process the above multiple two-dimensional matrices to predict the I / O feature vectors of the currently running job for the next 7 days.

[0140] Then, the job scheduling system 100 uses a second machine learning model to predict the I / O feature vectors of multiple jobs (jobs 1 to n as shown in the figure) in the job queue to be scheduled for the next 7 days according to the order of job submission time, and summarizes them to obtain the 3360-dimensional I / O feature vectors of multiple jobs over 7 days.

[0141] Finally, the job scheduling system 100 uses the 3360-dimensional I / O feature vectors of multiple jobs to determine in sequence whether loading a certain job will cause the system throughput and metadata read / write capabilities of the high-performance computing cluster to exceed the limits.

[0142] Taking job n (the first job) as an example, if loading the first job will not exceed the limit, then the job will be loaded; otherwise, the first job will be delayed until the next round of re-evaluation and scheduling.

[0143] In some possible implementations, the second machine learning model can be deployed in high-performance computing task profiling and cluster design systems. By predicting and collecting the I / O characteristics of user-submitted jobs, it determines the application distribution and job characteristics of users in the high-performance computing cluster based on these I / O characteristics. The aggregated application distribution and job characteristics of system users then guide the optimization of the user's HPC design. For example, it can select the front-end and back-end server ratio in the storage system based on I / O characteristics, and adjust the ratio of solid-state drives (SSDs) to hard disk drives (HDDs).

[0144] As one possible implementation, the second machine learning model can also be deployed in the scheduling engine 210 of the intelligent storage system 200 to provide auxiliary basis for hot and cold data identification. Currently, industry-wide hot and cold data identification is based on a "stream" granularity, lacking awareness of higher-level business processes. The second machine learning model can provide the intelligent storage system scheduling engine with a complete time series of I / O characteristics of jobs to be run, thereby providing a more accurate basis for classifying hot and cold data in the working directory, adjusting caching strategies, and data layout.

[0145] To illustrate, for a high-performance job about to start, "hot" data can be automatically moved to hot storage in advance to improve read and write speeds and shorten runtime; while for a low-performance job, its "cold" data can be kept on cold storage without needing to be moved in advance, thus shortening startup time.

[0146] In some possible implementations, the second machine learning model can also be deployed in a job fault diagnosis system to diagnose whether jobs and the system are operating normally from the perspective of I / O in the high-performance computing cluster. The second machine learning model provides a sequence of I / O throughput patterns for jobs under ideal conditions, and the I / O patterns of normally running jobs should be similar. Conversely, if the actual running results of multiple jobs in the high-performance computing cluster deviate significantly from the predictions (e.g., abnormally shortened or extended running times, abnormally high or low I / O values), it can be checked whether the node where the job resides or the storage system is operating normally.

[0147] Firstly, based on the content described above, the I / O feature prediction method provided in the embodiments of this application will be introduced. It is understood that this method is proposed based on the content described above, and some or all of the content of this method can be found in the description above.

[0148] For example, Figure 8This is a flowchart illustrating the I / O feature prediction method. As shown in the figure, I / O feature prediction for a job can be implemented through steps S810 to S830. In this embodiment, the I / O feature prediction method can be executed by an electronic device or server in a high-performance computing cluster. The specific form of the electronic device or server can be a supercomputer, desktop computer, laptop computer, tablet computer, or mobile phone; this application does not limit the specific type.

[0149] S810: Retrieve the job submission script for the first job.

[0150] As mentioned earlier, the job scheduling system 100 in the high-performance computing cluster includes multiple jobs input by the user. These jobs are submitted through job submission scripts. Each job submission script corresponds to a job and includes the job's configuration information, such as the job name, required resources (number of CPU cores, memory size, running time, etc.), the program to be executed, and input / output files.

[0151] The high-performance computing cluster can obtain the job submission script of the first job, which is one of the plurality of jobs.

[0152] S820: Convert the job submission script into at least one grayscale image.

[0153] When it is necessary to predict the IO characteristics of the first job, the high-performance computing cluster converts the job submission script of the first job into at least one grayscale image.

[0154] In some possible implementations, the job submission script for the first job can be converted into at least one grayscale image.

[0155] To illustrate, the assignment submission script consists of K lines of text characters, with a grayscale image corresponding to a two-dimensional matrix of M rows × N columns.

[0156] A high-performance computing cluster can retrieve a single line of text from K lines of a job submission script and convert it into a row of integer elements in a two-dimensional matrix corresponding to a grayscale image, until all K lines of text have been converted into at least M rows of two-dimensional matrices. In this case, each text character in the job submission script corresponds to an integer element in the two-dimensional matrix.

[0157] By converting the parameters in each row of the job submission script—representing the number of nodes, cores, graphics processors, job instance usage, job queue, program startup parameters, job start time, and job end time—into a row of integer elements in a two-dimensional matrix, the I / O feature prediction model can be processed using grayscale images, enabling more accurate and efficient prediction of the I / O features of jobs in high-performance computing clusters.

[0158] S830: Input at least one grayscale image into the I / O feature prediction model to obtain the I / O features of the first job.

[0159] High-performance computing clusters can input one less grayscale image into the I / O feature prediction model to obtain the I / O features for the first job.

[0160] Schematic illustration: The high-performance computing cluster can input at least one M-row two-dimensional matrix obtained in step S820 into the I / O feature prediction model. Using the aforementioned I / O feature prediction model, the I / O features of the first job are predicted based on the grayscale values ​​of multiple integer elements in the at least one M-row two-dimensional matrix. The aforementioned I / O features include one or more of the following: read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate of the first job.

[0161] In some possible implementations, the I / O feature prediction model is deployed in the job scheduling system 100.

[0162] The high-performance computing cluster's job scheduling system 100 can load the job submission script of the first job or delay the loading of the first job based on the I / O characteristics of the first job.

[0163] As one possible implementation, this is in the case where the I / O feature prediction model is deployed in the intelligent storage system 200.

[0164] The intelligent storage system 200 of the high-performance computing cluster can store the working data of the first job in a cold storage device or a hot storage device in the computing cluster 300 based on the I / O characteristics of the first job.

[0165] In some possible implementations, this applies when the I / O feature prediction model is deployed in the job fault diagnosis system.

[0166] A high-performance computing cluster job fault diagnosis system can determine the working status of a high-performance computing cluster based on the I / O characteristics of multiple jobs.

[0167] As one possible implementation, this applies to the deployment of I / O feature prediction models in task profiling and cluster design systems.

[0168] The task profiling and cluster design system for high-performance computing clusters can determine the ratio of front-end and back-end servers, solid-state drives, and hard disk drives in a high-performance computing cluster based on the I / O characteristics of multiple jobs.

[0169] This application proposes an I / O feature prediction method, model training method, and related apparatus. By converting the job submission script into a grayscale image, a deep learning method based on image recognition is employed to predict I / O features. In predicting I / O features, existing processing methods in the deep learning field, including various classic image recognition network structures, can be reused to process the grayscale image and obtain the job's I / O features. Compared to existing methods that use the "job name-username-number of compute nodes" triple as I / O features and employ clustering algorithms to process large-scale datasets for prediction, this method predicts I / O features based on the job submission script, reducing the prediction time and achieving accurate and efficient prediction of job I / O features in high-performance computing clusters.

[0170] Secondly, based on the content described above, the training method of the I / O feature prediction model provided in the embodiments of this application will be introduced. It is understood that this method is proposed based on the content described above, and some or all of the content of this method can be found in the description above.

[0171] For example, Figure 9 This is a flowchart illustrating the training method for an I / O feature prediction model. As shown in the figure, the training of the I / O feature prediction model can be implemented through steps S910 to S980. In this embodiment, the training method for this model can be executed by an electronic device or a server. The specific form of the electronic device or server can be a supercomputer, desktop computer, laptop computer, tablet computer, or mobile phone; this application does not limit the specific type.

[0172] S910: Obtain the job submission script I / O performance data and job start and end times for the current job.

[0173] When pre-training a neural network model is required, the job scheduling system 100 of the high-performance computing cluster can be used to obtain the job submission script of the currently running job on the high-performance computing cluster; the performance monitoring system 400 of the high-performance computing cluster can be used to obtain the I / O performance data and start and end times of the current job. The job submission script of the current job includes multiple text characters describing the configuration information of the current job, such as the job name, required resources (number of CPU cores, memory size, running time, etc.), the program to be executed, and input / output files; the I / O performance data of the current job includes information on the read bandwidth performance, write bandwidth performance, read metadata performance, and write metadata performance of at least one node executing the job.

[0174] S920: Obtain the first grayscale image based on the job submission script of the current job.

[0175] The high-performance computing cluster converts the job submission script of the first job obtained in step S910 into a first grayscale image. The first grayscale image includes at least one two-dimensional matrix. Each text character in the job submission script corresponds to an integer element in the two-dimensional matrix.

[0176] S930: Feature vector of the current job determined based on I / O performance data.

[0177] The high-performance computing cluster can determine the feature vector of the current job based on the I / O performance data of the current job obtained in step S910.

[0178] As an illustration, a high-performance computing cluster can encode the I / O feature vectors in a sample into four floating-point numerical vectors based on four attributes: read throughput performance, write throughput performance, read metadata performance, and write metadata performance.

[0179] S940: Determine the feature dataset based on the first grayscale image, the feature vector of the current job, and the start and end times of the job.

[0180] High-performance computing clusters can correlate the first grayscale image, the feature vector of the current job, and the start and end times of the job in the time and node dimensions to determine the feature dataset used for training neural network models.

[0181] S950: Input the feature dataset into the neural network model for pre-training to obtain the first machine learning model.

[0182] High-performance computing clusters can input feature datasets into neural network models for pre-training to obtain a first machine learning model. The training objective of this first machine learning model can be set as the "job runtime".

[0183] In some possible implementations, the training dataset can be input into a neural network model, which can then be used to output the prediction results for the current task.

[0184] The high-performance computing cluster can train the neural network model based on the prediction results of the current job and a loss function to obtain a trained first machine learning model. The loss function indicates the deviation between the prediction results of the current job and the start and end times of the current job.

[0185] S960: Retrieve the job submission script for the target job.

[0186] When model migration is required, the high-performance computing cluster can obtain the job submission script of the target job. This script contains multiple text characters describing the target job's configuration information, such as the job name, required resources (number of CPU cores, memory size, runtime, etc.), the program to be executed, and input / output files.

[0187] S970: Convert the job submission script of the target job into a second grayscale image.

[0188] High-performance computing clusters can convert the job submission script of a target job into a second grayscale image. This second grayscale image comprises at least one two-dimensional matrix. Each text character in the job submission script corresponds to an integer element in this two-dimensional matrix.

[0189] S980: Input the second grayscale image into the first machine model, and retrain the output layer of the first machine model to obtain the I / O feature prediction model.

[0190] First, high-performance computing clusters can change the training objective of the first machine model from "job runtime" to "job I / O characteristics".

[0191] Then, the high-performance computing cluster can input the second grayscale image into the first machine model and retrain the output layer of the first machine model to obtain the I / O feature prediction model. The aforementioned I / O features include one or more of the target job's read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate.

[0192] In this application, the job submission script is converted into a grayscale image, and a deep learning method for image recognition is used to train a neural network model. The neural network model automatically extracts I / O features related to the I / O feature vector from the current job submission script, eliminating the need for manual feature engineering. This method fully preserves the information in the current job submission script, reducing information loss and noise interference caused by human factors, and achieving more accurate feature learning. Furthermore, based on the first machine learning model, this training method retrains the output layer based on the target job's submission script to adapt to prediction tasks with different I / O features. This optimizes the generalization ability of the neural network model, improving its adaptability and flexibility, allowing the model to quickly transfer to new prediction tasks while maintaining training efficiency.

[0193] Thirdly, embodiments of this application disclose an I / O feature prediction device. For example, Figure 10 This is a schematic diagram of the I / O feature prediction device. As shown in the figure, the I / O feature prediction device 10 includes a communication module 11 and a processing module 12.

[0194] The communication module 11 is used to obtain the job submission script of the first job input by the user.

[0195] Processing module 12 is used to convert the job submission script into at least one grayscale image.

[0196] In some possible implementations, the processing module 12 is also used to input at least one grayscale image into the I / O feature prediction model to obtain the I / O features of the target job.

[0197] As one possible implementation, the job submission script for the target job includes K lines of text characters, with each grayscale image corresponding to an M-line two-dimensional matrix.

[0198] Processing module 12 is also used to obtain a line of text characters from K lines, convert the line of text characters into a line of integer elements in a two-dimensional matrix, until the K lines of text characters are converted into at least one two-dimensional matrix with M lines.

[0199] In some possible implementations, the processing module 12 is further configured to input at least one M-row two-dimensional matrix into the I / O feature prediction model, and use the I / O feature prediction model to predict the I / O features of the first job based on the grayscale values ​​of the integer elements. The I / O features of the first job include one or more of the following: read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate.

[0200] Thirdly, embodiments of this application disclose a high-performance computing cluster. Figure 11 This is a schematic diagram of the architecture of a high-performance computing cluster. As shown in the figure, the high-performance computing cluster 20 includes at least one computing device. Schematic, the high-performance computing cluster 20 may include computing device 210 and computing device 220.

[0201] The computing device 210 includes a processor 211 and a memory 212, and the computing device 220 includes a processor 221 and a memory 222.

[0202] Taking either computing device 210 or computing device 220 as an example,

[0203] The processor 211 of the computing device 210 is used to execute instructions stored in the memory 212 so that the high-performance computing cluster 20 performs the aforementioned I / O feature prediction method.

[0204] In addition to the methods, apparatus, and electronic devices described above, embodiments of this application may also provide a computer program product, comprising computer program instructions. When executed by a processor, the computer program instructions cause the processor to perform the steps of the methods described in the "Methods" section of this specification. The computer program product can be written in any combination of one or more programming languages ​​to execute the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The computer program code can be in source code form, object code form, executable file, or some intermediate form. The computer program code can be executed entirely on a user's computing device, partially on a user's device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0205] Furthermore, embodiments of this application may also provide a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the methods described in the "Method" section of this specification according to the various embodiments of this disclosure. The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. It should be noted that the content contained in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0206] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0207] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0208] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of the various embodiments of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the specific details described above.

[0209] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0210] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0211] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations therein.

[0212] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application. The specific embodiments described above have further detailed the purpose, technical solutions, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

[0213] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above description is only a specific embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for predicting I / O features, characterized in that, Applied to a high-performance computing cluster for running multiple jobs, the method includes: Obtain the job submission script for the first job, where the first job is one of the plurality of jobs; Convert the job submission script into at least one grayscale image; The at least one grayscale image is input into the I / O feature prediction model to obtain the I / O features of the first job.

2. The method according to claim 1, characterized in that, The job submission script includes K lines of text characters, each grayscale image corresponds to an M-line two-dimensional matrix, and converting the job submission script into at least one grayscale image includes: Obtain a line of text characters from K lines, and convert the line of text characters into a row of integer elements in the two-dimensional matrix, until the K lines of text characters are converted into at least one two-dimensional matrix with M lines; each text character corresponds to one integer element.

3. The method according to claim 2, characterized in that, The step of inputting the grayscale image into the I / O feature prediction model to obtain the I / O features of the first job includes: The at least M-row two-dimensional matrix is ​​input into the I / O feature prediction model. Using the I / O feature prediction model, the I / O features of the first job are predicted based on the gray values ​​of the plurality of integer elements. The I / O features include one or more of the following: read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate of the first job.

4. The method according to claim 1, characterized in that, The high-performance computing cluster includes a job scheduling system. When the I / O feature prediction model is deployed in the job scheduling system, the method further includes: loading the job submission script or delaying the loading of the first job based on the I / O features of the first job.

5. The method according to claim 1, characterized in that, The high-performance computing cluster includes an intelligent storage system. When the I / O feature prediction model is deployed in the intelligent storage system, the method further includes: Based on the I / O characteristics of the first job, the working data of the first job is stored in a cold storage device or a hot storage device.

6. The method according to claim 1, characterized in that, The high-performance computing cluster includes a job fault diagnosis system. When the I / O feature prediction model is deployed within the job fault diagnosis system, the method further includes: Based on the I / O characteristics and performance data of the multiple jobs, the working status of the high-performance computing cluster is determined.

7. The method according to claim 1, wherein the high-performance computing cluster includes a task profiling and cluster design system, and the I / O feature prediction model is deployed in the task profiling and cluster design system, the method further includes: Based on the I / O characteristics and performance data of the multiple jobs, the ratio of front-end and back-end servers, solid-state drives and hard disk drives in the high-performance computing cluster is determined.

8. A training method for an I / O feature prediction model, characterized in that, The method includes: Obtain the current job's submission script, I / O performance data, and job start and end times; the I / O performance data includes performance information for read bandwidth, write bandwidth, read metadata, and write metadata; The first grayscale image is obtained based on the job submission script of the current job; The feature vector of the current job is determined based on the I / O performance data; A feature dataset is determined based on the first grayscale image, the feature vector of the current job, and the start and end times of the job. The feature dataset is input into a neural network model for pre-training to obtain a first machine learning model, which is used to predict the runtime of the current job. Retrieve the job submission script for the target job; The task submission script of the target task is converted into a second grayscale image; The second grayscale image is input into the first machine model, and the output layer of the first machine model is retrained to obtain an I / O feature prediction model. The I / O feature prediction model is used to predict the I / O features of the target job. The I / O features include one or more of the target job's read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate.

9. The method according to claim 8, characterized in that, The step of inputting the training dataset into the neural network model for pre-training to obtain the first machine learning model includes: The training dataset is input into the neural network model, and the neural network model is used to output the prediction result of the current task. The neural network model is trained based on the prediction results of the current task and the loss function to obtain the first machine learning model after training. The loss function is used to indicate the deviation between the prediction results of the current task and the start and end times of the current task.

10. A device for predicting I / O characteristics, characterized in that, The device includes: The communication module is used to obtain the job submission script for the first job from the user's input. The processing module is used to convert the job submission script into at least one grayscale image; The processing module is also used to input the at least one grayscale image into the I / O feature prediction model to obtain the I / O features of the operation.

11. The apparatus according to claim 10, characterized in that, When the job submission script includes K lines of text characters, and each grayscale image corresponds to an M-line two-dimensional matrix, the processing module is specifically used for: Obtain a line of text characters from K lines, and convert the line of text characters into a row of integer elements in the two-dimensional matrix, until the K lines of text characters are converted into at least one two-dimensional matrix with M lines.

12. The apparatus according to claim 11, characterized in that, The processing module is specifically used for: The at least M-row two-dimensional matrix is ​​input into the I / O feature prediction model. Using the I / O feature prediction model, the I / O features of the first job are predicted based on the gray values ​​of the integer elements. The I / O features include one or more of the read bandwidth rate, write bandwidth rate, read metadata operation rate, and write metadata operation rate of the first job.

13. A high-performance computing cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device to cause the high-performance computing cluster to perform the method as described in claims 1-7.

14. A computer program product containing instructions, characterized in that, When the instruction is executed by the high-performance computing cluster, the high-performance computing cluster performs the method as described in claims 1-7.

15. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a high-performance computing cluster, perform the method as described in claims 1-7.

Images