An HPC job semantic representation and classification method based on online drift adaptation

Abstract

The application relates to an HPC job semantic representation and classification method based on online drift self-adaption, and steps are as follows: firstly, job information description is collected, semantic coding is completed by means of a pre-training language model, and job semantic features are output. Secondly, job running indexes are acquired, performance analysis is carried out in combination with a Roofline model, job types are divided, and real labels are generated. Then, semantic features and user historical job behavior features are fused to construct a comprehensive feature vector, an offline classification model is obtained through supervised learning training, and prediction is completed. Subsequently, an exponential weighted moving average method is adopted, and job type distribution probability is estimated according to recent job real labels. Finally, the probability estimation value and the model prediction result are fused to output the final job classification result. Compared with the prior art, the application has the advantages of significantly improving the accuracy and stability of job type classification.

Description

Technical Field

[0001] This invention relates to the field of online prediction methods for HPC job classification, and in particular to a semantic representation and classification method for HPC jobs based on online drift adaptation. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology across various industries, large-scale data processing and complex model training are placing increasingly higher demands on computing power. High-performance computing (HPC) clusters, with their powerful parallel computing capabilities, have become a crucial infrastructure supporting AI model training, large-scale data analysis, and scientific computing. In practical applications, numerous users submit different types of computing tasks to HPC clusters, and these tasks vary significantly in terms of computing patterns, resource requirements, and operational characteristics. For example, some tasks are computationally intensive, requiring high CPU or GPU computing power; while others are memory-intensive, relying more on memory bandwidth and data access efficiency. Without effective task identification and resource management mechanisms, unreasonable resource allocation can easily occur, reducing overall system utilization and computing efficiency, and even causing some resources to remain underutilized for extended periods. Therefore, accurately identifying the types of jobs submitted by users and achieving reasonable and efficient resource scheduling has become a key issue in HPC system management and scheduling optimization.

[0003] Currently, a common method to improve the resource utilization efficiency of HPC clusters is to collect resource utilization data offline through monitoring devices, including some operational performance indicators. Based on this, historical operational data is statistically analyzed, and algorithms are designed to determine the rationality of user resource requests, thereby assisting in the coordinated scheduling of computing and storage resources. This type of method can summarize task operation patterns from historical data and analyze resource demands through empirical rules or machine learning models. However, this method is usually an offline analysis approach, and the model mainly relies on historical data for training, making it difficult to respond promptly to constantly changing task demands. Furthermore, even if the offline-trained model is deployed to an online environment, user job patterns may change over time, such as algorithm updates, changes in data scale, or adjustments to task types. These factors can lead to changes in job characteristic distribution, resulting in concept drift. When the model cannot adapt to these changes in a timely manner, its predictive performance will gradually decline, affecting the accuracy of resource scheduling decisions and increasing system operating overhead and management complexity. Therefore, how to reduce model complexity while ensuring prediction accuracy, and how to adapt to the dynamic changes in user job behavior to achieve a lightweight, online deployable job classification method with drift adaptive capability has become an important technical problem that urgently needs to be solved in the field of HPC job classification and resource scheduling. Summary of the Invention

[0004] The purpose of this invention is to propose an online drift-adaptive semantic representation and classification method for HPC jobs, which effectively alleviates the concept drift problem caused by changes in HPC system job load and significantly improves the accuracy and stability of job type classification.

[0005] The objective of this invention can be achieved through the following technical solutions: A semantic representation and classification method for HPC jobs based on online drift adaptation, comprising the following steps: S1: Obtain the job information description of the user-submitted task, and use a pre-trained language model to perform semantic encoding on the job information description to obtain the semantic feature representation of the job; S2: Obtain the runtime metrics information of user-submitted tasks in the HPC cluster after completion, perform performance characteristic analysis on the completed jobs based on the Roofline model, classify the job types based on the analysis results, and generate real labels for job classification. S3: The semantic feature representation is fused with the behavioral features of users' historical job submissions to construct a comprehensive feature vector. The comprehensive feature vector is then input into the machine learning model for training. At the same time, the real labels of job classification are used to supervise the machine learning model and verify the results, thereby obtaining the HPC job offline classification model. Based on the HPC job offline classification model, the model prediction results corresponding to the actual tasks are obtained. S4: Using the real tag information of the user's recently completed tasks, the distribution change of the user's task category is estimated by the exponentially weighted moving average method (EWMA), and the recent probability estimation results of the corresponding task type are obtained. S5: Integrate the recent probability estimation results with the model prediction results online to obtain the final predicted probability of the job type, and output the HPC job classification results of the actual task.

[0006] Further, step S1 includes: S11. Obtain the initial job information description when the user submits the task from the database, and preprocess it to obtain the preprocessed job information description. S12. Use a pre-trained language model to encode the semantic features of the pre-processed job information description to obtain the semantic feature representation of the job.

[0007] Further, step S2 includes: S21. Obtain the running metric information after the user-submitted task in the HPC cluster is completed, and calculate the actual peak running intensity of the task based on the Roofline model; S22. Compare the actual peak intensity of operation with the theoretically calculated intensity threshold corresponding to the hardware indicators of the HPC cluster, and determine the true label of the job classification based on the comparison results.

[0008] Further, step S3 includes: S31. Generate a daily task template based on the user's historical completed task records, organized by date. S32. Extract user behavior characteristics from the N days prior to the current task submission time based on the daily task template; S33. The semantic features and behavioral features of the assignment are fused to construct a comprehensive feature vector. The comprehensive feature vector is then input into the machine learning model for training. At the same time, the real labels of the assignment classification are used to supervise the machine learning model and verify the results, thereby obtaining the HPC assignment offline classification model. The actual task is then input into the HPC assignment offline classification model to obtain the model prediction results.

[0009] Further, step S4 includes: S41. Initialize the EWMA drift tracker and establish the drift state; S42. Read the real tag information of the user's recently completed tasks, and use EWMA to recursively update the category probability based on the drift state to obtain the EWMA probability estimate of the user's short-term task category distribution, i.e. the recent probability estimate result. S43. Calculate the drift weight based on the preheating mechanism.

[0010] Furthermore, the specific steps of S41 are as follows: Initialize the EWMA drift tracker and set the smoothness coefficient. and initial probability For each user or user-task name combination template key, initialize the corresponding drift state, which includes the current probability estimate. and the number of observed missions And set the update factor for EWMA. .

[0011] Furthermore, the specific steps of S42 are as follows: Read the current probability estimate This represents the current short-term category distribution state; once an assignment is completed and the true label information is obtained, the current probability estimate is updated using EWMA to obtain a new current probability estimate. That is, the recent probability estimation results, and then update the number of observed tasks. .

[0012] Furthermore, the new current probability estimate for: in, This indicates the actual label information of the currently completed tasks. This represents the new current probability estimate and the updated coefficients of EWMA. .

[0013] Further, step S5 includes: S51. Based on the drift weight, the recent probability estimation results and the model prediction results are weighted and fused online to obtain the final job type prediction probability. S52. Based on the final job type, predict the probability and output the HPC job classification results.

[0014] Furthermore, the method also includes: When the job execution is completed and the real tag information is obtained, the end time and real tag information of the job are recorded according to the user or user-task name combination template, and the EWMA drift status is updated accordingly.

[0015] Compared with the prior art, the present invention has the following beneficial effects: This invention proposes an online drift-adaptive method for HPC job semantic representation and classification. By combining pre-trained language models to encode job semantic features with users' historical job behavior features, the model can simultaneously consider job semantic information and user behavior patterns. Through weighted fusion of the prediction probabilities of the offline-trained classification model and the EWMA drift probabilities, it can track changes in user behavior in real time, achieving online awareness of job category distribution drift, thereby improving job category prediction capabilities in dynamic environments. This invention improves the robustness of online model classification results and possesses good real-time performance and engineering deployability, making it suitable for practical HPC cluster online scheduling environments. Attached Figure Description

[0016] Figure 1 A flowchart illustrating the overall framework method of an embodiment of the present invention; Figure 2 A flowchart illustrating the exponentially weighted moving average method according to an embodiment of the present invention; Figure 3 This is a flowchart illustrating the offline HPC classification model training method according to an embodiment of the present invention. Figure 4 This is a flowchart illustrating the online deployment HPC job prediction method according to an embodiment of the present invention. Detailed Implementation

[0017] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0018] This invention proposes a semantic representation and classification method for HPC jobs based on online drift adaptation. The method first uses a pre-trained language model to semantically encode the job description information of user-submitted tasks, obtaining semantic feature representations of the jobs. Then, it fuses the semantic features of the current job with the behavioral features of the user's historical task submissions to construct a comprehensive feature vector, which is then input into a machine learning model for training, establishing an offline classification model for HPC job types. During the online deployment phase, it utilizes the real label information of recently completed tasks to construct an EWMA drift estimation mechanism, and fuses this estimation result with the job type probability predicted by the model, thereby achieving online drift adaptive correction of the prediction results. This invention can effectively alleviate the concept drift problem caused by changes in HPC system job load and significantly improve the accuracy and stability of job type classification.

[0019] To address the problems existing in the prior art, the purpose of this invention is to propose an online drift-adaptive semantic representation and classification method for HPC jobs, which effectively alleviates the concept drift problem caused by changes in HPC system job load and significantly improves the accuracy and stability of job type classification.

[0020] To solve the above problems, the present invention can be implemented by adopting the following solution: A semantic representation and classification method for HPC jobs based on online drift adaptation, comprising the following steps: S1: Obtain the job information description of the user-submitted task, and use a pre-trained language model to semantically encode the above job information to obtain the semantic feature representation of the job; S2: Obtain the runtime metrics information of user-submitted tasks in the HPC cluster after completion, and perform performance characteristic analysis on the completed jobs based on the Roofline model. Based on this, classify the job types and generate real labels for job classification. S3: The semantic features of the job are fused with the behavioral features of the user's historical job submissions to construct a comprehensive feature vector. This feature vector is then input into the machine learning model for training. At the same time, the job classification labels are used to supervise the learning of the model and verify the results, thereby obtaining an offline classification model for HPC jobs. S4: During the online deployment phase of the model, the distribution changes of user task categories are estimated using the real label information of tasks recently completed by users through the EWMA method, and the recent probability estimation results of the corresponding task types are obtained. S5: Integrate the recent probability estimation results with the model prediction results online to obtain the final job type prediction probability, and output the HPC job classification result of the user's current submitted task.

[0021] Specifically, step S1 includes: S11. Obtain the job information description when the user submits the task from the database, sort the job information according to the task submission time, and perform time aggregation and data deduplication. S12. Use a pre-trained language model to encode the semantic features of each job information submitted by the user to obtain a high-dimensional semantic representation of the job.

[0022] Specifically, step S2 includes: S21. Obtain the computational performance metrics and memory access metrics of the user's completed jobs, and calculate the actual peak intensity of the job based on the Roofline model. S22. Calculate the intensity threshold based on the theoretical values ​​corresponding to the hardware indicators of the HPC cluster, and determine the actual peak intensity of the job.

[0023] Specifically, step S3 includes: S31. Generate a daily task template based on the user's historical completed task records by date. The template includes the username, task name, date, task submission time, completion time, and the number of tasks of type 0 and type 1. S32. Extract the historical behavioral characteristics of users in the N days prior to the current task submission time based on the daily statistics template, including the proportion of type 1 tasks, the daily average purity of different tasks, and the stability of user-submitted tasks. S33. The semantic features of the job are fused with the extracted historical user behavior features to construct a comprehensive feature vector. This feature vector is then input into the machine learning model for training. Simultaneously, supervised learning and model validation are performed using the real labels of the job categories to obtain an offline HPC job classification model.

[0024] Specifically, step S4 includes: S41. Initialize the EWMA drift tracker, establish a drift state for each user or user-task name combination, and set an initial probability estimate for cold start; S42. Before predicting the current submitted task, read the real tag information of the user's completed tasks, and use EWMA to recursively update the category probability according to the corresponding user or user-task name combination template key, so as to obtain the EWMA probability estimate of the user's short-term job category distribution. S43. Obtain the EWMA probability and observation count corresponding to the current user or user-task name, and calculate the drift weight according to the warm-up mechanism.

[0025] Specifically, step S41 includes: For each user or user-task name combination template key, initialize the corresponding drift state and set the initial category probability estimate. and the number of observed missions And maintain independent drift statistics for each key; Set EWMA update factor This is to adjust the update magnitude of the current true label to the historical probability estimation results, where a larger value... More sensitive to recent category changes, smaller This makes the probability estimation process smoother and more stable.

[0026] Specifically, step S42 includes: When a new job arrives, read the current drift estimate for that user or user-task name combination template. This represents the current short-term category distribution state; When the task is completed and the true labels are obtained, EWMA is used to update the probability estimate; Also update the observation count for that user or user-task name combination template.

[0027] Specifically, step S5 includes: S51, Based on drift weight The classification model output probability is weighted and fused with the EWMA probability to obtain the final prediction probability; S52, Based on the final predicted probability Output the HPC job classification results (class 0 or 1); S53. When the job execution is completed and the real label is obtained, record the job's end time, the corresponding user or user-task name combination template key, and the real label, and use this information to update the EWMA drift status.

[0028] Figure 1 A flowchart illustrating the overall framework of an online drift-adaptive HPC job semantic representation and classification method provided in this embodiment of the invention includes the following steps: A semantic representation and classification method for HPC jobs based on online drift adaptation, characterized by the following steps: S1: Obtain the job information description of the user-submitted task, and use a pre-trained language model to perform semantic encoding on the above job information to obtain the semantic feature representation of the job.

[0029] S11. Obtain the job information description when the user submits the task from the database, sort the job information according to the task submission time, and perform time aggregation and data deduplication. Specifically, HPC cluster performance metrics and user job logs are typically stored in relational databases such as MySQL and PostgreSQL, with raw data resources obtained through database interfaces. Performance metrics include job submission time, start time, end time, actual number of nodes used, memory access bandwidth, and floating-point computation volume. User job logs include username, user ID, task name, and hardware resource information requested by the user when submitting the task, such as the number of CPU cores, CPU frequency, number of nodes, operating environment, and memory configuration. In this embodiment, the F-DATA job execution dataset from the Fugaku supercomputer is used, covering the period from May 2023 to April 2024. The obtained raw data undergoes preprocessing, including filtering empty string records in specified fields, converting performance metric fields to numeric types, and deleting duplicate records. The job information is then sorted by recording time and grouped by job number. Time information generated during the submission, start, and end phases of the same job is aggregated, ensuring each group corresponds to a complete job instance. This generates job-level performance metrics and user job log data at the job level, which are then summarized and saved monthly in Parquet format. Ultimately, this results in users submitting the same type of task in different jobs or different tasks in different jobs, with each job number being unique.

[0030] S12. Use a pre-trained language model to encode the semantic features of each job information submitted by the user to obtain a high-dimensional semantic representation of the job.

[0031] Specifically, a pre-trained language model is used to semantically encode the description information of each user-submitted job. This description information includes the username, user ID, task name, number of CPU cores, CPU frequency, number of nodes, and operating environment. This information is then used to construct a unified text representation, which is input into the pre-trained language model to obtain the job's semantic feature vector. The pre-trained language model can be gte-base-en-v1.5, bge-small-en-v1.5, bge-base-en-v1.5, or e5-base-v2, etc. In this embodiment, bge-base-en-v1.5 is used. Proposed by the Beijing Academy of Artificial Intelligence, this model is a general text embedding model based on the Transformer encoding structure and pre-trained on large-scale corpus data using a contrastive learning method. It can map input text into a fixed-dimensional semantic vector representation, effectively capturing the semantic relationships and potential features between texts. Through this encoding method, a unified semantic representation can be achieved for string and numerical data in user-submitted job information, enabling the model to comprehensively reflect the job's task attributes, resource requirements, and potential computational patterns. The resulting semantic vectors can be used as input features for subsequent machine learning models to predict whether the task is computationally intensive or memory-intensive.

[0032] S2: Obtain the runtime metrics information of user-submitted tasks in the HPC cluster after completion, perform performance characteristic analysis on the completed jobs based on the Roofline model, classify the job types accordingly, and generate real labels for job classification.

[0033] S21. Obtain the computational performance metrics and memory access metrics of the user's completed jobs, and calculate the actual peak intensity of the job based on the Roofline model.

[0034] Specifically, based on the job-level data generated in step S11, relevant performance metrics of the job during its execution are obtained, including actual job execution time, number of nodes used, memory access bandwidth, and floating-point computation load. Based on these performance metrics, the actual computational intensity of the job is calculated using the Roofline model, and the calculation formula is as follows: in, This indicates the actual computational intensity of the user-submitted job. This indicates the total number of floating-point calculations performed during the operation. This indicates the total number of bytes accessed during the execution of this job.

[0035] S22. Calculate the intensity threshold based on the theoretical values ​​corresponding to the hardware indicators of the HPC cluster, and determine the actual peak intensity of the job.

[0036] Specifically, the result calculated in step S21 Comparison with the theoretical computational intensity threshold corresponding to HPC cluster hardware: when When the value is greater than or equal to the theoretical threshold, the task is classified as computationally intensive and the task type is marked as 1; when... If the value is less than or equal to the theoretical threshold, the job is classified as a memory-intensive task and its type is marked as 0. The obtained job type label is used as a supervision label in subsequent machine learning model training and classification tasks.

[0037] S3: The semantic features of the job are fused with the behavioral features of the user's historical job submissions to construct a comprehensive feature vector. This feature vector is then input into the machine learning model for training. At the same time, the real labels of the job types are used to supervise the learning of the model and verify the results, thereby obtaining the HPC job classification model.

[0038] S31. Generate a daily task template based on the user's historical completed task records by date. The template includes the username, task name, date, task submission time, completion time, and the number of tasks of type 0 and type 1.

[0039] Specifically, firstly, job-level data files from multiple months are read, and the username, task name, date, job submission time, completion time, and the number of type 0 and type 1 jobs are extracted. Then, the time field is normalized and parsed, and a completion date identifier is generated based on the job completion time. On this basis, the job records are grouped by username, job name, and completion date as keys, and the total number of jobs completed in each group on the same day, the number of type 1 jobs, the earliest submission time, the earliest completion time, and the latest completion time are counted. Finally, a statistical result file for daily template behavior modeling is generated and saved in NPZ or CSV format.

[0040] S32. Extract the user's historical behavioral characteristics within N days prior to the current task submission time based on the daily statistics template, including the proportion of type 1 tasks, the daily average purity of different tasks, and the stability of user-submitted tasks.

[0041] Specifically, based on the daily statistics template in this example, historical data from 30 days to 1 day prior to the current job submission date is selected as the statistical window to ensure that future information is not introduced during feature calculation. Furthermore, for historical records on the same day as the current sample, their final completion time must be earlier than the current job submission time to ensure that the feature construction process adheres to strict temporal causality. After selecting the historical data that meets the criteria, user behavior features are calculated based on these records, including the proportion of task type 1, the daily average purity of different tasks, and the stability of user-submitted tasks. The percentage of Category 1 tasks represents the proportion of Category 1 tasks in the historical assignments. The calculation formula is as follows: The daily average purity for different tasks represents the average level of category concentration each day, calculated using the following formula: The stability of user-submitted tasks is represented by the proportion of stable days out of the total number of days and the proportion of jobs generated on stable days out of all jobs. The calculation formula is as follows: Where N represents the number of days in the history window, This represents the total number of assignments on day d. This represents the number of Category 1 assignments on day d. This represents the purity of the task category distribution on day d, used to measure whether the tasks on that day are mainly concentrated in a single category; This indicates whether the category distribution of user-submitted jobs on day d satisfies the stability condition (i.e., the number of jobs is not less than 8 and the category purity is not less than 0.9).

[0042] This step utilizes historical daily template statistics to construct user behavior characteristics over a past time window, given the current job submission time, and outputs a user feature matrix for subsequent machine learning model input.

[0043] S33. The semantic features of the job are fused with the extracted historical user behavior features to construct a comprehensive feature vector. This feature vector is then input into the machine learning model for training. Simultaneously, supervised learning and model validation are performed using the real labels of the job categories to obtain an offline HPC job classification model.

[0044] like Figure 3 The flowchart illustrates the offline HPC classification model training method according to an embodiment of the present invention. The specific process is as follows.

[0045] Specifically, the semantic features of the job and the user's historical behavior features are aligned based on the username and job submission time, and then feature fusion is performed. The job semantic features are 768-dimensional semantic vectors generated by a pre-trained language model, and the user's historical behavior features are 4-dimensional statistical feature vectors. A comprehensive feature vector is constructed through feature concatenation, resulting in a 772-dimensional job feature representation. This comprehensive feature vector is then used as input data for training the machine learning model. The machine learning model can be a classification model such as KNN, Random Forest (RF), Logistic Regression, XGBoost, Multilayer Perceptron (MLP), or Linear Support Vector Machine (Linear SVM). In this embodiment, the XGBoost model is preferably used for training. During model training, the job type label obtained in step S22 is used as a supervision signal for model training and validation, thereby obtaining a trained offline HPC job classification model.

[0046] S4: During the online deployment phase of the model, the distribution changes of user task categories in the short term are estimated using the real label information of tasks recently completed by users through the EWMA method, and the short-term probability estimation results of the corresponding task types are obtained.

[0047] like Figure 2 The flowchart of the exponentially weighted moving average method provided in this embodiment of the invention is as follows.

[0048] S41. Initialize the EWMA drift tracker, establish a drift state for each user or user-task name combination, and set an initial probability estimate for cold start; Specifically, the online drift tracker is initialized, and key parameters in the EWMA algorithm are set, including the smoothing coefficient. and initial probability Among them, the smoothing coefficient Used to control the weighting ratio between historical probability estimates and the latest observations; initial probability Used to provide a default probability estimate when historical observation data is lacking.

[0049] For each user or user-task name combination template key, initialize the corresponding drift state, which includes the current probability estimate. and the number of observed missions When it's necessary to retrieve the drift state of an object (such as a user or a composite template), the corresponding state information is retrieved using the key. If the object does not already exist in the state table, a new state record is automatically created, and its probability estimate is initialized. (In this embodiment, it is set to 0.5), and the number of observations is 0. The drift state of each key is maintained independently to track behavioral changes of different users or task types.

[0050] Set EWMA update factor (Set to 0.01 in this embodiment) to adjust the update magnitude of the current true label to the historical probability estimation result, where a larger value... More sensitive to recent category changes, smaller This makes the probability estimation process smoother and more stable.

[0051] S42. Before predicting the current submitted task, read the real tag information of the user's completed tasks, and use EWMA to recursively update the category probability according to the corresponding user or user-task name combination template key, so as to obtain the EWMA probability estimate of the user's short-term job category distribution.

[0052] Specifically, when a new job arrives, the current drift estimate for that user or user-task name combination template is read. This represents the current short-term category distribution state; once an assignment is completed and the true label is obtained, the program uses EWMA to update the probability estimate of that object, calculated as follows: in, This indicates the actual label of the currently completed task, with a value of 0 or 1. This represents the EWMA probability estimate of the updated distribution of user short-term job categories.

[0053] This update method allows the latest observations to influence probability estimates while retaining historical information, thus achieving smooth tracking of job type probabilities. After the update is complete, the observation count for that user or user-task name combination template is also updated. The drift probability must be updated using the true labels of completed jobs to ensure that only historical information is used in the prediction process and that future data is not introduced.

[0054] S43. Obtain the EWMA probability and observation count corresponding to the current user or user-task name, and calculate the drift weight according to the warm-up mechanism.

[0055] Specifically, obtain the EWMA probability estimate corresponding to the current user or the user-task name combination. and the number of observations To avoid the drift signal having too much influence on the final prediction results when there is limited observation data in the early stages of the system, a preheating mechanism is introduced. The drift weights are dynamically adjusted based on the number of observations. The weight of the drift signal in the prediction fusion is gradually increased, and the calculation formula is as follows: in, This indicates the number of observations during the preheating phase. This represents the maximum weight of the drift signal. (When the number of observations...) Less than the preset threshold At that time, drift weight The number of observations gradually increases proportionally; when the number of observations reaches or exceeds the threshold... At that time, the drift weight is fixed at its maximum value. This mechanism can reduce the impact of drift signals on prediction results in the early stages of the system, and gradually enhance the role of drift estimation in prediction fusion as the amount of observation data increases.

[0056] S5: Integrate the recent probability estimation results with the model prediction results online to obtain the final job type prediction probability, and output the HPC job classification result of the user's current submitted task.

[0057] like Figure 4 The flowchart illustrates the online deployment HPC job prediction method according to an embodiment of the present invention. The specific process is as follows.

[0058] S51, Based on drift weight The final predicted probability is obtained by weighting and fusing the output probability of the classification model with the EWMA probability.

[0059] Specifically, after a user submits a computation task online, the system first obtains the task description information, including the username, task name, and resource request information. Then, it uses a pre-trained language model (bge-base-en-v1.5) to semantically encode the task description information, resulting in a semantic feature vector. Simultaneously, based on the current task submission time, it extracts the user's historical behavioral features from the N days prior to the submission date, including statistical features such as the user's task category ratio, task purity, and behavioral stability. Subsequently, it fuses the task semantic features with the user's historical behavioral features to construct a comprehensive feature vector. This comprehensive feature vector is then input into an online HPC task classification model for prediction, obtaining the predicted probability of the task belonging to different categories. Further, it obtains the probability estimate calculated based on EWMA for the user's currently completed tasks and weights it with the output probability of the classification model to obtain the final predicted probability of the submitted task, used to determine whether the task is a computationally intensive or memory-intensive task. The calculation formula is as follows: in, This represents the output probability of the classification model. This represents the final predicted probability of a user submitting a job online.

[0060] S52, Based on the final predicted probability Output the HPC job classification results (class 0 or 1).

[0061] Specifically, in obtaining the final predicted probability Then, it is compared with a preset classification threshold to determine the job category. In this embodiment, the classification threshold is set to 0.5: when the final predicted probability... When the probability is greater than or equal to 0.5, the user-submitted online job is classified as a Class 1 task, i.e., a computationally intensive task; when the final predicted probability... When the value is less than 0.5, the jobs submitted online by users are classified as type 0 tasks, i.e., memory-intensive tasks.

[0062] S53. When the job execution is completed and the real label is obtained, record the job's end time, the corresponding user or user-task name combination template key, and the real label, and update the EWMA drift status accordingly.

[0063] Specifically, when the system detects that a user-submitted job has been completed and obtains its actual label... At that time, based on the user or user-task name combination key to which the job belongs, the end time and real label of the job are recorded, and the above information is added to the update queue for cache management. When the user continues to submit tasks, the corresponding job record is retrieved from the update queue, and the EWMA state in the drift tracker is updated using its real label, thereby correcting the short-term class probability estimate of the current user-task name combination.

[0064] In summary, this invention presents an online drift-adaptive framework for HPC job semantic representation and classification. By combining pre-trained language models that encode job semantic features with users' historical job behavior features, the model can simultaneously consider job semantic information and user behavior patterns. Through weighted fusion of the prediction probabilities of the offline-trained classification model and the EWMA drift probabilities, it can track changes in user behavior in real time, achieving online awareness of job category distribution drift, thereby improving job category prediction capabilities in dynamic environments. This invention improves the robustness of online model classification results and possesses good real-time performance and engineering deployability, making it suitable for practical HPC cluster online scheduling environments.

[0065] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A semantic representation and classification method for HPC jobs based on online drift adaptation, characterized in that, The method includes the following steps: S1: Obtain the job information description of the user-submitted task, and use a pre-trained language model to perform semantic encoding on the job information description to obtain the semantic feature representation of the job; S2: Obtain the runtime metrics information of user-submitted tasks in the HPC cluster after completion, perform performance characteristic analysis on the completed jobs based on the Roofline model, classify the job types based on the analysis results, and generate real labels for job classification. S3: The semantic feature representation is fused with the behavioral features of users' historical job submissions to construct a comprehensive feature vector. The comprehensive feature vector is then input into the machine learning model for training. At the same time, the real labels of job classification are used to supervise the machine learning model and verify the results, thereby obtaining the HPC job offline classification model. Based on the HPC job offline classification model, the model prediction results corresponding to the actual tasks are obtained. S4: Using the real tag information of the user's recently completed tasks, the distribution change of the user's task category is estimated by the exponentially weighted moving average method (EWMA), and the recent probability estimation results of the corresponding task type are obtained. S5: Integrate the recent probability estimation results with the model prediction results online to obtain the final predicted probability of the job type, and output the HPC job classification results of the actual task.

2. The HPC job semantic representation and classification method based on online drift adaptation according to claim 1, characterized in that, Step S1 includes: S11. Obtain the initial job information description when the user submits the task from the database, and preprocess it to obtain the preprocessed job information description. S12. Use a pre-trained language model to encode the semantic features of the pre-processed job information description to obtain the semantic feature representation of the job.

3. The HPC job semantic representation and classification method based on online drift adaptation according to claim 1, characterized in that, Step S2 includes: S21. Obtain the running metric information after the user-submitted task in the HPC cluster is completed, and calculate the actual peak running intensity of the task based on the Roofline model; S22. Compare the actual peak intensity of operation with the theoretically calculated intensity threshold corresponding to the hardware indicators of the HPC cluster, and determine the true label of the job classification based on the comparison results.

4. The HPC job semantic representation and classification method based on online drift adaptation according to claim 1, characterized in that, Step S3 includes: S31. Generate a daily task template based on the user's historical completed task records, organized by date. S32. Extract user behavior characteristics from the N days prior to the current task submission time based on the daily task template; S33. The semantic features and behavioral features of the assignment are fused to construct a comprehensive feature vector. The comprehensive feature vector is then input into the machine learning model for training. At the same time, the real labels of the assignment classification are used to supervise the machine learning model and verify the results, thereby obtaining the HPC assignment offline classification model. The actual task is then input into the HPC assignment offline classification model to obtain the model prediction results.

5. The HPC job semantic representation and classification method based on online drift adaptation according to claim 1, characterized in that, Step S4 includes: S41. Initialize the EWMA drift tracker and establish the drift state; S42. Read the real tag information of the user's recently completed tasks, and use EWMA to recursively update the category probability based on the drift state to obtain the EWMA probability estimate of the user's short-term task category distribution, i.e. the recent probability estimate result. S43. Calculate the drift weight based on the preheating mechanism.

6. The HPC job semantic representation and classification method based on online drift adaptation according to claim 5, characterized in that, The specific steps of S41 are as follows: Initialize the EWMA drift tracker and set the smoothness coefficient. and initial probability For each user or user-task name combination template key, initialize the corresponding drift state, which includes the current probability estimate. and the number of observed missions And set the update factor for EWMA. .

7. The HPC job semantic representation and classification method based on online drift adaptation according to claim 6, characterized in that, The specific steps of S42 are as follows: Read the current probability estimate This represents the current short-term category distribution state; once an assignment is completed and the true label information is obtained, the current probability estimate is updated using EWMA to obtain a new current probability estimate. That is, the recent probability estimation results, and then update the number of observed tasks. .

8. The HPC job semantic representation and classification method based on online drift adaptation according to claim 7, characterized in that, New current probability estimate for: in, This indicates the actual label information of the currently completed tasks. This represents the new current probability estimate, and the updated coefficients of EWMA. .

9. The HPC job semantic representation and classification method based on online drift adaptation according to claim 8, characterized in that, Step S5 includes: S51. Based on the drift weight, the recent probability estimation results and the model prediction results are weighted and fused online to obtain the final job type prediction probability. S52. Based on the final job type, predict the probability and output the HPC job classification results.

10. The HPC job semantic representation and classification method based on online drift adaptation according to claim 9, characterized in that, The method also includes: When the job execution is completed and the real tag information is obtained, the end time and real tag information of the job are recorded according to the user or user-task name combination template, and the EWMA drift status is updated accordingly.

Images