I / O characteristic prediction method, device, equipment and storage medium

Through the analysis of description information of supercomputer jobs and pre-trained model processing, the prediction of I/O characteristics is automatically solved, and the analysis complexity and resource allocation problems are solved, and computer performance is improved.

CN118312398BActive Publication Date: 2024-11-05国家超级计算天津中心
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410733289.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-07
Publication Date
2024-11-05
Estimated Expiration
2044-06-07

AI Technical Summary

Technical Problem

In a supercomputer environment, analyzing the I/O characteristics of different jobs is complex and difficult to accurately predict, resulting in the inability to effectively allocate storage resources and affecting computer performance.

Method used

By performing path analysis on the description information of the current transmission job, we determine whether the work path has recently submitted jobs. If not, we use the pre-trained I/O feature prediction model to extract, fusion and prediction to automatically predict the job's job. I/O features.

Benefits of technology

It realizes accurate I/O characteristic prediction for different jobs, optimizes storage resource allocation, and improves the performance of supercomputers.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118312398B_ABST
    Figure CN118312398B_ABST
Patent Text Reader

Abstract

The present disclosure relates to an I / O characteristic prediction method, device, equipment and storage medium. Perform path analysis on the description information of the current transmission job to determine the working path of the current transmission job; determine whether there is the first most recently submitted job under the working path of the current transmission job; if there is no first most recently submitted job under the working path of the current transmission job, use a pre-trained I / O characteristic prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job. In this way, for different jobs in a supercomputer, when the most recently submitted job cannot be found under the working path of the current transmission job, the pre-trained I / O characteristic prediction model is used to predict the I / O characteristics of the current transmission job. Thus, different methods are integrated to automatically and accurately predict the I / O characteristics of different jobs, so that appropriate storage resources are allocated based on the I / O characteristics of different jobs, and the performance optimization effect of the supercomputer is finally improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the field of data analysis, and in particular to an I / O characteristic prediction method, device, equipment and storage medium. Background Art

[0002] In a supercomputer environment, analyzing the I / O characteristics of different jobs becomes a complex task because the I / O system is complex, its performance is highly dependent on the network system, and it is shared by all users.

[0003] Therefore, for different jobs in supercomputers, it is necessary to propose an I / O characteristic prediction method so that appropriate storage resources can be allocated based on the I / O characteristics of different jobs to achieve the optimal performance of the supercomputer. Summary of the invention

[0004] In order to solve the above technical problems, the present disclosure provides an I / O characteristic prediction method, device, equipment and storage medium.

[0005] In a first aspect, the present disclosure provides an I / O characteristic prediction method, the method comprising:

[0006] Performing path analysis on the description information of the current transmission job to determine the working path of the current transmission job;

[0007] Determine whether there is a first most recently submitted job under the working path of the current transmission job;

[0008] If the first most recently submitted job does not exist under the working path of the current transmission job, the current transmission job is processed using a pre-trained I / O feature prediction model to predict the I / O characteristics of the current transmission job.

[0009] In some embodiments of the present disclosure, performing path analysis on the description information of the current transmission job to determine the working path of the current transmission job includes:

[0010] Acquire identification information of the current transmission job from description information of the current transmission job;

[0011] The identification information is transmitted to a target storage area, and a working path of the current transmission job in the target storage area is obtained.

[0012] In some embodiments of the present disclosure, the using of a pre-trained I / O feature prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job includes:

[0013] Based on the feature extraction module in the I / O feature prediction model, feature extraction processing is performed on the current transmission job to obtain multimodal feature data of the current transmission job;

[0014] Based on the feature fusion module in the I / O feature prediction model, feature fusion processing is performed on the multimodal feature data of the current transmission job to obtain feature fusion data of the current transmission job;

[0015] Based on the prediction module in the I / O feature prediction model, label information corresponding to the feature fusion data is predicted, and the label information is used as the I / O characteristic of the current transmission job.

[0016] In some embodiments of the present disclosure, the feature extraction module in the I / O feature prediction model performs feature extraction processing on the current transmission job to obtain multimodal feature data of the current transmission job, including:

[0017] Based on the enumeration type coding network in the feature extraction module, the user features and description information of the current transmission job are encoded to obtain the enumeration type coding of the user features and the enumeration type coding of the description information;

[0018] Building a network based on the job subject features in the feature extraction module, performing text processing on the subject character string containing the working path and description information of the current transmission job to obtain the job subject features;

[0019] Building a network based on the job script program features in the feature extraction module, performing text processing on the job script program of the current transmission job to obtain the job script program features;

[0020] Among them, the enumeration type code of the user characteristics, the enumeration type code of the description information, the job subject characteristics, and the job script program characteristics constitute the job script program characteristics.

[0021] In some embodiments of the present disclosure, it also includes:

[0022] If the first most recently submitted job exists in the working path of the current transmission job, the I / O characteristics of the first most recently submitted job are used as the I / O characteristics of the current transmission job.

[0023] In some embodiments of the present disclosure, it also includes:

[0024] If the first most recently submitted job does not exist under the working path of the current transmission job, determining whether a second most recently submitted job of the user who submitted the current transmission job is found;

[0025] If the second most recently submitted job is found, the I / O characteristics of the second most recently submitted job are used as the I / O characteristics of the current transmission job.

[0026] In some embodiments of the present disclosure, it also includes:

[0027] Obtaining a sample job and a label of an I / O characteristic of the sample job;

[0028] Based on the sample jobs and the labels of the I / O characteristics, the initial model is iteratively trained, and the hyperparameters in the model training process are updated until the currently trained model meets the cutoff training condition, thereby obtaining a trained I / O feature prediction model.

[0029] In a second aspect, the present disclosure provides an I / O characteristic prediction device, the device comprising:

[0030] An analysis module, used to perform path analysis on the description information of the current transmission job to determine the working path of the current transmission job;

[0031] A determination module, configured to determine whether there is a first recently submitted job under the working path of the current transmission job;

[0032] The prediction module is used to process the current transmission job using a pre-trained I / O feature prediction model to predict the I / O characteristics of the current transmission job if the first most recently submitted job does not exist under the working path of the current transmission job.

[0033] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the device comprising:

[0034] one or more processors;

[0035] a storage device for storing one or more programs,

[0036] When one or more programs are executed by one or more processors, the one or more processors implement the method provided in the first aspect.

[0037] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0038] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0039] An I / O characteristic prediction method, device, equipment and storage medium of an embodiment of the present disclosure performs path analysis on the description information of the current transmission job to determine the working path of the current transmission job; determines whether there is the first most recently submitted job under the working path of the current transmission job; if there is no first most recently submitted job under the working path of the current transmission job, uses a pre-trained I / O characteristic prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job. In this way, for different jobs in a supercomputer, when the most recently submitted job cannot be found under the working path of the current transmission job, the pre-trained I / O characteristic prediction model is used to predict the I / O characteristics of the current transmission job. As a result, the I / O characteristics of different jobs can be accurately predicted automatically in combination with different methods, so that appropriate storage resources are allocated based on the I / O characteristics of different jobs, and the performance optimization effect of the supercomputer is finally improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0042] Figure 1 A flowchart of an I / O characteristic prediction method provided by an embodiment of the present disclosure;

[0043] Figure 2 A schematic diagram of the structure of a supercomputer provided in an embodiment of the present disclosure;

[0044] Figure 3 A schematic diagram of the process of "using a pre-trained I / O feature prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job" in S130 provided in an embodiment of the present disclosure;

[0045] Figure 4 A flowchart of a training process of an I / O feature prediction model provided in an embodiment of the present disclosure;

[0046] Figure 5 A schematic diagram of the structure of an I / O characteristic prediction device provided in an embodiment of the present disclosure;

[0047] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0049] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0050] The new generation of supercomputers are usually equipped with multiple layers of storage resources with different capabilities, making it possible to provide appropriate storage resources for computing jobs with different I / O characteristics. The current high-performance computing storage software stack has a wealth of configurable parameters. High-performance computing (HPC) users achieve optimal performance by configuring different I / O software stack parameters for computing jobs with different I / O characteristics. However, HPC users lack HPC system knowledge and cannot accurately analyze the I / O characteristics of different jobs. At the same time, the complex structure of the I / O system increases the difficulty of HPC users to analyze I / O characteristics, making it impossible to take advantage of the diversified I / O resources in the supercomputer and ultimately unable to achieve the optimal performance of the supercomputer.

[0051] In order to improve the accuracy of I / O characteristic prediction, the following Figures 1 to 5 The I / O characteristic prediction method provided in the embodiment of the present disclosure is described. In the embodiment of the present disclosure, the I / O characteristic prediction method can be executed by an electronic device or a server. The specific form of the electronic device or the server can be a supercomputer.

[0052] Figure 1 A schematic flow chart of an I / O characteristic prediction method provided by an embodiment of the present disclosure is shown.

[0053] like Figure 1 As shown, the I / O characteristic prediction method may include the following steps.

[0054] S110: Perform path analysis on the description information of the current transmission job to determine the working path of the current transmission job.

[0055] In this embodiment, when it is necessary to predict the I / O characteristics of a high performance computing job, the supercomputer obtains the current transmission job and reads the description information of the current transmission job, and analyzes the working path of the current transmission job based on the description information.

[0056] The current transmission job refers to a high-performance computing job that requires I / O characteristic prediction. Different transmission jobs correspond to different parameters such as user characteristics, job work paths, and job script program characteristics. The description information of the current transmission job carries parameters such as identification information, job partition, and job name.

[0057] The job path refers to the storage path of the current transmission job.

[0058] The specific implementation method of S110 includes but is not limited to the following methods: obtaining identification information of the current transmission job from the description information of the current transmission job; transmitting the identification information to the target storage area, and obtaining the working path of the current transmission job in the target storage area.

[0059] For details, see Figure 2 The input module 210 of the supercomputer obtains the current transmission job and reads the description information of the current transmission job. Then, the input module 210 transmits the description information of the current transmission job to the heuristic prediction module 230. The heuristic prediction module 230 transmits the description information to the target storage area 220 and queries the working path of the current transmission job from the target storage area 220 through a specific query method (for example, a query method based on json or sql).

[0060] The target storage area includes but is not limited to a memory in a supercomputer, and may also be other forms of storage space. Optionally, the target storage area may also store other data, such as intermediate results generated by the I / O characteristic prediction process, current transmission operations, and other data.

[0061] S120: Determine whether there is a first most recently submitted job under the working path of the current transmission job.

[0062] In this embodiment, the supercomputer searches the working path of the current transmission job for other jobs submitted recently, that is, searches the working path for the first most recently submitted job. If not, continue to execute S130 and continue to predict the I / O characteristics of the current transmission job in other ways.

[0063] It is understandable that if there is a first recently submitted job under the working path of the current transmission job, the supercomputer considers that the I / O characteristics of the first recently submitted job are consistent with the I / O characteristics of the current transmission job. Accordingly, the method further includes: if there is a first recently submitted job under the working path of the current transmission job, using the I / O characteristics of the first recently submitted job as the I / O characteristics of the current transmission job.

[0064] In this way, when predicting the I / O characteristics of a high-performance computing job, first, based on the description information of the current transmission job, find out whether there are other jobs submitted recently under the working path of the job. If so, the I / O characteristics of the other jobs submitted recently are directly used as the I / O characteristics of the current transmission job. If not, continue to predict the I / O characteristics of the current transmission job through other methods, thereby realizing automatic and accurate prediction of the I / O characteristics of the high-performance computing job. In addition, this type of I / O characteristic prediction saves labor costs and is fast, and is suitable for I / O characteristic prediction of a large number of high-performance computing jobs.

[0065] S130: If there is no first recently submitted job under the working path of the current transmission job, use a pre-trained I / O feature prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job.

[0066] Continue to see Figure 2 Specifically, when the supercomputer determines that there is no first recently submitted job under the working path of the current transmission job, the supercomputer calls the pre-trained I / O feature prediction model from the machine learning module 240. Since the I / O feature prediction model is trained using a large number of high-performance computing jobs and their corresponding feature labels, the current transmission job and its description information are input into the I / O feature prediction model to predict the I / O characteristics of the current transmission job.

[0067] Furthermore, after the supercomputer determines the I / O characteristics of the current transmission job, it outputs and displays the I / O characteristics of the current transmission job to the user through the output module 250, or allows other modules to allocate appropriate storage resources based on the I / O characteristics of different jobs.

[0068] Optionally, the I / O characteristics of the current transmission job include but are not limited to I / O sensitivity, I / O ratio, maximum memory usage, meta-operation intensity, throughput intensity, etc. I / O sensitivity can also be specifically divided into read sensitivity and write sensitivity.

[0069] In other cases, the method also includes: if the first most recently submitted job does not exist under the working path of the current transmission job, determining whether the second most recently submitted job of the user who submitted the current transmission job is found; if the second most recently submitted job is found, using the I / O characteristics of the second most recently submitted job as the I / O characteristics of the current transmission job.

[0070] Specifically, if the first most recently submitted job is not found under the working path of the current transmission job, the supercomputer first obtains the identification information of the current transmission job from the description information of the current transmission job; then, the identification information is transmitted to the target storage area, and the target storage area is searched for the most recently submitted job by the user with the identification information, that is, it is determined whether the second most recently submitted job is found; finally, if the supercomputer finds the second most recently submitted job, it is considered that the I / O characteristics of the second most recently submitted job are consistent with the I / O characteristics of the current transmission job, and the I / O characteristics of the second most recently submitted job are used as the I / O characteristics of the current transmission job. If the supercomputer does not find the second most recently submitted job, the I / O characteristics of the current transmission job are predicted using S130.

[0071] An I / O characteristic prediction method of an embodiment of the present disclosure performs path analysis on the description information of the current transmission job to determine the working path of the current transmission job; determines whether there is the first most recently submitted job under the working path of the current transmission job; if there is no first most recently submitted job under the working path of the current transmission job, uses a pre-trained I / O characteristic prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job. In this way, for different jobs in a supercomputer, when the most recently submitted job cannot be found under the working path of the current transmission job, the pre-trained I / O characteristic prediction model is used to predict the I / O characteristics of the current transmission job. As a result, the I / O characteristics of different jobs can be accurately predicted automatically by combining different methods, so that appropriate storage resources are allocated based on the I / O characteristics of different jobs, and the performance optimization effect of the supercomputer is finally improved.

[0072] In another embodiment of the present disclosure, a process of predicting the I / O characteristics of the current transmission job using the I / O feature prediction model is explained in detail.

[0073] Figure 3 A flow chart of "using a pre-trained I / O feature prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job" in S130 provided by an embodiment of the present disclosure is shown.

[0074] like Figure 3 As shown, in S130, "using a pre-trained I / O feature prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job" may include the following steps.

[0075] S310: Based on the feature extraction module in the I / O feature prediction model, feature extraction processing is performed on the current transmission job to obtain multimodal feature data of the current transmission job.

[0076] In this embodiment, the supercomputer can use an integrated model as an I / O feature prediction model. First, based on the feature extraction module in the I / O feature prediction model, multiple features are extracted from the current transmission job to form multimodal feature data.

[0077] Specifically, the feature extraction module includes a coding network, a job theme feature construction network, and a job script program feature construction network. Accordingly, the specific implementation methods of S310 include but are not limited to the following methods: based on the enumeration type coding network in the feature extraction module, the user features and description information of the current transmission job are encoded to obtain the enumeration type coding of the user features and the enumeration type coding of the description information; based on the job theme feature construction network in the feature extraction module, the subject string containing the work path and description information of the current transmission job is subjected to text processing to obtain the job theme feature; based on the job script program feature construction network in the feature extraction module, the current transmission job is subjected to text processing to obtain the job script program feature; wherein, the enumeration type coding of the user features, the enumeration type coding of the description information, the job theme feature, and the job script program feature constitute the job script program feature.

[0078] The user characteristics of the current transmission job include but are not limited to the user's research field, city, unit, etc.; the description information of the current transmission job includes but is not limited to the job name, submission time, number of requested cores, number of requested nodes, job partition, etc. In this way, the user characteristics and description information of the current transmission job are converted into enumeration codes through the enumeration coding network in the feature extraction module.

[0079] Among them, the supercomputer uses the job subject features in the feature extraction module to build a network. First, the working path and description information of the current transmission job are concatenated with quotation marks to form a subject string. Then, the subject string is processed by text processing such as replacing punctuation and stop words to obtain the subject short sentence as the job subject feature.

[0080] Among them, the supercomputer uses the job script program features in the feature extraction module to build a network, and performs text processing on the subject string such as replacing punctuation and stop words to form different texts as job script program features.

[0081] In this way, the feature extraction module is used to perform feature extraction processing on the current transmission job to obtain multimodal features such as enumeration code, job subject features, job script program features, etc., which facilitates other networks of the I / O feature prediction model to accurately predict the I / O characteristics of the current transmission job based on multimodal features.

[0082] S320: Based on the feature fusion module in the I / O feature prediction model, feature fusion processing is performed on the multimodal feature data of the current transmission operation to obtain feature fusion data of the current transmission operation.

[0083] In this embodiment, the supercomputer uses an early fusion method to perform fusion processing on the generated multimodal features. Specifically, after the supercomputer uses the feature extraction module in the I / O feature prediction model to perform feature extraction processing, the feature fusion module in the I / O feature prediction model can be used to perform feature fusion processing on the extracted multimodal features, thereby generating feature fusion data for the current transmission operation. In this way, by fusing multimodal features, it is helpful to improve the accuracy of I / O feature prediction.

[0084] S330: Based on the prediction module in the I / O feature prediction model, predict the label information corresponding to the feature fusion data, and use the label information as the I / O characteristics of the current transmission job.

[0085] In this embodiment, the supercomputer can pre-select a classification network as a prediction module in the I / O feature prediction model, and then use the prediction module in the I / O feature prediction model to perform classification processing based on feature fusion data to output label information, and use the label information as the I / O characteristics of the current transmission job.

[0086] In this way, different modules of the I / O feature prediction model are used to process the current transmission job, so as to accurately and quickly predict the I / O characteristics of the current transmission job, realize automatic prediction of I / O characteristics, and reduce the prediction cost of I / O characteristics.

[0087] In yet another embodiment of the present disclosure, the training process of the I / O feature prediction model is explained in detail.

[0088] Figure 4 A flowchart of the training process of the I / O feature prediction model provided by an embodiment of the present disclosure is shown.

[0089] like Figure 4 As shown, the training process of the I / O feature prediction model may include the following steps.

[0090] S410: Obtain a sample job and a label of an I / O characteristic of the sample job.

[0091] In this embodiment, before the supercomputer predicts the I / O characteristics of the high performance computing job, it obtains a plurality of sample jobs through an input device and obtains labels of the I / O characteristics generated for different sample jobs.

[0092] Optionally, labels of I / O characteristics of different sample jobs include but are not limited to I / O sensitivity, I / O proportion, meta-operation intensity, throughput intensity, and maximum memory usage. I / O sensitivity can be divided into read sensitivity and write sensitivity, and read sensitivity and write sensitivity correspond to different labels. I / O proportion, meta-operation intensity, throughput intensity, and maximum memory usage can be calculated using statistical methods (such as k-sigma) and marked as 1 when greater than a preset value (such as µ+kσ), and marked as 0 when less than a preset value (such as µ+kσ).

[0093] S420, based on the labels of the sample jobs and I / O characteristics, iteratively train the initial model, and update the hyperparameters in the model training process until the currently trained model meets the cutoff training condition, thereby obtaining a trained I / O feature prediction model.

[0094] In order to ensure that the model training process proceeds in an orderly manner, before model training, encoding or feature construction can be performed on sample jobs and their description information.

[0095] Optionally, for the user features and job partitions of the sample job description information, enumerated data is constructed, that is, these data are encoded in binary encoding, specifically including the following steps: (1) Obtain a list of all job partitions of the user features and description information of the sample job, for example: ['partition 1', 'partition 2', 'partition 3']; (2) Set partition 1 to 0, partition 2 to 1, and partition 3 to 2; (3) Convert the values ​​to binary partition 1 to 00, partition 2 to 01, and partition 3 to 10; (4) Split each bit of the binary number into independent columns, each bit representing a feature, for example, 00 is split into [0,0]. Thus, in this way, an enumerated job partition is constructed, so that the machine learning module of the supercomputer performs model training based on the enumerated job partition.

[0096] Optionally, the job name in the working path and description information of the sample job is concatenated with quotation marks to form a subject string, and the subject string is preprocessed by replacing punctuation marks and stop words, etc. to form text corresponding to the subject string, so that the machine learning module of the supercomputer performs model training based on the text corresponding to the subject string.

[0097] Optionally, the job script program of the sample job is preprocessed by replacing punctuation marks, stop words, etc. to form text corresponding to the job script program, so that the machine learning module of the supercomputer performs model training based on the text corresponding to the job script program.

[0098] In this embodiment, when the supercomputer uses the machine learning module to perform model training, it first sets the hyperparameters of the initial model, and divides the labels of sample jobs and I / O characteristics into training sets and test sets. Then, based on the labels of sample jobs and I / O characteristics, it iterates the initial model for regular incremental training, and updates the hyperparameters during the model training process until the hyperparameters are optimal and the loss value of the current training model is less than or equal to a preset threshold. It is determined that the currently trained model meets the cutoff training conditions, and the currently trained model is used as the I / O feature prediction model.

[0099] In the above manner, based on the labels of sample jobs and I / O characteristics, and combined with the limiting condition of hyperparameters, the I / O feature prediction model is iteratively trained, thereby improving the generalization ability of the I / O feature prediction model, so that when the I / O feature prediction model is used to perform I / O feature prediction, it is beneficial to improve the I / O feature prediction accuracy of the supercomputer.

[0100] The present disclosure also provides an I / O characteristic prediction device for implementing the above-mentioned I / O characteristic prediction method. Figure 5 In the embodiment of the present disclosure, the I / O characteristic prediction device may be an electronic device or a server. The electronic device or the server may specifically be a supercomputer.

[0101] Figure 5 A schematic diagram of the structure of an I / O characteristic prediction device provided by an embodiment of the present disclosure is shown.

[0102] like Figure 5 As shown, the I / O characteristic prediction device 500 may include:

[0103] The analysis module 510 is used to perform path analysis on the description information of the current transmission job to determine the working path of the current transmission job;

[0104] A determination module 520, configured to determine whether there is a first recently submitted job under the working path of the current transmission job;

[0105] The prediction module 530 is used to process the current transmission job using a pre-trained I / O feature prediction model to predict the I / O characteristics of the current transmission job if the first most recently submitted job does not exist under the working path of the current transmission job.

[0106] An I / O characteristic prediction device of an embodiment of the present disclosure performs path analysis on the description information of the current transmission job to determine the working path of the current transmission job; determines whether there is the first most recently submitted job under the working path of the current transmission job; if there is no first most recently submitted job under the working path of the current transmission job, uses a pre-trained I / O characteristic prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job. In this way, for different jobs in a supercomputer, when the most recently submitted job cannot be found under the working path of the current transmission job, the pre-trained I / O characteristic prediction model is used to predict the I / O characteristics of the current transmission job. Thus, different methods are integrated to automatically and accurately predict the I / O characteristics of different jobs, so that appropriate storage resources are allocated based on the I / O characteristics of different jobs, and ultimately the performance optimization effect of the supercomputer is improved.

[0107] In some embodiments of the present disclosure, the analysis module 510 includes:

[0108] A first acquisition unit, configured to acquire identification information of the current transmission job from description information of the current transmission job;

[0109] The second acquisition unit is configured to transmit the identification information to a target storage area and acquire a working path of the current transmission job in the target storage area.

[0110] In some embodiments of the present disclosure, the prediction module 530 includes:

[0111] A first determining unit, configured to perform feature extraction processing on the current transmission job based on a feature extraction module in the I / O feature prediction model to obtain multimodal feature data of the current transmission job;

[0112] A second determining unit is used to perform feature fusion processing on the multimodal feature data of the current transmission job based on the feature fusion module in the I / O feature prediction model to obtain feature fusion data of the current transmission job;

[0113] The third determining unit is used to predict the label information corresponding to the feature fusion data based on the prediction module in the I / O feature prediction model, and use the label information as the I / O characteristic of the current transmission job.

[0114] In some embodiments of the present disclosure, the first determining unit is specifically configured to:

[0115] Based on the enumeration type coding network in the feature extraction module, the user features and description information of the current transmission job are encoded to obtain the enumeration type coding of the user features and the enumeration type coding of the description information;

[0116] Building a network based on the job subject features in the feature extraction module, performing text processing on the subject character string containing the working path and description information of the current transmission job to obtain the job subject features;

[0117] Building a network based on the job script program features in the feature extraction module, performing text processing on the job script program of the current transmission job to obtain the job script program features;

[0118] Among them, the enumeration type code of the user characteristics, the enumeration type code of the description information, the job subject characteristics, and the job script program characteristics constitute the job script program characteristics.

[0119] In some embodiments of the present disclosure, the device further includes:

[0120] The first determining module is configured to use the I / O characteristics of the first recently submitted job as the I / O characteristics of the current transmission job if the first recently submitted job exists in the working path of the current transmission job.

[0121] In some embodiments of the present disclosure, the device further includes:

[0122] A second determination module, configured to determine whether a second most recently submitted job of a user who submitted the current transmission job is found if the first most recently submitted job does not exist under the working path of the current transmission job;

[0123] The third determining module is configured to use the I / O characteristics of the second most recently submitted job as the I / O characteristics of the current transmission job if the second most recently submitted job is found.

[0124] In some embodiments of the present disclosure, the device further includes:

[0125] A sample acquisition module, used to acquire a sample job and a label of an I / O characteristic of the sample job;

[0126] The iterative training module is used to iteratively train the initial model based on the sample job and the label of the I / O characteristic, and update the hyperparameters in the model training process until the currently trained model meets the cutoff training condition, thereby obtaining a trained I / O feature prediction model.

[0127] It should be noted that Figure 5 The I / O characteristic prediction device 500 shown can execute Figures 1 to 4 The various steps in the method embodiment shown in the figure are implemented Figures 1 to 4 The various processes and effects in the method embodiment shown are not described in detail here.

[0128] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown. The electronic device may be the supercomputer mentioned in the above embodiment.

[0129] like Figure 6 As shown, the electronic device may include a processor 601 and a memory 602 storing computer program instructions.

[0130] Specifically, the processor 601 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0131] The memory 602 may include a large capacity memory for information or instructions. By way of example and not limitation, the memory 602 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive or a combination of two or more of these. Where appropriate, the memory 602 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 602 may be inside or outside the integrated gateway device. In a specific embodiment, the memory 602 is a non-volatile solid-state memory. In a specific embodiment, the memory 602 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (Electrically Erasable Programmable ROM, EEPROM), an electrically rewritable ROM (EAROM) or a flash memory, or a combination of two or more of these.

[0132] The processor 601 reads and executes the computer program instructions stored in the memory 602 to perform the steps of the I / O characteristic prediction method provided in the embodiment of the present disclosure.

[0133] In one example, the electronic device may further include a transceiver 603 and a bus 604. Figure 6 As shown, the processor 601, the memory 602 and the transceiver 603 are connected via a bus 604 and communicate with each other.

[0134] The bus 604 includes hardware, software, or both. For example, but not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a Memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, the bus 604 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0135] The following is an embodiment of a computer-readable storage medium provided in an embodiment of the present disclosure. The computer-readable storage medium and the I / O characteristic prediction method of the above-mentioned embodiments belong to the same inventive concept. For details not described in detail in the embodiment of the computer-readable storage medium, reference can be made to the embodiment of the above-mentioned I / O characteristic prediction method.

[0136] This embodiment provides a storage medium including computer executable instructions. When the computer executable instructions are executed by a computer processor, they are used to perform an I / O characteristic prediction method. The method includes:

[0137] Performing path analysis on the description information of the current transmission job to determine the working path of the current transmission job;

[0138] Determine whether there is a first most recently submitted job under the working path of the current transmission job;

[0139] If the first most recently submitted job does not exist under the working path of the current transmission job, the current transmission job is processed using a pre-trained I / O feature prediction model to predict the I / O characteristics of the current transmission job.

[0140] Of course, the computer executable instructions of a storage medium including computer executable instructions provided in an embodiment of the present disclosure are not limited to the above method operations, and can also execute related operations in the I / O characteristic prediction method provided in any embodiment of the present disclosure.

[0141] Through the above description of the implementation method, the technicians in the relevant field can clearly understand that the present disclosure can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer cloud platform (which can be a personal computer, a server, or a network cloud platform, etc.) to execute the I / O characteristic prediction method provided by each embodiment of the present disclosure.

[0142] Note that the above are only preferred embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the present disclosure is not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present disclosure. Therefore, although the present disclosure is described in more detail through the above embodiments, the present disclosure is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present disclosure, and the scope of the present disclosure is determined by the scope of the appended claims.

Claims

1. A method for predicting I / O characteristics, characterized in that: include: Performing path analysis on description information of a current transmission job to determine a working path of the current transmission job, wherein the current transmission job refers to a high-performance computing job that requires I / O characteristic prediction; Determine whether there is a first most recently submitted job under the working path of the current transmission job; If the first most recently submitted job does not exist under the working path of the current transmission job, the current transmission job is processed by using a pre-trained I / O feature prediction model to predict the I / O characteristics of the current transmission job, wherein the I / O characteristics of the current transmission job include I / O sensitivity, I / O proportion, maximum memory usage, meta-operation intensity, and throughput intensity; The using of the pre-trained I / O feature prediction model to process the current transmission job and predict the I / O characteristics of the current transmission job includes: Based on the feature extraction module in the I / O feature prediction model, feature extraction processing is performed on the current transmission job to obtain multimodal feature data of the current transmission job; Based on the feature fusion module in the I / O feature prediction model, feature fusion processing is performed on the multimodal feature data of the current transmission job to obtain feature fusion data of the current transmission job; Based on the prediction module in the I / O feature prediction model, predict the label information corresponding to the feature fusion data, and use the label information as the I / O characteristics of the current transmission job; The feature extraction module in the I / O feature prediction model performs feature extraction processing on the current transmission job to obtain multimodal feature data of the current transmission job, including: Based on the enumeration type coding network in the feature extraction module, the user features and description information of the current transmission job are encoded to obtain the enumeration type coding of the user features and the enumeration type coding of the description information; Building a network based on the job subject features in the feature extraction module, performing text processing on the subject character string containing the working path and description information of the current transmission job to obtain the job subject features; Building a network based on the job script program features in the feature extraction module, performing text processing on the job script program of the current transmission job to obtain the job script program features; Among them, the enumerated code of the user characteristics, the enumerated code of the description information, the job subject characteristics, and the job script program characteristics constitute the multimodal feature data.

2. The method according to claim 1, characterized in that The performing path analysis on the description information of the current transmission job to determine the working path of the current transmission job includes: Acquire identification information of the current transmission job from description information of the current transmission job; The identification information is transmitted to a target storage area, and a working path of the current transmission job in the target storage area is obtained.

3. The method according to claim 1, characterized in that Also includes: If the first recently submitted job exists in the working path of the current transmission job, the I / O characteristics of the first recently submitted job are used as the I / O characteristics of the current transmission job.

4. The method according to claim 1, characterized in that: Also includes: If the first most recently submitted job does not exist under the working path of the current transmission job, determining whether a second most recently submitted job of the user who submitted the current transmission job is found; If the second most recently submitted job is found, the I / O characteristics of the second most recently submitted job are used as the I / O characteristics of the current transmission job.

5. The method according to claim 1, characterized in that Also includes: Obtaining a sample job and a label of an I / O characteristic of the sample job; Based on the sample jobs and the labels of the I / O characteristics, the initial model is iteratively trained, and the hyperparameters in the model training process are updated until the currently trained model meets the cutoff training condition, thereby obtaining a trained I / O feature prediction model.

6. An I / O characteristic prediction device, characterized in that: include: An analysis module, configured to perform path analysis on description information of a current transmission job to determine a working path of the current transmission job, wherein the current transmission job refers to a high performance computing job requiring I / O characteristic prediction; A determination module, configured to determine whether there is a first recently submitted job under the working path of the current transmission job; A prediction module, configured to process the current transmission job using a pre-trained I / O feature prediction model if the first most recently submitted job does not exist under the working path of the current transmission job, and predict the I / O characteristics of the current transmission job, wherein the I / O characteristics of the current transmission job include I / O sensitivity, I / O proportion, maximum memory usage, meta-operation intensity, and throughput intensity; The prediction module comprises: A first determining unit, configured to perform feature extraction processing on the current transmission job based on a feature extraction module in the I / O feature prediction model to obtain multimodal feature data of the current transmission job; A second determining unit is used to perform feature fusion processing on the multimodal feature data of the current transmission job based on the feature fusion module in the I / O feature prediction model to obtain feature fusion data of the current transmission job; a third determining unit, configured to predict label information corresponding to the feature fusion data based on a prediction module in the I / O feature prediction model, and use the label information as the I / O characteristic of the current transmission job; The first determining unit is specifically configured to: Based on the enumeration type coding network in the feature extraction module, the user features and description information of the current transmission job are encoded to obtain the enumeration type coding of the user features and the enumeration type coding of the description information; Building a network based on the job subject features in the feature extraction module, performing text processing on the subject character string containing the working path and description information of the current transmission job to obtain the job subject features; Building a network based on the job script program features in the feature extraction module, performing text processing on the job script program of the current transmission job to obtain the job script program features; Among them, the enumerated code of the user characteristics, the enumerated code of the description information, the job subject characteristics, and the job script program characteristics constitute the multimodal feature data.

7. An electronic device, characterized in that: include: processor; A memory for storing executable instructions; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method described in any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the processor implements the method according to any one of claims 1 to 5.