Performance Calculation Method, System and Medium for High-Performance Computer Workflow Scheduling

Abstract

The present invention discloses a performance calculation method, system and medium for high-performance computer workflow scheduling. The present invention includes initializing a set X and a set Y of tasks that can start running, and then iterating: if the termination time of the task vi in X is equal to the variable k, then update X and Y; if Y is not empty, select a set of tasks Z, update the start time of the tasks therein to k and add them to X, calculate the estimated end time of the k-th stage for the tasks in X, and use the shortest estimated end time as the completion time of the k-th stage; repeat the iteration until both X and Y are empty, and output the sum of the completion times of each stage as the total completion time of the workflow. The present invention can realize the performance quantitative calculation of high-performance computer workflow scheduling, quickly determine the influence of each variable on the workflow scheduling performance during the operation of the workflow scheduling, and thus can conveniently determine the minimum resource quantity required for the best workflow scheduling performance.

Description

Technical Field

[0001] The present invention relates to the workflow scheduling technology of high-performance computers, and particularly to a performance calculation method, system and medium for high-performance computer workflow scheduling. Background Art

[0002] A workflow (scientific workflow) is a sequence of tasks defined to achieve various scientific research goals. Driven by the development of service-oriented architecture and its loose coupling characteristics, workflows have become a key technology in the current distributed and dynamic environment. Workflows have prominent advantages in describing complex scientific problems, which makes them commonly used to solve large-scale scientific problems in fields such as bioinformatics, astronomy, and physics. Specifically, a workflow usually consists of multiple independent computing tasks with strict dependencies. A directed acyclic graph is an effective tool for representing a workflow. As Figure 1 shown, the nodes in the figure represent independent tasks in the workflow, and the directed edges represent the dependencies between tasks. The weight of a node represents the amount of computing resources (number of cores or number of nodes) required for that task, and the weight of a directed edge represents the data dependency between two tasks. For example, see Figure 1 , where the weight of task v1 being 2 means that task v1 needs to use 2 computing resources (2 cores or 2 computing nodes), and the directed edge from task v1 to task v3 means that task v3 needs to start running after task v1 and needs to read 10 GB of data generated by task v1.

[0003] The purpose of workflow scheduling is to maintain good overall performance or throughput of the computing system while meeting user requirements and resource provider management metrics. For a single workflow, minimizing the completion time of the workflow is a common scheduling goal. For a given workflow, the shorter its completion time, the higher the workflow scheduling performance. The amount of computing and I / O resources allocated to the workflow and the scheduling strategy will both affect the scheduling performance of the workflow. In recent years, with the continuous improvement of the parallel performance of large-scale high-performance computers, high-performance computers have gradually become an important operating platform for workflows. The scenario of scheduling workflows on high-performance computers is relatively complex. Figure 2This paper demonstrates a workflow scheduling scenario on a high-performance computer (HPC). First, a temporary, independent resource partition is allocated on the HPC. All tasks in the workflow run within this partition, where the total resources within the partition should be greater than the resource requirements of any one task. Each workflow task is submitted to the HPC as a separate batch task at an appropriate time. A shared file system serves as the storage medium for data transmission between tasks. Each workflow task reads and writes data to the shared file system. During workflow execution, I / O interference occurs between concurrently running tasks, and the read and write rates of the resource partition to the file system may vary over time. A burst buffer is a storage technology proposed to meet user requirements for improved I / O performance. Using a burst buffer can increase the total bandwidth available to applications. Therefore, for HPCs with a certain burst buffer capacity, depending on the size of the burst buffer, some or all tasks can be allowed to use the burst buffer to improve application I / O efficiency. When a task is allowed to use the burst buffer, all its output is directed to the burst buffer. However, for each task, where it reads data (from the shared file system or the burst buffer) depends on whether its predecessor task uses the burst buffer. As can be seen, scheduling workflows on high-performance computers (HPCs) is a complex scenario. Currently, there is a lack of effective tools to study the impact of various variables on workflow scheduling performance during workflow scheduling, particularly the impact of various scheduling strategies. Furthermore, for a given workflow, it is difficult to determine the minimum number of resources required to achieve optimal workflow scheduling performance. Therefore, achieving performance calculation for workflow scheduling on high-performance computers has become a key technical issue that needs to be addressed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is as follows: In response to the above-mentioned problems of the prior art, a performance calculation method, system and medium for high-performance computer workflow scheduling are provided. The present invention can realize the performance quantitative calculation of high-performance computer workflow scheduling, so as to quickly determine the impact of various variables on workflow scheduling performance during the operation of workflow scheduling, thereby conveniently determining the minimum number of resources required for optimal workflow scheduling performance.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is:

[0006] A performance calculation method for high-performance computer workflow scheduling, comprising:

[0007] S1, initialize variable k to 1, completion time of stage k T kis 0; set the start time and end time of all tasks in the workflow to 0; initialize the set X for recording currently running tasks as empty and the set Y for tasks that can start running as empty;

[0008] S2. For all tasks in the set X, if the end time of a certain task v i is equal to the variable k, then update the set X and the set Y;

[0009] S3. If the set Y is not empty, then select the set Z of tasks that start running now from the set Y. If the set Z is not empty, then jump to S4; otherwise, jump to S5;

[0010] S4. For each task in the set Z, update the start time of this task to the variable k, and add this task to the set X:

[0011] S5. For each task v i in the set X, calculate the read / write bandwidth of task v i for the shared file system, and calculate the estimated end time of task v i in the k-th stage based on the read / write bandwidth of this task for the shared file system

[0012] S6. Determine the estimated end time [[ID= twenty-five ]] of the shortest task v n in the k-th stage according to all tasks in the set X, n assign the corresponding estimated end time to the completion time T of the k-th stage k , and set the end time of task v n to k + 1;

[0013] S7. If both the set X and the set Y are empty, then take the sum of all completion times T k of the k-th stage as the total completion time of the workflow, end and exit; otherwise, increment the variable k by 1, and jump to step S2 to enter the next stage.

[0014] Optionally, the function expressions for updating the set X and the set Y in step S2 are:

[0015] X = X - v i ,

[0016]

[0017] In the above formula, v i is the task whose end time is equal to the variable k, v j is the task in the successor task set succ(v i ) of task v i , and v mFor task v j The tasks in the set of predecessor tasks pred(v j ), e m For task v m The end time.

[0018] Optionally, in step S5, the estimated end time of task v i In the k-th stage is The function expression is:

[0019]

[0020] In the above formula, Is the estimated end time of task v i In the (k + 1)-th stage, T k Is the completion time of the k-th stage, C i Is the total computation volume of task v i , Is the computation volume of task v i In the m-th stage, p i Is the number of computing nodes used by task v i , s is the computing speed of the computing node, And Are respectively the read and write bandwidths of task v i To the shared file system in the k-th stage, FI i And FO i Respectively represent the total read and write data sizes of task v i To the shared file system, BI i And BO i Respectively represent the total read and write data sizes of task v i To the burst buffer, F is the size of the shared file system, B is the size of the burst buffer, I i And O i Are respectively the read and write data sizes of task v i , And W i (k) Are respectively the read and write data sizes of task v i In the k-th stage.

[0021] Optionally, step S5 also includes, for each task v in set X i , calculating the computation volume of task v i [[ID=7�]]In the k-th stage, and the function expression for calculating the computation volume of task v i In the k-th stage is:

[0022]

[0023] In the above formula, Ci (k) For task v i The computational workload at the k-th stage, T k+1 Is the completion time of the (k + 1)-th stage, min represents taking the minimum value, R B And W B Are the read and write bandwidths of the burst buffer respectively.

[0024] Optionally, the task v i The read bandwidth at the k-th stage, task v i The calculation function expressions for the write bandwidth at the k-th stage are:

[0025]

[0026]

[0027] In the above formula, And Are the read and write bandwidths of task v i At the k-th stage respectively, task v j Is the set X of tasks that are currently running at the k-th stage (k) Of the tasks in it, min represents taking the minimum value, And Respectively represent that task v i Is the read and write density for the shared file system, R s Is the read bandwidth of the compute node for the parallel file system, W s Is the write bandwidth of the compute node for the parallel file system, R f Is the read bandwidth of the node partition for the parallel file system, W f Is the write bandwidth of the node partition for the parallel file system, p i For task v i The number of compute nodes used.

[0028] Optionally, the calculation function expression for the read and write density of task v i For the shared file system is:

[0029]

[0030]

[0031] In the above formula, And Respectively represent that task v i Is the read and write density for the shared file system, FI i And FO i Respectively represent that task v i The total sizes of the read and write data for the shared file system, t i For task vi The time required to run.

[0032] Optionally, task v i The calculation function expression for the total size of read and write data of the shared file system is:

[0033]

[0034]

[0035] Mission v i The calculation function expression for the total size of read and write data in the burst buffer is:

[0036]

[0037]

[0038] Among them, FI i and FO i Represents task v i The total size of data read and written to the shared file system, BI i and BO i Represents task v i The total size of read and write data to the burst buffer, v j For task v i The successor task set succ(v i ), To adopt the preset storage resource scheduling strategy To schedule task v j Whether the burst buffer is allowed to be used, e ji Represents task v j and task v i The amount of data transmitted between i For task v i The total size of the written data, and there is task v i The calculation function expression of the total calculation amount is:

[0039] C i =p i *s,

[0040] Mission v i The calculation function expression of the total size of read and write data is:

[0041]

[0042]

[0043] In the above formula, e ij For task v iand task v j The amount of data transferred to and from task v, succ(v i ) is the set of successor tasks of task v i , and pred(v i ) is the set of predecessor tasks of task v i .

[0044] In addition, the present invention also provides a performance calculation method for high-performance computer workflow scheduling, including:

[0045] S101. For the four factors affecting the workflow scheduling performance in a high-performance computer, namely system computing resources, system storage resources, computing resource scheduling policies, and storage resource scheduling policies, by fixing three of the factors and changing the resource configuration of the remaining one factor, multiple resource configuration schemes are generated. For a given workflow, for each resource configuration scheme, the total completion time of the corresponding workflow is obtained by calling the aforementioned performance calculation method for high-performance computer workflow scheduling, so as to obtain the relationship curve between each influencing factor and the total completion time of the workflow;

[0046] S102. Based on the relationship curve between each influencing factor and the total completion time of the workflow, the optimal resource configuration is selected from the four influencing factors respectively to obtain the optimal resource configuration scheme for the high-performance computer for the given workflow.

[0047] In addition, the present invention also provides a performance calculation system for high-performance computer workflow scheduling, including a microprocessor and a memory connected to each other, and the microprocessor is programmed or configured to execute the performance calculation method for high-performance computer workflow scheduling.

[0048] In addition, the present invention also provides a computer-readable storage medium, in which a computer program is stored, and the computer program is used to be programmed or configured by a microprocessor to execute the performance calculation method for high-performance computer workflow scheduling.

[0049] Compared with the prior art, the present invention mainly has the following advantages: The present invention can realize the quantitative calculation of the performance of high-performance computer workflow scheduling, quickly determine the influence of each variable on the workflow scheduling performance during the operation of the workflow scheduling, and thus can conveniently determine the minimum amount of resources required for the best workflow scheduling performance. Description of the Drawings

[0050] Figure 1 It is a schematic diagram of an existing workflow.

[0051] Figure 2 It is a schematic diagram of the scenario of scheduling a workflow on an existing high-performance computer.

[0052] Figure 3 This is a schematic diagram of the basic process of the method according to an embodiment of the present invention.

[0053] Figure 4 This is a multi-stage schematic diagram of the workflow running process in an embodiment of the present invention. Detailed implementation manners

[0054] As Figure 3 shown, the performance calculation method for the workflow scheduling of the high-performance computer in this embodiment includes:

[0055] S1. Initialize the variable k to 1 and the completion time T of the k-th stage to 0; set the start time and end time of all tasks in the workflow to 0; initialize the set X for recording the currently running tasks to be empty and the set Y for the tasks that can start running to be empty; k For all tasks in the set X, if the end time of a certain task v

[0056] equals the variable k, then update the set X and the set Y; i

[0057] S3. If the set Y is not empty, then select the set Z of tasks that start running now from the set Y. If the set Z is not empty, then jump to S4; otherwise, jump to S5;

[0058] S4. For each task in the set Z, update the start time of this task to the variable k, and add this task to the set X;

[0059] S5. For each task v i in the set X, calculate the read / write bandwidth of the task v i to the shared file system, and calculate the estimated end time of the task v i in the k-th stage according to the read / write bandwidth of this task to the shared file system

[0060] S6. Determine, according to all tasks in the set X, the task v with the shortest estimated end time in the k-th stage n , assign the corresponding estimated end time n of the task v to the completion time T of the k-th stage k , and set the end time of the task v n to k + 1;

[0061] S7. If both the set X and the set Y are empty, then take the sum of all the completion times T k of the k-th stage as the total completion time of the workflow, end and exit; otherwise, increment the variable k by 1, and jump to step S2 to enter the next stage.

[0062] In step S1 of this embodiment, setting the start time and end time of all tasks in the workflow to 0 can be expressed as:

[0063]

[0064] In the above formula, G represents the workflow, and v i is a task in the workflow G, and s i and e i represent the start time and end time of v i respectively. Specifically, in this embodiment, a directed acyclic graph is used to represent the workflow. In a directed acyclic graph G=(V, E), the node set V={v1, v2,..., v n} are independent tasks in the workflow. Each task v i has two attributes: one is the number of computing nodes p i used by the task v i , and the other is the time t i required for the task v i to run. Therefore, the total computing amount of the task can be defined as:

[0065] C i =p i *s,

[0066] where s is a constant representing the operation speed of a single computing node in the system. The edge set represents the dependency relationship between tasks in the workflow. If (v1, v2) belongs to the edge set E, it means there is a dependency relationship between v1 and v2, that is, v2 must start running after v1 finishes running. The weight of each edge represents the data dependency relationship between two tasks with a dependency relationship. For example, e ij =10 means that the task v i writes 10GB of data to the file system or burst buffer, and the task vj then reads the 10GB of data written by the task v i from the file system or burst buffer. The set of predecessor tasks of the task v i is pred(v i ), and the set of successor tasks is succ(v i ). Add an empty task v0. If a task has no predecessor tasks, use the empty task v0 as the predecessor of this task to form an initial task. Add an empty task v n+1 . If a task has no successor tasks, use the task v n+1 as the successor task of this task (end task v n+1 ). The initial task v0 and the end task v n+1The number of computing nodes used is 0, and the running time is also 0. In step S1 of this embodiment, initializing the set X for recording currently running tasks as empty and the set Y of tasks that can start running is expressed as:

[0067] X = φ,

[0068] Y = {v i s.t. pred(v i ) = v0},

[0069] In the above formula, φ represents the empty set, pred(v i ) is the set of predecessor tasks of task v i , and v0 is the initial task, that is, a task that does not depend on other task data and can be executed independently.

[0070] In step S2 of this embodiment, the function expressions for updating set X and set Y are:

[0071] X = X - v i ,

[0072]

[0073] In the above formula, v i is the task whose termination time is equal to variable k, v j is the task in the successor task set succ(v i ) of task v i , v m is the task in the predecessor task set pred(v j ) of task v j , e m is the end time of task v m .

[0074] In step S3 of this embodiment, if the set Y is non-empty (Y ≠ φ), then the set Z of tasks that start running now is selected from the set Y, which is implemented by calling the preset computing resource scheduling policy η. Generally speaking, the computing resource scheduling policy η is selected and scheduled according to the workflow G, the number of computing nodes P, and the set Y, which can be expressed as Z = η(G, P, Y), such as random or round-robin, etc. The specific implementation is not the content concerned by the method of this embodiment, so the detailed implementation details are not described in detail here.

[0075] In step S4 of this embodiment, for each task in the set Z update the start time of this task to variable k, and add this task to the set X, which can be expressed as: s i = k, X = X + v i .

[0076] In step S5 of this embodiment, calculate task v iThe estimated end time F in the k-th stage i (k) The function expression is as follows:

[0077]

[0078] In the above formula, is the estimated end time of task v i in the (k + 1)-th stage, T k is the completion time of the k-th stage, C i is the total computation volume of task v i is the computation volume of task v i in the m-th stage, p i is the number of computing nodes used by task v i s is the operation speed of the computing nodes, and are the read and write bandwidths of task v i to the shared file system in the k-th stage, FI i and FO i respectively represent the total read and write data sizes of task v i to the shared file system, BI i and BO i respectively represent the total read and write data sizes of task v i to the burst buffer, F is the size of the shared file system, B is the size of the burst buffer, I i and O i are the total read and write data sizes of task v i and W i (k) are the read and write data sizes of task v i in the k-th stage. It should be noted that the parameters involved in the above function calculation can be calculated in this step or in the appropriate steps in the previous text as needed.

[0079] In this embodiment, step S5 further includes, for each task v in set X i , calculating the computation volume of task v i in the k-th stage, and the function expression for calculating the computation volume of task v i in the k-th stage is:

[0080]

[0081] In the above formula, is the computation volume of task v i in the k-th stage, T k+1 is the completion time of the (k + 1)-th stage, min represents taking the minimum value, R​​B and W B are the read and write bandwidths of the burst buffer, respectively. The burst buffer usually has a communication link and a storage device independent of the shared file system. Therefore, the read and write bandwidths of the burst buffer are not affected by the shared file system. The probability of I / O interference in the burst buffer is lower than that in the shared file system, and there are two reasons for this. One is that the read and write bandwidths of the burst buffer are usually much larger than those of the shared file system. Using the burst buffer can complete the read and write requirements of tasks in a shorter time and avoid I / O contention. The other is that due to the capacity limitation of the burst buffer, usually only some discrete tasks can use the burst buffer. Combining the above analysis, for the sake of simplicity of the model, task v i has a fixed read bandwidth of R B for the burst buffer at any stage, and a fixed write bandwidth of W B for the burst buffer. When modeling the workflow running process, the start or termination of each task is defined as an event. The start running time of task v i is s i , and the termination running time is e i . The time period between two consecutive events is defined as a stage. Obviously, the running cycle of the workflow consists of multiple stages, and the completion time of the workflow is the time when the last event occurs. As Figure 4 shown, event p is the start of tasks v i and v j , and its time is T p . Event p + 1 is the termination of task v i and the start of task v j , and its time is T p+1 . The time period between them is stage p. Within a certain stage, the running tasks are stable. Therefore, the read and write bandwidths of the file system are also constant. This is the key property of the stage and also the original intention of our definition of the concept of the stage. The set of tasks running in stage k is X (k) . For task v i , the amount of computation completed within stage k is . Analyzing an already started stage k, for any task v i ∈X (k) , the following five constraints need to be satisfied:

[0082]

[0083]

[0084]

[0085]

[0086]

[0087] Among the above five restrictions, the first restriction describes that all computing tasks must be completed within phase k for this task, and the second and third restrictions describe that the read and write tasks to the shared file system must be completed within phase k for this task. It should be noted that according to the previous assumption, the read and write requirements of task v i for the shared file system within phase k account for the proportion of the total read and write requirements of task v i for the shared file system, which is equal to the proportion of the computing amount of task v i within phase k to the total computing amount of task v i . Similarly, the fourth and fifth restrictions describe that the read and write tasks to the burst buffer must be completed within phase k for this task. According to the above five restrictions, the functional expression of the computing amount of task v i in the k-th phase is as shown in the above formula. According to the functional expression of the computing amount of task v i in the k-th phase, for each task v i running in phase k, assuming that no events occur after the start of phase k, the estimated end time of task v i can be determined Take the minimum task end time as the occurrence time of the next event, that is:

[0088]

[0089] Assume that the read and write operations of the task are evenly distributed during the task execution. In the absence of I / O interference, the amount of data read and written by the task to the file system is proportional to the progress of the task execution. To describe the I / O interference generated by multiple parallel tasks running simultaneously in the same resource partition in the shared file system, the read bandwidth of task v i in the k-th phase and the write bandwidth of task v i in the k-th phase are defined. In this embodiment, the calculation function expressions of the read bandwidth of task v i in the k-th phase and the write bandwidth of task v i in the k-th phase are:

[0090]

[0091]

[0092] In the above formula, and are the read and write bandwidths of task v i in the k-th phase respectively, and task v jThe set X of tasks that are currently running in the k-th stage (k) For the tasks in and respectively represent the task v i is the read-write density of the shared file system, R s is the read bandwidth of the compute node for the parallel file system, W s is the write bandwidth of the compute node for the parallel file system, R f is the read bandwidth of the node partition for the parallel file system, W f is the write bandwidth of the node partition for the parallel file system, p i is the task v i The number of compute nodes used.

[0093] The read bandwidth of a single compute node for the parallel file system is R s and the write bandwidth is W s . The read bandwidth of the node partition for the parallel file system is R f and the write bandwidth of the node partition for the parallel file system is W f . The read bandwidth of a single compute node for the burst buffer is R b and the write bandwidth is W b . In an actual scenario, the shared file system usually has a large enough capacity. In this embodiment, the calculation function expression of the read-write density of task v i for the shared file system is:

[0094]

[0095]

[0096] In the above formula, and respectively represent the task v i is the read-write density of the shared file system, FI i and FO i respectively represent the total read and write data sizes of task v i for the shared file system, t i is the time required for task v i to run. According to the above function expression, it can be seen that the occupancy of the read-write bandwidth of task v i in stage k is positively correlated with the proportion of the read-write density of this task in the total read-write density, and cannot exceed the maximum read-write bandwidth corresponding to the computing resources it uses.

[0097] In this embodiment, the calculation function expression of the total read and write data sizes of task v i for the shared file system is:

[0098]

[0099]

[0100] Task v i The calculation function expression for the total read and write data sizes of the burst buffer is:

[0101]

[0102]

[0103] where FI i and FO i respectively represent the total read and write data sizes of Task v i to the shared file system, BI i and BO i respectively represent the total read and write data sizes of Task v i to the burst buffer, v j is a task in the successor task set succ(v i ) of Task v i ), is whether Task v is allowed to use the burst buffer when scheduling according to the preset storage resource scheduling policy j (value 1 means allowed, value 0 means not allowed, and if not allowed, only the shared file system can be used), e ji represents the amount of data transferred between Task v j and Task v i , O i is the total write data size of Task v i , and the calculation function expression for the total calculation amount of Task v i is:

[0104] C i = p i * s,

[0105] The calculation function expression for the total read and write data sizes of Task v i is:

[0106]

[0107]

[0108] In the above formula, e ij is the amount of data transferred between Task v i and Task v j (the edge between the nodes corresponding to Task v i and Task v j in the directed acyclic graph G), succ(vi ) is the set of successor tasks of task v i , and pred(v i ) is the set of predecessor tasks of task v i . And there is: I i = FI i + BI i , O i = FO i + BO i .

[0109] In this embodiment, step S6 determines the estimated end time at the k-th stage according to all tasks in set X The task v with the shortest time n can be expressed as:

[0110]

[0111] Assign the estimated end time n corresponding to task v to the completion time T at the k-th stage k , and set the end time of task v n to k + 1, which can be expressed as: e n = k + 1.

[0112] Finally, in step S7, sets X and set Y are judged. If both sets X and Y are empty, then the sum of all completion times T at the k-th stage k is used as the total completion time of the workflow, and the process ends and exits; otherwise, the variable k is incremented by 1 (k = k + 1), and it jumps to step S2 to enter the next stage.

[0113] In addition, on the basis of the foregoing method, this embodiment further provides a performance calculation method for high-performance computer workflow scheduling, including:

[0114] S101. For the four influencing factors of system computing resources, system storage resources, computing resource scheduling policies, and storage resource scheduling policies in high-performance computers on workflow scheduling performance, by fixing three of the influencing factors and changing the resource configuration of the remaining one influencing factor, multiple resource configuration schemes are generated, and for a given workflow, for each resource configuration scheme, the foregoing performance calculation method for high-performance computer workflow scheduling is called to obtain the corresponding total completion time of the workflow, so as to obtain the relationship curve between each influencing factor and the total completion time of the workflow;

[0115] S102. Based on the relationship curves between each influencing factor and the total completion time of the workflow, select the optimal resource allocation among the four influencing factors respectively to obtain the optimal resource allocation plan for the given workflow by the high-performance computer.

[0116] Through step S101, the performance calculation model mentioned above in this embodiment can be used to study the influence of system computing resources, system storage resources, computing resource scheduling strategies, and storage resource scheduling strategies on the workflow scheduling performance. For example, with the system storage resources, computing resource scheduling strategies, and storage resource scheduling strategies fixed and the system computing resources changed, the scheduling performance of the workflow under different system computing resources can be obtained by invoking the performance model, so as to obtain the influence of the quantity of system computing resources on the workflow scheduling performance. Using the above performance calculation model, the minimum resource allocation combination that can achieve the best scheduling performance for a given workflow can be calculated, that is, the minimum resource partition size and the minimum quantity of storage resources allocated to the workflow. Specifically, the value ranges of two variables, namely the resource partition size and the quantity of storage resources, can be set, and the combination of the two variables is used as the search space. The performance model is used to search for the minimum completion time of the given workflow within the search space, and the resource partition size and the quantity of storage resources corresponding to the minimum completion time are the optimal resource allocation.

[0117] In addition, this embodiment also provides a performance calculation system for high-performance computer workflow scheduling, including a microprocessor and a memory connected to each other. The microprocessor is programmed or configured to execute the aforementioned performance calculation method for high-performance computer workflow scheduling. In addition, this embodiment also provides a computer-readable storage medium, in which a computer program is stored. The computer program is used to be programmed or configured by the microprocessor to execute the aforementioned performance calculation method for high-performance computer workflow scheduling.

[0118] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk memory, CD-ROM, optical memory, etc.) that contain computer-usable program code. The present application is described with reference to the flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, such that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for implementing the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks. These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, such that the instructions stored in the computer-readable memory generate a manufactured article including an instruction device that implements the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks. These computer program instructions can also be loaded onto a computer or other programmable data processing device, such that a series of operation steps are executed on the computer or other programmable device to generate a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in the process Figure 1 one process or multiple processes and / or blocks Figure 1 or a device for implementing the functions specified in multiple blocks.

[0119] The above is only the preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the idea of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present invention, several improvements and modifications should also be regarded as the protection scope of the present invention.

Claims

1. A performance calculation method for high-performance computer workflow scheduling, characterized in that, including: S1, Initialize variables as 1, the completion time of the k phase is 0; set the start time and end time of all tasks in the workflow to 0; initialize the set used to record the currently running tasks as empty and the set of tasks that can start running ; S2, for the set of all tasks, if the termination time of a certain task is equal to the variable then update the set and the set ; S3, if the set is non-empty, then select from the set the set that starts running the task now , if the set is non-empty, then jump to S4; otherwise jump to S5; S4. For each task in the set , update the start time of the task to the variable , and add the task to the set ; S5, for each task in the set , calculate the read / write bandwidth of the task to the shared file system, and calculate the estimated end time of the task in the k phase based on the read / write bandwidth of the task to the shared file system ; S6, determine the estimated end time at the phase according to all tasks in the set k ; select the task with the shortest estimated end time in the set , and assign the estimated end time corresponding to the task to the completion time at the k phase, and set the end time of the task to ;​ S7, if both the set and the set are empty, then take the sum of all the completion times k at the stage as the total completion time of the workflow, end and exit; Otherwise, increment the variable by 1 and jump to step S2 to enter the next stage; The update set in step S2 and the set have the following function expressions: ， ， In the above formula, is the task with the end time equal to the variable . is the task in the successor task set of the task. is the task in the predecessor task set of the task. is the end time of the task . The computing task in step S5 at the k estimated end time of the phase is given by the function expression: ， In the above formula, is the task at the k+ estimated end time of the 1st stage, is the completion time of the k stage, is the total computation volume of the task , is the computation volume of the task at the m stage, is the number of computing nodes used by the task , is the operation speed of the computing node, and are respectively the read and write bandwidths of the shared file system by the task at the k stage, and respectively represent the total read and write data sizes of the shared file system by the task , and respectively represent the total read and write data sizes of the burst buffer by the task , is the size of the shared file system, is the size of the burst buffer, and are respectively the total read and write data sizes of the task , and are respectively the read and write data sizes of the task at the k stage.

2. The performance calculation method for high-performance computer workflow scheduling according to claim 1, characterized in that Step S5 also includes for the set each task in , calculating the computational amount of task for the k phase, and the function expression of the computational amount of task for the k phase is: ， In the above formula, is the task at the k computation volume in the stage, k+ is the completion time in the first stage, and are the read and write bandwidths of the burst buffer respectively.

3. The performance calculation method for high-performance computer workflow scheduling according to claim 2, characterized in that, The said task In the k read bandwidth of the stage, the task In the k calculation function expression of the write bandwidth of the stage is as follows: ， ， In the above formula, and are respectively the read and write bandwidths of task in the k stage. Task is the set of tasks that are currently running in the k stage. The min represents taking the minimum value. In the formula, and respectively represent that task is the read / write density of the shared file system. is the read bandwidth of the compute node for the parallel file system. is the write bandwidth of the compute node for the parallel file system. is the read bandwidth of the node partition for the parallel file system. is the write bandwidth of the node partition for the parallel file system. is the number of compute nodes used by task .

4. The performance calculation method for high-performance computer workflow scheduling according to claim 3, characterized in that Task The calculation function expression for the read-write density of the shared file system is as follows: ， ， In the above formula, and respectively represent the read-write density of the task on the shared file system, and respectively represent the total sizes of the read and write data of the task on the shared file system, is the time required for the task to run.

5. The performance calculation method for high-performance computer workflow scheduling according to claim 1, wherein Task The calculation function expression for the total read and write data size of the shared file system is as follows: ， ， Task The calculation function expression for the total size of read and write data in the burst buffer is as follows: ， ， Among them, and respectively represent the total read and write data sizes of task to the shared file system. and respectively represent the total read and write data sizes of task to the burst buffer. is the set of successor tasks of task in . is whether task is allowed to use the burst buffer when scheduling tasks using a preset storage resource scheduling policy . represents the amount of data transferred between task and task . is the total write data size of task , and the calculation function expression for the total computation amount of task is: ， Task The calculation function expression for the total read and write data size is as follows: ， ， In the above formula, is the amount of data transmitted between tasks and task . is the set of successor tasks of task , and is the set of predecessor tasks of task 6. A performance calculation method for high-performance computer workflow scheduling, characterized in that, including: S101. For the four influencing factors on the workflow scheduling performance, namely the system computing resources, system storage resources, computing resource scheduling strategy, and storage resource scheduling strategy in a high-performance computer, by fixing three of the influencing factors and changing the resource configuration of the remaining one influencing factor, multiple resource configuration schemes are generated. For a given workflow, for each resource configuration scheme, the performance calculation method of the high-performance computer workflow scheduling described in any one of claims 1 to 5 is called to obtain the total completion time of the corresponding workflow, thereby obtaining the relationship curve between each influencing factor and the total completion time of the workflow; S102. Based on the relationship curve between each influencing factor and the total completion time of the workflow, the best resource configuration is respectively selected among the four influencing factors to obtain the optimal resource configuration scheme of the high-performance computer for the given workflow.

7. A performance calculation system for high-performance computer workflow scheduling, including a microprocessor and a memory connected to each other, characterized in that, The microprocessor is programmed or configured to execute the performance calculation method of the high-performance computer workflow scheduling described in any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program therein, characterized in that, The computer program is used to be programmed or configured by the microprocessor to execute the performance calculation method of the high-performance computer workflow scheduling described in any one of claims 1 to 5.

Images