Job scheduler and job scheduling methods

Abstract

To determine the number of nodes for efficiently executing a job.SOLUTION: A login node creates a performance model M based on designation of parameters (Nnode, tcmpt, tcomm, pabn, αabn, tbench). The login node uses the performance model M to determine Nspare that minimizes E[C] using the performance model M. For example, it is assumed that Nnode is determined with "Nnode=3" and Nspare is determined with "Nspare=1". In this case, a management node allocates a job to four nodes 1101-1104, which are the sum of Nspare and Nnode. The node 1101 causes each of the nodes 1101 to 1104 to execute a benchmark. The node 1101 selects the nodes 1101, 1102, and 1104 relevant to the three shortest of the benchmark time based on "Nnode=3", and causes the nodes 1101, 1102, and 1104 to execute an application.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to a job scheduler and a job scheduling method. [Background technology]

[0002] Traditionally, there are cluster-type supercomputers that consist of numerous high-performance computers. In cluster-type supercomputers, for example, a job scheduler assigns computing jobs submitted by users to available nodes to perform application calculations. Supercomputers are used for large-scale, advanced scientific and technological calculations such as weather forecasting, space development, and genetic analysis.

[0003] Prior art includes methods for dynamically adjusting performance management and application deployment management tasks. Additionally, there is a technique that, based on operational tests of each processor unit, distributes data processing programs to the processor units confirmed to be functioning correctly, and then assigns the divided data to each processor unit.

[0004] Furthermore, there is a technology that sequentially substitutes performance specification information into a quantitative model to calculate throughput for each pool server, selects the pool server corresponding to the throughput that is greater than the throughput change and closest to it, and instructs the system to execute configuration change control. There is also a technology that predicts the likelihood of failure of nodes running applications in parallel and migrates computing nodes whose likelihood of failure exceeds a threshold to a spare computing node at the next scheduled checkpoint. In addition, there is a technology for job management in an HPC (High Performance Computing) environment. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Special Publication No. 2008-515106 [Patent Document 2] Japanese Patent Application Publication No. 10-162130 [Patent Document 3] International Publication No. 2007 / 034826 [Patent Document 4] U.S. Patent Application Publication No. 2010 / 0223379 [Patent Document 5] U.S. Patent Application Publication No. 2020 / 0004648 [Patent Document 6] U.S. Patent Application Publication No. 2018 / 0121253 [Overview of the project] [Problems that the invention aims to solve]

[0006] However, with conventional technology, if an anomaly occurs in a node within a supercomputer that the system cannot detect, the faulty node may be assigned to the job, potentially degrading the application's computing performance. For example, submitting a job with a redundant number of nodes could mitigate this performance degradation, but too many nodes would lead to decreased supercomputer utilization efficiency and increased usage costs. Conversely, too few nodes would still result in reduced computing performance.

[0007] In one aspect, the present invention aims to determine the number of nodes required to efficiently execute a job. [Means for solving the problem]

[0008] In one embodiment, a job scheduler is provided that, when executing a job on one or more nodes in a system, accepts specifications for the number of nodes used by the application for executing the job, the probability of node failure in the system, the ratio of processing time of abnormal nodes to normal nodes in the system, and the benchmark time for executing a benchmark that is performed before the application in the job. Based on the accepted specifications, it creates a performance model that outputs an expected value of resource consumption for executing the job from the expected value of execution time for executing the job, the number of nodes used, and the number of spare nodes in the job. Using the created performance model, it determines the number of spare nodes that minimize the expected value of resource consumption. [Effects of the Invention]

[0009] According to one aspect of the present invention, it is possible to determine the number of nodes necessary to efficiently execute a job. [Brief explanation of the drawing]

[0010] [Figure 1] Figure 1 is an explanatory diagram showing one embodiment of the job scheduling method according to the embodiment. [Figure 2] Figure 2 is an explanatory diagram showing an example of the system configuration of the job scheduling system 200. [Figure 3] Figure 3 is an explanatory diagram showing an example of a network topology. [Figure 4] Figure 4 is a block diagram showing an example of hardware configuration, including login node 201. [Figure 5] Figure 5 is a block diagram showing an example of the functional configuration of login node 201. [Figure 6] Figure 6 is an explanatory diagram showing an example of how E[C] is calculated. [Figure 7] Figure 7 is a block diagram showing an example of the functional configuration of node Ni. [Figure 8]Figure 8 is an explanatory diagram showing an example of the contents of the benchmark execution time table 800. [Figure 9] Figure 9 is an explanatory diagram showing an example of the operation of the job scheduling system 200. [Figure 10] Figure 10 is an explanatory diagram showing an example of connections between nodes. [Figure 11] Figure 11 is an explanatory diagram showing an example of job execution. [Figure 12] Figure 12 is a flowchart (part 1) showing an example of the job submission process procedure for login node 201. [Figure 13] Figure 13 is a flowchart (part 2) showing an example of the job submission process procedure for login node 201. [Figure 14] Figure 14 is a flowchart showing an example of the specific processing steps for EC calculation. [Figure 15] Figure 15 is a flowchart showing an example of the job execution control processing procedure for node Ni. [Figure 16A] Figure 16A is an explanatory diagram (part 1) showing a specific example of the benchmark time for each node. [Figure 16B] Figure 16B is an explanatory diagram (part 2) showing a specific example of the benchmark time for each node. [Figure 17] Figure 17 is an explanatory diagram showing an example of a prediction for E[C]. [Modes for carrying out the invention]

[0011] Embodiments of the job scheduler and job scheduling method according to the present invention will be described in detail below with reference to the drawings.

[0012] (Embodiment) Figure 1 is an explanatory diagram showing one embodiment of a job scheduling method according to an embodiment. In Figure 1, the information processing device 101 is a computer that determines the number of spare nodes when executing a job using one or more nodes in the system. The system includes multiple nodes that can communicate with each other. The system is, for example, a cluster-type supercomputer.

[0013] A node is a computer that has communication capabilities and can perform various processes. A node may be, for example, a physical server or a virtual machine. A job is a unit of processing work in a computer, for example, a unit of computation specified by a user. The processes performed within a job are, for example, user-dependent processes.

[0014] For example, the processes executed within a job are often performed by all nodes working together to compute using parallelized programs (applications) such as MPI (Message Passing Interface). In parallelized programs, each node performs calculations and communicates with the other nodes.

[0015] For example, in deep learning, collective communication (parameter synchronization) and node-by-node calculations (forward and backward calculations) are performed alternately. Similarly, in fluid dynamics analysis, collective communication and P2P communication (inner product and sparse matrix-vector product of the CG method) are performed alternately with node-by-node calculations.

[0016] Spare nodes are nodes that are prepared in addition to the number of nodes required to run a job. The number of spare nodes is the number of redundant nodes prepared when the number of nodes required is greater than the number of nodes used by the application running the job.

[0017] Here, a job scheduler is software that schedules user-specified computing units (jobs) and assigns them to nodes such as supercomputers. Each job has information such as the computation content, the number of nodes to use, and the maximum usage time (wall-time). Normally, users cannot select and use a specific node.

[0018] In a typical job scheduler, for example, jobs that are added to the queue first are executed first. For example, suppose jobs A, B, and C are added to the queue in the order "Job A → Job B → Job C". Also, assume that the total number of nodes in the supercomputer is "8 nodes". Furthermore, assume that job A uses "3" nodes, job B uses "4" nodes, and job C uses "4" nodes.

[0019] In this case, since Job A was added to the queue before Job C, Job A takes priority over Job C, even if it means creating an idle node. For example, if nodes 1-3 are assigned to Job A and nodes 4-7 are assigned to Job B, then node 8 becomes an idle node. Note that 1-8 correspond to node IDs.

[0020] As a result, nodes with non-consecutive node IDs may be assigned to jobs. For example, if job A completes execution while job B is running, making job C available, job C will be assigned to nodes 1-3 and 8. Nodes 1-3 and Node 8 are nodes with non-consecutive node IDs.

[0021] Furthermore, the nodes assigned to each job are immediately released, for example, when the job's calculation is completed or when the wall-time is exceeded. For example, if the wall-time for job A is set to "1 hour," and job A's calculation is completed in 45 minutes, nodes 1-3 assigned to job A will be released without waiting for the wall-time (1 hour) to expire and will be assigned to the next job. Similarly, if the wall-time for job C is set to "1 hour," even if job C's calculation is not completed in 1 hour, nodes 1-3 and 8 assigned to job C will be released when the wall-time (1 hour) is exceeded.

[0022] Furthermore, in typical job schedulers, a mechanism called backfill may allow idle nodes to be overtaken in the queue. For example, suppose job D is submitted after job C. Let's assume job D uses "1" node. In this case, out of nodes 1-8, nodes 1-3 are assigned to job A, nodes 4-7 are assigned to job B, and node 8 is assigned to job D, which was submitted after job C. Backfill reduces idle nodes and improves the overall utilization efficiency of the supercomputer.

[0023] In supercomputers, hardware and process (software) failures can occur in the nodes. If the system fails to detect such failures, users may be assigned to the faulty nodes, resulting in a decrease in the computational performance of their applications. For example, if a job is assigned to a group of nodes that includes a faulty node, the faulty node will bottleneck the process, reducing computational performance and ultimately leading to decreased supercomputer utilization efficiency and increased user fees.

[0024] Anomalies that can cause application performance degradation include those resulting from jobs previously executed on the node. For example, anomalies may occur if processes or local files created by a previous job are not deleted or initialized by the system. Also, anomalies may occur if settings that affect performance (e.g., clock frequency) were changed by a previous job but were not restored by the system.

[0025] Furthermore, some abnormalities can occur due to malfunctions or bugs in processes or daemons running at the OS (Operating System) level. Additionally, some abnormalities can arise from individual hardware differences, such as variations in the clock frequencies used due to differences in processor power consumption characteristics.

[0026] Furthermore, the network (interconnect) between nodes is shared with other jobs, and latency may occur due to communication from other jobs. In addition, in supercomputers that have the capability to logically share a single node among multiple jobs, hardware such as processors and memory may be subject to competition with other jobs.

[0027] Such anomalies are often only discovered when users submit jobs and check the results, but it is difficult for users to identify the cause. For example, the execution time of an application may not be known before execution, making it difficult to determine whether the performance degradation is due to a specific node.

[0028] Furthermore, if an application is forcibly terminated due to exceeding the wall-time limit, and the termination occurs before the application can output logs to confirm the processing results, it becomes difficult for the user to identify the cause of the performance degradation. Also, since administrators often have no involvement with the applications that users run, it is difficult to resolve the issue through discussions between the user and the administrator.

[0029] Furthermore, due to the nature of the problem, performance degradation is likely to occur in jobs using a large number of nodes, requiring the task of narrowing down the problematic node from among those nodes. However, this process of narrowing down the nodes is time-consuming and resource-intensive. In addition, typical job schedulers do not allow users to specify a particular node when submitting a job. Therefore, users cannot precisely identify and verify the node suspected to be the cause of the problem.

[0030] Furthermore, in typical job schedulers, the backfill mechanism means that if you resubmit a job when the wall-time is exceeded, or manually resubmit it when an anomaly is detected, the faulty node causing the problem may be assigned again. Therefore, resubmitting a job is not a solution to the problem.

[0031] Therefore, one approach is to initially submit the job with a redundant number of nodes, and then remove the slowest nodes from that set to perform the application's calculations, thereby mitigating the degradation of computing performance. However, having too many nodes leads to decreased utilization efficiency and increased usage costs. On the other hand, having too few nodes still results in a degradation of computing performance.

[0032] Therefore, in this embodiment, when submitting jobs with a redundant number of nodes considering the occurrence of abnormal nodes, a job scheduling method for determining the number of nodes necessary to efficiently execute jobs will be described. Here, an example of processing by the information processing device 101 (corresponding to the processing in (1) to (3) below) will be described.

[0033] (1) The information processing device 101 accepts the specification of parameter 110 when executing a job on one or more nodes in the system. The specification of parameter 110 is made, for example, by the user submitting the job. Parameter 110 includes the number of nodes used by the application for job execution. The number of nodes used is a value of 1 or more and is determined, for example, by considering the nature of the application and the computing speed.

[0034] Parameter 110 also includes the probability of anomaly occurring in a node within the system. This probability is a common value across all nodes in the system and is between 0 and 1. The system is, for example, a supercomputer containing multiple nodes (high-performance computers). Parameter 110 also includes the ratio of processing time for abnormal nodes to normal nodes within the system.

[0035] An abnormal node is a node experiencing an anomaly that can degrade application performance. A normal node is any node other than an abnormal node. Processing time refers to, for example, the processing time required for calculations in an application or the processing time required for running a benchmark. The processing time ratio is a value greater than 1, expressed, for example, as the rate of increase in the processing time of an abnormal node compared to the processing time of a normal node.

[0036] Parameter 110 also includes the benchmark time required to run the benchmark. The benchmark is software used to evaluate the performance of nodes that run before the application within a job. The benchmark is run to determine which nodes should be excluded from the group of nodes assigned to the job with a redundant number of nodes.

[0037] Furthermore, parameter 110 may include, for example, a first processing time within the application's execution time that is affected by performance degradation due to abnormal nodes, and a second processing time that is not affected by performance degradation due to abnormal nodes. The first processing time is, for example, the computation time required for calculations at each node in the application. The second processing time is, for example, the communication time required for communication between nodes in the application.

[0038] However, the first processing time and the second processing time may be values ​​specified by the system. For example, the information processing device 101 may set the first processing time to a value determined from the job's wall-time, etc., and the second processing time to a fixed value (for example, 0).

[0039] (2) The information processing device 101 creates a performance model 120 based on the parameters 110 it has received. The performance model 120 is a model that outputs the expected value of resource consumption required for job execution from the expected value of the execution time required for job execution, the number of nodes used, and the number of spare nodes in the job.

[0040] Resource consumption represents the amount of system resources consumed when submitting a job with redundant nodes. Resource consumption is equivalent to the cost that comes into play when considering the increased number of nodes and node usage time during job execution due to the presence of spare nodes and benchmarks.

[0041] Specifically, for example, the information processing device 101 creates a performance model 120 from predetermined model equations (for example, the first, second, third, fourth, and fifth model equations described later) based on the specifications of the received parameters 110. A specific example of the process for creating the performance model 120 will be described later with reference to Figure 5.

[0042] (3) The information processing device 101 uses the created performance model 120 to determine the number of spare nodes that minimize the expected value of resource consumption required for job execution. Specifically, for example, the information processing device 101 uses the performance model 120 to calculate the expected value C of resource consumption while sequentially changing the number of spare nodes from 0 to the number of nodes used by the application.

[0043] The information processing device 101 may then determine the number of spare nodes corresponding to the minimum of the calculated expected resource consumption values ​​C as the number of spare nodes that minimizes the expected resource consumption. The determined number of spare nodes is used as the number of redundant nodes when submitting a job with a redundant number of nodes.

[0044] Thus, according to the information processing device 101, when submitting a job with a redundant number of nodes considering the occurrence of abnormal nodes, it is possible to search for the number of spare nodes that minimize the expected value C of resource consumption required for job execution, and to determine the number of nodes necessary to efficiently execute the job. As a result, the information processing device 101 can submit a job by specifying the number of spare nodes that minimize the expected value of resource consumption required for job execution.

[0045] (Example of system configuration for job scheduling system 200) Next, we will describe an example of a system configuration of the job scheduling system 200, which includes the information processing device 101 shown in Figure 1. Here, we will explain using the example of applying the information processing device 101 shown in Figure 1 to the login node 201 within the job scheduling system 200. The job scheduling system 200 is applied to a supercomputer for executing jobs such as fluid analysis, structural analysis, and electromagnetic field analysis.

[0046] Figure 2 is an explanatory diagram showing an example of the system configuration of the job scheduling system 200. In Figure 2, the job scheduling system 200 includes a login node 201, a management node 202, a client terminal 203, a storage server 204, and computing nodes N1 to Nn (n: a natural number greater than or equal to 2). In the job scheduling system 200, the login node 201, the management node 202, the client terminal 203, the storage server 204, and the computing nodes N1 to Nn are connected via a wired or wireless network 210. The network 210 is, for example, the Internet, a LAN (Local Area Network), or a WAN (Wide Area Network).

[0047] In the following explanation, any computation node among computation nodes N1 to Nn may be referred to as "computation node Ni" (i=1,2,…,n). Also, computation nodes may sometimes be simply referred to as "nodes".

[0048] Here, login node 201 is a computer that the user can directly operate. Login node 201 executes, for example, the submission script P1 shown in Figure 9 below. Submission script P1 is an information processing program for submitting jobs. Login node 201 is, for example, a server.

[0049] The management node 202 is a computer for operating the job scheduling system 200. The management node 202 runs, for example, the job scheduler P2 as shown in Figure 9 below. The job scheduler P2 is a program for job scheduling. The management node 202 is, for example, a server.

[0050] The client terminal 203 is a computer used by users of the job scheduling system 200. For example, a user can submit jobs by operating the login node 201 from the client terminal 203. The client terminal 203 can be, for example, a PC (Personal Computer) or a tablet PC.

[0051] The storage server 204 is a computer that has a file system FS and stores the main program files (executable files) and data of various programs executed by the various nodes 201, 202, N1 to Nn. The various nodes 201, 202, N1 to Nn access the file system FS of the storage server 204, for example, to obtain information about the various programs.

[0052] Computing nodes N1 to Nn are computers to which jobs are assigned. On one of the nodes Ni to which a job is assigned, a job script P3, as shown in Figure 9 below, is executed. Job script P3 is an information processing program for executing the application related to job execution. Each of the computing nodes N1 to Nn is, for example, a server.

[0053] In the job scheduling system 200, for example, compute nodes can communicate with each other, as well as the login node 201, the management node 202, and the storage server 204, using interconnects with a network topology (communication architecture) as shown in Figure 3. A specific example of an interconnect within the job scheduling system 200 is a fat tree type network.

[0054] Here, the login node 201, the management node 202, and the computing node Ni are provided as separate entities, but this is not limited to this configuration. For example, the login node 201, the management node 202, and the computing node Ni may all be implemented on a single computer. The login node 201 may also be implemented on the management node 202. The management node 202 may also be implemented on the computing node Ni. Furthermore, the submission script P1 may be implemented, for example, as a function of the job scheduler P2. Similarly, the job script P3 may also be implemented, for example, as a function of the job scheduler P2.

[0055] (Network topology) Here, we will use Figure 3 to describe the network topology of the interconnects within the job scheduling system 200.

[0056] Figure 3 is an explanatory diagram showing an example of a network topology. In Figure 3, nodes 301 to 308 are an example of the computing nodes N1 to Nn shown in Figure 2. Nodes 301 to 308 are connected via switches 311 to 313 (network devices). Here, the upstream path of the tree-like network structure is redundant. As a result, nodes 301 to 308 can communicate with each other at high performance even when nodes are physically located in discontinuous locations.

[0057] (Example hardware configuration for login node 201, etc.) Next, we will describe an example hardware configuration of the login node 201, management node 202, storage server 204, and compute nodes N1 to Nn shown in Figure 2. Here, the login node 201, management node 202, storage server 204, and compute nodes N1 to Nn will be referred to as "login node 201, etc."

[0058] Figure 4 is a block diagram showing an example of the hardware configuration of a login node 201, etc. In Figure 4, the login node 201, etc., includes a CPU (Central Processing Unit) 401, memory 402, disk drive 403, disk 404, communication interface 405, portable recording medium interface 406, and portable recording medium 407. Each component is connected by a bus 400.

[0059] Here, CPU 401 is responsible for the overall control of the login node 201 and other components. CPU 401 may have multiple cores. Memory 402 includes, for example, ROM (Read Only Memory), RAM (Random Access Memory), and flash ROM. Specifically, for example, flash ROM stores the OS program, ROM stores application programs, and RAM is used as the work area for CPU 401. Programs stored in memory 402 are loaded into CPU 401, causing CPU 401 to execute the coded processes.

[0060] The disk drive 403 controls the reading and writing of data to the disk 404 according to the control of the CPU 401. The disk 404 stores the data written under the control of the disk drive 403. Examples of disks 404 include magnetic disks and optical disks.

[0061] The communication interface 405 is connected to the network 210 via a communication line, and then connected to an external computer via the network 210. The communication interface 405 manages the interface between the network 210 and the inside of the device, and controls the input and output of data from the external computer. For example, a modem or LAN adapter can be used as the communication interface 405.

[0062] The portable recording medium interface 406 controls the reading and writing of data to the portable recording medium 407 according to the control of the CPU 401. The portable recording medium 407 stores the data written under the control of the portable recording medium interface 406. Examples of portable recording media 407 include CD (Compact Disc)-ROM, DVD (Digital Versatile Disk), and USB (Universal Serial Bus) memory.

[0063] Furthermore, the login node 201, etc., may have, in addition to the components described above, an input device, a display, etc. Also, the login node 201, etc., may omit, for example, the portable recording medium I / F 406 and the portable recording medium 407 from the components described above. The client terminal 203 shown in Figure 2 can also be implemented with the same hardware configuration as the login node 201, etc. However, the client terminal 203 may have, in addition to the components described above, an input device, a display, etc.

[0064] (Example of a functional configuration for login node 201) Next, we will describe a functional configuration example for login node 201.

[0065] FIG. 5 is a block diagram showing a functional configuration example of the login node 201. In FIG. 5, the login node 201 includes a reception unit 501, a creation unit 502, a determination unit 503, and an input unit 504. The reception unit 501 to the input unit 504 are functions of the control unit 500. Specifically, for example, a program (input script P1 as shown in FIG. 9 described later) stored in a storage device such as the memory 402, the disk 404, and the portable recording medium 407 of the login node 201 shown in FIG. 4 is executed by the CPU 401, or the function is realized by the communication I / F 405. The processing results of each functional unit are stored in a storage device such as the memory 402 and the disk 404 of the login node 201, for example.

[0066] When executing a job by one or more nodes in the job scheduling system 200, the reception unit 501 receives the specification of parameters. The parameters include, for example, N node , p abn , α abn , t bench . Here, N node represents the number of nodes used by the application for job execution. N node is determined by the user considering, for example, the nature of the application and the calculation speed.

[0067] In the following description, the application for job execution may be simply referred to as "app".

[0068] p abn represents the probability of occurrence of an abnormality of a node in the job scheduling system 200. p abn is a common value for all nodes of the job scheduling system 200 and is a value between 0 and 1. Each node is assumed to be abnormal with p abn and the state does not change during job execution.

[0069] α abn is a coefficient (abnormal node calculation time coefficient) representing the ratio (increase rate) of the processing time of an abnormal node to a normal node in the job scheduling system 200. αabn The value is greater than 1. An abnormal node is calculated in t bench ,t cmpt ) is α abn Assume that it is multiplied by t. For example, an abnormal node is t compared to a normal node. bench is α abn It doubles.

[0070] t bench This represents the benchmark time required to run the benchmark. A benchmark is software used to evaluate the performance of nodes that run before the application within a job. For benchmarks, lightweight software that is computationally limiting, such as LINPACK, is used.

[0071] Here, we assume that when benchmarks are run on all nodes, and the benchmark times for all nodes are sorted in descending order, the abnormal nodes will be at the top. In this case, if the number of abnormal nodes is less than or equal to the number of spare nodes in the job, the abnormal nodes can be excluded from the application execution. On the other hand, if the number of abnormal nodes is greater than the number of spare nodes, the abnormal nodes cannot be excluded from the application execution.

[0072] Also, the parameter is, for example, t cmpt , t comm It may also include t. cmpt This is the computation time required for each node in the application (where t cmpt >0). t cmpt This is an example of the first processing time within the application's execution time that is affected by performance degradation due to abnormal nodes.

[0073] t comm This is the communication time required for communication between nodes in the application (however, t comm ≥0). t comm This is an example of a second processing time within the application's execution time that is not affected by performance degradation due to abnormal nodes. cmpt ,t comm For example, the user makes the decision considering factors such as the nature of the application and the processing speed.

[0074] Furthermore, some applications have a dominant time that does not fall under both computation and communication, such as I / O (Input / Output). In this case, t cmpt Specify a value for the first processing time within the application's execution time that is affected by performance degradation due to abnormal nodes, and t comm You may specify a value for this as a second processing time that is not affected by performance degradation due to abnormal nodes.

[0075] Specifically, for example, the reception unit 501 may accept the specification of parameters included in the job submission request by receiving the job submission request from the client terminal 203 shown in Figure 2. In addition to the parameters mentioned above, the job execution request may include information such as the calculation details of the job and the maximum usage time (wall-time).

[0076] The creation unit 502 creates a performance model M based on the parameters specified. The performance model M is E[T total ] and N total This includes a model equation that outputs E[C]. E[T total ] is T total This represents the expected value of T. total This represents job time. Job time is the execution time required to run a job.

[0077] N total This represents the total number of nodes required to execute the job. total is, N node and N spare It is the sum of (where N total (where N is an integer between 1 and the maximum number of nodes). node This represents the number of nodes the application uses to execute the job (where N is the number of nodes used). node (where N is an integer between 1 and the maximum number of nodes). spare This represents the number of spare nodes in the job (where N is the number of spare nodes). spare (where is an integer between 1 and the maximum number of nodes).

[0078] E[C] represents the expected value of node time (cost). Node time is an indicator that represents the resource consumption required to execute a job, and is equivalent to, for example, the value obtained by multiplying the (number of nodes) and (node ​​usage time) required to execute the job (for example, the area of ​​the dotted line frame 1110 shown in Figure 11 below). Performance model 120 shown in Figure 1 corresponds to, for example, performance model M.

[0079] Specifically, for example, the creation unit 502 is N total and N abn Based on this, we create a first model equation that represents the probability (probability of existence) of an abnormal node being present in the job. Here, N total This is expressed by the following equation (1).

[0080] N total =N node +N spare ...(1)

[0081] Also, N abn This represents the number of abnormal nodes in the job. abn For example, N total and p abn Using these, it can be expressed as shown in equation (2) below. Here, B(n,p) represents a binomial distribution with n trials and p probability. Also, ~ means that it follows a probability distribution.

[0082] N abn ~B(N total ,p abn ) ···(2)

[0083] Then, the creation unit 502 can use the above equations (1) and (2) to create a first model equation like the following equation (3). However, P[N abn >0] is the probability that an abnormal node exists within the job (P[N abn >0]∈[0,1]). The exponential part "Ntotal" in equation (3) below is "N total This represents ".

[0084] P[N abn >0]=1-(1-p abn )Ntotal ···(3)

[0085] Also, the creation unit 502 creates a second model formula representing the benchmark time required for benchmark execution in a job based on P[N abn >0], α abn , and t bench . The second model formula can be represented, for example, by the following formulas (4) and (5).

[0086] However, T bench is the benchmark time required for benchmark execution in a job. P[T bench =α abn ·t bench represents the probability that T bench becomes "T bench =α abn ·t bench ". P[T bench =t bench represents the probability that T bench becomes "T bench =t bench ". If there is even one abnormal node, then "T bench =α abn ·t bench ", and in other cases, "T bench =t bench ".

[0087] P[T bench =α abn ·t bench =P[N abn >0] ···(4) P[T bench =t bench =1-P[N abn >0] ···(5)

[0088] Also, the creation unit 502 creates a third model formula representing the probability (exclusion probability) of being able to exclude abnormal nodes from app execution based on N node , N spare , and p abn . The third model formula can be represented, for example, by the following formula (6). However, P[N abn ≦Nspare is the probability of excluding abnormal nodes from the execution of the application (P[N abn ≤ N spare ∈ [0, 1]). N total is represented by the above formula (1).

[0089]

Number

[0090] Also, the creation unit 502 creates a fourth model formula representing the application time in the job based on t cmpt and t comm and α abn and P[N abn ≤ N spare . The application time is the execution time required for the execution of the application. The fourth model formula can be represented, for example, by the following formulas (7) and (8).

[0091] However, T app is the application time in the job (T app > 0). P[T app = α abn · t cmpt + t comm represents the probability that T app is "T app = α abn · t cmpt + t comm ". P[T app = t cmpt + t comm represents the probability that T app is "T app = t cmpt + t comm ". When the number of abnormal nodes exceeds the number of spare nodes, "T<00​​​​​​​​​​​​​​​​​​​​·t cmpt +t comm ]=1-P[N abn ≤N spare ]···(7) P[T app =t cmpt +t comm ]=P[N abn ≤N spare ]···(8)

[0093] Furthermore, the creation unit 502 creates a fifth model equation representing the expected job time based on the benchmark time and application time of the job. Job time is the execution time required to run the job. Job time is the sum of the benchmark time and application time of the job, and is expressed by the following equation (9). However, T total This is the job time.

[0094] T total =T bench +T app ...(9)

[0095] More specifically, for example, the creation unit 502 can create a fifth model equation such as equation (10) below from the above equations (4), (5), (7), (8) and (9). However, E[T total E[T] is the expected value of the job time (>0). bench E[T] is the expected benchmark time for the job. app ] represents the expected application time for the job.

[0096]

number

[0097] Then, the creation unit 502 creates the fifth model equation and N total Based on this, we create a performance model M. totalThis is expressed by equation (1) above. The performance model M can be expressed, for example, by equation (11) below, where E[C] is the expected value of node time (cost) (C>0).

[0098] E[C]=N total ·E[T total ] ···(11)

[0099] Note that the expected node time when this method is not used (corresponding to As-is described later) is "t bench =0,N spare This is equivalent to E[C] when "=0" is set (because the benchmark is not executed and spare nodes are not used). In this case, T bench is "T bench =0」, T total is "T total =T app This is what it becomes.

[0100] The decision unit 503 uses the created performance model M to minimize N E[C]. spare The number of spare nodes is determined. Specifically, for example, the determination unit 503 uses the performance model M to determine the number of spare nodes. spare From 0 to N node The E[C] is calculated by sequentially changing it up to the specified value. Then, the determination unit 503 determines the N corresponding to the minimum value among the calculated E[C]. spare To minimize E[C], N spare It may be decided that way.

[0101] Furthermore, the determination unit 503 sets the range from 0 to N. node Limited to only odd or even numbers up to N spare E[C] may be calculated while varying . Also, the determination unit 503 is set from 0 to N node Up to a predetermined number of intervals N spare E[C] may be calculated while varying the value. The predetermined interval can be set arbitrarily. This allows N spare This can reduce the computational complexity required to make that decision.

[0102] Here, we will explain an example of calculating E[C] using Figure 6. Here, t cmpt =10, t comm =5, N node =100, p abn =0.005, α abn =10, t bench Let's assume it equals 0.1. Also, we will perform numerical calculations using double-precision floating-point numbers.

[0103] Figure 6 is an explanatory diagram showing an example of E[C] calculation. In Figure 6, line graph 601 represents N spare This shows the change in E[C] calculated by sequentially changing from 1 to 10. However, in Figure 6, the right vertical axis represents E[C], and the horizontal axis represents N. spare This shows the result. Also, As-is shows E[C] when this method is not used.

[0104] Also, bar graph 602 is N spare E[T] is calculated by sequentially changing the value from 1 to 10. total This shows the change in [T]. However, in Figure 6, the left vertical axis is E[T]. total This shows ]. The horizontal axis is N spare This shows that As-is is the case when this method is not used. total This indicates ].

[0105] In line graph 601, "N spare The minimum value "E[C]=1671" is obtained when "=3". This minimum value is 0.33 times the As-is value, indicating that the cost is reduced compared to when this method is not applied. According to line graph 601 and bar graph 602, N spare If the value is less than 3, the number of redundant nodes decreases, but it is not possible to completely eliminate abnormal nodes, resulting in E[T] total It can be seen that the expected job time increases.

[0106] Also, N spare If it is 4 or greater, E[T total Although ] continues to take the optimal value, it can be seen that as the number of nodes increases, E[C] (expected value of node time) gradually increases. In this case, the decision unit 503 determines "N spareN = 3" minimizes E[C] spare The number of spare nodes will be determined.

[0107] Returning to the explanation of Figure 5, the input section 504 is determined N spare The job is submitted by specifying the number of spare nodes. Specifically, for example, the submission unit 504 submits a job to the management node 202 shown in Figure 2, N node (Number of nodes used by the app) and N spare Specify the number of spare nodes and submit the job.

[0108] As a result, for example, a job is submitted to the queue at the management node 202. Then, the management node 202 retrieves the job from the queue using the job scheduler P2, as shown in Figure 9 below, and assigns the job to one of the available nodes from nodes N1 to Nn. The node group is N node (Number of nodes used by the application) and N spare This is a group of nodes consisting of the sum of (the number of spare nodes) and the number of other nodes.

[0109] Furthermore, the functional parts of the login node 201 described above (for example, the reception unit 501 to the input unit 504) may be implemented by the management node 202 or node Ni. Alternatively, the login node 201 may have the functions of the management node 202 (for example, the job scheduler P2) or the functions of node Ni (for example, the job script P3). For example, if the login node 201 has the functions of the management node 202, the input unit 504 will determine N spare and N node The sum of (N) total Alternatively, you could assign the job to a group of nodes.

[0110] (Supplementary information on performance model M) Here, I will explain the supplementary information for performance model M.

[0111] In the explanation above, the specified parameter is t cmpt ,t comm It was stated that it may be included, but before the job was executed tcmpt ,t comm In some cases, one or both of these may not be clear. Also, the app's execution time may be clear, but... cmpt and t comm In some cases, the ratio is unknown.

[0112] Therefore, the user can use t as a parameter. cmpt ,t comm It may not be possible to specify this. In this case, the creation unit 502 will, for example, t cmpt This may be a constant multiple of the application execution time or the maximum usage time (wall-time) of the job. The constant is a value less than 1. Also, the creation unit 502 may, for example, t comm You may set it to 0.

[0113] In typical parallel computing applications, when computational performance is significantly reduced, the computation becomes rate-limiting, resulting in "α abn ·t cmpt ≫t comm This is because it is expected that... Note that the execution time of the application may be included in the job submission request, for example, or it may be stored in association with the application on the system side. The maximum usage time (wall-time) of the job is included in the job submission request, for example.

[0114] Also, p abn and α abn This may be calculated by the user, for example, from statistical information published by the system, or estimated from the results obtained by running an appropriate benchmark job against the job scheduling system 200.

[0115] Furthermore, generally, the failure rate of a node follows a path similar to the so-called failure rate curve (bathtub curve). Therefore, the failure interval and anomaly interval of a node behave probabilistically. However, in this embodiment, we focus on "the probability that an anomaly occurs at the moment a node is assigned to a job," and thus this is expressed as a single value "p abn Expressing it as "" does not diminish its generality.

[0116] For example, as shown in equations (12) and (13) below, the failure interval t of each node flt (The interval at which an event occurs that is detected and resolved by the system) and the abnormal interval t abn Assume that the interval between events that cause performance degradation but are not detected by the system follows an exponential distribution, where λ abn <λ flt Let's assume that.

[0117] t flt ~Exp(λ flt ) ···(12) t abn ~Exp(λ abn ) ···(13)

[0118] Under this assumption, the probability p that an anomaly occurs when a certain node is secured is that an anomaly has occurred. abn _ exp This is expressed by the following equation (14), where f(x|λ) and F(x|λ) represent the density function and distribution function of the exponential distribution Exp(λ), respectively.

[0119]

number

[0120] "λ flt When it goes →∞, it becomes "(node ​​operating time) →∞", and an anomaly is bound to occur, therefore, "p abn_exp →1" and also "λ abn When it goes →∞, the anomaly no longer occurs, so "p abn_exp →0. Similarly, if the distribution of failure intervals and anomaly intervals is constant with respect to the overall system uptime and is equivalent for each node, then p abn It can be represented as a single value.

[0121] It should be noted that the distribution of failure intervals and anomaly intervals may not be uniform across nodes; for example, there may be nodes with an extremely high number of failures or anomalies. In such cases, in a system with many nodes having the same configuration, such as the job scheduling system 200, it is expected that the cause will be eliminated by replacing parts or other means during recovery. Therefore, it is generally not expected that such an event will occur.

[0122] (Example of functional configuration of node Ni) Next, we will describe an example of the functional configuration of node Ni. Node Ni is one of the nodes (computation nodes) from nodes N1 to Nn.

[0123] Figure 7 is a block diagram showing an example of the functional configuration of node Ni. In Figure 7, node Ni includes a first execution unit 701, a selection unit 702, and a second execution unit 703. The first execution unit 701 to the second execution unit 703 function as a control unit 700, and specifically, they realize their function by having the CPU 401 execute a program (such as the job script P3 shown in Figure 9, which will be described later) stored in a storage device such as the memory 402, disk 404, or portable recording medium 407 of node Ni shown in Figure 4, or by using a communication interface 405. The processing results of each functional unit are stored in a storage device such as the memory 402 or disk 404 of node Ni.

[0124] The first execution unit 701 is N total After assigning jobs to the node groups, each node group is made to run the benchmark. total For example, N is determined by login node 201. spare And N specified by the user node It is the sum of and .

[0125] A benchmark is software (e.g., LINPACK) used to evaluate the performance of nodes that run before the application within a job. Specifically, for example, the first execution unit 701 uses the mpirun command, which is included with various MPI libraries, to request each node (including its own node) to run the benchmark.

[0126] In the following explanation, the group of nodes to which a job is assigned may be referred to as "node group N[1]~N[m]" (where m is a natural number greater than or equal to 2).

[0127] Furthermore, the first execution unit 701 collects the benchmark execution time for each node group N[1] to N[m]. The benchmark execution time is the time required to execute the benchmark at each node. Specifically, for example, the first execution unit 701 collects the benchmark execution time for each node group N[1] to N[m] from the standard output of mpirun. However, the content of the benchmark is adjusted so that the time for each node is output to the standard output of mpirun.

[0128] Alternatively, benchmark logs for each node may be output to a unique path on the file system FS shown in Figure 2. In this case, the first execution unit 701 may, for example, collect the benchmark execution time for each node group N[1] to N[m] from the file system FS.

[0129] The collected benchmark execution times are stored in a benchmark execution time table 800, for example, as shown in Figure 8. The benchmark execution time table 800 is implemented using storage devices such as the memory 402 and disk 404 of node Ni.

[0130] Figure 8 is an explanatory diagram showing an example of the contents of the benchmark execution time table 800. In Figure 8, the benchmark execution time table 800 has fields for node ID and benchmark execution time, and by setting information in each field, benchmark execution time information 800-1 to 800-m is stored as records.

[0131] Here, the node ID is an identifier that uniquely identifies a node included in the node group N[1]~N[m]. The benchmark execution time is the benchmark execution time of the node identified by the node ID. For example, benchmark execution time information 800-1 shows the benchmark time t1 of node group N[1].

[0132] The selection unit 702 selects the collected benchmark execution time and N node Based on this, the node on which to run the application for job execution is selected. Specifically, for example, the selection unit 702 refers to the benchmark execution time table 800 shown in Figure 8 and selects the nodes from the node group N[1] to N[m] with the shortest benchmark execution time. node Select a node for a few minutes.

[0133] The second execution unit 703 causes the selected node to run the application. Specifically, for example, the second execution unit 703 causes the selected N node The second execution unit 703 then creates a hostfile listing the hostnames, using the node ID of each node as the hostname. node Specify the row using arguments.

[0134] As a result, the second execution unit 703 selects N from the node group N[1] to N[m]. node You can run jobs on nodes that take several minutes to complete.

[0135] The functional parts of node Ni described above (for example, the first execution unit 701 to the second execution unit 703) may be implemented by the login node 201 or the management node 202. Alternatively, node Ni may have the functions of the login node 201 (for example, the submission script P1) or the functions of the management node 202 (for example, the job scheduler P2).

[0136] (Example of operation of job scheduling system 200) Next, an operation example of the job scheduling system 200 will be described.

[0137] FIG. 9 is an explanatory diagram showing an operation example of the job scheduling system 200. In FIG. 9, a login node 201, a management node 202, nodes N1 to Nn, and a file system FS within the job scheduling system 200 are shown. Here, it is assumed that node N1 among nodes N1 to Nn executes a job script P3.

[0138] First, the login node 201 receives the specification of parameter 900 from the user U by the input script P1. The user U is a user who operates the input script P1 to request the execution of a job, and corresponds to the client terminal 203 shown in FIG. 2. The parameter 900 includes N node , t cmpt , t comm , t bench , p abn , α abn .

[0139] Then, the login node 201 creates a performance model M by the input script P1 based on the specification of the parameter 900. Next, the login node 201 determines N that minimizes E[C] using the performance model M by the input script P1. Then, the login node 201 designates N spare and N node and N spare to the management node 202 by the input script P1 to submit a job.

[0140] The management node 202 allocates the submitted job to an available node group among nodes N1 to Nn by the job scheduler P2 and executes the job script P3. The path for accessing the main body of the job script P3 in the file system FS is specified, for example, from the input script P1. Also, all information that the job script P3 has is passed from the input script P1 via the job scheduler P2, for example.

[0141] Incidentally, information for scheduling such as a list of job nodes is held by, for example, the job scheduler P2. Also, for the process of identifying an available node group from nodes N1 to Nn, any existing technology may be used. For example, the job scheduler P2 may identify nodes to which jobs are not assigned or nodes with a margin in CPU usage rate or the like.

[0142] Node N1 causes the job script P3 to execute a benchmark on each of the node groups to which jobs are assigned, and creates a node list 901 of the nodes used for application execution. Then, node N1 uses the node list 901 by the job script P3 to select node several minutes of nodes and execute the application. Information necessary for the execution of the application or benchmark (such as the path and arguments of the execution format of the application or benchmark) is passed from the input script P1 to the job script P3 via, for example, the job scheduler P2.

[0143] Here, an example of the connection between the nodes for executing the application will be described using FIG. 10.

[0144] FIG. 10 is an explanatory diagram showing an example of the connection between nodes. In FIG. 10, nodes N1, N2, N3, and N4 are an example of a node group N[1] to N[m] secured for the execution of a job. Nodes N1, N3, and N4 are node an example of several minutes of nodes selected as nodes for executing the application.

[0145] Node N1 requests the execution of the application for nodes N1, N3, and N4 by the job script P3. The request for the execution of the application to each of nodes N1, N3, and N4 is realized by, for example, a command (mpiexec or mpirun) implemented by the MPI library.

[0146] Furthermore, inter-node communication conducted by the application is performed, for example, via switch 1001 (Figure 10 assumes a tree structure using switch 1001). This enables high-performance communication even between nodes in discontinuous physical locations within the job scheduling system 200.

[0147] (Example of job execution) Next, we will explain an example of job execution using Figure 11.

[0148] Figure 11 is an explanatory diagram showing an example of job execution. Login node 201 has parameter (N node , t cmpt , t comm , p abn , α abn , t bench Based on the specifications, a performance model M is created. Login node 201 uses the performance model M to minimize N[C]. spare Determine N node to "N node Let N =3 spare is "N spare Assume that it is determined to be "=1". In this case, management node 202 is N spare and N node Assign the job to the four nodes, which are a total of these four nodes.

[0149] In Figure 11, nodes 1101 to 1104 are nodes included in nodes N1 to Nn, and represent an example of the node group N[1] to N[m] reserved for job execution. Here, node 1101 is designated as node Ni, which executes job script P3 (see, for example, Figure 9).

[0150] In Figure 11, "Benchmark" shows the benchmark execution time for each node 1101-1104. "Collection" shows the time taken to collect the benchmark execution times for each node 1101-1104. However, it is assumed that the time taken to collect the benchmark execution times is negligibly small. "Computation" shows the computation time for the application. cmptThis represents the total computation time for the entire application. "Communication" indicates the communication time between nodes in the application. comm This represents the total communication time for the entire app.

[0151] Node 1101 runs the benchmark on each of the nodes 1101 through 1104. Then, node 1101 collects the benchmark execution time for each of the nodes 1101 through 1104. In this case, an anomaly occurred on node 1103, and the benchmark execution time for node 1103 is longer than that of nodes 1101, 1102, and 1104.

[0152] Node 1101 is "N node Based on "=3", the three nodes with the shortest benchmark times, 1101, 1102, and 1104, are selected as the nodes to run the application. Here, the number of abnormal nodes is "1" for N. spare For the following reasons, node 1103 can be excluded as an abnormal node.

[0153] Then, node 1101 instructs the selected nodes 1101, 1102, and 1104 to run the application. This allows node 1101 to prevent the application required for job execution from running on a faulty node, thus preventing a decrease in computing performance. The node time is (number of nodes: 4) × (usage time: Tx) (corresponding to the area of ​​the dotted frame 1110 in Figure 11).

[0154] (Job submission procedure for login node 201) Next, we will describe the job submission process for login node 201. The job submission process is, for example, part of the job scheduling process.

[0155] Figures 12 and 13 are flowcharts illustrating an example of the job submission process procedure for login node 201. In the flowchart of Figure 12, first, login node 201 determines whether or not it has received a job submission request from client terminal 203 (step S1201).

[0156] For example, the job submission request includes the parameter (N node , t cmpt , t comm , p abn , α abn , t bench This includes information such as the job specifications, job calculation details, and maximum usage time (wall-time). Here, login node 201 waits to receive a job submission request (step S1201: No).

[0157] If login node 201 accepts a job submission request (step S1201: Yes), N spare to "N spare Set "=0" (step S1202) and execute the EC calculation process to calculate E[C] (step S1203). The specific procedure for the EC calculation process will be described later using Figure 14.

[0158] And login node 201 is E C_best Let E[C] be calculated in step S1203 (step S1204), N spare_best to "N spare_best Set i to "i=0" (step S1205). Next, login node 201 sets i to "i=1" (step S1206).

[0159] And login node 201 is N spare to "N spare Set to "=i" (step S1207), and execute the EC calculation process based on the parameter specifications included in the job submission request (step S1208). The specific processing procedure for the EC calculation process will be described later using Figure 14.

[0160] Next, login node 201 determines that E[C] calculated in step S1208 is E C_best Determine whether it is smaller or not (step S1209). Here, E[C] is E C_best If it is smaller (step S1209: Yes), login node 201 is E C_bestLet it be E[C] calculated in step S1208 (step S1210).

[0161] Then, the login node 201 sets N spare_best to "N spare_best =i" (step S1211) and proceeds to step S1301 shown in FIG. 13. Also, in step S1209, if E[C] is greater than or equal to E C_best (step S1209: No), the login node 201 proceeds to step S1301 shown in FIG. 13.

[0162] In the flowchart of FIG. 13, first, the login node 201 increments i (step S1301) and determines whether i is greater than N node (step S1302). Here, if i is less than or equal to N node (step S1302: No), the login node 201 proceeds to step S1207.

[0163] On the other hand, if i is greater than N node (step S1302: Yes), the login node 201 sets N spare to "N spare =N spare_best " (step S1303). Then, the login node 201 specifies N spare and submits a job (step S1304), ending the series of processes according to this flowchart.

[0164] Thereby, the login node 201 can submit a job by specifying the number of preliminary nodes that minimizes the expected value of the node time (cost).

[0165] Next, the specific processing procedures of the EC calculation processes in steps S1203 and S1208 shown in FIG. 12 will be described.

[0166] FIG. 14 is a flowchart showing an example of the specific processing procedure of the EC calculation process. In the flowchart of FIG. 14, first, the login node 201 sets N nodeand N spare Based on this, the above formula (1) is created (step S1401). Then, login node 201 is N total and N abn Based on this, formula (3) is created from formulas (1) and (2) above (step S1402).

[0167] Next, login node 201 sets s to "s=0" (step S1403) and i to "i=0" (step S1404). Then, login node 201 sets N total and p abn Based on this, s is calculated from the following formula (15) (step S1405). The following formula (15) corresponds to the above formula (6).

[0168]

number

[0169] Next, login node 201 increments i (step S1406) until i is N spare Determine whether it is greater than or equal to (step S1407). Here, i is N spare In the following case (step S1407: No), login node 201 returns to step S1405.

[0170] On the other hand, i is N spare If it is greater than (step S1407: Yes), login node 201 is [N abn ≤N spare ] to "P[N abn ≤N spare Set ]=s" (step S1408). Next, login node 201 uses the above equation (10) to calculate E[T total Calculate ] (step S1409).

[0171] Specifically, for example, login node 201 is P[N abn >0] and α abn and t bench Based on this, equations (4) and (5) above are created. Also, login node 201 is tcmpt and t comm and α abn and P[N abn ≤N spare Based on ], the above equations (7) and (8) are created. Then, the login node 201 creates the above equation (10) from the above equations (4), (5), (7), (8) and (9), and E[T total [As described in ].

[0172] Then, login node 201 calculates E[T total Using ], E[C] is calculated from the above formula (11) (step S1410), and the process returns to the step that called the EC calculation process.

[0173] This allows login node 201 to calculate the expected value of node time (cost).

[0174] (Job execution control processing procedure for node Ni) Next, the job execution control processing procedure for node Ni will be described. Node Ni is a node among the node group N[1]~N[m] that has job script P3. The job execution control processing corresponds to, for example, a part of the job scheduling processing.

[0175] Figure 15 is a flowchart showing an example of the job execution control processing procedure for node Ni. In the flowchart of Figure 15, first, node Ni causes each node in the group of nodes N[1] to N[m] to which the job is assigned to run the benchmark (step S1501).

[0176] Next, node Ni collects the benchmark execution time for each node (step S1502). Then, node Ni sorts the node IDs of each node group N[1] to N[m] in ascending order so that the collected benchmark execution times for each node are in ascending order (step S1503).

[0177] Next, node Ni refers to the sorted node IDs and selects the nodes with the shortest benchmark execution times, N in descending order. nodeSelect the node for a few minutes (step S1504). Then, node Ni is selected N node Run the application in a few minutes using the node (step S1505) to complete the series of processes according to this flowchart.

[0178] This allows node Ni to prevent users from being assigned abnormal nodes, which can degrade the computational performance of applications and thus suppress increases in node time.

[0179] (Example of E[C] reduction) Next, we will explain an example of E[C] reduction when this method is applied. First, using Figures 16A and 16B, we will explain the p specified as a parameter. abn , α abn , t bench Let's explain an example of how to calculate it.

[0180] Figures 16A and 16B are explanatory diagrams illustrating specific examples of benchmark times for each node. In Figure 16A, bar graph 1601 (a graph of 96 bars) shows the benchmark times for each node when Job A is executed on 96 of the nodes N1 to Nn, sorted in descending order. According to bar graph 1601, the first two nodes can be considered abnormal nodes.

[0181] In Figure 16B, bar graph 1602 (a graph of 96 bars) shows the benchmark time for each node when Job B is executed on 96 nodes out of nodes N1 to Nn, sorted in descending order. According to bar graph 1602, the first three nodes can be considered abnormal nodes.

[0182] t bench This can be calculated, for example, from the average benchmark time of non-anomalous nodes. Here, t bench is, "t bench =0.0167[s]. Also, α abn For example, it can be calculated from the ratio of the average benchmark times of abnormal nodes and non-abnormal nodes. Here, α abn is, "α abn =3.53. Also, pabn This can be calculated, for example, by maximum likelihood estimation. Here, p abn is, "p abn ={(2+3) / 2} / 96=0.026.

[0183] Next, we will explain an example of prediction for E[C]. Here, N node Assuming = 100, there are three cases with different computational loads "(t cmpt ,t comm Let's take the example of predicting the expected node time (E[C]) from equation (11) above for the values ​​)=(100 seconds, 0 seconds), (50 seconds, 50 seconds), and (10 seconds, 90 seconds).

[0184] Figure 17 is an explanatory diagram showing an example of a prediction for E[C]. In Figure 17, line graph 1701 is "(t cmpt ,t comm Let ) = (100 seconds, 0 seconds), N spare This shows the change in E[C] when the value is sequentially changed from 1 to 15. In Figure 17, the vertical axis represents E[C], and the horizontal axis represents N. spare This shows the result. Also, As-is shows E[C] when this method is not used.

[0185] Line graph 1702 is "(t cmpt ,t comm Let ) = (50 seconds, 50 seconds), N spare This shows the change in E[C] when the value is changed sequentially from 1 to 15. Line graph 1703 shows "(t cmpt ,t comm Let ) = (10 seconds, 90 seconds), N spare This shows how E[C] changes when the value is sequentially changed from 1 to 15.

[0186] In line graph 1701, N spare is "N spare When "=8", E[C] is minimized, and it is estimated that E[C] can be reduced by about (1 / 3.1) times compared to the As-is E[C]. Note that the optimal value is "N spare"=8" means that by securing 108 nodes and removing 8 nodes, it is possible to eliminate almost all abnormal nodes, and the expected value of node time will be minimized.

[0187] In line graph 1702, N spare is "N spare When N = 7, E[C] is minimized, and it is estimated that E[C] can be reduced by about (1 / 2) times compared to the As-is E[C]. In line graph 1703, spare is "N spare When E[C] = 5, it is estimated that E[C] can be minimized by about (1 / 1.2) times compared to the As-is E[C].

[0188] The above explanation used the example of running a benchmark within a user-submitted job, but it is not limited to this. For example, the operations from benchmark execution to the exclusion of abnormal nodes could be performed by the administrator of the job scheduling system 200.

[0189] In this case, for example, when multiple nodes are assigned to a single job, the administrator (e.g., management node 202) performs operations from benchmarking to excluding abnormal nodes before the user's application is executed. Then, by passing a list of nodes from which abnormal nodes have been removed to the user's application, it is expected that performance degradation of the application will be suppressed and the efficiency of node utilization will be improved.

[0190] Furthermore, if the above operations are performed on the administrator side (for example, the management node 202), the management node 202 may, for example, use a certain metric to detect abnormal nodes when collecting benchmark execution times for each node. If, for example, a large number of abnormal nodes are detected, the management node 202 may exclude the abnormal nodes from the job scheduling system 200. If it is difficult to exclude abnormal nodes, a mechanism may be put in place to prevent users from suffering any disadvantages, such as by notifying the user from the administrator side. In addition, the management node 202 may, for example, use the benchmark execution time of each node to determine the parameters of the performance model M that are invariant to the application (p abn , α abn , t bench You may also calculate (etc.).

[0191] Furthermore, for supercomputers employing mesh or torus topologies, where the physical node arrangement has a relatively strong impact on communication performance, excluding abnormal nodes on the user side can lead to discontinuities in the arrangement, potentially increasing communication latency between specific nodes. However, when applying the system-level exclusion of abnormal nodes described above to supercomputers with high-dimensional mesh topologies, it is possible to provide users with a contiguous group of nodes on the network using methods similar to those used for excluding normal faulty nodes.

[0192] As described above, the login node 201 of the job scheduling system 200 according to this embodiment can accept parameter specifications when executing a job. For example, N node , p abn , α abn , t bench This includes. Also, according to login node 201, based on the parameters specified, E[T total ] and N total A performance model M can be created that outputs E[C] from this. Then, according to login node 201, N minimizes E[C] using the created performance model M. spare The number of spare nodes can be determined.

[0193] This allows login node 201 to determine the number of nodes necessary to efficiently execute a job by searching for the number of spare nodes that minimizes the expected node time (cost) when submitting a job with a redundant number of nodes, taking into account the occurrence of abnormal nodes. For example, login node 201 can submit a job by specifying the number of spare nodes that minimizes the expected node time (cost).

[0194] Furthermore, login node 201 can accept the specification of parameters including a first processing time and a second processing time. The first processing time is the processing time of the application execution time that is affected by performance degradation due to abnormal nodes. The second processing time is the processing time of the application execution time that is not affected by performance degradation due to abnormal nodes.

[0195] This allows login node 201 to create a performance model M that considers both the processing time affected by performance degradation due to abnormal nodes and the processing time not affected by performance degradation due to abnormal nodes as part of the application execution time. Therefore, login node 201 can accurately predict E[C] by taking into account the characteristics of the application.

[0196] Furthermore, according to login node 201, t cmpt t comm It can accept parameter specifications that include t. cmpt This is an example of the first processing time. comm This is an example of the second processing time.

[0197] This allows login node 201 to arbitrarily specify the computation time for each node in the application and the communication time between nodes in the application as parameters determined by the application, when all nodes cooperate to perform calculations for the application executed within the job. Therefore, login node 201 can create a performance model M that takes into account the characteristics of the application, and can improve the prediction accuracy of E[C].

[0198] Furthermore, according to login node 201, N node and N spare and p abn Based on this, the probability of an abnormal node existing in the job is P[N] abn Create a first model equation that represents >0), and compare the first model equation with α abn and t bench Based on this, the benchmark time in the job (P[T bench =α abn ·t bench ], P[T bench =t bench A second model equation representing ]) can be created.

[0199] This allows login node 201 to predict the benchmark time for a job by taking into account the probability of an abnormal node being present in the job.

[0200] Furthermore, according to login node 201, N node and N spare and p abn Based on this, the elimination probability (P[N[N]) that can eliminate abnormal nodes from the application's execution is calculated. abn ≤N spare A third model equation representing ]) can be created. And according to login node 201, t cmpt and t comm and α abn Based on the third model equation, the application time in the job (P[T app =α abn ·t cmpt +t comm ], P[T app =t cmpt +t comm A fourth model equation representing ]) can be created.

[0201] This allows login node 201 to predict application time in a job by considering the elimination probability that will allow it to exclude abnormal nodes from application execution.

[0202] Furthermore, according to login node 201, based on the second and fourth model equations, the expected job time (E[T) can be calculated. total Create a fifth model equation representing ]), and the fifth model equation, N node and N spare Based on this, a performance model M can be created.

[0203] This allows login node 201 to accurately predict the expected job time, thereby improving the accuracy of node time predictions.

[0204] Furthermore, according to node Ni of the job scheduling system 200 according to the embodiment, N total As a result of assigning jobs to the node groups N[1]~N[m], each node group can run the benchmark. total For example, N is determined by login node 201. spare And N specified by the user node This is the sum of the two numbers. According to node Ni, among the node group N[1] to N[m], the ones with the shortest benchmark execution time are N. node You can run the application on a node.

[0205] This allows node Ni to run the application by excluding nodes from the node group N[1] to N[m] that take a long time to run the benchmark. Therefore, node Ni can prevent users from being assigned abnormal nodes, which would degrade the computational performance of the application, and thus suppress the increase in node time.

[0206] Based on these considerations, the job scheduling system 200 according to this embodiment makes it possible to determine the minimum number of nodes that can suppress the increase in node time even when abnormal nodes are excluded, without modifying the job or hardware environment, and thus enable efficient job execution.

[0207] The scheduling method described in this embodiment can be implemented by executing a pre-prepared program on a computer such as a personal computer or workstation. This scheduler is recorded on a computer-readable recording medium such as a hard disk, flexible disk, CD-ROM, DVD, or USB memory, and is executed when read from the recording medium by a computer. This scheduler may also be distributed via a network such as the Internet.

[0208] With regard to the embodiments described above, the following additional information is disclosed.

[0209] (Note 1) When executing a job using one or more nodes in the system, the system accepts specifications for the number of nodes used by the application for executing the job, the probability of node failure in the system, the ratio of processing time of abnormal nodes to normal nodes in the system, and the benchmark time for the benchmark executed before the application in the job. Based on the received specifications, a performance model is created that outputs the expected value of resource consumption for executing the job, taking into account the expected execution time for the job, the number of nodes used, and the number of spare nodes in the job. Using the performance model created, the number of spare nodes that minimizes the expected value of resource consumption is determined. A job scheduler characterized by having a computer execute the processing.

[0210] (Note 2) The job scheduler according to Note 1, characterized in that the above specification includes specifying a first processing time within the execution time of the application that is affected by the performance degradation caused by the abnormal node, and a second processing time that is not affected by the performance degradation caused by the abnormal node.

[0211] (Note 3) The first processing time is the computation time required for each node in the application, The second processing time is the communication time required for communication between nodes in the application. The job scheduler described in Appendix 2, characterized by the features described herein.

[0212] (Note 4) As a result of assigning the job to a group of nodes equal to the sum of the determined number of spare nodes and the number of nodes in use, the benchmark was executed on each of the node groups. From the group of nodes, the application is executed on the nodes with the shortest processing time required to run the benchmark, up to the number of nodes used. A job scheduler according to any one of the appendices 1 to 3, characterized in that it causes the computer to perform the processing.

[0213] (Note 5) The process to be created above is: Based on the number of nodes in use, the number of spare nodes, and the probability of an anomaly occurring, a first model equation is created that represents the probability of an anomaly being present in the job. Based on the first model equation, the ratio, and the benchmark time, a second model equation representing the benchmark time in the job is created. Based on the number of nodes in use, the number of spare nodes, and the probability of anomaly occurrence, a third model equation is created that represents the elimination probability that the abnormal node can be excluded from the execution of the application. Based on the first processing time, the second processing time, the ratio, and the third model equation, a fourth model equation representing the application time in the job is created. Based on the second and fourth model equations, a fifth model equation is created that represents the expected execution time for the execution of the job. Based on the created fifth model equation, the number of nodes used, and the number of spare nodes, the performance model is created. A job scheduler characterized by any one of the features described in Appendix 2 to 4.

[0214] (Note 6) When executing a job using one or more nodes in the system, the system accepts specifications for the number of nodes used by the application for executing the job, the probability of node failure in the system, the ratio of processing time of abnormal nodes to normal nodes in the system, and the benchmark time for the benchmark executed before the application in the job. Based on the received specifications, a performance model is created that outputs the expected value of resource consumption for executing the job, taking into account the expected execution time for the job, the number of nodes used, and the number of spare nodes in the job. Using the performance model created, the number of spare nodes that minimizes the expected value of resource consumption is determined. A job scheduling method characterized by having a computer perform the processing.

[0215] (Note 7) When executing a job using one or more nodes in the system, the system includes a receiving unit that accepts specifications for the number of nodes used by the application for executing the job, the probability of node failure in the system, the ratio of processing time of abnormal nodes to normal nodes in the system, and the benchmark time for executing the benchmark that is performed before the application in the job. Based on the specifications received by the reception unit, a creation unit creates a performance model that outputs the expected value of resource consumption for executing the job from the expected value of execution time for executing the job, the number of nodes used, and the number of spare nodes in the job. A determination unit that determines the number of spare nodes that minimize the expected value of resource consumption using the performance model created by the creation unit, A first execution unit, which assigns the job to a group of nodes equal to the sum of the number of spare nodes and the number of nodes in use determined by the decision unit, causes each of the node groups to execute the benchmark, A second execution unit that causes the number of nodes used to run the application, starting with the node with the shortest processing time required to run the benchmark, A job scheduling system characterized by including the following: [Explanation of Symbols]

[0216] 101 Information Processing Device 110,900 parameters 120,M performance model 200 Job Scheduling Systems 201 Login Node 202 Management Nodes 203 Client terminals 204 Storage Server 210 Network 301, 302, 303, 304, 305, 306, 307, 308, 1101, 1102, 1103, 1104, N1~Nn, Ni node 311, 312, 313, 1001 switches 400 bus 401 CPU 402 memory 403 Disk Drive 404 disk 405 Communication I / F 406 Portable recording medium interface 407 Portable recording media 500,700 Control Unit 501 Reception Department 502 Creation Department 503 Decision Section 504 Input section 701 First Execution Unit 702 Selection Section 703 Second Executive Unit 800 Benchmark Execution Time Table 901 Node List

Claims

1. When executing a job using one or more nodes in the system, the system accepts the specification of parameters including the number of nodes used by the application for executing the job, the probability of node failure in the system, the assumed ratio of processing time of abnormal nodes to normal nodes, and the benchmark time for executing the benchmark before the application in the job. Based on the received specifications, a performance model is created that outputs the expected value of resource consumption for executing the job, taking into account the expected execution time for the job, the number of nodes used, and the number of spare nodes in the job. Using the performance model created, the number of spare nodes that minimizes the expected value of resource consumption is determined. A job scheduler characterized by having a computer execute the processing.

2. The job scheduler according to claim 1, characterized in that the specification includes specifying a first processing time within the execution time of the application that is affected by the performance degradation caused by the abnormal node, and a second processing time that is not affected by the performance degradation caused by the abnormal node.

3. The first processing time is the computation time required for each node in the application, The second processing time is the communication time required for communication between nodes in the application. The job scheduler according to feature 2.

4. As a result of assigning the job to a group of nodes equal to the sum of the determined number of spare nodes and the number of nodes in use, the benchmark is executed on each of the node groups. From the group of nodes, the application is executed on the nodes with the shortest processing time required to run the benchmark, up to the number of nodes used. The job scheduler according to claim 1, characterized in that it causes the computer to perform the processing.

5. When executing a job using one or more nodes in the system, the system accepts the specification of parameters including the number of nodes used by the application for executing the job, the probability of node failure in the system, the assumed ratio of processing time of abnormal nodes to normal nodes, and the benchmark time for executing the benchmark before the application in the job. Based on the received specifications, a performance model is created that outputs the expected value of resource consumption for executing the job, taking into account the expected execution time for the job, the number of nodes used, and the number of spare nodes in the job. Using the performance model created, the number of spare nodes that minimizes the expected value of resource consumption is determined. A job scheduling method characterized by having a computer perform the processing.

Images