Method for allocating I / O forwarding nodes to jobs in I / O forwarding architecture

Abstract

The present invention provides a method for allocating I / O forwarding nodes for jobs under an I / O forwarding architecture, comprising: step 1) obtaining the number of default I / O forwarding nodes for a job and a threshold value of the number of I / O forwarding nodes for the job; wherein the default I / O forwarding node for the job is an I / O forwarding node allocated to the job by a job scheduling device when performing job scheduling; step 2) comparing the number of default I / O forwarding nodes for the job with the threshold value of the number of I / O forwarding nodes for the job to determine whether to allocate a standby I / O forwarding node as the actual I / O forwarding node for the job. The present invention dynamically allocates I / O forwarding nodes for jobs, effectively avoiding I / O performance interference of job granularity on I / O forwarding nodes, and improving the utilization rate of I / O forwarding nodes and the I / O performance of jobs.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing and storage, and in particular to a method for allocating I / O forwarding nodes to jobs under an I / O forwarding architecture. Background Art

[0002] With the development of high-performance computing, the demand for adaptive, application-aware parallel I / O services is increasing. Currently, an I / O forwarding architecture that adds an additional forwarding layer between computing nodes and storage nodes is widely used in large-scale computing systems to solve the scalability problem in high-performance computing. Among them, the forwarding layer includes multiple I / O forwarding nodes, each of which is responsible for forwarding I / O requests from computing nodes to storage nodes, thereby helping computing nodes decouple I / O operations. The I / O forwarding architecture narrows the performance gap between CPU and I / O components, makes I / O resource configuration more flexible, improves the I / O performance that users can feel, and also reduces the amount of backend data.

[0003] Due to its many advantages, the I / O forwarding architecture is very popular in high-performance computing technology, including nearly half of the current top supercomputers. However, most of these supercomputers currently use a static forwarding mechanism, that is, forwarding I / O requests based on the I / O forwarding nodes assigned by the job scheduling device by default. This static forwarding mechanism has the following problems: when the default assigned I / O forwarding nodes meet the I / O requirements of the application, most of the I / O forwarding nodes in the forwarding layer are often underutilized; in addition, the I / O forwarding nodes are shared by jobs of different sizes, and jobs with conflicting I / O requirements may cause serious performance interference to each other, resulting in low I / O performance of the application. Summary of the invention

[0004] In order to solve the above problems in the prior art, according to an embodiment of the present invention, a method for allocating I / O forwarding nodes to jobs in an I / O forwarding architecture is provided, comprising:

[0005] Step 1) obtaining the number of default I / O forwarding nodes of a job and a threshold value of the number of I / O forwarding nodes of the job; wherein the default I / O forwarding node of the job is an I / O forwarding node allocated to the job by the job scheduling device when performing job scheduling;

[0006] Step 2) Compare the number of default I / O forwarding nodes of the job with the I / O forwarding node number threshold of the job to determine whether to allocate a spare I / O forwarding node as the actual I / O forwarding node of the job.

[0007] In the above method, step 2) includes: comparing the number of default I / O forwarding nodes of the job with the threshold number of I / O forwarding nodes of the job; if the number of default I / O forwarding nodes of the job is less than the threshold number of I / O forwarding nodes of the job, allocating a spare I / O forwarding node as the actual I / O forwarding node of the job; otherwise, determining whether there is job-level I / O performance interference with the default I / O forwarding node of the job, if so, allocating a spare I / O forwarding node as the actual I / O forwarding node of the job, if not, using the default I / O forwarding node of the job as the actual I / O forwarding node of the job.

[0008] In the above method, the I / O forwarding node number threshold of the job is calculated according to the following formula:

[0009]

[0010] Where i represents the i-th job; TH_node i Indicates the I / O forwarding node number threshold for the i-th job; NUM_io i represents the number of I / O processes when the i-th job was last executed; α represents the expansion factor, which is the quotient of the peak I / O bandwidth of the I / O forwarding node and the peak I / O bandwidth of the computing node.

[0011] In the above method, in step 2), determining whether the default I / O forwarding node of the job has job-level I / O performance interference includes: for each I / O forwarding node in the default I / O forwarding node of the job, obtaining all jobs including the default I / O forwarding node; if among all the obtained jobs, there is a job whose average aggregate bandwidth during the last execution is greater than a predetermined threshold, then the default I / O forwarding node of the job has job-level I / O performance interference.

[0012] In the above method, in step 2), determining whether there is job-level I / O performance interference in the default I / O forwarding node of the job includes: for each I / O forwarding node in the default I / O forwarding node of the job, obtaining all jobs including the default I / O forwarding node; if among all the obtained jobs, there is a job whose average aggregate IOPS during the last execution is greater than a predetermined threshold, then there is job-level I / O performance interference in the default I / O forwarding node of the job.

[0013] In the above method, in step 2), determining whether there is job-level I / O performance interference in the default I / O forwarding node of the job includes: for each I / O forwarding node in the default I / O forwarding node of the job, obtaining all jobs including the default I / O forwarding node; if among all the obtained jobs, there is a job whose average metadata waiting queue length during the last execution is greater than a predetermined threshold, then there is job-level I / O performance interference in the default I / O forwarding node of the job.

[0014] In the above method, determining whether there is job-level I / O performance interference in the default I / O forwarding node of the job includes: for each I / O forwarding node in the default I / O forwarding node of the job, obtaining all jobs whose default I / O forwarding node includes the I / O forwarding node; if among all the obtained jobs, there is a job whose I / O mode is N-1 mode, then there is job-level I / O performance interference in the default I / O forwarding node of the job.

[0015] In the above method, allocating a standby I / O forwarding node as the actual I / O forwarding node of the job includes: calculating the larger value M between the number of default I / O forwarding nodes of the job and the I / O forwarding node number threshold of the job; comparing the number of standby I / O forwarding nodes in the standby I / O forwarding node pool with the larger value M; if the number of standby I / O forwarding nodes in the standby I / O forwarding node pool is greater than or equal to the larger value M, allocating M standby I / O forwarding nodes from the standby I / O forwarding node pool to the job as the actual I / O forwarding nodes of the job; otherwise, allocating all standby I / O forwarding nodes in the standby I / O forwarding node pool to the job as the actual I / O forwarding nodes of the job.

[0016] In the above method, the standby I / O forwarding node is an exclusive I / O forwarding node.

[0017] The above method may further include: obtaining I / O information when the job was last executed.

[0018] According to one embodiment of the present invention, there is also provided an electronic device, comprising a processor and a memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the electronic device implements a method for allocating I / O forwarding nodes to jobs under the above-mentioned I / O forwarding architecture.

[0019] The embodiments of the present invention have the following beneficial effects:

[0020] Compared with the existing static forwarding mechanism, the present invention realizes the dynamic allocation of I / O forwarding nodes for jobs. Among them, if the number of default I / O forwarding nodes allocated to the job is less than the optimal usage of I / O forwarding nodes calculated based on the historical I / O information of the job, a spare I / O forwarding node that meets the optimal usage of the I / O forwarding node is allocated to the job as much as possible; in addition, even if the number of default I / O forwarding nodes of the job meets (i.e., is greater than or equal to) the optimal usage of the I / O forwarding nodes of the job, it is necessary to further determine whether there is I / O performance interference in the default I / O forwarding node of the job, and if there is I / O performance interference, a spare I / O forwarding node will be allocated to the job. The present invention can effectively avoid I / O performance interference at the job granularity on the I / O forwarding node, improve the utilization rate of the I / O forwarding node, and improve the I / O performance of the job. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The exemplary embodiments will be described in detail below with reference to the accompanying drawings, which are intended to depict exemplary embodiments and should not be interpreted as limiting the intended scope of the claims. Unless explicitly noted, the accompanying drawings are not to be considered to be drawn to scale.

[0022] Figure 1 A flowchart of a method for allocating I / O forwarding nodes to jobs in an I / O forwarding architecture according to an embodiment of the present invention is shown;

[0023] Figure 2 A flow chart of a method for calculating the optimal usage of I / O forwarding nodes of a job according to an embodiment of the present invention is shown;

[0024] Figure 3 A flow chart of a method for determining whether an I / O forwarding node has job-level I / O performance interference according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail by specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0026] According to one embodiment of the present invention, a method for allocating I / O forwarding nodes to jobs in an I / O forwarding architecture is provided. The method can be used to dynamically allocate I / O forwarding resources according to job granularity in a supercomputer adopting an I / O forwarding architecture.

[0027] Figure 1 The flowchart of the method for allocating I / O forwarding nodes to jobs in the I / O forwarding architecture according to an embodiment of the present invention is schematically shown. Figure 1, taking the allocation of I / O forwarding nodes for job A as an example to describe the various steps of this method:

[0028] Step S1. Obtain the default I / O forwarding node allocated to job A during job scheduling.

[0029] Those skilled in the art should understand that the main function of job scheduling is to examine whether the system can meet the resource requirements of user jobs based on the information in the job control block, wherein certain jobs are selected from the backup queue of the external memory and loaded into the memory according to a certain algorithm, and processes are created for these jobs and necessary resources are allocated; then, the newly created processes are inserted into the ready queue for execution. In a supercomputer that uses an I / O forwarding architecture, the job scheduling device will use an algorithm to allocate one or more I / O forwarding nodes to the job when performing job scheduling. The one or more I / O forwarding nodes are called the default I / O forwarding nodes of the job.

[0030] Step S2: Compare the number of default I / O forwarding nodes of job A with the optimal usage of I / O forwarding nodes of job A (or I / O forwarding node number threshold).

[0031] According to step S1, the number of default I / O forwarding nodes for job A is the number of I / O forwarding nodes allocated to job A by the job scheduling device when performing job scheduling. The optimal usage of I / O forwarding nodes for job A refers to the number of I / O forwarding nodes used when job A achieves optimal I / O performance without interference. Generally speaking, if the number of actual I / O forwarding nodes allocated to job A is greater than or equal to the optimal usage of I / O forwarding nodes for job A, the I / O performance of job A is better during its execution. The optimal usage of I / O forwarding nodes for each job may be different. Figure 2 A flowchart of a method for calculating the optimal usage of I / O forwarding nodes for a job according to an embodiment of the present invention is shown. Figure 2 , the method comprises the following steps:

[0032] ① Obtain the I / O information of the last time the job was executed (in the supercomputer that applies the I / O forwarding architecture).

[0033] The I / O information includes the number of I / O processes when the job was last executed. Those skilled in the art should understand that various existing I / O performance monitoring tools can be used to monitor the job-level I / O information in the supercomputer that uses the I / O forwarding architecture. These tools include but are not limited to: Beacon, Darshan I / O performance monitoring tools, etc.

[0034] ② Obtain the peak I / O bandwidth of the I / O forwarding nodes and the peak I / O bandwidth of the computing nodes in the supercomputer (ie, the supercomputer to which the I / O forwarding architecture is applied), and calculate the expansion factor.

[0035] The expansion factor represents the maximum number of computing nodes that an I / O forwarding node can support in a supercomputer that uses the I / O forwarding architecture. Specifically, the expansion factor is obtained by dividing the peak I / O bandwidth of the I / O forwarding node by the peak I / O bandwidth of the computing node. In other words, the expansion factor is the quotient of the peak I / O bandwidth of the I / O forwarding node and the peak I / O bandwidth of the computing node.

[0036] ③ Calculate the optimal usage of the I / O forwarding nodes for the job based on the data obtained in steps ① and ②.

[0037] According to one embodiment of the present invention, the calculation formula is as follows:

[0038]

[0039] Where i represents the i-th job; TH_node i Indicates the optimal usage of I / O forwarding nodes for the i-th job; NUM_io i represents the number of I / O processes when the i-th job was last executed in the supercomputer to which the I / O forwarding architecture is applied; α represents the expansion factor.

[0040] Step S3. If the number of default I / O forwarding nodes of job A is less than the optimal usage of I / O forwarding nodes of job A, execute step S5; otherwise, execute step S4.

[0041] Step S4. Determine whether the default I / O forwarding node of job A has job-level I / O performance interference (i.e., whether there is a default I / O forwarding node among the default I / O forwarding nodes of job A that has job-level I / O performance interference); if so, execute step S5; if not, assign the default I / O forwarding node of job A to job A as the actual I / O forwarding node of job A, and the process of assigning I / O forwarding nodes to job A ends here.

[0042] The existence of job-level I / O performance interference in an I / O forwarding node (e.g., a default I / O forwarding node of job A) means that there is I / O performance interference between multiple jobs whose default I / O forwarding node includes the I / O forwarding node. In other words, when the job scheduling device performs job scheduling, it allocates the I / O forwarding node as the default I / O forwarding node for some jobs, and there is I / O performance interference between these jobs.

[0043] Figure 3The flowchart of a method for determining whether an I / O forwarding node has job-level I / O performance interference according to an embodiment of the present invention is schematically shown. i For example, the method includes the following steps:

[0044] A. Get the default I / O forwarding nodes including Node i All assignments.

[0045] As mentioned above, when the job scheduling device performs job scheduling, it will assign default I / O forwarding nodes to multiple jobs. If a default I / O forwarding node of a job is Node i , it means that the default I / O forwarding nodes for this job include Node i .

[0046] B. Determine whether there is a high-bandwidth job among all the jobs obtained in step A. If there is a high-bandwidth job, the I / O forwarding node Node i There is I / O performance interference at the job level, and the judgment process ends; otherwise, execute step C.

[0047] According to an embodiment of the present invention, determining whether a job is a high-bandwidth job includes:

[0048] ① Obtain the average aggregate bandwidth of the last execution of the job (in the supercomputer using the I / O forwarding architecture). For example, obtain the average aggregate bandwidth of the last execution of the job through an I / O performance monitoring tool.

[0049] The aggregate bandwidth refers to the total amount of I / O operations performed per unit time at a certain moment during the I / O operation of the job (i.e., the amount of I / O operation data of all processes of the job), and its unit is GB / S. The average aggregate bandwidth of the job refers to the average value of the aggregate bandwidth during the I / O operation of the job.

[0050] ② If the average aggregate bandwidth of the job is greater than a predetermined threshold, the job is determined as a high-bandwidth job.

[0051] C. Determine whether there is a high IOPS (i.e., read and write times per second) job among all the jobs obtained in step A. If there is a high IOPS job, the I / O forwarding node Node i There is I / O performance interference at the job level, and the judgment process ends; otherwise, execute step D.

[0052] In one embodiment, determining whether a job is a high IOPS job includes:

[0053] ① Obtain the average aggregate IOPS of the last execution of the job (in the supercomputer using the I / O forwarding architecture). For example, obtain the average aggregate IOPS of the last execution of the job through an I / O performance monitoring tool.

[0054] ② If the average aggregate IOPS of the job is greater than a predetermined threshold, the job is determined as a high IOPS job.

[0055] D. Determine whether there is a metadata access-intensive job among all the jobs obtained in step A. If there is a metadata access-intensive job, the I / O forwarding node Node i There is I / O performance interference at the job level, and the judgment process ends; otherwise, execute step E.

[0056] According to an embodiment of the present invention, the metadata waiting queue is used as a standard for determining whether a job is a metadata access intensive job, because metadata operations will occupy thread resources on the I / O forwarding node, thereby interfering with normal data operation requests.

[0057] Those skilled in the art should understand that the metadata waiting queue generally refers to a queue specifically storing metadata operation requests, and each job has a corresponding metadata waiting queue on the I / O forwarding node actually assigned to it. In some cases, there may not be a queue specifically storing metadata operation requests on the I / O forwarding node. In this case, the metadata waiting queue refers to the I / O request queue (which stores data operation requests and metadata operation requests), and the length of the metadata waiting queue in this case refers to the number of metadata operation requests in the I / O request queue.

[0058] In one embodiment, determining whether a job is a metadata access intensive job includes:

[0059] ① Obtain the average metadata waiting queue length when the job was last executed (in the supercomputer using the I / O forwarding architecture).

[0060] Among them, firstly, the length of the metadata waiting queue on all I / O forwarding nodes actually allocated to the job when it was last executed is obtained, and then the average value of these metadata waiting queue lengths is taken as the average metadata waiting queue length of the job. According to one embodiment of the present invention, the average metadata waiting queue length when the job was last executed can be obtained through an I / O performance monitoring tool.

[0061] ② If the average metadata waiting queue length of the job is greater than a predetermined threshold, the job is determined as a metadata access intensive job.

[0062] E. Determine whether there is a job with I / O mode N-1 among all the jobs obtained in step A. If there is a job with I / O mode N-1, the I / O forwarding node Node i There is job-level I / O performance interference; otherwise, the I / O forwarding node i There is no I / O performance interference at the job level; the determination process ends.

[0063] Among them, the I / O performance monitoring tool can be used to obtain the I / O mode of the job when the job was last executed. The I / O mode can include: N-1 mode, that is, N I / O processes in the job read and write a file at the same time; NN mode, that is, N I / O processes in the job read and write N independent files; 1-1 mode, that is, only one I / O process in the job reads and writes a file. If the I / O mode used by the job is N-1 mode, that is, the job has N I / O processes operating a file at the same time, the job may cause I / O performance interference.

[0064] Step S5: Allocate a standby I / O forwarding node from the standby I / O forwarding node pool as the actual I / O forwarding node for job A.

[0065] Those skilled in the art should understand that the default I / O forwarding node assigned by the job scheduling device to the job is usually a shared I / O forwarding node, that is, multiple jobs can share the I / O forwarding node. According to one embodiment of the present invention, the standby I / O forwarding node is an unused I / O forwarding node and is an exclusive forwarding node. In one embodiment, step S5 includes the following sub-steps:

[0066] Step S51. If the number of default I / O forwarding nodes of job A is less than the optimal usage of I / O forwarding nodes of job A, execute step S52; otherwise, execute step S53.

[0067] Step S52. Compare the number of standby I / O forwarding nodes in the standby I / O forwarding node pool with the optimal usage S of the I / O forwarding nodes of job A. If the number of standby I / O forwarding nodes in the standby I / O forwarding node pool is greater than or equal to the optimal usage S of the I / O forwarding nodes of job A, then allocate S standby I / O forwarding nodes from the standby I / O forwarding node pool to job A as the actual I / O forwarding nodes of job A. Otherwise, allocate all standby I / O forwarding nodes in the standby I / O forwarding node pool to job A. The process of allocating I / O forwarding nodes to job A ends here.

[0068] Step S53. Compare the number of standby I / O forwarding nodes in the standby I / O forwarding node pool with the number L of default I / O forwarding nodes for job A; if the number of standby I / O forwarding nodes in the standby I / O forwarding node pool is greater than or equal to the number L of default I / O forwarding nodes for job A, then allocate L standby I / O forwarding nodes from the standby I / O forwarding node pool to job A as the actual I / O forwarding nodes for job A, otherwise allocate all standby I / O forwarding nodes in the standby I / O forwarding node pool to job A, and the process of allocating I / O forwarding nodes to job A ends here.

[0069] In the above-mentioned embodiment, the dynamic allocation of I / O forwarding nodes for jobs is realized. Among them, if the number of default I / O forwarding nodes allocated to the job is less than the optimal usage of I / O forwarding nodes calculated based on the historical I / O information of the job, a spare I / O forwarding node that meets the optimal usage of I / O forwarding nodes is allocated to the job as much as possible; in addition, even if the number of default I / O forwarding nodes of the job meets (i.e., is greater than or equal to) the optimal usage of I / O forwarding nodes of the job, it is necessary to further determine whether there is I / O performance interference in the default I / O forwarding node of the job, and if there is I / O performance interference, a spare I / O forwarding node will be allocated to the job. The present invention can effectively avoid I / O performance interference at the job granularity on the I / O forwarding node, improve the utilization rate of the I / O forwarding node, and improve the I / O performance of the job.

[0070] To verify the effectiveness of the method provided by the present invention, the inventors conducted experiments on the Sunway TaihuLight supercomputer. The experimental results show that the method provided by the present invention can bring more than 20 times I / O performance improvement to applications.

[0071] In the above-mentioned embodiment, the optimal usage of the I / O forwarding nodes of the job is calculated based on the number of I / O processes when the job was last executed in a supercomputer (to which the I / O forwarding architecture is applied); in addition, the I / O information when the job was last executed is used to determine whether the job is a high-bandwidth job, whether it is a high-IOPS job, and whether it is a metadata-intensive job. Those skilled in the art should understand that in other embodiments, the I / O information obtained by monitoring when the job is executed multiple times can also be used to perform the above-mentioned calculations or judgments. For example, the number of I / O processes when the job was executed n times before (n is an integer ≥2) is averaged or weighted averaged, and the optimal usage of the I / O forwarding nodes of the job is calculated using the average or weighted average; in addition, the average aggregate bandwidth, average aggregate IOPS, and average metadata waiting queue length of the job can be calculated in a similar manner.

[0072] In the above-mentioned embodiment, the I / O information of the job execution is monitored and obtained by the I / O performance monitoring tool. In a further embodiment, the I / O information of the job can be recorded after each job execution, so as to facilitate the subsequent search and calculation of the data such as the optimal usage of the I / O forwarding node of the job. It should be understood by those skilled in the art that the I / O information of the job includes but is not limited to: the number of I / O processes of the job, the total amount of I / O data, the average metadata waiting queue length of the I / O forwarding node of the job, the I / O mode of the job (including N-1 mode, NN mode and 1-1 mode), etc.

[0073] In the above-mentioned embodiment, (in step S4), in order to determine whether an I / O forwarding node has job-level I / O performance interference, it is first determined whether there is a high-bandwidth job in the job whose default I / O forwarding node is the I / O forwarding node. If there is no high-bandwidth job, it is then determined in turn whether there is a high-IOPS job, a metadata access intensive job, and an N-1 mode job in these jobs. In other embodiments, other sequences may be used to determine whether an I / O forwarding node has job-level I / O performance interference, for example, first determining whether there is a high-IOPS job in the job, etc. In other embodiments, one or some of the steps may be skipped, for example, only determining whether there is a high-bandwidth job in the job; or, first determining whether there is a high-bandwidth job in the job, and if there is no high-bandwidth job, determining whether there is a metadata access intensive job in the job.

[0074] In the above embodiment, sub-steps S51-S53 of step S5 are shown. In other embodiments, for step S5, the larger value M between the number of default I / O forwarding nodes of job A and the optimal usage of I / O forwarding nodes of job A can be calculated first, and then the number of standby I / O forwarding nodes in the standby I / O forwarding node pool can be compared with the larger value M, and then the actual I / O forwarding nodes are allocated according to the comparison result. In one embodiment, step S5 can include the following sub-steps:

[0075] Step S51 ′. Calculate the larger value M between the number of default I / O forwarding nodes of job A and the optimal usage of I / O forwarding nodes of job A.

[0076] Step S52 ′: Compare the number of standby I / O forwarding nodes in the standby I / O forwarding node pool with the larger value M.

[0077] Step S53'. If the number of standby I / O forwarding nodes in the standby I / O forwarding node pool is greater than or equal to the larger value M, then M standby I / O forwarding nodes are allocated from the standby I / O forwarding node pool to job A as the actual I / O forwarding nodes of job A; otherwise, all standby I / O forwarding nodes in the standby I / O forwarding node pool are allocated to job A as the actual I / O forwarding nodes.

[0078] It should be noted that some exemplary methods are depicted as flow charts. Although the flow charts describe the operations as being performed sequentially, it is understood that many operations may be performed in parallel, simultaneously, or synchronously. In addition, the order of the operations may be rearranged. The process may terminate when the operations are completed, but may also have additional steps not included in the figures or in the embodiments.

[0079] The above method can be implemented by hardware, software, firmware, middleware, pseudocode, hardware description language or any combination thereof. When implemented in software, firmware, middleware or pseudocode, the program code or code segment used to perform the task can be stored in a computer-readable medium, such as a storage medium, and the processor can perform the task.

[0080] It should be understood that exemplary embodiments of software implementations are typically encoded on some form of program storage medium or implemented on some type of transmission medium. The program storage medium can be any non-transient storage medium, such as a disk (e.g., a floppy disk or hard disk) or an optical disk (e.g., a compact disk read-only memory or "CD ROM"), and can be read-only or randomly accessed. Similarly, the transmission medium can be a twisted pair, a coaxial cable, an optical fiber, or some other suitable transmission medium known in the art.

[0081] Although the present invention has been described through preferred embodiments, the present invention is not limited to the embodiments described herein but includes various changes and modifications that may be made without departing from the scope of the present invention.

Claims

1. A method for allocating an I / O forwarding node to a job in an I / O forwarding architecture, comprising: Step 1) obtaining the number of default I / O forwarding nodes of a job and a threshold value of the number of I / O forwarding nodes of the job; wherein the default I / O forwarding node of the job is an I / O forwarding node allocated to the job by the job scheduling device when performing job scheduling; Step 2) compares the number of default I / O forwarding nodes of the job with the threshold number of I / O forwarding nodes of the job to determine whether to allocate a spare I / O forwarding node as the actual I / O forwarding node of the job, wherein step 2) includes: comparing the number of default I / O forwarding nodes of the job with the threshold number of I / O forwarding nodes of the job; if the number of default I / O forwarding nodes of the job is less than the threshold number of I / O forwarding nodes of the job, allocating a spare I / O forwarding node as the actual I / O forwarding node of the job; otherwise, determining whether there is job-level I / O performance interference with the default I / O forwarding node of the job, if so, allocating a spare I / O forwarding node as the actual I / O forwarding node of the job, if not, using the default I / O forwarding node of the job as the actual I / O forwarding node of the job; The I / O forwarding node number threshold of the job is calculated according to the following formula: Where i represents the i-th job; TH_node i Indicates the I / O forwarding node number threshold for the i-th job; NUM_io i represents the number of I / O processes when the i-th job was last executed; α represents the expansion factor, which is the quotient of the peak I / O bandwidth of the I / O forwarding node and the peak I / O bandwidth of the computing node.

2. According to the method of claim 1, in step 2), determining whether the default I / O forwarding node of the job has job-level I / O performance interference comprises: For each I / O forwarding node in the default I / O forwarding nodes of the job, obtaining all jobs where the default I / O forwarding node includes the I / O forwarding node; If, among all the obtained jobs, there is a job whose average aggregate bandwidth during the last execution is greater than a predetermined threshold, then the default I / O forwarding node of the job has job-level I / O performance interference.

3. According to the method of claim 1, in step 2), determining whether the default I / O forwarding node of the job has job-level I / O performance interference comprises: For each I / O forwarding node in the default I / O forwarding nodes of the job, obtaining all jobs where the default I / O forwarding node includes the I / O forwarding node; If, among all the obtained jobs, there is a job whose average aggregate IOPS during the last execution is greater than a predetermined threshold, then the default I / O forwarding node of the job has job-level I / O performance interference.

4. According to the method of claim 1, in step 2), determining whether the default I / O forwarding node of the job has job-level I / O performance interference comprises: For each I / O forwarding node in the default I / O forwarding nodes of the job, obtaining all jobs where the default I / O forwarding node includes the I / O forwarding node; If, among all the obtained jobs, there is a job whose average metadata waiting queue length during the last execution is greater than a predetermined threshold, then the default I / O forwarding node of the job has job-level I / O performance interference.

5. According to the method of claim 1, in step 2), determining whether the default I / O forwarding node of the job has job-level I / O performance interference comprises: For each I / O forwarding node in the default I / O forwarding nodes of the job, obtaining all jobs where the default I / O forwarding node includes the I / O forwarding node; If, among all the obtained jobs, there is a job whose I / O mode is the N-1 mode, then the default I / O forwarding node of the job has job-level I / O performance interference.

6. The method according to claim 1, wherein: Allocating a standby I / O forwarding node as an actual I / O forwarding node for the job includes: Calculate the larger value M between the number of default I / O forwarding nodes of the job and the I / O forwarding node number threshold of the job; Comparing the number of standby I / O forwarding nodes in the standby I / O forwarding node pool with the larger value M; If the number of standby I / O forwarding nodes in the standby I / O forwarding node pool is greater than or equal to the larger value M, M standby I / O forwarding nodes are allocated from the standby I / O forwarding node pool to the job as actual I / O forwarding nodes for the job; otherwise, all standby I / O forwarding nodes in the standby I / O forwarding node pool are allocated to the job as actual I / O forwarding nodes for the job.

7. The method according to claim 1, wherein: The standby I / O forwarding node is an exclusive I / O forwarding node.

8. The method according to claim 1, further comprising: Get the I / O information of the last job execution.

9. An electronic device, comprising a processor and a memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the electronic device implements the method according to any one of claims 1 to 8.

Images