Workflow processing method, scheduler and storage medium

Abstract

The invention discloses a workflow processing method, a scheduler and a storage medium, and relates to the technical field of data processing. The scheduler obtains a workflow comprising a plurality of parallel operation paths and schedules the workflow to a computing cluster, and the computing cluster is used for executing the workflow; in the process that a computing cluster executes a first job included in a key job path, a scheduler writes target data required by a second job into a first computing node in the computing cluster where the second job is located from a storage cluster, the total time consumed for executing multiple jobs included in the key job path is the maximum, and the target data required by the second job are written into a second computing node in the computing cluster where the second job is located from the storage cluster. The first job is performed prior to the second job in the critical job path. Therefore, the scheduler writes the input data required for executing the part of the jobs in the key job path into the first computing node for executing the part of the jobs in advance, so that the efficiency of executing the second job by the first computing node can be effectively improved, and the overall efficiency of executing the workflow can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a workflow processing method, a scheduler and a storage medium. Background Art

[0002] Workflow is a computer model of a workflow that is used to describe the business rules between the various operation steps in the workflow in a general way. It usually includes multiple jobs, and the expected goals can be achieved by executing multiple jobs in the workflow according to predefined rules. For example, in a genetic computing scenario, genetic data analysis can be achieved by executing multiple jobs in the workflow in sequence.

[0003] Currently, multiple jobs in a workflow are usually scheduled by a scheduler to at least one computing node in a computing cluster. At this time, the computing cluster can obtain the input data required to execute the multiple jobs by remotely accessing the storage cluster, and execute the jobs according to the obtained input data. However, the latency of the computing cluster remotely accessing the storage cluster is usually high, which affects the efficiency of the computing cluster in executing the workflow.

[0004] Therefore, there is an urgent need for a method to improve the efficiency of computing clusters in executing workflows. Summary of the invention

[0005] The present application provides a workflow processing method to improve the efficiency of a computing cluster in acquiring data required for executing a workflow, thereby improving the efficiency of the computing cluster in executing the workflow. In addition, the present application also provides a scheduler, a computer-readable storage medium, and a computer program product.

[0006] In a first aspect, the present application provides a workflow processing method, which is executed by a scheduler. Specifically, the scheduler obtains a workflow to be processed, which may be, for example, a workflow for analyzing genetic data, etc., including multiple parallel job paths, each of which includes multiple jobs executed in sequence; then, the scheduler schedules the workflow to a computing cluster, which is used to execute the workflow, and specifically may be scheduled to a corresponding computing node in the computing cluster, and the workflow is executed by the computing node, wherein the computing cluster includes multiple computing nodes, each of which may be, for example, a server, and the multiple computing nodes include a first computing node; then, in the process of the computing cluster executing the first job included in the critical job path, the scheduler writes target data from the storage cluster to the first computing node where the second job included in the critical job path is located, the target data is the input data required for executing the second job, and the total time consumed for executing the multiple jobs included in the critical job path among the multiple parallel job paths is the largest, and in the critical job path, the first job is executed before the second job, that is, before executing the second job, the input data required for executing the second job is written from the storage cluster to the first computing node where the second job is located in advance.

[0007] Since the total time consumed to execute multiple jobs in the critical job path determines the overall time consumed to execute the workflow, when the computing cluster is executing the workflow, the scheduler writes the input data required to execute some jobs (that is, the second job) in the critical job path from the storage cluster to the first computing node used to execute the part of the job in advance. This allows the first computing node to directly obtain the input data locally without remotely accessing the storage cluster, which can effectively improve the efficiency of the first computing node in executing the second job, thereby improving the overall efficiency of executing the workflow.

[0008] In a possible implementation, when the scheduler writes the target data from the storage cluster to the first computing node, it may specifically first obtain the available bandwidth of the computing cluster and the required bandwidth of the first job, and determine the remaining bandwidth based on the available bandwidth and the required bandwidth, and the remaining bandwidth may not be currently occupied by other jobs, so that, in the process of the computing cluster executing the first job, the scheduler uses the remaining bandwidth to write the target data from the storage cluster to the first computing node. In this way, when the target data required for executing the second job is preloaded for the first computing node, the use of bandwidth by the first job does not need to be affected, which can avoid the impact of the preloaded target data on the efficiency of the computing cluster executing the first job.

[0009] In a possible implementation, the scheduler can also obtain historical operation information when the computing cluster executes the workflow in the historical time period, and the historical operation information includes the execution time of each job in the workflow, so that the scheduler can determine the execution time corresponding to each job path in multiple parallel job paths based on the historical operation information, and determine the key job path with the longest total time consumption based on the execution time corresponding to each job path. In this way, the scheduler can estimate the execution time of each job in the workflow based on the historical operation information, so as to determine the key job path that affects the overall time consumption of the workflow.

[0010] In a possible implementation, when the scheduler schedules the workflow to the computing cluster, the scheduler may specifically schedule the first job to the first computing node, the output data of the first job is stored in the first computing node, and when the output data of the first job is used as the input data of the second job, the scheduler schedules the second job to the first computing node. In this way, the scheduler can schedule multiple jobs with data dependencies to be executed on the same computing node, which can speed up the efficiency of the first computing node in obtaining the input data of the second job, thereby improving the efficiency of executing the second job.

[0011] In a possible implementation, the multiple parallel job paths in the workflow also include a first job path, the first job path includes a third job, and the input data of the second job includes the output data of the first job and the output data of the third job; then, when the scheduler schedules the workflow to the computing cluster, it can be specifically to first schedule the first job to the first computing node, the output data of the first job is stored in the first computing node, and then, the scheduler schedules the third job to the second computing node among the multiple computing nodes, and the output data of the third job is stored in the second computing node; then, when scheduling the second job, it can be specifically when the start time of executing the second job caused by the output data of the third job being transmitted to the first computing node is earlier than the start time of executing the second job caused by the output data of the first job being transmitted to the second computing node, the second job is scheduled to the first computing node, and the output data of the third job is transmitted from the second computing node to the first computing node. In this way, when scheduling the second job, it can be determined which computing node to schedule the second job to according to the earliest time of executing the second job, and how to schedule the input data required for executing the second job across computing nodes, which can further speed up the execution of jobs in the workflow and improve the execution efficiency of the workflow.

[0012] In a second aspect, the present application provides a scheduler, which includes modules for executing the workflow processing method in the first aspect or any possible implementation of the first aspect.

[0013] In a third aspect, the present application provides a scheduler, the scheduler comprising a processor and a memory. The processor and the memory communicate with each other. The processor is used to execute instructions stored in the memory so that the scheduler executes the workflow processing method in the first aspect or any one of the implementations of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The scheduler may also include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.

[0014] In a fourth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computing device, the computing device executes the operating steps of the workflow processing method described in the first aspect or any one of the implementations of the first aspect.

[0015] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed on a computing device, enables the computing device to execute the operating steps of the workflow processing method described in the first aspect or any one of the implementations of the first aspect.

[0016] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A schematic diagram of the architecture of an exemplary data processing system provided by the present application;

[0018] Figure 2 is a schematic diagram of a workflow including five jobs;

[0019] Figure 3 A flowchart of a workflow processing method provided for this application;

[0020] Figure 4 A schematic diagram for accelerating the execution of tasks in a critical task path;

[0021] Figure 5 is a schematic diagram of using the remaining bandwidth to transmit the input data of operation 4;

[0022] Figure 6 A schematic diagram of the structure of a scheduler provided for this application;

[0023] Figure 7 A schematic diagram of the hardware structure of a scheduler provided in this application. DETAILED DESCRIPTION

[0024] In order to improve the efficiency of computing clusters in executing workflows, the present application provides a workflow processing method. During the execution of the workflow, the input data required for executing part of the job (which can affect the execution efficiency of the workflow) is written from the storage node 200 to the computing cluster in advance. This allows the computing cluster to directly obtain the input data locally when executing the part of the job, without the need to remotely access the storage cluster, thereby improving the efficiency of the computing cluster in executing the workflow.

[0025] To facilitate understanding of the technical solution of the present application, the technical terms involved in the present application are first explained below.

[0026] A workflow is a computer model of a workflow that is used to provide a general description of the business rules between the various operation steps in the workflow. It usually includes multiple jobs, and the desired goal can be achieved by executing multiple jobs in the workflow according to predefined rules.

[0027] A job refers to a collection of work that a computing device is required to do in a transaction, which may include programs, corresponding data, and job instructions.

[0028] A job path refers to multiple jobs that are executed in sequence. In each job path, you can define multiple jobs that need to be executed and the order in which each job is executed.

[0029] In this application, the critical job path refers to the job path with the longest total time consumption after executing all jobs in multiple parallel job paths included in the workflow. Therefore, when jobs in multiple job paths are executed in parallel, the overall time consumption of the workflow depends on the execution time consumption of the critical job path.

[0030] The required bandwidth is used in this application to refer to the bandwidth required to execute a job.

[0031] Historical operation information, in this application, refers to information recorded when a computing cluster executes a workflow in the past period of time, and may include information such as the time required for the computing cluster to execute each job in the workflow.

[0032] The technical solution in the present application will be described below in conjunction with the drawings in the embodiments of the present application.

[0033] See also Figure 1 , is a schematic diagram of the architecture of an exemplary data processing system provided in an embodiment of the present application. Figure 1 As shown, the data processing system 10 includes a computing cluster 100, a storage cluster 200 and a scheduler 300, and the computing cluster 100, the storage cluster 200 and the scheduler 300 can be connected through a high-speed Internet network.

[0034] The computing cluster 100 includes a plurality of computing nodes, each of which may be a server, etc. The plurality of computing nodes are used to execute workflows, such as executing workflows for gene analysis, etc. For ease of understanding and description, Figure 1 In the example, a computing cluster 100 including computing nodes 101 and 102 is used for exemplary description. Different computing nodes are connected through a high-speed interconnection network. Each computing node may include multiple processor cores, and each processor core may be configured with a memory. In addition, each computing node may also be configured with a cache, and the cache located in the computing node may be referred to as a local cache. Different processor cores, memories, and caches in the computing node are interconnected through an on-chip bus. Generally, different processor cores located in the same computing node can share data in the local cache.

[0035] The storage cluster 200 includes multiple storage nodes, which may be storage servers, etc. Figure 1 In the example, a storage cluster 200 including a storage node 201 and a storage node 202 is used for exemplary description. Each storage node can be used for persistent storage of data, such as persistent storage of temporary data or execution results generated when the data processing system 10 executes a job. The storage medium in the storage node can be, for example, a hard disk drive (HDD), a hybrid hard drive (HHD), a solid-state drive (SSD), or other types of storage media.

[0036] The scheduler 300 is used to schedule corresponding resources for each job in the workflow submitted by the user, and the resources may include computing resources, storage resources, and network resources. Among them, computing resources, for example, may be processor cores in the computing cluster 100, or other types of processors such as graphics processing units (GPUs). Storage resources, for example, may be memory and cache in computing nodes, or storage areas, input / output operations per second (IOPS) in the storage cluster 200. Network resources, for example, may be routing, network bandwidth, etc.

[0037] Since it takes a long time for the computing nodes in the computing cluster 100 to remotely access the data in the storage cluster 200, when the computing nodes execute a job in the workflow, if it takes a long time to access the input data required to execute the job from the storage cluster 200, the job will be in a state of waiting for execution for a long time, which will cause the total delay generated by the computing node executing the job to be longer, thereby resulting in lower overall efficiency of the computing node in executing the workflow.

[0038] Based on this, the present application provides a workflow processing method, which aims to improve the overall efficiency of executing workflows. In specific implementation, the workflow to be processed obtained by the scheduler 300 may include multiple parallel job paths, wherein each job path includes multiple jobs executed in sequence, and the jobs included in different job paths are different, and the jobs with differences in different job paths can be executed in parallel. For example, assuming that the workflow includes Figure 2 The 5 jobs shown, among which, Figure 2 The arrows in the figure indicate the execution sequence of the jobs. For example, jobs 2 and 3 need to wait until job 1 is completed before they can start executing. Figure 2 The workflow shown may include two parallel job paths, where job path 1 is "job 1 → job 2 → job 5" and job path 2 is "job 1 → job 3 → job 4 → job 5". Job 2 in job path 1 and job 3 (and job 4) in job path 2 may be executed in parallel.

[0039] After obtaining the workflow, the scheduler 300 can schedule the workflow to the computing cluster 100 so as to use the computing nodes 101 and 102 in the computing cluster 100 to execute each job in the workflow. In the process of the computing cluster 100 executing the workflow, the scheduler 300 can determine the key job path among the multiple parallel job paths, where the total time consumed in executing multiple jobs in the key job path is the largest, so that the execution time of multiple jobs in the key job path determines the overall execution time of the workflow. Figure 2 Taking the workflow shown as an example, assuming that the execution time of job 1 to job 5 is 1 hour, 4 hours, 3 hours, 5 hours, and 2 hours respectively, the total time consumed to execute multiple jobs in job path 1 is 7 hours (i.e. 1 hour + 4 hours + 2 hours), and the total time consumed to execute multiple jobs in job path 2 is 11 hours (i.e. 1 hour + 3 hours + 5 hours + 2 hours). Therefore, the scheduler 300 can take job path 2 as the critical job path.

[0040] Then, in the process of executing the first job in the critical job path, the scheduler 300 can write the input data required for executing the second job from the storage cluster 200 to the computing node where the second job is located, assuming it is computing node 102, wherein the execution order of the first job is before the execution order of the second job. In this way, when executing the second job, computing node 102 can directly obtain the input data required for executing the second job locally without remotely accessing the storage cluster 200. Since the rate at which computing node 102 obtains data locally is higher than the rate at which computing node 102 remotely accesses the storage cluster 200 to obtain data, the efficiency of computing node 102 in executing the second job can be improved, thereby reducing the total time consumed in executing multiple jobs in the critical job path, thereby improving the overall efficiency of computing cluster 100 in executing the workflow. Still with Figure 2 Taking the workflow shown as an example, during the execution of job 3, the scheduler 300 can write the input data required for executing job 4 from the storage cluster 200 to the computing node executing job 4 in advance, so as to speed up the efficiency of the computing node in obtaining the input data of job 4, thereby accelerating the execution of the workflow by the computing cluster 100.

[0041] Exemplarily, the above scheduler 300 may be implemented by software or hardware.

[0042] When implemented by software, the scheduler 300 can be implemented by program code in a computing device, for example, by at least one of a virtual machine, a container, a bare metal server (BMS), a computing engine, etc.

[0043] In the second example, when implemented by hardware, the scheduler 300 can be implemented by at least one physical device including a processor, such as a server, etc. Among them, the processor can be a central processing unit (CPU), and an application-specific integrated circuit (ASIC), a programmable logic device (PLD), a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a system on chip (SoC), a software-defined infrastructure (SDI) chip, an artificial intelligence (AI) chip, a data processing unit (DPU), and any processor core or any combination thereof. In addition, the number of processors included in the scheduler 300 can be one or more, and the type of processors included can be one or more. The number and type of processors can be set according to the business requirements of the actual application, and this embodiment does not limit this.

[0044] It is worth noting that the above Figure 1 The data processing system 10 shown is only used as an exemplary description and is not intended to be limiting. For example, in other possible data processing systems, other devices may also be included, such as a cache pool, which can be used to cache data for computing nodes 101 and computing nodes 102, such as caching data that interacts with storage cluster 200. Alternatively, the data processing system may also include more functional devices to support the data processing system 10 with more functions. For example, the data processing system may also include a log server for storing logs of resource scheduling operations performed by the scheduler 300 for multiple workflows, etc. The present application does not limit the specific architecture of the data processing system 10.

[0045] For ease of understanding, an embodiment of the workflow processing method provided by the present application is described below in conjunction with the accompanying drawings.

[0046] See also Figure 3 , Figure 3 A flowchart of a workflow processing method provided in an embodiment of the present application, which can be applied to Figure 1 The data processing system 10 described above may be applied to other applicable data processing systems. Figure 1 The data processing system 10 shown is taken as an example for illustrative description.

[0047] in, Figure 3 The workflow processing method shown may specifically include:

[0048] S301: The scheduler 300 obtains a workflow to be processed, where the workflow includes multiple parallel job paths, and each job path includes multiple jobs executed in sequence.

[0049] In actual application, there are usually some jobs in the workflow acquired by the scheduler 300 that can be executed in parallel, so based on these jobs, multiple parallel job paths can be formed, such as Figure 2 The working path 1 and the working path 2 are shown.

[0050] As some implementation examples of obtaining workflows, the data processing system 10 may provide a client to the outside, so that a user may submit a workflow including multiple jobs on the client, and the client may send the workflow to the scheduler 300 .

[0051] Alternatively, the data processing system 10 may further include a login node, on which a user may log in to an account, and after successfully completing the login, submit a job flow to the scheduler 300 through the login node.

[0052] Typically, the workflow submitted by the user may define resource requirements for each job, and the resource requirements are used to indicate the resources required to execute the job, such as the number of processor cores required, the number of processes required, the size of memory space required, etc., so that the scheduler 300 schedules resources for the job to meet the resource requirements of the job.

[0053] S302: The scheduler 300 schedules the workflow to the computing cluster 100, which is used to execute the workflow.

[0054] In specific implementation, the scheduler 300 may obtain available resources of some or all computing nodes in the computing cluster 100 to perceive the remaining available resources in the computing cluster 100. Available resources refer to resources on the computing nodes that are idle and can be scheduled to execute jobs, such as idle processor cores, unallocated memory, cache, etc.

[0055] Then, the scheduler 300 can schedule multiple jobs in the workflow to corresponding computing nodes according to the resource requirements of each job in the workflow and the available resources of multiple computing nodes in the computing cluster 100. Figure 2 Taking the job flow shown as an example, the scheduler 300 can schedule jobs 1, 3, 4 and 5 to the computing node 101, and schedule job 2 to the computing node 102, etc. In actual application, the scheduler 300 will also schedule corresponding storage resources and network resources for each job scheduled to the computing node to meet the storage resource and network resource requirements when the computing node executes the job.

[0056] S303: During the process of the computing cluster 100 executing the first job included in the critical job path, the scheduler 300 writes the target data from the storage cluster 200 to the computing node 102 where the second job included in the critical job path is located, wherein the target data is the input data required to execute the second job, the total time consumed to execute the multiple jobs included in the critical job path among the multiple parallel job paths included in the job flow is the largest, and the first job is executed before the second job in the critical job path.

[0057] In this embodiment, during the process of computing cluster 100 executing the workflow, scheduler 300 can speed up the efficiency of computing cluster 100 acquiring the input data by writing the required input data for some jobs into computing cluster 100 in advance, thereby accelerating the overall execution efficiency of the workflow.

[0058] In specific implementation, the scheduler 300 can first estimate the execution time of each job in the workflow, and calculate the total time required to execute all jobs in each job path based on the execution time required for each job. Then, the scheduler 300 can determine the job path with the longest total time consumption from multiple parallel job paths. For ease of description, the job path with the longest total time consumption is referred to as the critical job path. For example, suppose Figure 2 In the workflow shown, the execution time of job 1 to job 5 is 1 hour, 4 hours, 3 hours, 5 hours, and 2 hours, respectively. The scheduler 300 can calculate that the total time consumed to execute multiple jobs in job path 1 is 7 hours, and the total time consumed to execute multiple jobs in job path 2 is 11 hours. Therefore, the scheduler 300 can determine that the critical job path with the largest total time consumption is specifically job path 2.

[0059] Exemplarily, the present embodiment provides the following two implementation methods for estimating the duration of each job in the workflow as exemplary illustrations.

[0060] In the first implementation example, since the same workflow may be repeatedly executed in the actual application scenario, the scheduler 300 can obtain the historical operation information when the computing cluster 100 executes the workflow in the historical time period, such as obtaining the historical operation information generated when the computing cluster 100 executes the workflow last time. The historical operation information may include the execution time of each job in the workflow, such as the computing cluster 100 takes 1 hour to execute job 1 in the workflow in the historical time period, and takes 4 hours to execute job 2 in the workflow. Further, the historical operation information may also include the identifier of the input data / output data of each job (such as the file name), the data volume, and the source of the input data; and the historical operation information may also include the input / output (IO) characteristics of each job, such as the bandwidth allocated to each job in the historical time period. In actual application, the scheduler 300 may also record the identifier of the workflow to which the historical operation information belongs, the dependency relationship between different jobs in the workflow (or it may be called a workflow processing instance), etc. while recording the historical operation information.

[0061] In this way, when the scheduler 300 can receive the workflow again, the identifier of the workflow and the dependencies between different jobs in the workflow can be matched with the identifier and dependencies of the recorded workflow. And, when the match is successful, the scheduler 300 can estimate the execution time of each job in the workflow based on the historical operation information of the workflow (that is, the execution time of each job before is used as the estimated execution time of the job), so that the scheduler 300 can determine the critical job path with the longest total time consumption based on the execution time of each job.

[0062] Alternatively, when recording historical operation information, the scheduler 300 can analyze the historical operation information to obtain the critical job path of the workflow, so that when the scheduler 300 receives the workflow again, it can determine the critical job path in the workflow when it is determined that the workflow identifier and the dependencies between different jobs in the workflow match the recorded information.

[0063] In the second implementation example, for each job in the workflow, the scheduler 300 can derive the execution time of the job through theoretical calculation based on the number of input data, the number of output data, the processing logic for the input data, and the resource requirements of the job defined by the job, and thereby estimate the execution time corresponding to each job. In this way, the scheduler 300 can determine the critical job path with the longest total time consumption based on the estimated execution time for each job.

[0064] For multiple job paths in a workflow, jobs with differences between different job paths can usually be executed in parallel, and the overall execution time of the workflow depends on the job path with the longest total execution time among the multiple job paths, that is, the critical job path. To this end, in this embodiment, the scheduler 300 can specifically improve the overall efficiency of the workflow by accelerating the execution of jobs on the critical job path.

[0065] For example, Figure 2 Take the workflow shown as an example. The two different jobs in the workflow include job 2 in job path 1, and job 3 and job 4 in job path 2 (i.e., the key job path). Job 2 can be executed in parallel with job 3 and job 4. Since job 5 needs to wait for job 2 and job 4 to be completed before it can start execution, after job 2 (execution time is 4 hours) is completed, it needs to wait for job 3 and job 4 (execution time is 8 hours) to be completed. Therefore, by accelerating the execution of job 3 and job 4, job 5 can be executed in advance, that is, the execution of the workflow can be accelerated.

[0066] Illustratively, this embodiment provides the following two implementation methods for accelerating the execution of jobs in a critical job path.

[0067] In a first possible implementation, taking the critical job path including the first job and the second job as an example, the first job is executed before the second job, assuming that the scheduler 300 schedules the computing node 101 to execute the first job, and schedules the second job to the computing node 102. Then, in the process of the computing node 101 executing the first job in the critical job path, the scheduler 300 can determine the target data as the input of the second job, so that the scheduler 300 can access the storage cluster 200, and write the target data from the storage cluster 200 to the computing node 102 scheduled for the second job, for example, it can be written to the memory or local cache in the computing node 102. In this way, after the first job is executed, the computing node 102 can directly read the target data from the local, and use the target data to execute the second job, thereby reducing the latency of the computing node 102 obtaining the target data, thereby improving the efficiency of the computing node 102 in executing the second job.

[0068] For example, Figure 4 As shown above, before accelerating the execution of jobs in the critical job path, after executing job 3, the computing node 102 may need to spend 10 minutes to obtain the input data required for executing job 4 by remotely accessing the storage cluster 200. Figure 4As shown below, after accelerating the execution of jobs in the critical job path, during the execution of job 3, the scheduler 300 will schedule the input data required for executing job 4 to the computing node 102, so that after executing job 3, the computing node 102 can spend 1 minute to read the input data required for executing job 4 locally, which allows job 4 and its subsequent jobs to be executed 9 minutes (10 minutes - 1 minute) in advance, thereby reducing the overall execution time of the workflow by 9 minutes and shortening the time waiting for data transmission during the execution of the workflow.

[0069] In the process of writing the target data to the computing node 102, the scheduler 300 may first create a temporary working directory for the local storage area in the computing node 102, and write the target data into the temporary working directory. Accordingly, when the working node 102 obtains the target data, the directory to be accessed may be modified from the directory where the target data is stored in the storage cluster 200 to the temporary working directory, so that the computing node 102 may obtain the target data by accessing the local cache area under the temporary working directory.

[0070] In a second possible implementation, since the bandwidth resources of the computing cluster are limited and each job usually needs to occupy part of the bandwidth resources in the computing cluster during execution, the scheduler 300 can use the unoccupied bandwidth resources in the computing cluster 100 to load the target data to the computing node 102 in advance for the second job, so as to avoid competition for bandwidth resources in the computing cluster 100 between the scheduler 300 and the job currently executed by the computing cluster 100 during the process of loading the target data in advance.

[0071] In specific implementation, the scheduler 300 may first determine the required bandwidth of the first job and the second job, and the required bandwidth may be the bandwidth allocated to the first job and the second job according to the above historical operation information, or may be the required bandwidth of each job manually specified by the user when submitting the workflow. In addition, the scheduler 300 also determines the available bandwidth of the computing cluster 100. In this way, the scheduler 300 may determine the available bandwidth (hereinafter referred to as the remaining bandwidth) that is not occupied by the computing cluster 100 (the computing node 101 in it) during the execution of the first job according to the available bandwidth of the computing cluster 100 and the required bandwidth of the first job. Then, during the execution of the first job, the scheduler 300 may use the remaining bandwidth to write the target data from the storage cluster 100 to the computing node 102. In this way, when the computing node 102 loads the target data in advance, the efficiency of the computing cluster 100 in executing the first job may be avoided from being affected.

[0072] by Figure 2Taking the workflow shown in FIG. 1 as an example, if the scheduler 300 loads the target data from the storage cluster 200 to the computing node 102 based on the fixed-size bandwidth resources, the scheduler 300 may compete with the job 3 for the bandwidth resources of the computing cluster 100 during the execution of job 3, resulting in the bandwidth resources allocated to job 3 not meeting the required bandwidth of job 3. This makes job 3 acquire data more slowly due to the smaller bandwidth allocated, thereby increasing the execution time of job 3. Figure 5 Therefore, the scheduler 300 can determine the remaining bandwidth according to the available bandwidth of the computing cluster 100 and the required bandwidth of job 1 and job 3, and use the unoccupied remaining bandwidth to write the input data required for executing job 4 in the storage cluster 200 to the computing node 102 during the execution of job 1 and job 3. In this way, it can avoid that the insufficient bandwidth resources allocated to job 1 and job 3 will increase the execution time, as shown in FIG. Figure 5 shown.

[0073] In actual application, the scheduler 300 can start the process of loading the target data from the storage cluster 200 to the computing node 102 at an appropriate time to reduce the duration of the target data occupying the storage resources in the computing node 102. Specifically, the scheduler 300 can calculate the target duration required to load the target data from the storage cluster 200 to the computing node 102 according to the remaining bandwidth and the amount of target data, and estimate the end time of the first job (which can be calculated according to the start execution time of the first job and the execution time of the first job), so as to use the end time of the first job as the end time of loading the target data, calculate the start time of loading the target data using the remaining bandwidth according to the target duration required to load the target data, and start loading the target data to the computing node 102 at the start time. In this way, in the time before the start time, it is not necessary to occupy the storage resources in the computing node 102 to store the target data, and when the target data is loaded, the computing node 102 can execute the second job by reading the target data, so as to avoid the target data from meaninglessly occupying the storage resources in the computing node 102 for a long time, thereby reducing the waste of storage resources.

[0074] It is worth noting that in this embodiment, the critical job path includes two jobs, the first job and the second job, for example. When the critical job path includes more jobs, the scheduler 300 can load the input data of the job in advance for more jobs, wherein the data loaded in advance can be data stored in the storage cluster 200 or data on other computing nodes. For example, assuming that the critical job path also includes a third job, the execution order of the third job is after the second job, and the output data of the first job is used as the input data of the third job, then, when the computing node 102 executes the second job, the scheduler 300 can load the data output by the computing node 101 when executing the first job from the computing node 101 to the computing node 102, so as to speed up the efficiency of the computing node 102 in obtaining the data, thereby accelerating the execution of the third job.

[0075] Furthermore, when the critical job path includes more jobs, the scheduler 300 can determine, based on the remaining bandwidth when each job is executed, to use the remaining bandwidth to preload the input data required by other jobs during the execution of multiple jobs. Figure 5 As shown, the scheduler 300 can use the remaining bandwidth to preload data for job 4 during the execution of job 1 and job 3.

[0076] In actual application, the input data of some jobs in the critical job path may be the output data of other jobs. Therefore, when the scheduler 300 performs resource scheduling for the workflow, it can schedule multiple jobs with data dependencies to the same computing node, so that the computing node can directly use the output data of other jobs stored locally to execute the part of the job. For example, the scheduler 300 can determine the data dependency between different jobs in the critical job path based on historical operation information, wherein the data dependency refers to the output data of one job as the input data of another job, and there is a data dependency between the two jobs. For example, assuming that the input data of the second job includes not only the target data stored in the storage cluster 200, but also the output data generated by executing the first job, then the scheduler 300 can schedule the first job and the second job with data dependencies to the same computing node, that is, both to the computing node 102. In this way, after executing the first job, the computing node 102 can directly use the output data of the first job to execute the second job without obtaining the output data of the first job from other computing nodes or from the storage cluster 200. This can speed up the efficiency of the computing node 102 in obtaining the input data of the second job and improve the resource utilization of the computing cluster 100.

[0077] Furthermore, there may be input data of some jobs in the critical job path, including output data of other jobs in the critical job path and output data of jobs in other job paths. At this time, the scheduler 300 can schedule the jobs in the workflow to appropriate computing nodes based on the cost of data migration between different computing nodes to improve the overall efficiency of executing the workflow.

[0078] As an implementation example, it is assumed that the workflow includes job path 1 and job path 2, and job path 2 is the above-mentioned key job path. Among them, the output data of the first job in job path 2 and the output data of the third job in job path 1 (and the above-mentioned target data) are the input data required to execute the second job, and it is assumed that the scheduler 300 schedules the computing node 102 to execute the first job and schedules the computing node 101 to execute the third job.

[0079] Then, the scheduler 300 can calculate the first starting time of executing the second job caused by the process of transmitting the output data of the third job from the computing node 101 to the computing node 102. Specifically, the scheduler 300 can calculate the time required to transmit the output data of the third job from the computing node 101 to the computing node 102 according to the amount of the output data of the third job and the remaining bandwidth in the computing cluster 100, and determine the time when the computing node 102 obtains the complete output data of the third job according to the time and the end time of the third job. When the time is earlier than the end time of the first job, the end time of the first job can be used as the starting time of the second job (the computing node 102 can store all the input data required for executing the second job), and when the time is later than the end time of the first job, the time can be used as the starting time of the second job. For the convenience of description and distinction, it is referred to as the first starting time below.

[0080] Similarly, the scheduler 300 may calculate the second starting time of executing the second job caused by the process of transmitting the output data of the first job from the computing node 102 to the computing node 101 based on the above similar manner.

[0081] Finally, the scheduler 300 can compare the timing of the first start time and the second start time. When the first start time is earlier than the second start time, the scheduler 300 can transmit the output data of the third job to the computing node 102, and also schedule the second job to the computing node 102. When the first start time is later than the second start time, the scheduler 300 can transmit the output data of the first job to the computing node 101, and also schedule the second job to the computing node 101. In this way, the second job can start execution as early as possible, thereby shortening the execution time of the workflow and accelerating the overall execution efficiency of the workflow.

[0082] Still Figure 2 Taking the workflow shown as an example, it is assumed that the input data of job 5 (i.e., the second job) in job path 2 includes the output data of job 2 (i.e., the third job) and the output data of job 4 (i.e., the first job), wherein computing node 101 executes job 2 and computing node 102 executes job 4. The end time of executing job 2 is 8:00:00 and the end time of executing job 4 is 10:00:00. According to the amount of output data and the remaining bandwidth, it can be calculated that it takes 1 hour to transmit the output data of job 2 from computing node 101 to computing node 102, and it takes 15 minutes to transmit the output data of job 4 from computing node 102 to computing node 101.

[0083] If the output data of job 2 is transmitted to computing node 102, the end time of the output data being transmitted to computing node 102 is 9:00:00, which is later than the end time 10:00:00 of executing job 4. Therefore, the earliest start time of job 5 that can start execution is 10:00:00.

[0084] If the output data of job 4 is transmitted to computing node 101, the end time of the output data transmission to computing node 101 is 10:15:00, which is later than the end time 8:00:00 of executing job 2. Therefore, the earliest second start time that job 5 can start executing is 10:15:00.

[0085] Since the first start time 10:00:00 is earlier than the second start time 10:15:00, after the output data of job 2 is transmitted to the computing node 102 and job 5 is also scheduled to the computing node 102, the computing node 102 can execute job 5 15 minutes in advance. In this way, the scheduler 300 can reserve resources for job 5 on the computing node 102, so that after job 4 is executed, the computing node 102 can use the reserved resources to start executing job 5, thereby further accelerating the execution efficiency of the workflow.

[0086] It is worth noting that other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0087] Combination of the above Figures 1 to 5 The workflow processing method provided in the embodiment of the present application is introduced, and then the structure of the scheduler and computing device provided in the embodiment of the present application is introduced in conjunction with the accompanying drawings.

[0088] See also Figure 6 , showing a schematic diagram of the structure of a scheduler. Figure 6 The scheduler 600 shown may correspond to the scheduler 300 in the above-mentioned embodiment. Figure 6 As shown, the scheduler 600 includes:

[0089] The acquisition module 601 is used to acquire a workflow to be processed, where the workflow includes multiple parallel job paths, and each job path includes multiple jobs executed in sequence;

[0090] A scheduling module 602 is used to schedule the workflow to a computing cluster, where the computing cluster is used to execute the workflow, and the computing cluster includes multiple computing nodes, where the multiple computing nodes include a first computing node;

[0091] The writing module 603 is used to write the target data from the storage cluster to the first computing node where the second job included in the critical job path is located during the process of the computing cluster executing the first job included in the critical job path, the target data is the input data required to execute the second job, the total time consumed to execute the multiple jobs included in the critical job path among the multiple parallel job paths is the largest, and the first job in the critical job path is executed before the second job.

[0092] In a possible implementation, the writing module 603 is specifically configured to:

[0093] Obtaining the available bandwidth of the computing cluster and the required bandwidth of the first job;

[0094] Determine the remaining bandwidth based on the available bandwidth and the required bandwidth;

[0095] During the process of executing the first job in the computing cluster, the target data is written from the storage cluster to the first computing node by using the remaining bandwidth.

[0096] In a possible implementation, the acquisition module 601 is further used to acquire historical operation information when the computing cluster executes the workflow in a historical time period, and the historical operation information includes the execution time of each job in the workflow;

[0097] The scheduler 600 further includes: a determination module 604, which is used to determine the execution time corresponding to each operation path in multiple parallel operation paths according to historical operation information; and determine the key operation path according to the execution time corresponding to each operation path.

[0098] In a possible implementation, the scheduling module 602 is configured to:

[0099] Scheduling a first job to a first computing node, where output data of the first job is stored in the first computing node;

[0100] When the output data of the first job is used as the input data of the second job, the second job is scheduled to the first computing node.

[0101] In a possible implementation, the plurality of parallel operation paths further include a first operation path, the first operation path includes a third operation, and the input data of the second operation includes output data of the first operation and output data of the third operation;

[0102] The scheduling module 602 is used to:

[0103] Scheduling a first job to a first computing node, where output data of the first job is stored in the first computing node;

[0104] Scheduling the third job to a second computing node among the plurality of computing nodes, and storing output data of the third job on the second computing node;

[0105] When the start time of executing the second job caused by the output data of the third job being transmitted to the first computing node is earlier than the start time of executing the second job caused by the output data of the first job being transmitted to the second computing node, the second job is scheduled to the first computing node, and the output data of the third job is transmitted from the second computing node to the first computing node.

[0106] because Figure 6 The scheduler 600 shown corresponds to the above Figure 3 The scheduler 300 in the illustrated embodiment, therefore Figure 6 For the specific implementation of the scheduler 600 and its technical effects, see the above Figure 3 The description of the relevant parts in the illustrated embodiment will not be repeated here.

[0107] Figure 7 The hardware structure diagram of a scheduler 700 provided in this application is as follows. The scheduler 700 can implement the above Figure 3 The scheduler 300 etc. in the illustrated embodiment.

[0108] like Figure 7 As shown, the scheduler 700 includes a processor 701, a memory 702, and a communication interface 703. The processor 701, the memory 702, and the communication interface 703 communicate via a bus 704, and may also communicate via other means such as wireless transmission. The memory 702 is used to store instructions, and the processor 701 is used to execute instructions stored in the memory 702. Furthermore, the scheduler 700 may also include a memory unit 705, and the memory unit 705 may be connected to the processor 701, the storage medium 702, and the communication interface 703 via a bus 704. The memory 702 stores program code, and the processor 701 may call the program code stored in the memory 702 to perform the following operations:

[0109] Acquire a workflow to be processed, wherein the workflow includes multiple parallel job paths, and each job path includes multiple jobs executed in sequence;

[0110] Scheduling the workflow to a computing cluster, the computing cluster being used to execute the workflow, the computing cluster comprising a plurality of computing nodes, the plurality of computing nodes comprising a first computing node;

[0111] During the process of the computing cluster executing the first job included in the critical job path, target data is written from the storage cluster to the first computing node where the second job included in the critical job path is located, the target data is input data required to execute the second job, the total time consumed to execute the multiple jobs included in the critical job path among the multiple parallel job paths is the largest, and the first job in the critical job path is executed before the second job.

[0112] It should be understood that in the embodiment of the present application, the processor 701 may be a CPU, and the processor 701 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0113] The memory 702 may include a read-only memory and a random access memory, and provides instructions and data to the processor 701. The memory 702 may also include a nonvolatile random access memory.

[0114] The memory 702 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0115] The communication interface 703 is used to communicate with other devices connected to the scheduler 700. The bus 704 may include a power bus, a control bus, a status signal bus, etc. in addition to the data bus. However, for the sake of clarity, various buses are labeled as bus 704 in the figure.

[0116] It should be understood that the scheduler 700 according to the embodiment of the present application may correspond to the scheduler 600 in the embodiment of the present application, and may correspond to executing the scheduler 600 in the embodiment of the present application. Figure 3 The method executed by the scheduler 300 in the method shown, the above and other operations and / or functions implemented by the scheduler 700 are respectively to implement Figure 3 For the sake of brevity, the process of the corresponding method in will not be repeated here.

[0117] The embodiment of the present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk). The computer-readable storage medium includes instructions that instruct the computing device to execute the above-mentioned workflow processing method.

[0118] The embodiment of the present application also provides a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part.

[0119] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from one website, computer or data center to another website, computer or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0120] The computer program product may be a software installation package. When any of the aforementioned workflow processing methods is required, the computer program product may be downloaded and executed on a computing device.

[0121] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0122] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A workflow processing method, characterized in that: The method comprises: The scheduler obtains a workflow to be processed, wherein the workflow includes multiple parallel job paths, and each job path includes multiple jobs executed in sequence; The scheduler schedules the workflow to a computing cluster, the computing cluster is used to execute the workflow, the computing cluster includes a plurality of computing nodes, and the plurality of computing nodes includes a first computing node; During the process of the computing cluster executing the first job included in the critical job path, the scheduler writes target data from the storage cluster to the first computing node where the second job included in the critical job path is located, the target data is the input data required to execute the second job, the total time consumed to execute the multiple jobs included in the critical job path among the multiple parallel job paths is the largest, and the first job in the critical job path is executed before the second job.

2. The method according to claim 1, characterized in that The scheduler writes the target data from the storage cluster to the first computing node where the second job included in the critical job path is located, including: The scheduler obtains the available bandwidth of the computing cluster and the required bandwidth of the first job; The scheduler determines the remaining bandwidth according to the available bandwidth and the required bandwidth; During the process of the computing cluster executing the first job, the scheduler uses the remaining bandwidth to write the target data from the storage cluster to the first computing node.

3. The method according to claim 1 or 2, characterized in that: The method further comprises: The scheduler obtains historical operation information of the computing cluster when executing the workflow in a historical time period, wherein the historical operation information includes the execution time of each job in the workflow; The scheduler determines, according to the historical operation information, the execution duration corresponding to each of the multiple parallel operation paths; The scheduler determines the critical operation path according to the execution time corresponding to each operation path.

4. The method according to any one of claims 1 to 3, characterized in that: The scheduler schedules the workflow to the computing cluster, including: The scheduler schedules the first job to the first computing node, and the output data of the first job is stored in the first computing node; When the output data of the first job is used as the input data of the second job, the scheduler schedules the second job to the first computing node.

5. The method according to any one of claims 1 to 3, characterized in that: The plurality of parallel operation paths further include a first operation path, the first operation path includes a third operation, and the input data of the second operation includes output data of the first operation and output data of the third operation; The scheduler schedules the workflow to the computing cluster, including: The scheduler schedules the first job to the first computing node, and the output data of the first job is stored in the first computing node; The scheduler schedules the third job to a second computing node among the plurality of computing nodes, and output data of the third job is stored in the second computing node; When the start time of executing the second job caused by the output data of the third job being transmitted to the first computing node is earlier than the start time of executing the second job caused by the output data of the first job being transmitted to the second computing node, the second job is scheduled to the first computing node, and the output data of the third job is transmitted from the second computing node to the first computing node.

6. A scheduler, characterized in that: The scheduler comprises: An acquisition module, used to acquire a workflow to be processed, wherein the workflow includes multiple parallel job paths, and each job path includes multiple jobs executed in sequence; A scheduling module, used for scheduling the workflow to a computing cluster, the computing cluster is used for executing the workflow, the computing cluster includes a plurality of computing nodes, and the plurality of computing nodes includes a first computing node; A writing module is used to write target data from a storage cluster to the first computing node where a second job included in a critical job path is located during the process of the computing cluster executing a first job included in the critical job path, wherein the target data is input data required to execute the second job, the total time consumed to execute the multiple jobs included in the critical job path among the multiple parallel job paths is the largest, and the first job in the critical job path is executed before the second job.

7. The scheduler according to claim 6, characterized in that The writing module is used for: Obtaining the available bandwidth of the computing cluster and the required bandwidth of the first job; Determining a remaining bandwidth according to the available bandwidth and the required bandwidth; During the process of the computing cluster executing the first job, the target data is written from the storage cluster to the first computing node by using the remaining bandwidth.

8. The scheduler according to claim 6 or 7, characterized in that: The acquisition module is further used to acquire historical operation information when the computing cluster executes the workflow in a historical time period, wherein the historical operation information includes the execution time of each job in the workflow; The scheduler also includes: A determination module is used to determine the execution time corresponding to each operation path in the multiple parallel operation paths according to the historical operation information; and determine the key operation path according to the execution time corresponding to each operation path.

9. The scheduler according to any one of claims 6 to 8, characterized in that: The scheduling module is used to: Scheduling the first job to the first computing node, where output data of the first job is stored in the first computing node; When the output data of the first job is used as the input data of the second job, the second job is scheduled to the first computing node.

10. The scheduler according to any one of claims 6 to 8, characterized in that: The plurality of parallel operation paths further include a first operation path, the first operation path includes a third operation, and the input data of the second operation includes output data of the first operation and output data of the third operation; The scheduling module is used to: Scheduling the first job to the first computing node, where output data of the first job is stored in the first computing node; Scheduling the third job to a second computing node among the plurality of computing nodes, and storing output data of the third job on the second computing node; When the start time of executing the second job caused by the output data of the third job being transmitted to the first computing node is earlier than the start time of executing the second job caused by the output data of the first job being transmitted to the second computing node, the second job is scheduled to the first computing node, and the output data of the third job is transmitted from the second computing node to the first computing node.

11. A scheduler, characterized in that: Including processor and memory; The processor is used to execute instructions stored in the memory so that the scheduler is prepared to perform the steps of the method according to any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that: The method comprises instructions which, when executed on a computing device, cause the computing device to perform the steps of the method according to any one of claims 1 to 5.

Images