Resource scheduling method, scheduler, data processing system and related equipment
By scheduling related jobs to the same computing node to share data based on the data flow description information of the job flow in the data processing system, the problems of low IO performance and waste of resources in the data processing system are solved, and the overall efficiency and resources of the system are improved Utilization.
Patent Information
- Application Number
- CN202310912355.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-05-25
- Filing Date
- 2023-07-21
- Publication Date
- 2024-11-26
AI Technical Summary
When processing job flows, data processing systems are prone to problems of low IO performance and waste of resources. This is mainly due to the frequent migration of data by burst caches created by each job, which increases the inefficiency of scheduling of computing resources.
By obtaining data flow description information in the job flow, the scheduler schedules the relevant jobs to the same computing node, sharing input or output data, avoiding unnecessary migration of data between computing nodes, cache pools and storage clusters, and optimizing computing resources Scheduling.
It improves the IO performance and resource utilization of the data processing system, reduces the number of data migrations, reduces the job execution delay, and alleviates the problem of resource waste.
Smart Images

Figure CN119025015A_ABST
Abstract
Description
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on May 25, 2023, with application number 202310605586.3 and application name “A Resource Management Method”, all contents of which are incorporated by reference in this application. Technical Field
[0002] The present application relates to the field of Internet technology, and in particular to a resource scheduling method, a scheduler, a data processing system and related equipment. Background Art
[0003] As people's demand for computing power in data processing systems continues to increase, the scale of computing nodes in data processing systems is also expanding. At present, a scheduler is deployed in the data processing system, which is used to schedule the corresponding computing resources for each job submitted by the user and configure the storage resources and network resources required to execute the job.
[0004] In order to meet the user's low latency requirements for job processing, the scheduler will create at least one burst buffer (BB) for each job in the data processing system. The BB is used to load data from the file system (external filesystem) before the job is executed, so that data can be quickly obtained from the BB during the execution of the job, and the data in the BB will be persistently stored in the file system after the job is executed, thereby reducing the latency of the data processing system in processing jobs by improving the efficiency of reading and writing data.
[0005] In actual application scenarios, users often submit a job flow consisting of multiple jobs to a data processing system. When the data processing system executes each job in the job flow based on the BB created separately for each job, the input / output (IO) performance of the data processing system is low and resources are wasted. Summary of the invention
[0006] The present application provides a resource scheduling method for improving the IO performance of a data processing system and alleviating the problem of resource waste. In addition, the present application also provides a scheduler, a data processing system, a computer-readable storage medium, and a computer program product.
[0007] In a first aspect, the present application provides a resource scheduling method, which is applied to a scheduler in a data processing system, wherein the data processing system also includes a computing cluster and a storage cluster, wherein the computing cluster includes multiple computing nodes, and the computing cluster is connected to the storage cluster through a network; when performing resource scheduling, the scheduler obtains data flow description information and a job flow submitted by a user, wherein the job flow submitted by the user includes multiple jobs, and the data flow description information is used to indicate the relationship between the input data or output data of each job in the multiple jobs, such as indicating that the output data of job A is used as the input data of job B, etc.; then, when the input data or output data of the first job in the multiple jobs is stored in the first computing node among the multiple computing nodes, the scheduler schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, and the second job shares the input data or output data of the first job.
[0008] In this way, after the first job and the second job are scheduled to the same computing node, one of the jobs can directly obtain the input data it needs in the local cache area in the computing node, without having to access the cache area in the cache pool to obtain data. For example, during the execution of the first job, the input data or output data of the first job can be stored in the local cache area, so that when the second job is executed, the input data or output data of the first job can be directly obtained in the local cache area as the input data of the second job, which can effectively improve the efficiency of obtaining input data when executing the second job, thereby improving the efficiency of the data processing system in executing the second job. Moreover, during the execution of the first job and the second job, the input data required by the second job can be continuously stored in the local cache area in the computing node, which can avoid unnecessary migration of the data between the computing node, the cache pool and the storage cluster, that is, it can avoid the same data from migrating multiple times between the computing node, the storage cluster and the cache pool, thereby optimizing the scheduling of computing resources at the job flow level, thereby effectively improving the IO performance of the data processing system and alleviating the problem of resource waste.
[0009] In a possible implementation, the relationship between the input data or output data of each job in a plurality of jobs may be a dependency relationship between the input data or output data of each job in a plurality of jobs. For example, the data output by executing job 1 is used as the input data required when executing job 2. Alternatively, the relationship between the input data or output data of each job in a plurality of jobs may be a sharing relationship between the input data or output data of each job in a plurality of jobs. For example, the input data required to execute job 1 is the same as the input data required to execute job 2, that is, multiple jobs can share the same input data. In this way, scheduling multiple jobs in a job flow according to the dependency relationship or sharing relationship between the input data or output data of different jobs can improve the overall execution efficiency of the job flow and avoid unnecessary migration of data between data processing systems.
[0010] In a possible implementation, when the scheduler schedules the first job and the second job to the first computing node, specifically, when the available resources of the first computing node meet the resource requirements of the first job and the second job, the scheduler schedules the first job and the second job to the first computing node according to the data flow description information. In this way, according to the available resources on the computing node and the resource requirements of each job, the job flow is scheduled, which can avoid the resource requirements of multiple jobs scheduled to the computing node exceeding the number of available resources of the computing node, thereby avoiding the subsequent impact of job execution due to insufficient available resources of the computing node.
[0011] In a possible implementation, when the scheduler obtains the data flow description information, it can first schedule the first resource for multiple jobs according to the job flow, and collect the operation information generated when the multiple jobs are running based on the first resource, and the operation information includes the input data and output data of each of the multiple jobs, so as to analyze the operation information to obtain the data flow description information. In this way, by executing a job flow once, the dependency or sharing relationship between the input data or output data of different jobs in the job flow can be analyzed, so as to automatically generate the data flow description information, so that when the job flow is executed next time, the automatic optimization of resource scheduling can be realized according to the data flow description information, and the IO performance of the data processing system can be improved.
[0012] In a possible implementation, when scheduling the first job and the second job to the first computing node, the scheduler can generate inter-job data flows corresponding to the multiple jobs according to the data flow description information. Figure XJDFG, the XJDFG includes a job node, a data node, and a connection line, wherein the job node is used to indicate the job in the job flow, the data node is used to indicate the input data or output data of the job in the job flow, and the connection line is used to indicate the relationship between the job node and the data node, so that the scheduler schedules the first job and the second job related to the input data or output data of the first job to the first computing node according to the XJDFG. In this way, the scheduler can perceive the dependency or sharing relationship between the input data or output data of different jobs according to the XJDFG, and thereby schedule the first job and the second job in the job flow to the same computing node, which can optimize the scheduling of computing resources at the level of the job flow.
[0013] In a possible implementation, when the scheduler obtains the data flow description information, it may specifically obtain a configuration file submitted by a user and written in a domain-specific language DSL, where the configuration file includes multiple configuration items, and the multiple configuration items are used to configure the input data and output data of multiple jobs. In this way, the scheduler can directly use the data flow description information submitted by the user to optimize the scheduling of computing resources at the job flow level.
[0014] In a possible implementation, the computing cluster also includes a cache pool, and the multiple jobs in the job flow also include a third job. The scheduler can also schedule the third job related to the input data or output data of the second job to the second computing node among the multiple computing nodes (such as the amount of available resources on the first computing node is insufficient to support the running of the third job, etc.) according to the data flow description information, and schedule the target cache area in the cache pool to the second job and the third job; wherein the third job shares the input data or output data of the second job through the target cache area. In this way, the input data or output data of the second job can be shared with the third job through the target cache area in the storage pool, which enables the input data or output data of the second job to be continuously stored in the target cache area, avoiding the frequent migration of the input data or output data of the second job between the cache pool, the storage cluster and the computing nodes.
[0015] In a possible implementation, the cache pool includes multiple available cache areas, the multiple cache areas include a target cache area, and the sum of the latency of the first computing node and the second computing node accessing the target cache area is less than the sum of the latency of the first computing node and the second computing node accessing other cache areas in the multiple cache areas. In this way, the latency of the first computing node and the second computing node reading / writing data shared between the second job and the third job from the cache area in the cache pool when executing the job can be reduced.
[0016] In a second aspect, the present application provides a scheduler, which is applied to a data processing system, wherein the data processing system also includes a computing cluster and a storage cluster, wherein the computing cluster includes multiple computing nodes, and the computing cluster and the storage cluster are connected through a network; the scheduler includes: an acquisition module, which is used to obtain data flow description information and a job flow submitted by a user, wherein the job flow includes multiple jobs, and the data flow description information is used to indicate the relationship between the input data or output data of each job in the multiple jobs; a scheduling module, which is used to schedule a second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information when the input data or output data of the first job in the multiple jobs is stored in the first computing node among the multiple computing nodes, and the second job shares the input data or output data of the first job.
[0017] In a possible implementation, the relationship between the input data or output data of each job in a plurality of jobs includes: a dependency relationship between the input data or output data of each job in a plurality of jobs; or a sharing relationship between the input data or output data of each job in a plurality of jobs.
[0018] In a possible implementation, when the scheduling module schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, it is specifically used to:
[0019] When the available resources of the first computing node meet the resource requirements of the first job and the second job, the first job and the second job are scheduled to the first computing node according to the data flow description information.
[0020] In one possible implementation, when acquiring data flow description information, the acquisition module is specifically used to: schedule a first resource for multiple jobs based on the job flow; collect operation information generated when multiple jobs are running based on the first resource, the operation information including input data and output data of each job in the multiple jobs; analyze the operation information to obtain data flow description information.
[0021] In a possible implementation, when the scheduling module schedules the second job and the first job related to the input data or output data of the first job to the first computing node according to the data flow description information, it is specifically used to: generate the data flows between the jobs corresponding to the multiple jobs according to the data flow description information; Figure XJDFG, XJDFG includes job nodes, data nodes and connecting lines. The job nodes are used to indicate jobs in the job flow, the data nodes are used to indicate the input data or output data of the jobs in the job flow, and the connecting lines are used to indicate the relationship between the job nodes and the data nodes. According to XJDFG, the first job and the second job related to the input data or output data of the first job are scheduled to the first computing node.
[0022] In a possible implementation, when acquiring data flow description information, the acquisition module is specifically used to: acquire a configuration file submitted by a user and written in a domain-specific language DSL, where the configuration file includes multiple configuration items, and the multiple configuration items are used to configure input data and output data of multiple jobs.
[0023] In one possible implementation, the computing cluster also includes a cache pool, and the multiple jobs also include a third job. Then, the scheduling module is further used to schedule the third job related to the input data or output data of the second job to the second computing node among the multiple computing nodes based on the data flow description information, and schedule the target cache area in the cache pool to the second job and the third job; wherein the third job shares the input data or output data of the second job through the target cache area.
[0024] In one possible implementation, the cache pool includes multiple available cache areas, the multiple cache areas include a target cache area, and the sum of the delays of the first computing node and the second computing node accessing the target cache area is less than the sum of the delays of the first computing node and the second computing node accessing other cache areas in the multiple cache areas.
[0025] The scheduler provided in the second aspect corresponds to the resource scheduling method provided in the first aspect. Therefore, the technical effects of the scheduler in the second aspect and any possible implementation method of the second aspect can refer to the technical effects of the first aspect and the corresponding implementation method of the first aspect, and will not be repeated here.
[0026] In a third aspect, the present application provides a scheduler, the scheduler comprising a processor and a memory; the memory is used to store instructions, and when the scheduler is running, the processor executes the instructions stored in the memory, so that the scheduler executes the method described in the first aspect or any possible implementation of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The scheduler may also include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.
[0027] In a fourth aspect, the present application provides a data processing system, which includes a computing cluster, a storage cluster, and a scheduler, wherein the scheduler is used to execute the method described in the first aspect or any possible implementation of the first aspect.
[0028] In a fifth aspect, the present application provides a computer-readable storage medium, wherein instructions are stored in the computer-readable storage medium, which, when executed on a scheduler, enables the scheduler to execute the method described in the first aspect or any one of the implementations of the first aspect.
[0029] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a scheduler, enables the scheduler to execute the method described in the first aspect or any one of the implementations of the first aspect.
[0030] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS
[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0032] Figure 1 A schematic diagram of the structure of an exemplary data processing system provided in an embodiment of the present application;
[0033] Figure 2 A schematic diagram of multiple migrations of temporary data between the cache pool 120 and the storage cluster 200;
[0034] Figure 3 A schematic diagram of a resource scheduling method provided in an embodiment of the present application;
[0035] Figure 4 A schematic diagram of the structure of a scheduler 300 provided in an embodiment of the present application;
[0036] Figure 5A A schematic diagram of an exemplary configuration file in json format provided in an embodiment of the present application;
[0037] Figure 5B A schematic diagram of another exemplary configuration file in json format provided in an embodiment of the present application;
[0038] Figure 6 A schematic diagram of an exemplary data flow diagram between jobs provided in an embodiment of the present application;
[0039] Figure 7 A schematic diagram of the cache pool 120 scheduling three cache areas for jobs 1 to 4;
[0040] Figure 8 A flowchart of another resource scheduling method provided in an embodiment of the present application;
[0041] Fig. 9 A schematic diagram of the structure of a scheduler provided in an embodiment of the present application;
[0042] Fig.10 A schematic diagram of the structure of a scheduler provided in an embodiment of the present application. DETAILED DESCRIPTION
[0043] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.
[0044] See also Figure 1 , which shows a schematic diagram of the structure of a data processing system. Figure 1 As shown, the data processing system 10 includes a computing cluster 100, a storage cluster 200 and a scheduler 300, and the computing cluster 100, the storage cluster 200 and the scheduler 300 can be connected through a high-speed Internet network.
[0045] The computing cluster 100 includes a plurality of computing nodes, each of which may be a server, etc. The plurality of computing nodes are used to execute jobs submitted by users. Figure 1 In the example, a computing cluster 100 including computing nodes 101 and 102 is used for exemplary description. Different computing nodes are connected via a high-speed interconnection network. Each computing node may include multiple processor cores, such as Figure 1 The computing node 101 includes four processor cores, and the processor cores in the computing node are configured with memory. In addition, each computing node can be configured with a cache, and the cache located in the computing node can be called a local cache. Then, the storage area in the local cache is the local cache area. Figure 1 As shown, computing node 101 includes a local cache area 1011, computing node 1021 includes a local cache area 1021, and different processor cores, memories, and caches in the computing nodes are interconnected via an on-chip bus ( Figure 1 The connection between the cache and the processor core and the memory is not shown in the figure). Generally, different processor cores located in the same computing node can share data in the local cache area.
[0046] The computing cluster 100 also includes a cache pool 120 for caching data. Exemplarily, the cache pool 120 can be implemented by one or more storage media, such as a solid state drive (SSD), a static random access memory (SRAM), etc., which is not limited to this.
[0047] The storage cluster 200 includes multiple storage nodes, which may be storage servers, etc. Figure 1 In the example, a storage cluster 200 including a storage node 201 and a storage node 202 is used for exemplary description. Each storage node can be used for persistent storage of data, such as persistent storage of temporary data or execution results generated when the data processing system 10 executes a job. The storage medium in the storage node can be, for example, a hard disk drive (HDD), a hybrid hard drive (HHD), a solid-state drive (SSD), or other types of storage media.
[0048] The scheduler 300 is used to schedule corresponding resources for the jobs submitted by the user, and the resources may include at least computing resources, storage resources, and network resources. Among them, computing resources, for example, may be processor cores in the computing cluster 100, or other types of processors such as graphics processing units (GPUs). Storage resources, for example, may be cache areas (such as BBs, etc.) in the cache pool 102, or storage areas in the storage cluster 200, input / output operations per second (IOPS), etc. Network resources, for example, may be routing, network bandwidth, etc.
[0049] In actual application scenarios, users usually submit a job stream to the data processing system 10, and the job stream includes multiple jobs that have a dependency or sharing relationship with each other. Among them, there is a dependency relationship between job 1 and job 2, which means that the execution of job 2 depends on the previous execution of job 1. There is a sharing relationship between job 1 and job 2, which means that the same data will be used as input data for job 1 and job 2. At this time, if the scheduler 300 only considers the computing power level and schedules resources for each job in the job stream one by one, then the data processing system 10 will frequently migrate data between the cache pool 120 and the storage cluster 200, which will result in low IO performance of the data processing system and prominent waste of resources.
[0050] Take the job flow including job 1 and job 2 as an example. Assuming that job 1 and job 2 have a dependency relationship, then, Figure 2 As shown, the scheduler 300 will not only schedule the corresponding computing nodes for job 1 and job 2 to execute the jobs, but also schedule the cache area 1 in the cache pool 120 to job 1, and schedule the cache area 2 in the cache pool 120 to job 2, so that before executing job 1 and job 2, the data stored persistently in the storage cluster 200 is migrated to the cache area in the cache pool 120, so as to use the cache area in the cache pool 120 to speed up job 1 and job 2 to obtain input data. Figure 2 As shown, during the execution of job 1, the data processing system 10 will obtain input data from the cache area 1 allocated for job 1 (the cache pool 120 migrates the input data from the storage cluster 200 to the cache area 1 in advance), and write the temporary data generated by executing job 1 into the cache area 1. After executing job 1, the temporary data in the cache area 1 will be written into the storage cluster 200 for persistent storage. Then, before executing job 2, the cache pool 120 will write the temporary data stored in the storage cluster 200 into the cache area 2 allocated for job 2 in advance, so that job 2 can be executed according to the temporary data stored in the cache area 2 as input data, thereby using the cache area 2 to speed up the execution of job 2. In this way, the same temporary data is frequently migrated between the cache pool 120 and the storage cluster 200, which will cause a large number of unnecessary IOs to be generated between the cache pool 120 and the storage cluster 200, thereby affecting the overall IO performance of the data processing system 10.
[0051] Based on this, the present application provides a resource scheduling method. For a job stream submitted by a user, the scheduler 300 will obtain the data stream description information corresponding to the job stream. The data stream description information is used to indicate the relationship between the input data or output data of each job in the multiple jobs included in the job stream. The relationship may be, for example, a dependency relationship or a sharing relationship. Among them, the dependency relationship between input data or output data refers to the data output by one job, which is the input data of another job. The sharing relationship between input data or output data refers to the input data of two jobs being the same data. Then, the scheduler 300 schedules resources for multiple jobs according to the data stream description information. Specifically, the first job and the second job in the job stream may be scheduled to the same computing node. Taking scheduling to computing node 101 as an example, the available resources on computing node 101 can simultaneously meet the resource requirements of the first job and the second job. The input data / output data of the first job has a relationship with the input data / output data of the second job, such as a dependency relationship or a sharing relationship. In this way, the second job can share the input data or output data of the first job on the computing node 101. For example, the input and output or output data of the first job can be written into the local cache area 1011 on the computing node 101, so that when the computing node 101 executes the second job, it can obtain the input data of the second job by accessing the local cache area 1011. When the input data or output data of the two jobs have a sharing relationship, the input data of the second job is also the input data of the first job, and the input data can be loaded from the storage cluster 200. When the input data or output data of the two jobs have a dependency relationship, the input data of the second job is temporary data generated after the first job is executed.
[0052] In this way, the scheduler 300 schedules the first job and the second job having a relationship between input data or output data to the same computing node 10, that is, computing node 101, so that one of the jobs can directly obtain the input data it needs from the local cache area 1011 in the computing node 101, without having to obtain data by accessing the cache area in the cache pool 120. Since the rate at which the computing node 101 reads and writes data from the local cache area is usually higher than the rate at which it reads and writes data from the cache pool 120, this can effectively improve the efficiency of obtaining input data when executing the second job, thereby improving the efficiency of the data processing system 10 in executing the second job. Moreover, during the execution of the first job and the second job, the input data required for the second job can be continuously stored in the local cache area 1011 in the computing node 101, which can avoid unnecessary migration of the data between the computing node 101, the cache pool 120 and the storage cluster 200, that is, it can avoid the same data from being migrated multiple times between the computing node 101, the storage cluster 200 and the cache pool 120, thereby optimizing the scheduling of computing resources at the job flow level, and further effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0053] Still taking the above-mentioned scheduling of resources for jobs 1 and 2 in the job flow as an example, the scheduler 300 can determine the output data of job 1 as the input data of job 2 according to the data flow description information submitted by the user, then the scheduler 300 can schedule jobs 1 and 2 to the computing node 101. In this way, when executing job 2, the computing node 101 can directly read the temporary data generated by executing the job from the local cache area 1011. In this way, in the process of executing the job flow, the temporary data generated by executing job 1 can be continuously stored in the local cache area 1011, and there is no need to migrate the temporary data back and forth between the computing node 101, the cache pool 120 and the storage cluster 200 (that is, there is no need to migrate the temporary data from the computing node 101 to the cache pool 120, and then from the cache pool 120 to the storage cluster 200, and when executing job 2, the temporary data is migrated from the storage cluster 200 to the storage pool 120, and then from the storage pool 120 to the computing node 101), so that the IO performance of the data processing system 10 can be improved and the problem of resource waste can be alleviated. When the input data of job 1 is consistent with the input data of job 2 (that is, job 1 and job 2 share the same input data), computing node 101 can store the input data in local cache area 1011 when executing job 1, so that computing node 101 can directly obtain the input data from local cache area 1011 when executing job 2, without having to access cache pool 120 or storage cluster 200 to obtain the input data. This can reduce the number of times the input data is migrated from storage cluster 200 to cache pool 120 and computing node 101, thereby improving the efficiency of computing node 101 in executing job 2.
[0054] In addition, the scheduler 300 can also schedule the cache area in the cache pool 120 for each job in the job flow, so that each computing node can obtain data by accessing the corresponding cache area in the cache pool 102 when executing the job, which can speed up data acquisition compared to accessing data in the storage cluster 200.
[0055] It is worth noting that the above Figure 1 The data processing system 10 shown is only used as an exemplary illustration and is not intended to limit the implementation of the data processing system 10 to Figure 1Example shown. For example, in other possible data processing systems, the cache pool 120 may not be included, such as each computing node can directly obtain the input data required to execute the job by accessing the storage cluster 200. Alternatively, in other possible data processing systems, more functional devices may be included, such as one or more login nodes, so that users can log in to their accounts and submit jobs through the login nodes. Alternatively, the data processing system may also include a log server for storing logs of scheduling operations performed by the scheduler 300 for multiple job streams, etc., to support the data processing system 10 with more functions. In addition, the above-mentioned resource scheduling method can be applied to data processing systems with other architectures, such as a data processing system in which a computing cluster 100 and a storage cluster 200 are integrated (i.e., storage and computing are integrated). The specific architecture of the data processing system is not limited in this application.
[0056] For ease of understanding, the resource scheduling process of the scheduler for the user's job flow is described in detail below with reference to the accompanying drawings.
[0057] See also Figure 3 , shows a schematic diagram of a resource scheduling method provided in an embodiment of the present application, which can be applied to Figure 1 The data processing system 10 shown in FIG. 1 may be used in other applicable data processing systems. Figure 1 The data processing system 10 shown in FIG. Figure 3 As shown, the method may specifically include:
[0058] S301: The scheduler 300 obtains data flow description information and a job flow submitted by a user, wherein the job flow submitted by the user includes multiple jobs, and the data flow description information is used to indicate the relationship between input data or output data of each job in the multiple jobs in the job flow.
[0059] The relationship between the input data / output data of different jobs may be a dependency relationship or shared data, or may be other types of relationships, which are not limited in this embodiment.
[0060] Taking the job flow including the first job and the second job as an example, when the input data / output data of the first job and the input data / output data of the second job satisfy the dependency relationship, the data generated and output by executing the first job is used as the input data when executing the second job.
[0061] When the input data / output data of the first job and the input data / output data of the second job satisfy a sharing relationship, the input data required to execute the first job and the input data required to execute the second job are the same, such as both are data loaded from the storage cluster 200 .
[0062] Typically, the job flow submitted by the user may define resource requirements for each job, and the resource requirements are used to indicate the resources required to execute the job, such as the number of processor cores required, the number of processes required, the size of memory space required, etc., so that the scheduler 300 schedules resources for the job to meet the resource requirements of the job.
[0063] In this embodiment, when submitting a job flow including multiple jobs, the user also submits data flow description information for indicating the dependency relationship or sharing relationship between the input data / output data of the multiple jobs.
[0064] As some implementation examples, the data processing system 10 may provide a client to the outside, so that the user can submit the job flow and data flow description information to the data processing system 10 on the client, and the client sends it to the scheduler 300. Alternatively, the data processing system 10 may also include a login node, and the user may log in to the account on the login node, and submit the job flow and data flow description information to the scheduler 300 on the login node after successfully completing the login.
[0065] In actual application, the scheduler 300 may be configured with middleware, such as Figure 4 The middleware 301 shown, so that the scheduler 301 can use the middleware 301 to receive the job flow and data flow description information submitted by the user.
[0066] The configuration file submitted by the user can be written based on a domain specific language (DSL). A domain specific language refers to a programming language designed for a specific problem domain or application scenario, which aims to provide a higher level of abstraction to make programming simpler, clearer and more effective. The configuration file includes multiple configuration items, and the multiple configuration items are used to configure the input data and output data of multiple jobs in the job flow, so as to describe the dependency and sharing relationship between the input data / output data of the multiple jobs.
[0067] In a specific implementation, some configuration items included in the configuration file can be used to describe the relationship between the job and the data, and another part of the configuration items included in the configuration file can be used to describe relevant information of the data, such as attribute information of the data, IO information for the data, etc. Exemplarily, the types of multiple configuration items in the configuration file can be shown in Table 1 below.
[0068] Table 1
[0069]
[0070] In actual application, the types of configuration items defined in the configuration file are not limited to the examples shown in Table 1 above.
[0071] The configuration file submitted by the user may be in any of the following formats: json, xml, yaml, or other file formats.
[0072] For example, a user can submit Figure 5A The json-formatted configuration file shown in the figure can directly describe the input data and output data of each job. For example, for a job with "jobID" (i.e., job ID) of "2", its input data includes data with data names (i.e., data ID) of "tmpData1" and "srcData2", and its output data includes data with data name of "outData1", and describes information such as data type, data operation, number of times it is opened, and temporary type.
[0073] Alternatively, the user can submit Figure 5B The json-formatted configuration file shown in the figure can define the input data and output data of each job through labels. For example, for the job with "jobID" (i.e., job name) "2", the label defined for the input data and output data of the job is "label2", and the label "label2" defines the data name of the job's input data as "tmpData1" and "srcData2", the data name of the output data as "outData1", and describes information such as data type, data operation, number of opens, and temporary type.
[0074] S302: When the input data or output data of a first job among multiple jobs is stored in the computing node 101, the scheduler 300 schedules a second job related to the input data or output data of the first job and the first job to the computing node 101 according to the data flow description information, and the second job shares the input data or output data of the first job.
[0075] In a possible implementation, when scheduling resources for multiple jobs in a job stream, the scheduler 300 may first obtain available resources of some or all computing nodes in the computing cluster 100, so as to perceive the available resource situation on the computing node, so as to subsequently schedule resources for multiple jobs in the job stream according to the available resources on the acquired computing node. Available resources refer to resources on the computing node that are in an idle state and can be scheduled to execute jobs, such as idle processor cores, unallocated memory, cache, etc.
[0076] Then, the scheduler 300 schedules the first job and the second job whose input data or output data have relationship to the computing node 101 with sufficient available resources according to the data flow description information, the available resources of the computing node 101 and the resource requirements of each job in the multiple jobs.
[0077] Generally, a job flow may include more than 2 (including 2) jobs. To facilitate understanding and explanation of the resource scheduling of the job by the scheduler 300, in this embodiment, the example of the scheduler 300 scheduling resources for two jobs (i.e., the first job and the second job) in the job flow is used for exemplary explanation. The scheduled resources include computing resources, storage resources, and network resources. Among them, computing resources specifically refer to scheduling the job to the corresponding computing node, and allocating the processor / processor core on the computing node to the job, so as to use the processor / processor core to execute the job. Storage resources specifically refer to allocating the memory / local cache area on the computing node, or the cache area in the storage pool 120, or the persistent storage area in the storage cluster 200 to the job. Network resources refer to scheduling network-level resources such as bandwidth and communication ports in the network to the job. In this embodiment, the implementation process of the scheduler 300 scheduling computing resources and storage resources on the computing node 101 for the first job and the second job is mainly introduced.
[0078] For example, Figure 4 As shown, the scheduler 300 may be configured with a job scheduling plug-in 302 and a scheduling component 303, so as to schedule corresponding computing resources for each job in the job flow using the job scheduling plug-in 302 and the scheduling component 303. The following is an explanation of the implementation process of the scheduler 300 scheduling computing resources and storage resources for jobs.
[0079] In specific implementation, the middleware 301 can provide the job flow submitted by the user, the available resources on at least one computing node, and the data flow description information to the scheduling component 303, so that the scheduling component 303 can prioritize scheduling multiple jobs with dependency or sharing relationships between input data / output data to the same computing node with sufficient available resources based on the resource requirements of each job in the job flow, the available resources on each computing node, and the data flow description information.
[0080] Taking the scheduling of computing resources for the first job and the second job in the job flow as an example, assuming that the first job requires the use of 3 processor cores to execute the job, the second job requires 2 processor cores to execute the job, and the output data of the first job is used as the input data when executing the second job, then, when the processor cores 1 to 5 (and other processor cores) are available on the computing node 101, the scheduling component 303 can schedule the processor cores 1 to 3 on the computing node 101 to the first job, and schedule the processor cores 4 and 5 on the computing node 101 to the second job, so that the scheduled processor cores can be used to execute the job later. In addition, the scheduler 300 also schedules the local cache area 1011 on the computing node 101 to the first job and the second job. In this way, the temporary data generated by the subsequent computing node 101 after using the processor cores 1 to 3 to execute the first job can be stored in the local cache area 1011. When the processor cores 4 and 5 execute the second job, the input data can be directly accessed from the local cache area 1011 to execute the second job according to the input data. Referring to the above-mentioned similar method, the scheduling component 303 can, according to the size of available resources on the computing node, give priority to scheduling multiple jobs in the job flow having a dependency or sharing relationship between input data / output data to the same computing node (the available resources on the computing node are sufficient to support the execution of multiple jobs scheduled to the computing node), and allocate the local cache area on the computing node to the multiple jobs.
[0081] In actual application, since the resources on a single computing node are limited, the scheduling component 303 can schedule multiple jobs in the job flow to multiple computing nodes, such as Figure 7As shown, the scheduling component 303 can schedule job 1 and job 2 in the job flow to computing node 101, and schedule job 3 and job 4 to computing node 102, etc. Exemplarily, the scheduling component 303 can divide the multiple jobs in the job flow into multiple groups according to the data flow description information, each group includes at least one job, and schedule different groups to different computing nodes. Among them, the scheduling component 303 can preferentially divide multiple jobs with a dependency relationship between input data / output data into one group, and schedule multiple jobs in the group to the same computing node. For example, a job flow may include six jobs, namely, job 1 to job 6, wherein the input data / output data of jobs 1 to 3 have a dependency relationship, the input data / output data of jobs 4 to 6 have a dependency relationship, and the input data / output data of jobs 1 and 4 have a sharing relationship. Then, the scheduling component 303 may divide jobs 1 to 3 into a group, and schedule the three jobs in the group to the computing node 101, and schedule the local cache area 1011 to jobs 1 to 3; and the scheduling component 303 may divide jobs 4 to 6 into a group, and schedule the three jobs in the group to the computing node 102, and schedule the local cache area 1021 to jobs 4 to 6, and so on.
[0082] In this way, the second job located on the same computing node as the first job can directly obtain the input data it needs from the local cache area 1011 in the computing node 101, without having to access the cache in the cache pool 120 to obtain data. This can not only effectively improve the efficiency of obtaining input data when executing the second job, but also avoid unnecessary migration of the data between the computing node 101, the cache pool 120 and the storage cluster 200, thereby effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0083] In this embodiment, in addition to scheduling computing resources and local cache areas for multiple jobs in the job flow based on the above implementation process, the scheduler 300 can also schedule storage resources in the cache pool 120 for the job flow, specifically, it can schedule the cache area in the cache pool 120 for the job in the job flow, so that the computing node can obtain input data from the cache area when executing the job. Therefore, this embodiment can also include the following steps:
[0084] S303: The scheduler 300 schedules the cache areas in the cache pool 120 for the multiple jobs in the job flow.
[0085] In a first implementation example, the scheduler 300 may schedule a cache area in the cache pool 120 for each job, and the cache area allocated to each job can only be accessed by the job.
[0086] In a second implementation example, the scheduler 300 can schedule cache areas in the cache pool 120 for multiple jobs in the job flow based on the data flow description information, and a cache area scheduled by the scheduler 300 can support access by multiple jobs with a dependency or sharing relationship between input data / output data.
[0087] In specific implementation, the middleware 301 may call an interface provided by the job scheduling plug-in 302 so as to provide the data flow description information submitted by the user to the job scheduling plug-in 302 through the interface.
[0088] The job scheduling plug-in 302 generates a cross-job data flow graph (XJDFG) corresponding to multiple jobs based on the data flow description information. The XJDFG can specifically be a directed acyclic graph (DAG), including job nodes, data nodes, and connecting lines. Among them, the job node is used to indicate the job in the job flow, and each job node can indicate a job. The data node is used to indicate the input data or output data of the job in the job flow, and each job node is used to indicate the input data or output data of a job. The connecting line is used to indicate the relationship between the job node and the data node, such as indicating that the data represented by the data node is used as the input data or output data of the job represented by the job node.
[0089] For example, suppose the job flow includes Figure 5A The four jobs shown are jobs 1 to 4, respectively. Then, the job scheduling plug-in 302 generates the following according to the data flow description information corresponding to the job flow: Figure 6 XJDFG shown in Figure 1. XJDFG includes 4 job nodes and 7 data nodes, each of which corresponds to Figure 5A The data described in the . The job nodes and data nodes are connected by directed lines.
[0090] In this way, the XJDFG generated by the job scheduling plug-in 302 can not only indicate the data that each job depends on and the information of the generated data, but also indicate the dependency relationship between jobs. For example, Figure 6 In the XJDFG shown, job 2 depends on job 1, and job 4 depends on job 3.
[0091] After generating XJDFG, the job scheduling plug-in 302 can generate a data migration request based on the XJDFG and the amount of available cache resources remaining in the cache pool 120, and the data migration request carries the instruction information for creating a cache area. For example, the data migration request can carry the identifiers of multiple jobs in the job flow and the information of the data to be stored in the cache pool 120; or, the data migration request can carry XJDFG. In actual application, the data migration request can also include other information such as the execution time of multiple jobs, which is not limited to this.
[0092] Then, the job scheduling plug-in 302 sends the data migration request to the middleware 301 , and the middleware 301 sends the data migration request to the cache pool 120 .
[0093] The cache pool 120 (specifically, it can be a storage resource manager in the cache pool 120) responds to the data migration request, and schedules corresponding cache areas for the multiple jobs from the remaining available cache areas according to the dependency and sharing relationship between the input data / output data of the multiple jobs in the job flow. The scheduled cache areas can be, for example, BB (burst buffer) instances, or can be other types of caches.
[0094] Take the example of a job flow including the first job, the second job and the third job, wherein the input data / output data of the first job and the second job have a dependency relationship, and the input data / output data of the second job and the third job have a dependency relationship or a sharing relationship. Due to the limitation of available resources on the computing node 101, the scheduler 300 can schedule the first job and the second job to the computing node 101, and schedule the third job to the computing node 102. Then, the cache pool 120 can schedule a cache area for the second job and the third job, which is referred to as the target cache area in this embodiment for the convenience of distinction. The scheduled target cache area is used to store the target data shared between the second job and the third job. Among them, when the input data / output data of the second job and the third job have a dependency relationship, the target data is specifically temporary data generated by executing the second job, and the temporary data is also the input data required by the third job when it is executed. When there is a sharing relationship between the input data / output data of the second job and the third job, the target data is specifically the input data required for executing the second job and the third job; usually, the target data will be persistently stored in the storage cluster 200, so before executing the second job and the third job, the target data can be pre-loaded from the storage cluster 200 to the target cache area.
[0095] In addition, the scheduler 300 may also schedule a cache area for the first job to store input data required when executing the first job, so as to increase the efficiency of obtaining the input data when executing the first job.
[0096] For example, suppose that the dependency and sharing relationships between the input data / output data of multiple jobs included in the job flow are as follows: Figure 6 As shown, the scheduler 300 can schedule jobs 1 and 2 to computing node 101, and schedule jobs 3 and 4 to computing node 102, as shown in FIG. Figure 7 As shown. In this way, the temporary data "tmpData1" output when executing job 1 can be written into the local cache area 1011 on the computing node 101, so that job 2 can obtain the temporary data "tmpData1" from the local cache area 1011, thereby speeding up the rate at which job 2 obtains the temporary data "tmpData1" (job 3 is similar to job 4). At the same time, the scheduler 300 schedules cache area 2 for job 2, and the cache area 2 is used to store the input data required for executing job 2 and executing job 3, so that when executing job 2 and job 3, the input data can be read from cache area 2.
[0097] Moreover, the cache pool 120 does not need to schedule a cache area for the third job, which can effectively reduce the occupation of cache resources in the cache pool 120 by executing the job flow. Figure 7 As shown, the cache pool 120 may not need to schedule cache area 3 for job 3, thereby reducing resource usage in the cache pool 120.
[0098] Accordingly, before each job in the job flow is executed, the cache pool 120 can read the input data required for executing each job from the storage cluster 200 in advance to one or more cache areas allocated to the job flow, so as to improve the overall execution efficiency of the job flow by speeding up the acquisition of input data for each job.
[0099] In the third implementation example, the scheduler 300 first schedules computing resources for the job flow, and schedules the target cache area in the cache pool 120 according to the computing resources.
[0100] Typically, there may be differences in the latency for processor cores in different computing nodes to access different cache areas in cache pool 120. For example, when the network distance between cache area 1 in cache pool 120 and processor core A in computing node 101 is closer, and the network distance between cache area 1 and processor core B in computing node 102 is farther, the latency for processor core A to access cache area 1 will generally be less than the latency for processor core B to access cache area 1.
[0101] Based on this, in a further possible implementation, the scheduler 300 can first schedule the processor cores (i.e., schedule computing resources) and local cache areas on the corresponding computing nodes for the job flow, and then allocate the cache areas with smaller total access delays to the jobs in the job flow based on the latency of the scheduled processor cores accessing the cache areas in the cache pool 120.
[0102] In a specific implementation, the job flow includes not only the first job and the second job but also the third job, and there is a dependency or sharing relationship between the input data / output data of the second job and the third job. Due to the limitation of available resources on the computing node 101, the scheduler 300 can first use the scheduling component 303 to schedule the processor core 1 in the computing node 101 for the first job, the processor core 2 in the computing node 101 for the second job, and the processor core 5 in the computing node 102 for the third job. Then, the scheduler 300 uses the middleware 301 to instruct the cache pool 120 to schedule a cache area with a smaller access delay for the second job and the third job. For the second job and the third job with a dependency or sharing relationship between the input data / output data, since the second job and the third job share the same input data, or the temporary data output by the second job is used as the input data of the third job, the cache pool 120 can first determine multiple candidate cache areas, and for each candidate cache area, estimate the total delay of the processor core 2 in the computing node 101 and the processor core 5 in the computing node 102 to access the cache area. Next, the cache pool 120 can determine a target cache area from the multiple candidate cache areas according to the sum of the latencies corresponding to the multiple candidate cache areas, and schedule the target cache area to the second job and the third job. In this way, the latency of the processor core 2 in the computing node 101 and the processor core 5 in the computing node 102 reading / writing the data shared between the second job and the third job in the target cache area when executing the job can be reduced.
[0103] It is worth noting that the above various implementation examples mainly introduce that the scheduler 300 schedules computing resources, computing nodes and storage resources in the cache pool 120 for the job flow. In actual application, the resources scheduled by the scheduler 300 for the job flow may also include network resources and storage resources in the storage cluster 200. Among them, the specific implementation method of scheduling network resources and the storage resources has related applications in the existing technology and will not be repeated here. In addition, in actual application, the implementation process of the scheduler 300 scheduling computing resources and storage resources for the job flow according to the data flow description information is not limited to the above implementation examples. For example, the scheduler 300 can schedule resources for the job flow by combining the above multiple implementation examples.
[0104] Furthermore, after completing resource scheduling, the data processing system 10 can execute each job in the job flow based on the scheduled computing resources and storage resources.
[0105] In specific implementation, the job scheduling plug-in 302 can determine the dependency relationship between multiple jobs in the job flow based on the data flow description information provided by the middleware 301, such as the execution of job 2 depends on the execution and completion of job 1, and feed back the dependency relationship between the multiple jobs to the middleware 301. The scheduler 300 can also include a dependency management component 304, such as Figure 4 As shown. In addition, the middleware 301 can provide the dependency relationship to the dependency management component 304, so that the dependency management component 304 can control the processor core on the computing node to execute each job in the job flow in sequence according to the dependency relationship. In the process of executing each job, different jobs with a sharing relationship between input data / output data can obtain input data from the same cache area (the input data is pre-loaded from the storage cluster 200 to the cache area), or the second job with a sharing relationship between input data / output data can obtain input data from the local cache area 1011 (the input data is loaded from the cache area to the local cache area 1011 by the first job). In this way, the same copy of data can be avoided from being migrated from the storage cluster 200 to the cache pool 120 and the computing node multiple times. In addition, between job 1 and job 2 with a dependency relationship between input data / output data, the data output by job 1 can be shared with job 2 through the cache area in the cache pool 120 or the local cache area in the computing node, so that the data output by job 1 can be avoided from being frequently migrated between the cache pool 120, the storage cluster 200 and the computing node.
[0106] For example, the job flow includes a first job, a second job, and a third job, and there is a dependency relationship between the input data / output data of the first job and the second job, and there is a sharing relationship between the input data / output data of the first job and the third job, then the dependency management component 304 can control the processor cores on different computing nodes to execute the first job and the third job in parallel. In this process, multiple processor cores can read the data used as inputs of the first job and the third job from the same cache area. After the first job is executed, the temporary data output by the first job can be cached in the local cache area in the computing node. Then, the dependency management component 304 can control other processor cores on the computing node where the first job is located to execute the second job, etc. At this time, the processor core can read the temporary data output by the first job from the local cache area and use it as the input data of the second job to execute the second job. In this way, the execution of the job flow can be realized.
[0107] In a further possible implementation, after multiple jobs in the job flow are executed, the cache pool 120 may send the data in the cache area to the storage cluster 200 for persistent storage, such as Figure 7 The data "outDATA2" (and data "tmpData3") in the cache area 4 shown in the figure are sent to the storage cluster 200 for persistent storage, etc. Among them, the area in the storage cluster 200 for storing the data can be scheduled by the scheduler 300. After completing the persistent storage of the data, the scheduler 300 can instruct the cache pool 120 to release the cache area scheduled for the multiple jobs to reclaim the storage resources corresponding to the cache area. At this time, the life cycle of the cache area scheduled for the multiple jobs is bound to the job flow.
[0108] The cache pool 120 can release all cache areas scheduled for the job flow after the entire job flow is executed. Alternatively, in the process of executing the job flow, when part of the job is executed and the data in the cache area scheduled for the part of the job is not used as input data for other jobs, the cache pool 120 can write the data in the cache area to the storage cluster 200 and release the cache area. In this way, the cache pool 120 can recycle resources by reclaiming the cache area in advance without affecting the execution of the job flow, so as to improve the resource utilization of the cache area, such as scheduling the cache area to jobs in other job flows after it is reclaimed.
[0109] In addition, when each job in the job flow defines input data and output data based on a label, the scheduler 300 can also support a third-party component to query the label or the input data and output data defined by the label. For example, before submitting a job, the user can query the labels defined in the scheduler 300 through the third-party component.
[0110] In a specific implementation, the scheduler 300 may further include a tag management component 305, and the scheduler 300 may further be connected to a third-party component 306, such as Figure 4As shown. Among them, the tag management component 305 is used to record and manage the tags defined by the user when submitting the job. For example, the user can customize the tags in the data flow description information submitted for job A. Then, the tag management component 305 can verify whether the user-defined tags sent by the middleware 301 are legal based on all the recorded tags. Specifically, it can be verified whether there are duplications between the user-defined tags and all the tags recorded by the tag management component 305, and whether the naming format of the user-defined tags complies with the specifications. When the user-defined tags are legal, the tag management component 305 can record the tags and include them in the management. When the user-defined tags are illegal, the tag management component 305 can notify the user of the tag definition abnormality through the middleware 301. In addition, the tag management component 305 can also create new tags, such as automatically creating new tags to define the input and output data based on the input data and output data of the completed job.
[0111] When it is necessary to query the tags recorded in the scheduler 300, the user can use the third-party component 306 to generate a tag query request and send the tag query request to the tag management component 305. The tag management component 305 responds to the query request and feeds back all the recorded tags to the third-party component, or, based on the identifier of the target job carried in the query request, feeds back the tags used to define the input and output data of the target job to the third-party component.
[0112] After querying the label, the user can also use the third-party component 306 to query the dependency between different jobs based on the label. For example, the user can use the third-party component 306 to generate a data dependency query request, which may include labels of multiple jobs, and the multiple jobs may be jobs included in one or more job flows. Then, the third-party component 306 can send the data dependency query request to the job scheduling plug-in 302. The job scheduling plug-in 302 queries the XJDFG corresponding to the multiple jobs based on the labels of the multiple jobs included in the data dependency query request, thereby determining the dependency between the input data / output data of the multiple jobs based on the XJDFG, and feeding back the queried dependency to the third-party component 306.
[0113] In this embodiment, the scheduler 300 schedules at least two jobs to the same computing node based on the dependency / sharing relationship between the input data / output data of multiple jobs in the job flow, so that the at least two jobs can use the local cache area on the computing node to store the data shared between the jobs, which can effectively avoid unnecessary migration of the data shared between the jobs between the storage cluster 200, the cache pool 120 and the computing nodes, that is, it can avoid the same data from being migrated multiple times between the storage cluster, the cache pool 120, and the computing nodes, and optimize the scheduling of computing resources and storage resources at the job flow level, thereby effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0114] Above Figure 3 In the embodiment shown, the user submits data flow description information as an example for explanation. In other possible embodiments, the data processing system 10 may analyze and obtain the dependency relationship between the input data / output data of multiple jobs in the job flow. This will be described in detail below.
[0115] See also Figure 8 , shows a schematic diagram of another resource scheduling method provided by an embodiment of the present application. Figure 8 As shown, the method may specifically include:
[0116] S801: The scheduler 300 obtains a job flow submitted by a user, wherein the job flow submitted by the user includes multiple jobs, and the multiple jobs include at least a first job and a second job.
[0117] The specific implementation of step S801 can be found in the above Figure 3 The description of the relevant aspects of the illustrated embodiment will not be repeated here.
[0118] In this embodiment, after obtaining the job flow submitted by the user, the scheduler 300 can analyze and determine the dependency relationship between the input data / output data of the multiple jobs by first running the multiple jobs in the job flow. The specific implementation process can be shown in the following steps.
[0119] S802 : The scheduler 300 obtains available resources of each computing node in the computing cluster 100 .
[0120] S803: The scheduler 300 schedules first resources for multiple jobs in the job flow according to available resources of each computing node.
[0121] In specific implementation, the scheduler 300 can schedule the resources (including computing resources, storage resources, network resources, etc.) in the computing cluster 100 to the job flow according to the resource requirements of each job in the job flow and the currently available resources of each computing node in the computing cluster 100. Taking the scheduling of computing resources as an example, the scheduler 300 can schedule some processor cores from the currently available multiple processor cores in the computing cluster 100 to the job flow, so as to use the multiple processor cores to execute each job in the job flow.
[0122] In addition, the scheduler 300 will also schedule the storage resources and network resources required for executing the multiple jobs respectively.
[0123] For the convenience of distinction and description, the scheduler 300 preliminarily schedules computing resources, storage resources, and network resources for the job flow, which are collectively referred to as first resources in this embodiment.
[0124] S804: The scheduler 300 collects running information of multiple jobs in the job flow. The multiple jobs are run based on the scheduled first resource. The running information includes input data and output data of each of the multiple jobs.
[0125] In this embodiment, after the scheduler 300 schedules the first resource for the job flow, the data processing system 10 can use the first resource to execute each job in the job flow, so that the scheduler 300 can collect input data and output data of each job during the operation of the job flow.
[0126] Exemplarily, the collected input data may be, for example, an identifier of the input data of each job, and may also include information such as the data type, data length, and data storage location of the input data.
[0127] The collected output data may be, for example, an identifier of the output data of each job, and may also include information such as the data type, data operation, opening times, and temporary type of the output data.
[0128] S805: The scheduler 300 analyzes the collected operation information to obtain data flow description information, where the data flow description information is used to indicate the dependency or sharing relationship between the input data / output data of multiple jobs in the job flow.
[0129] In specific implementation, the scheduler 300 can determine job A and job B that use the same data as input data by analyzing the input data and output data of each job during execution. Then, the scheduler 300 can determine that there is a sharing relationship between the input data / output data of job A and job B.
[0130] Furthermore, by analyzing the input data and output data of each job during execution, the scheduler 300 can determine that the data generated and output by executing job C will be used as input data when executing job D. The scheduler 300 can then determine that there is a dependency relationship between the input data / output data of job C and job D.
[0131] In this way, the scheduler 300 can obtain the data flow description information corresponding to the job flow according to the sharing relationship and dependency relationship between the input data / output data of different jobs.
[0132] S806: The scheduler 300 re-schedules a second resource for multiple jobs in the job flow, where the second resource is different from the first resource; or, the scheduler 300 schedules a target resource for multiple jobs in the job flow, where the target resource is used to replace part of the first resources that have been scheduled for multiple jobs.
[0133] In this embodiment, the scheduler 300 can re-schedule resources for multiple jobs in the job flow according to the job flow submitted by the user and the data flow description information analyzed by the scheduler 300, that is, the second resource in step S805. The specific implementation of the scheduler 300 scheduling the second resource for multiple jobs according to the data flow description information can be found in the description of the relevant parts of step S302 in the above embodiment, which will not be repeated here.
[0134] Alternatively, since the scheduler 300 has already scheduled the first resource for the job flow, the scheduler 300 may schedule resources for multiple jobs in the job flow based on the job flow submitted by the user and the data flow description information analyzed by the scheduler 300, which may overlap with the scheduled first resource. Based on this, the scheduler 300 may only need to replace some of the scheduled first resources to optimize the resource scheduling of the job flow.
[0135] In specific implementation, the scheduler 300 can first determine the third resource that needs to be scheduled (resource scheduling has not yet been performed) based on the job flow and data flow description information, and compare the third resource that needs to be scheduled with the scheduled first resource to determine the target resource that is not included in the first resource, and determine the resource A that is included in the first resource but not included in the third resource, so that the scheduler 300 can schedule the target resource to the job flow, and instruct the job flow to use the target resource to replace part of the resources in the first resource (that is, replace resource A in the first resource).
[0136] In actual application scenarios, the job flow submitted by the user may be executed multiple times. For example, the user expects the data processing system 10 to periodically execute the job flow to regularly obtain the job execution results he needs. Therefore, after running the job flow once in the data processing system 10, the scheduler 300 can analyze the dependency and sharing relationship between the input data / output data of multiple jobs in the job flow, thereby optimizing the resource scheduling of the job flow according to the dependency and sharing relationship, thereby effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0137] Combined with the above Figures 1 to 8 , describes in detail the data processing system and resource scheduling method provided by the present application, and will be combined with Figures 9 and 10 , describing the scheduler provided according to the present application.
[0138] Fig. 9 A schematic diagram of the structure of a scheduler provided for this application. Fig. 9 The scheduler 900 shown is applied to a data processing system, which also includes a computing cluster and a storage cluster. The computing cluster includes multiple computing nodes, and the computing cluster and the storage cluster are connected via a network.
[0139] like Fig. 9 As shown, the scheduler 900 includes:
[0140] The acquisition module 901 is used to acquire data flow description information and a job flow submitted by a user, wherein the job flow includes multiple jobs, and the data flow description information is used to indicate the relationship between input data or output data of each job in the multiple jobs;
[0141] The scheduling module 902 is used to schedule a second job related to the input data or output data of the first job and the first job to the first computing node according to data flow description information when the input data or output data of the first job among multiple jobs is stored in the first computing node among multiple computing nodes, and the second job shares the input data or output data of the first job.
[0142] In a possible implementation, the relationship between the input data or output data of each of the multiple jobs includes:
[0143] The dependency between the input data or output data of each job in multiple jobs;
[0144] Or, the sharing relationship between the input data or output data of each job in multiple jobs.
[0145] In a possible implementation, when the scheduling module 902 schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, it is specifically configured to:
[0146] When the available resources of the first computing node meet the resource requirements of the first job and the second job, the first job and the second job are scheduled to the first computing node according to the data flow description information.
[0147] In a possible implementation manner, when acquiring the data flow description information, the acquisition module 901 is specifically used to:
[0148] Scheduling a first resource for a plurality of jobs according to the job flow;
[0149] Collecting operation information generated when multiple jobs are run based on the first resource, the operation information including input data and output data of each job in the multiple jobs;
[0150] Analyze the running information and obtain the data flow description information.
[0151] In a possible implementation, when the scheduling module 902 schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, it is specifically configured to:
[0152] Generate inter-job data flows corresponding to multiple jobs based on data flow description information Figure X JDFG and XJDFG include job nodes, data nodes, and connection lines. The job nodes are used to indicate jobs in a job flow, the data nodes are used to indicate input data or output data of jobs in a job flow, and the connection lines are used to indicate the relationship between the job nodes and the data nodes.
[0153] According to XJDFG, a first job and a second job related to input data or output data of the first job are scheduled to a first computing node.
[0154] In a possible implementation manner, when acquiring the data flow description information, the acquisition module 901 is specifically used to:
[0155] A configuration file written in a domain specific language (DSL) submitted by a user is obtained. The configuration file includes multiple configuration items. The multiple configuration items are used to configure input data and output data of multiple jobs.
[0156] In a possible implementation, the computing cluster further includes a cache pool, and the multiple jobs further include a third job. Then, the scheduling module 902 is further configured to:
[0157] According to the data flow description information, a third job related to the input data or output data of the second job is scheduled to the second computing node among the multiple computing nodes, and the target cache area in the cache pool is scheduled to the second job and the third job; wherein the third job shares the input data or output data of the second job through the target cache area.
[0158] In one possible implementation, the cache pool includes multiple available cache areas, the multiple cache areas include a target cache area, and the sum of the delays of the first computing node and the second computing node accessing the target cache area is less than the sum of the delays of the first computing node and the second computing node accessing other cache areas in the multiple cache areas.
[0159] Fig. 9 The scheduler 900 shown corresponds to Figures 1 to 8 The functions and technical effects of the scheduler 300 and the scheduler 900 in the embodiment shown can be found in Figures 1 to 8 The description of the relevant parts in the illustrated embodiment will not be repeated here.
[0160] Fig.10 A schematic diagram of the structure of a scheduler provided in this application. Fig.10 As shown, the scheduler 1000 includes a processor 1001, a memory 1002, a communication interface 1003 and a bus 1004. The processor 1001, the memory 1002 and the communication interface 1003 communicate through the bus 1004. The bus 1004 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.10 The communication interface 1003 is used for external communication, for example, receiving jobs (and target data) submitted by users through terminals or clients.
[0161] It should be understood that in the embodiment of the present application, the processor 1001 may be a CPU, and the processor 1001 may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0162] The memory 1002 may include a read-only memory and a random access memory, and provide instructions and data to the processor 1001. The memory 1002 may also include a non-volatile random access memory. For example, the memory 1002 may also store information about the device type.
[0163] The memory 1002 may be a volatile memory or a nonvolatile memory, or may include both volatile and nonvolatile memories. Among them, the nonvolatile memory may be a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0164] The memory 1002 stores executable codes, and the processor 1001 executes the executable codes to execute the method executed by the aforementioned scheduler 300 .
[0165] Specifically, in implementing Figures 1 to 8 In the case of the embodiment shown, and Figures 1 to 8 In the case where the scheduler 300 described in the illustrated embodiment is implemented by software, the execution Figures 1 to 8 The software or program code required for the functions of the scheduler 300 in the illustrated embodiment is stored in the memory 1002 , and the processor 1001 is used to execute the instructions in the memory 1002 to implement the method performed by the scheduler 300 .
[0166] It should be understood that the scheduler 1000 according to the embodiment of the present application may correspond to the scheduler 900 in the embodiment of the present application, and may correspond to the scheduler 900 executing the scheduler 900 according to the embodiment of the present application. Figures 1 to 8The scheduler 300 in the method shown in the figure and the above and other operations and / or functions implemented by the scheduler 1000 are respectively to implement Figures 1 to 8 For the sake of brevity, the corresponding processes in the illustrated method embodiment will not be described in detail here.
[0167] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a scheduler, the scheduler executes the method executed by the scheduler 300 in the above embodiment.
[0168] In addition, the embodiment of the present application further provides a computer program product, when the computer program product is executed by a scheduler, the one or more schedulers execute any of the aforementioned job scheduling methods. The computer program product may be a software installation package, and when any of the aforementioned job scheduling methods is required, the computer program product may be downloaded and executed on a computer.
[0169] It should also be noted that the above-described scheduler embodiment is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the accompanying drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0170] The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances. This is just a way of distinguishing objects with the same attributes when describing the embodiments of this application.
[0171] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0172] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).
[0173] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.
Claims
1. A resource scheduling method, It is characterized in that The method is applied to a scheduler in a data processing system, wherein the data processing system further comprises a computing cluster and a storage cluster, wherein the computing cluster comprises a plurality of computing nodes, and the computing cluster and the storage cluster are connected via a network; The method comprises: The scheduler obtains data flow description information and a job flow submitted by a user, wherein the job flow includes a plurality of jobs, and the data flow description information is used to indicate a relationship between input data or output data of each of the plurality of jobs; When the input data or output data of a first job among the multiple jobs is stored in a first computing node among the multiple computing nodes, the scheduler schedules a second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, and the second job shares the input data or output data of the first job.
2. The method according to claim 1, It is characterized in that The relationship between the input data or output data of each of the multiple jobs includes: A dependency relationship between input data or output data of each of the multiple jobs; Alternatively, the sharing relationship between the input data or output data of each job in the multiple jobs.
3. The method according to claim 1 or 2, It is characterized in that The scheduler schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, including: When available resources of the first computing node meet resource requirements of the first job and the second job, the scheduler schedules the first job and the second job to the first computing node according to the data flow description information.
4. The method according to any one of claims 1 to 3, It is characterized in that The scheduler obtains data flow description information, including: The scheduler schedules a first resource for the plurality of jobs according to the job flow; The scheduler collects operation information generated when the multiple jobs are run based on the first resource, wherein the operation information includes input data and output data of each of the multiple jobs; The scheduler analyzes the operation information to obtain the data flow description information.
5. The method according to any one of claims 1 to 4, It is characterized in that The scheduler schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, including: The scheduler generates an inter-job data flow graph XJDFG corresponding to the multiple jobs according to the data flow description information, wherein the XJDFG includes job nodes, data nodes, and connecting lines, wherein the job nodes are used to indicate jobs in the job flow, the data nodes are used to indicate input data or output data of jobs in the job flow, and the connecting lines are used to indicate the relationship between the job nodes and the data nodes; The scheduler schedules the first job and the second job related to input data or output data of the first job to the first computing node according to the XJDFG.
6. The method according to any one of claims 1 to 4, It is characterized in that The scheduler obtains data flow description information, including: The scheduler obtains a configuration file submitted by a user and written based on a domain specific language (DSL). The configuration file includes a plurality of configuration items, and the plurality of configuration items are used to configure input data and output data of the plurality of jobs.
7. The method according to any one of claims 1 to 6, It is characterized in that The computing cluster further includes a cache pool, the multiple jobs further include a third job, and the method further includes: The scheduler schedules the third job related to the input data or output data of the second job to a second computing node among the plurality of computing nodes according to the data flow description information, and schedules the target cache area in the cache pool to the second job and the third job; The third job shares the input data or output data of the second job through the target cache area.
8. The method according to claim 7, It is characterized in that The cache pool includes multiple available cache areas, the multiple cache areas include the target cache area, and the sum of the delays of the first computing node and the second computing node accessing the target cache area is less than the sum of the delays of the first computing node and the second computing node accessing other cache areas in the multiple cache areas.
9. A scheduler, It is characterized in that The scheduler is applied to a data processing system, the data processing system further comprising a computing cluster and a storage cluster, the computing cluster comprising a plurality of computing nodes, the computing cluster and the storage cluster being connected via a network; The scheduler comprises: An acquisition module, used to acquire data flow description information and a job flow submitted by a user, wherein the job flow includes a plurality of jobs, and the data flow description information is used to indicate the relationship between input data or output data of each job in the plurality of jobs; A scheduling module is used to schedule a second job related to the input data or output data of the first job among the multiple jobs and the first job to the first computing node according to the data flow description information when the input data or output data of the first job among the multiple jobs is stored in the first computing node among the multiple computing nodes, and the second job shares the input data or output data of the first job.
10. The scheduler according to claim 9, It is characterized in that The relationship between the input data or output data of each of the multiple jobs includes: A dependency relationship between input data or output data of each of the multiple jobs; Alternatively, the sharing relationship between the input data or output data of each job in the multiple jobs.
11. The scheduler according to claim 9 or 10, It is characterized in that When the scheduling module schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, the scheduling module is specifically used to: When the available resources of the first computing node meet the resource requirements of the first job and the second job, the first job and the second job are scheduled to the first computing node according to the data flow description information.
12. The scheduler according to any one of claims 9 to 11, It is characterized in that When acquiring the data stream description information, the acquisition module is specifically used to: Scheduling a first resource for the plurality of jobs according to the job flow; Collecting operation information generated when the multiple jobs are run based on the first resource, the operation information including input data and output data of each job in the multiple jobs; The operation information is analyzed to obtain the data flow description information.
13. A scheduler, It is characterized in that The scheduler includes a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method according to any one of claims 1 to 8.
14. A data processing system, It is characterized in that The data processing system includes a computing cluster, a storage cluster, and a scheduler, and the scheduler is used to execute the method according to any one of claims 1 to 8.
15. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores instructions, which, when executed on a scheduler, enable the scheduler to execute the method according to any one of claims 1 to 8.
Citation Information
Cited By
Job scheduling optimization method based on Burst Buffer resource reservation
CN119917298A
Task assignment method and apparatus, and device and computer program product
WO2026144250A1
