Resource scheduling method, scheduler, data processing system, and related device
By scheduling jobs with dependencies or shared relationships to the same computing node in the scheduler of the data processing system, low IO performance and waste of resources in the data processing system are solved by scheduling jobs with dependencies or shared relationships to the same computing node and sharing input and output data. The problem of the system improves the overall efficiency and resource utilization rate.
Patent Information
- Application Number
- PCT/CN2023/143692
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-07-21
- Filing Date
- 2023-12-30
- Publication Date
- 2024-11-28
AI Technical Summary
Data processing systems are prone to low IO performance and waste of resources when processing job flows. This is mainly due to the burst cache created by each job that causes frequent data migration, which increases data migration between computing nodes, cache pools and storage clusters. frequency.
By obtaining the data flow description information of the job flow in the scheduler, the scheduler schedules jobs with dependencies or shared relationships to the same computing node, sharing input and output data, avoiding unnecessary data migration, and optimizing the scheduling of computing resources.
It improves the IO performance of the data processing system, reduces resource waste, acquires data through local cache areas, improves job execution efficiency, and reduces the number of data migrations between computing nodes, cache pools and storage clusters.
Smart Images

Figure CN2023143692_28112024_PF_FP_ABST
Abstract
Description
Resource scheduling method, scheduler, data processing system and related equipment
[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on May 25, 2023, with application number 202310605586.3 and application name “A resource management method”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 21, 2023, with application number 202310912355.7 and invention name “A resource scheduling method, scheduler, data processing system and related equipment”, all contents of which are incorporated by reference into this application. Technical Field
[0002] The present application relates to the field of Internet technology, and in particular to a resource scheduling method, a scheduler, a data processing system, and related equipment. Background Art
[0003] As the demand for computing power in data processing systems continues to increase, the scale of computing nodes in these systems is also expanding. Currently, data processing systems are deployed with a scheduler that allocates computing resources for each job submitted by the user and configures the storage and network resources necessary to execute the job.
[0004] In order to meet the user's low-latency requirements for job processing, the scheduler will create at least one burst buffer (BB) for each job in the data processing system. The BB is used to load data from the file system (external filesystem) before the job is executed, so that data can be quickly obtained from the BB during the execution of the job. After the job is executed, the data in the BB is persistently stored in the file system, thereby reducing the latency of the data processing system in processing jobs by improving the efficiency of reading and writing data.
[0005] In real-world applications, users often submit job flows consisting of multiple jobs to data processing systems. When the data processing system executes each job in the job flow based on a dedicated backup (BB) created for each job, it can easily suffer from issues such as low input / output (IO) performance and significant resource waste.
[0006] Summary of the Invention
[0007] The present application provides a resource scheduling method for improving the IO performance of a data processing system and alleviating the problem of resource waste. In addition, the present application also provides a scheduler, a data processing system, a computer-readable storage medium, and a computer program product.
[0008] In a first aspect, the present application provides a resource scheduling method, which is applied to a scheduler in a data processing system, wherein the data processing system also includes a computing cluster and a storage cluster, wherein the computing cluster includes multiple computing nodes, and the computing cluster and the storage cluster are connected through a network; when performing resource scheduling, the scheduler obtains data flow description information and a job flow submitted by a user, wherein the job flow submitted by the user includes multiple jobs, and the data flow description information is used to indicate the relationship between the input data or output data of each job in the multiple jobs, such as indicating that the output data of job A is used as the input data of job B, etc.; then, when the input data or output data of the first job in the multiple jobs is stored in the first computing node among the multiple computing nodes, the scheduler schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, and the second job shares the input data or output data of the first job.
[0009] In this way, after the first job and the second job are scheduled to the same computing node, one of the jobs can directly obtain the input data it needs from the local cache area in the computing node, without having to access the cache area in the cache pool to obtain the data. For example, during the execution of the first job, the input data or output data of the first job can be stored in the local cache area, so that when the second job is executed, the input data or output data of the first job can be directly obtained from the local cache area as the input data of the second job. This can effectively improve the efficiency of obtaining input data when executing the second job, thereby improving the efficiency of the data processing system in executing the second job. Moreover, during the execution of the first job and the second job, the input data required by the second job can be continuously stored in the local cache area in the computing node, which can avoid unnecessary migration of the data between the computing node, the cache pool and the storage cluster, that is, it can avoid the same data from migrating multiple times between the computing node, the storage cluster and the cache pool, thereby optimizing the scheduling of computing resources at the job flow level, thereby effectively improving the IO performance of the data processing system and alleviating the problem of resource waste.
[0010] In one possible implementation, the relationship between the input data or output data of each job in a plurality of jobs may be a dependency relationship between the input data or output data of each job in a plurality of jobs. For example, the data output by executing job 1 is used as the input data required when executing job 2. Alternatively, the relationship between the input data or output data of each job in a plurality of jobs may be a sharing relationship between the input data or output data of each job in a plurality of jobs. For example, the input data required to execute job 1 is the same as the input data required to execute job 2, that is, multiple jobs can share the same input data. In this way, scheduling multiple jobs in a job stream based on the dependency relationship or sharing relationship between the input data or output data of different jobs can improve the overall execution efficiency of the job stream and avoid unnecessary migration of data between data processing systems.
[0011] In one possible implementation, when the scheduler schedules the first and second jobs to the first computing node, the scheduler may specifically schedule the first and second jobs to the first computing node based on the data flow description information when the available resources of the first computing node meet the resource requirements of the first and second jobs. In this way, by scheduling the job flow based on the available resources on the computing node and the resource requirements of each job, it is possible to avoid situations where the resource requirements of multiple jobs scheduled to the computing node exceed the available resources of the computing node, thereby preventing the subsequent impact of job execution due to insufficient available resources on the computing node.
[0012] In one possible implementation, when obtaining data flow description information, the scheduler may first schedule a first resource for multiple jobs based on the job flow, and then collect the runtime information generated by the multiple jobs when they execute based on the first resource. This runtime information includes the input data and output data of each of the multiple jobs. The scheduler then analyzes this runtime information to obtain the data flow description information. In this way, by executing a job flow once, the dependencies or shared relationships between the input data or output data of different jobs in the job flow can be analyzed, thereby automatically generating data flow description information. This data flow description information can then be used to automatically optimize resource scheduling during the next execution of the job flow, thereby improving the IO performance of the data processing system.
[0013] In one possible implementation, in the process of scheduling the first job and the second job to the first computing node, the scheduler can generate an inter-job data flow graph XJDFG corresponding to multiple jobs based on the data flow description information, and the XJDFG includes job nodes, data nodes, and connecting lines, wherein the job nodes are used to indicate jobs in the job flow, the data nodes are used to indicate the input data or output data of the jobs in the job flow, and the connecting lines are used to indicate the relationship between the job nodes and the data nodes, so that the scheduler schedules the first job and the second job related to the input data or output data of the first job to the first computing node according to the XJDFG. In this way, the scheduler can perceive the dependency or sharing relationship between the input data or output data of different jobs according to the XJDFG, and thereby schedule the first job and the second job in the job flow to the same computing node, thereby optimizing the scheduling of computing resources at the job flow level.
[0014] In one possible implementation, when the scheduler obtains data flow description information, it may specifically obtain a user-submitted configuration file written in a domain-specific language (DSL). This configuration file includes multiple configuration items used to configure the input and output data of multiple jobs. In this way, the scheduler can directly utilize the user-submitted data flow description information to optimize the scheduling of computing resources at the job flow level.
[0015] In one possible embodiment, the computing cluster further includes a cache pool, and the multiple jobs in the job flow further include a third job. The scheduler can also schedule the third job related to the input data or output data of the second job to the second computing node among the multiple computing nodes based on the data flow description information (e.g., the amount of available resources on the first computing node is insufficient to support the running of the third job, etc.), and schedule the target cache area in the cache pool to the second job and the third job; wherein the third job shares the input data or output data of the second job through the target cache area. In this way, the input data or output data of the second job can be shared with the third job through the target cache area in the storage pool, which enables the input data or output data of the second job to be continuously stored in the target cache area, avoiding the frequent migration of the input data or output data of the second job between the cache pool, the storage cluster, and the computing nodes.
[0016] In one possible implementation, the cache pool includes multiple available cache areas, including a target cache area. The total latency of the first and second computing nodes accessing the target cache area is less than the total latency of the first and second computing nodes accessing other cache areas in the multiple cache areas. This reduces the latency of the first and second computing nodes reading and writing data shared between the second and third jobs from the cache areas in the cache pool when executing jobs.
[0017] In a second aspect, the present application provides a scheduler, which is applied to a data processing system, which also includes a computing cluster and a storage cluster, the computing cluster including multiple computing nodes, and the computing cluster and the storage cluster are connected through a network; the scheduler includes: an acquisition module for acquiring data flow description information and a job flow submitted by a user, the job flow including multiple jobs, and the data flow description information is used to indicate the relationship between the input data or output data of each job in the multiple jobs; a scheduling module for scheduling a second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information when the input data or output data of the first job in the multiple jobs is stored in the first computing node among the multiple computing nodes, and the second job shares the input data or output data of the first job.
[0018] In one possible implementation, the relationship between the input data or output data of each job in a plurality of jobs includes: a dependency relationship between the input data or output data of each job in a plurality of jobs; or a sharing relationship between the input data or output data of each job in a plurality of jobs.
[0019] In one possible implementation, when the scheduling module schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, the scheduling module is specifically configured to:
[0020] When the available resources of the first computing node meet the resource requirements of the first job and the second job, the first job and the second job are scheduled to the first computing node according to the data flow description information.
[0021] In one possible implementation, when acquiring data flow description information, the acquisition module is specifically used to: schedule the first resource for multiple jobs based on the job flow; collect operation information generated when multiple jobs run based on the first resource, the operation information including input data and output data of each job in the multiple jobs; analyze the operation information to obtain data flow description information.
[0022] In one possible embodiment, when the scheduling module schedules the first job and the second job related to the input data or output data of the first job to the first computing node according to the data flow description information, it is specifically used to: generate an inter-job data flow graph XJDFG corresponding to multiple jobs according to the data flow description information, XJDFG includes job nodes, data nodes and connecting lines, the job nodes are used to indicate the jobs in the job flow, the data nodes are used to indicate the input data or output data of the jobs in the job flow, and the connecting lines are used to indicate the relationship between the job nodes and the data nodes; according to the XJDFG, the first job and the second job related to the input data or output data of the first job are scheduled to the first computing node.
[0023] In one possible implementation, when acquiring data flow description information, the acquisition module is specifically used to: acquire a configuration file submitted by a user and written in a domain-specific language DSL, where the configuration file includes multiple configuration items, and the multiple configuration items are used to configure input data and output data of multiple jobs.
[0024] In one possible embodiment, the computing cluster also includes a cache pool, and the multiple jobs also include a third job. Then, the scheduling module is further used to schedule the third job related to the input data or output data of the second job to the second computing node among the multiple computing nodes based on the data flow description information, and schedule the target cache area in the cache pool to the second job and the third job; wherein the third job shares the input data or output data of the second job through the target cache area.
[0025] In one possible embodiment, the cache pool includes multiple available cache areas, the multiple cache areas include a target cache area, and the sum of the delays of the first computing node and the second computing node accessing the target cache area is less than the sum of the delays of the first computing node and the second computing node accessing other cache areas in the multiple cache areas.
[0026] The scheduler provided in the second aspect corresponds to the resource scheduling method provided in the first aspect. Therefore, the technical effects of the scheduler in the second aspect and any possible implementation method of the second aspect can refer to the technical effects of the first aspect and the corresponding implementation method of the first aspect, and will not be repeated here.
[0027] In a third aspect, the present application provides a scheduler, comprising a processor and a memory; the memory is used to store instructions, and when the scheduler is running, the processor executes the instructions stored in the memory, so that the scheduler performs the method described in the first aspect or any possible implementation of the first aspect. It should be noted that the memory can be integrated into the processor or can be independent of the processor. The scheduler may also include a bus. The processor is connected to the memory via the bus. The memory may include a readable memory and a random access memory.
[0028] In a fourth aspect, the present application provides a data processing system, which includes a computing cluster, a storage cluster, and a scheduler, wherein the scheduler is used to execute the method described in the first aspect or any possible implementation of the first aspect.
[0029] In a fifth aspect, the present application provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium runs on a scheduler, it enables the scheduler to execute the method described in the first aspect or any implementation of the first aspect.
[0030] In a sixth aspect, the present application provides a computer program product comprising instructions, which, when executed on a scheduler, enables the scheduler to execute the method described in the first aspect or any one of the implementations of the first aspect.
[0031] Based on the implementation methods provided in the above aspects, this application can also be further combined to provide more implementation methods. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.
[0033] FIG1 is a schematic diagram of the structure of an exemplary data processing system provided in an embodiment of the present application;
[0034] FIG2 is a schematic diagram of multiple migrations of temporary data between the cache pool 120 and the storage cluster 200;
[0035] FIG3 is a flow chart of a resource scheduling method provided in an embodiment of the present application;
[0036] FIG4 is a schematic diagram of the structure of a scheduler 300 provided in an embodiment of the present application;
[0037] FIG5A is a schematic diagram of an exemplary configuration file in JSON format provided in an embodiment of the present application;
[0038] FIG5B is a schematic diagram of another exemplary configuration file in JSON format provided in an embodiment of the present application;
[0039] FIG6 is a schematic diagram of an exemplary inter-job data flow diagram provided in an embodiment of the present application;
[0040] FIG7 is a schematic diagram of the cache pool 120 scheduling three cache areas for jobs 1 to 4;
[0041] FIG8 is a flow chart of another resource scheduling method provided in an embodiment of the present application;
[0042] FIG9 is a schematic diagram of the structure of a scheduler provided in an embodiment of the present application;
[0043] FIG10 is a schematic diagram of the structure of a scheduler provided in an embodiment of the present application. DETAILED DESCRIPTION
[0044] The terms "first," "second," and so on, in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are distinguished in the embodiments of this application.
[0045] 1 , which shows a schematic diagram of the structure of a data processing system. As shown in FIG1 , the data processing system 10 includes a computing cluster 100, a storage cluster 200, and a scheduler 300. The computing cluster 100, the storage cluster 200, and the scheduler 300 can be connected via a high-speed Internet network.
[0046] Computing cluster 100 includes multiple computing nodes, each of which can be, for example, a server. These multiple computing nodes are used to execute user-submitted jobs. For ease of understanding and explanation, FIG1 illustrates computing cluster 100 as including computing nodes 101 and 102. Different computing nodes are connected via a high-speed interconnect network. Each computing node may include multiple processor cores. For example, computing node 101 in FIG1 includes four processor cores. The processor cores in each computing node are configured with memory. Furthermore, each computing node may be configured with a cache. The cache within the computing node can be referred to as a local cache. The storage area within the local cache is the local cache area. As shown in FIG1 , computing node 101 includes a local cache area 1011, and computing node 1021 includes a local cache area 1021. The different processor cores, memories, and caches within the computing nodes are interconnected via an on-chip bus (the connections between the cache and the processor cores and memories are not shown in FIG1 ). Typically, different processor cores within the same computing node can share data within the local cache area.
[0047] Computing cluster 100 also includes a cache pool 120 for caching data. Cache pool 120 can be implemented, for example, by one or more storage media, such as a solid state drive (SSD) or static random access memory (SRAM), though this is not a limitation.
[0048] The storage cluster 200 includes multiple storage nodes, which may be, for example, storage servers. FIG1 illustrates an example of a storage cluster 200 including storage nodes 201 and 202. Each storage node may be used to persistently store data, such as temporary data or execution results generated when the data processing system 10 executes a job. The storage medium in the storage node may be, for example, a hard disk drive (HDD), a hybrid hard drive (HHD), a solid-state drive (SSD), or other types of storage media.
[0049] The scheduler 300 is used to schedule corresponding resources for user-submitted jobs. The resources may include at least computing resources, storage resources, and network resources. Computing resources may be, for example, processor cores in the computing cluster 100, or other types of processors such as graphics processing units (GPUs). Storage resources may be, for example, cache areas (such as BBs) in the cache pool 102, or storage areas in the storage cluster 200, input / output operations per second (IOPS), etc. Network resources may be, for example, routing, network bandwidth, etc.
[0050] In actual application scenarios, users usually submit a job stream to the data processing system 10, which includes multiple jobs that have a dependency or sharing relationship with each other. Among them, there is a dependency relationship between job 1 and job 2, which means that the execution of job 2 depends on the prior execution of job 1. There is a sharing relationship between job 1 and job 2, which means that the same data will be used as input data for job 1 and job 2. At this time, if the scheduler 300 only considers the computing power level and schedules resources for each job in the job stream one by one, then the data processing system 10 will frequently migrate data between the cache pool 120 and the storage cluster 200, which will result in low IO performance of the data processing system and prominent resource waste.
[0051] Taking the example of a job flow including Job 1 and Job 2, and assuming that Job 1 and Job 2 have a dependency relationship, as shown in FIG2 , scheduler 300 not only schedules the corresponding compute nodes for Job 1 and Job 2 to execute the jobs, but also schedules cache area 1 in cache pool 120 to Job 1 and cache area 2 in cache pool 120 to Job 2. This allows data persistently stored in storage cluster 200 to be migrated to the cache areas in cache pool 120 before executing Job 1 and Job 2, thereby utilizing the cache areas in cache pool 120 to expedite the acquisition of input data for Job 1 and Job 2. As shown in FIG2 , during the execution of Job 1, data processing system 10 obtains input data from cache area 1 allocated to Job 1 (cache pool 120 pre-migrates this input data from storage cluster 200 to cache area 1), writes temporary data generated by executing Job 1 to cache area 1, and writes the temporary data in cache area 1 to storage cluster 200 for persistent storage after Job 1 completes. Then, before executing Job 2, cache pool 120 will pre-write the temporary data stored in storage cluster 200 into cache area 2 allocated for Job 2, so that Job 2 can be executed based on the temporary data stored in cache area 2 as input data. This utilizes cache area 2 to speed up the execution of Job 2. In this way, the same temporary data is frequently migrated between cache pool 120 and storage cluster 200, which will result in a large amount of unnecessary I / O between cache pool 120 and storage cluster 200, thereby affecting the overall I / O performance of data processing system 10.
[0052] Based on this, the present application provides a resource scheduling method. For a job stream submitted by a user, the scheduler 300 will obtain the data flow description information corresponding to the job stream. The data flow description information is used to indicate the relationship between the input data or output data of each job in the multiple jobs included in the job stream. The relationship can be, for example, a dependency relationship or a sharing relationship. Among them, the dependency relationship between input data or output data means that the data output by one job is the input data of another job. The sharing relationship between input data or output data means that the input data of two jobs are the same data. Then, the scheduler 300 schedules resources for multiple jobs based on the data flow description information. Specifically, it can be to schedule the first job and the second job in the job stream to the same computing node. Taking scheduling to computing node 101 as an example, the available resources on computing node 101 can simultaneously meet the resource requirements of the first job and the second job. The input data / output data of the first job has a relationship with the input data / output data of the second job, such as a dependency relationship or a sharing relationship. In this way, the second job can share the input data or output data of the first job on computing node 101. For example, the input and output data of the first job can be written to the local cache area 1011 on computing node 101. Thus, when computing node 101 executes the second job, it can obtain the input data of the second job by accessing the local cache area 1011. When the input data or output data of the two jobs are shared, the input data of the second job is also the input data of the first job, and the input data can be loaded from storage cluster 200. When the input data or output data of the two jobs are dependent on each other, the input data of the second job is temporary data generated after the execution of the first job.
[0053] In this way, scheduler 300 schedules a first job and a second job, whose input data or output data have a relationship, to the same computing node 10, namely, computing node 101. This allows one of the jobs to directly obtain its required input data from local cache area 1011 in computing node 101, without having to access the cache area in cache pool 120. Because computing node 101 typically reads and writes data from its local cache area at a higher rate than it reads and writes data from cache pool 120, this effectively improves the efficiency of obtaining input data when executing the second job, thereby improving the efficiency of data processing system 10 in executing the second job. Moreover, during the execution of the first job and the second job, the input data required for the second job can be continuously stored in the local cache area 1011 in the computing node 101, which can avoid unnecessary migration of the data between the computing node 101, the cache pool 120 and the storage cluster 200, that is, it can avoid the same data from being migrated multiple times between the computing node 101, the storage cluster 200 and the cache pool 120, thereby optimizing the scheduling of computing resources at the job flow level, and further effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0054] Still using the example of scheduling resources for Job 1 and Job 2 in the job flow, scheduler 300 can determine, based on the data flow description information submitted by the user, that the output data of Job 1 serves as the input data for Job 2. Scheduler 300 can then schedule Job 1 and Job 2 to compute node 101. Thus, when executing Job 2, compute node 101 can directly read the temporary data generated by the job execution from local cache area 1011. Thus, during the execution of the job flow, the temporary data generated by Job 1 can be continuously stored in local cache area 1011, eliminating the need to migrate the temporary data back and forth between compute node 101, cache pool 120, and storage cluster 200 (i.e., there is no need to migrate the temporary data from compute node 101 to cache pool 120, then from cache pool 120 to storage cluster 200, and then, when executing Job 2, migrate the temporary data from storage cluster 200 to storage pool 120, and then from storage pool 120 to compute node 101). This improves the IO performance of data processing system 10 and alleviates resource waste. When the input data of job 1 is consistent with the input data of job 2 (that is, job 1 and job 2 share the same input data), the computing node 101 can store the input data in the local cache area 1011 when executing job 1, so that the computing node 101 can directly obtain the input data from the local cache area 1011 when executing job 2, without having to access the cache pool 120 or the storage cluster 200 to obtain the input data. This can reduce the number of times the input data is migrated from the storage cluster 200 to the cache pool 120 and the computing node 101, thereby improving the efficiency of the computing node 101 in executing job 2.
[0055] In addition, the scheduler 300 can also schedule the cache area in the cache pool 120 for each job in the job flow, so that each computing node can obtain data by accessing the corresponding cache area in the cache pool 102 when executing the job, which can speed up the data acquisition speed compared to accessing the data in the storage cluster 200.
[0056] It is worth noting that the data processing system 10 shown in FIG1 is only used as an exemplary illustration and is not intended to limit the implementation of the data processing system 10 to the example shown in FIG1 . For example, in other possible data processing systems, the cache pool 120 may not be included, such as each computing node can directly obtain the input data required to execute the job by accessing the storage cluster 200. Alternatively, in other possible data processing systems, more functional devices may be included, such as one or more login nodes, so that users can log in to their accounts and submit jobs through the login nodes. Alternatively, the data processing system may also include a log server for storing logs of the scheduling operations performed by the scheduler 300 for multiple job streams, etc., to support the data processing system 10 with more functions. In addition, the above resource scheduling method can be applied to data processing systems with other architectures, such as a data processing system in which the computing cluster 100 and the storage cluster 200 are integrated (i.e., storage and computing are integrated). The specific architecture of the data processing system is not limited in this application.
[0057] For ease of understanding, the resource scheduling process performed by the scheduler for the user's job flow is described in detail below with reference to the accompanying drawings.
[0058] Referring to FIG3 , a schematic flow chart of a resource scheduling method provided in an embodiment of the present application is shown. This method can be applied to the data processing system 10 shown in FIG1 , or can be applied to other applicable data processing systems. For ease of understanding and explanation, the following exemplary description is given using the data processing system 10 shown in FIG1 as an example. As shown in FIG3 , the method may specifically include:
[0059] S301: The scheduler 300 obtains data flow description information and a job flow submitted by a user, wherein the job flow submitted by the user includes multiple jobs. The data flow description information is used to indicate the relationship between input data or output data of each job in the multiple jobs in the job flow.
[0060] The relationship between input data / output data of different jobs may be a dependency relationship or shared data, or other types of relationships, which are not limited in this embodiment.
[0061] Taking the job flow including the first job and the second job as an example, when the input data / output data of the first job and the input data / output data of the second job satisfy the dependency relationship, the data generated and output by executing the first job is used as the input data when executing the second job.
[0062] When the input data / output data of the first job and the input data / output data of the second job satisfy a sharing relationship, the input data required to execute the first job and the input data required to execute the second job are the same, such as data loaded from the storage cluster 200 .
[0063] Typically, the job stream submitted by the user may define resource requirements for each job. The resource requirements are used to indicate the resources required to execute the job, such as the number of processor cores required, the number of processes required, the size of memory space required, etc., so that the scheduler 300 can schedule resources for the job to meet the resource requirements of the job.
[0064] In this embodiment, when a user submits a job flow including multiple jobs, the user also submits data flow description information indicating a dependency relationship or a sharing relationship between input data / output data of the multiple jobs.
[0065] As some implementation examples, the data processing system 10 may provide a client so that the user can submit job flow and data flow description information to the data processing system 10 through the client, and the client sends it to the scheduler 300. Alternatively, the data processing system 10 may further include a login node, and the user can log in to the login node and, after successfully completing the login, submit the job flow and data flow description information to the scheduler 300 through the login node.
[0066] In actual application, the scheduler 300 may be configured with middleware, such as the middleware 301 shown in FIG. 4 , so that the scheduler 301 can use the middleware 301 to receive the job flow and data flow description information submitted by the user.
[0067] User-submitted configuration files can be written using a domain-specific language (DSL). A DSL is a programming language designed for a specific problem domain or application scenario, aiming to provide a higher level of abstraction to make programming simpler, clearer, and more efficient. The configuration file includes multiple configuration items, which are used to configure the input and output data of multiple jobs in a job flow. This allows for describing the dependencies and shared relationships between the input and output data of these multiple jobs.
[0068] In a specific implementation, some configuration items included in the configuration file can be used to describe the relationship between the job and the data, while other configuration items included in the configuration file can be used to describe relevant information about the data, such as data attribute information, I / O information specific to the data, etc. For example, the types of the multiple configuration items in the configuration file can be as shown in Table 1 below.
[0069] Table 1
[0070] In actual application, the types of configuration items defined in the configuration file are not limited to the examples shown in Table 1 above.
[0071] The configuration file submitted by the user may be in any of the following formats: json, xml, yaml, or other file formats.
[0072] For example, a user can submit a JSON-formatted configuration file as shown in Figure 5A. In this configuration file, the input and output data of each job can be directly described. For example, for a job with a "jobID" (i.e., job identifier) of "2," its input data includes data named "tmpData1" and "srcData2," and its output data includes data named "outData1," along with information such as the data type, data operation, number of times it is opened, and temporary type.
[0073] Alternatively, users can submit a JSON-formatted configuration file as shown in Figure 5B . In this configuration file, the input and output data for each job can be defined using tags. For example, for a job with a "jobID" (i.e., job name) of "2," the tag defined for the job's input and output data is "label2." Label "label2" defines the job's input data as "tmpData1" and "srcData2," and the output data as "outData1." It also describes information such as the data type, data operation, number of opens, and temporary type.
[0074] S302: When the input data or output data of a first job among multiple jobs is stored in the computing node 101, the scheduler 300 schedules a second job related to the input data or output data of the first job and the first job to the computing node 101 according to the data flow description information, and the second job shares the input data or output data of the first job.
[0075] In one possible implementation, when scheduling resources for multiple jobs in a job stream, the scheduler 300 may first obtain the available resources of some or all of the computing nodes in the computing cluster 100, thereby sensing the available resources on the computing nodes, and subsequently scheduling resources for the multiple jobs in the job stream based on the obtained available resources on the computing nodes. Available resources refer to resources on the computing nodes that are idle and can be scheduled to execute jobs, such as idle processor cores, unallocated memory, cache, and the like.
[0076] Then, the scheduler 300 schedules the first job and the second job whose input data or output data have relationship to the computing node 101 with sufficient available resources based on the data flow description information, the available resources of the computing node 101 and the resource requirements of each job in the multiple jobs.
[0077] Typically, a job stream may include more than two (including two) jobs. To facilitate understanding and explanation of the resource scheduling of jobs by the scheduler 300, this embodiment uses the example of the scheduler 300 scheduling resources for two jobs (i.e., the first job and the second job) in the job stream as an example for illustrative explanation. The scheduled resources include computing resources, storage resources, and network resources, etc. Computing resources specifically refer to scheduling jobs to corresponding computing nodes and allocating processors / processor cores on the computing nodes to the jobs so that the jobs can be executed using the processors / processor cores. Storage resources specifically refer to allocating memory / local cache areas on computing nodes, or cache areas in the storage pool 120, or persistent storage areas in the storage cluster 200 to jobs, etc. Network resources refer to scheduling network-level resources such as bandwidth and communication ports in the network to jobs. This embodiment focuses on the implementation process of the scheduler 300 scheduling computing resources and storage resources on the computing node 101 for the first job and the second job.
[0078] For example, as shown in FIG4 , scheduler 300 may be configured with a job scheduling plug-in 302 and a scheduling component 303 , thereby utilizing the job scheduling plug-in 302 and scheduling component 303 to schedule corresponding computing resources for each job in a job flow. The following describes the implementation process of scheduler 300 scheduling computing resources and storage resources for a job.
[0079] In specific implementation, the middleware 301 can provide the job flow submitted by the user, the available resources on at least one computing node, and the data flow description information to the scheduling component 303, so that the scheduling component 303 can prioritize scheduling multiple jobs with dependency or sharing relationships between input data / output data to the same computing node with sufficient available resources based on the resource requirements of each job in the job flow, the available resources on each computing node, and the data flow description information.
[0080] Taking the scheduling of computing resources for the first and second jobs in a job flow as an example, assuming that the first job requires the use of three processor cores to execute the job, and the second job requires two processor cores to execute the job, and that the output data of the first job serves as input data for executing the second job, then, when processor cores 1 to 5 (and other processor cores) are available on computing node 101, scheduling component 303 can schedule processor cores 1 to 3 on computing node 101 to the first job and processor cores 4 and 5 on computing node 101 to the second job, so that the scheduled processor cores can subsequently be used to execute the jobs. Furthermore, scheduler 300 also schedules local cache area 1011 on computing node 101 to the first and second jobs. In this way, temporary data generated by computing node 101 after executing the first job using processor cores 1 to 3 can be stored in local cache area 1011. When processor cores 4 and 5 execute the second job, they can directly access the input data from local cache area 1011 to execute the second job based on the input data. Referring to the above-mentioned similar method, the scheduling component 303 can prioritize scheduling multiple jobs in the job flow with dependency or sharing relationships between input data / output data to the same computing node (the available resources on the computing node are sufficient to support the execution of multiple jobs scheduled to the computing node) based on the size of available resources on the computing node, and allocate the local cache area on the computing node to the multiple jobs.
[0081] In actual application, due to the limited resources on a single computing node, the scheduling component 303 can schedule multiple jobs in the job stream to multiple computing nodes. As shown in Figure 7, the scheduling component 303 can schedule jobs 1 and 2 in the job stream to computing node 101, and jobs 3 and 4 to computing node 102, etc. For example, the scheduling component 303 can divide the multiple jobs in the job stream into multiple groups based on the data flow description information, each group including at least one job, and schedule different groups to different computing nodes. Among them, the scheduling component 303 can preferentially divide multiple jobs with a dependency relationship between input data / output data into one group, and schedule multiple jobs in the group to the same computing node. For example, a job flow may include six jobs, namely job 1 to job 6, where the input data / output data of jobs 1 to 3 have a dependency relationship, the input data / output data of jobs 4 to 6 have a dependency relationship, and the input data / output data of jobs 1 and 4 have a sharing relationship. Then, the scheduling component 303 can divide jobs 1 to 3 into a group, and schedule the three jobs in the group to the computing node 101, and schedule the local cache area 1011 to jobs 1 to 3; and the scheduling component 303 can divide jobs 4 to 6 into a group, and schedule the three jobs in the group to the computing node 102, and schedule the local cache area 1021 to jobs 4 to 6, etc.
[0082] In this way, the second job located on the same computing node as the first job can directly obtain the input data it needs in the local cache area 1011 in the computing node 101, without having to access the cache in the cache pool 120 to obtain data. This can not only effectively improve the efficiency of obtaining input data when executing the second job, but also avoid unnecessary migration of the data between the computing node 101, the cache pool 120 and the storage cluster 200, thereby effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0083] In this embodiment, in addition to scheduling computing resources and local cache areas for multiple jobs in a job flow based on the above implementation process, the scheduler 300 can also schedule storage resources in the cache pool 120 for the job flow. Specifically, the scheduler 300 can schedule cache areas in the cache pool 120 for the jobs in the job flow so that the computing nodes can obtain input data from the cache areas when executing the jobs. Therefore, this embodiment can also include the following steps:
[0084] S303: The scheduler 300 schedules cache areas in the cache pool 120 for multiple jobs in the job flow.
[0085] In a first implementation example, the scheduler 300 may schedule a cache area in the cache pool 120 for each job, and the cache area allocated to each job can only be accessed by the job.
[0086] In the second implementation example, the scheduler 300 can schedule cache areas in the cache pool 120 for multiple jobs in the job flow based on the data flow description information, and a cache area scheduled by the scheduler 300 can support access by multiple jobs with dependency or sharing relationships between input data / output data.
[0087] In a specific implementation, the middleware 301 may call an interface provided by the job scheduling plug-in 302 so as to provide the data flow description information submitted by the user to the job scheduling plug-in 302 through the interface.
[0088] The job orchestration plug-in 302 generates a cross-job data flow graph (XJDFG) corresponding to multiple jobs based on the data flow description information. The XJDFG can specifically be a directed acyclic graph (DAG) including job nodes, data nodes, and connecting lines. Among them, the job node is used to indicate the job in the job flow, and each job node can indicate one job. The data node is used to indicate the input data or output data of the job in the job flow, and each job node is used to indicate the input data or output data of one job. The connecting line is used to indicate the relationship between the job node and the data node, such as indicating that the data represented by the data node serves as the input data or output data of the job represented by the job node.
[0089] For example, assuming the job flow includes four jobs, Job 1 through Job 4, as shown in Figure 5A , the job orchestration plug-in 302 generates the XJDFG shown in Figure 6 based on the data flow description information corresponding to the job flow. The XJDFG includes four job nodes and seven data nodes, each corresponding to the data described in Figure 5A . The job nodes and data nodes are connected by directed lines.
[0090] In this way, the XJDFG generated by the job orchestration plug-in 302 can not only indicate the data that each job depends on and the information of the generated data, but also indicate the dependency relationship between jobs. For example, in the XJDFG shown in Figure 6, job 2 depends on job 1, and job 4 depends on job 3.
[0091] After generating the XJDFG, the job orchestration plug-in 302 can generate a data migration request based on the XJDFG and the remaining available cache resources in the cache pool 120. This data migration request carries information indicating the creation of a cache area. For example, the data migration request can include the identifiers of multiple jobs in the job flow and information about the data to be stored in the cache pool 120; alternatively, the data migration request can include the XJDFG. In actual applications, the data migration request can also include other information such as the execution times of multiple jobs, which is not limited to this.
[0092] Then, the job orchestration plug-in 302 sends the data migration request to the middleware 301 , and the middleware 301 sends the data migration request to the cache pool 120 .
[0093] The cache pool 120 (specifically, it can be the storage resource manager in the cache pool 120) responds to the data migration request and schedules corresponding cache areas for the multiple jobs from the remaining available cache areas based on the dependency and sharing relationship between the input data / output data of the multiple jobs in the job flow. The scheduled cache areas can be, for example, BB (burst buffer) instances, or can be other types of caches.
[0094] Take the example of a job flow including a first job, a second job, and a third job, wherein the input data / output data of the first job and the second job have a dependency relationship, and the input data / output data of the second job and the third job have a dependency relationship or a sharing relationship. Due to the limitation of available resources on computing node 101, the scheduler 300 can schedule the first job and the second job to computing node 101, and schedule the third job to computing node 102. Then, the cache pool 120 can schedule a cache area for the second job and the third job, which is referred to as the target cache area in this embodiment for ease of distinction. The scheduled target cache area is used to store the target data shared between the second job and the third job. When the input data / output data of the second job and the third job have a dependency relationship, the target data is specifically temporary data generated by executing the second job, and the temporary data is also the input data required by the third job when it is executed. When the input data / output data of the second job and the third job have a sharing relationship, the target data is specifically the input data required for executing the second job and the third job; usually, the target data will be persistently stored in the storage cluster 200. Therefore, before executing the second job and the third job, the target data can be pre-loaded from the storage cluster 200 to the target cache area.
[0095] In addition, the scheduler 300 may also schedule a cache area for the first job to store input data required when executing the first job, so as to increase the efficiency of obtaining the input data when executing the first job.
[0096] For example, assuming that the dependency and sharing relationships between the input data and output data of multiple jobs included in a job flow are as shown in FIG6 , scheduler 300 may schedule jobs 1 and 2 to compute node 101, and jobs 3 and 4 to compute node 102, as shown in FIG7 . In this way, the temporary data "tmpData1" output by job 1 during execution can be written to local cache area 1011 on compute node 101, allowing job 2 to retrieve the temporary data "tmpData1" from local cache area 1011, thereby accelerating the rate at which job 2 can retrieve the temporary data "tmpData1" (similar to jobs 3 and 4). Simultaneously, scheduler 300 schedules cache area 2 for job 2, and cache area 2 is used to store the input data required for executing jobs 2 and 3. This allows both jobs 2 and 3 to read the input data from cache area 2 during execution.
[0097] Furthermore, cache pool 120 may not schedule a cache area for the third job, which can effectively reduce the cache resource usage of cache pool 120 by executing the job flow. For example, as shown in FIG7 , cache pool 120 may not schedule cache area 3 for job 3, thereby reducing the resource usage of cache pool 120.
[0098] Accordingly, before each job in the job flow is executed, the cache pool 120 can read the input data required to execute each job from the storage cluster 200 in advance to one or more cache areas allocated to the job flow, so as to improve the overall execution efficiency of the job flow by speeding up the acquisition of the input data of each job.
[0099] In the third implementation example, the scheduler 300 first schedules computing resources for the job flow, and then schedules the target cache area in the cache pool 120 according to the computing resources.
[0100] Typically, there may be differences in the latency for processor cores in different computing nodes to access different cache areas in cache pool 120. For example, when the network distance between cache area 1 in cache pool 120 and processor core A in computing node 101 is shorter, and the network distance between cache area 1 and processor core B in computing node 102 is longer, the latency for processor core A to access cache area 1 will usually be shorter than the latency for processor core B to access cache area 1.
[0101] Based on this, in a further possible implementation, the scheduler 300 can first schedule the processor cores (i.e., schedule computing resources) and local cache areas on the corresponding computing nodes for the job flow, and then allocate the cache areas with smaller total access delays to the jobs in the job flow based on the delays of the scheduled processor cores accessing the cache areas in the cache pool 120.
[0102] In a specific implementation, the job flow includes not only the first and second jobs but also a third job. The input / output data of the second and third jobs are dependent or shared. Due to the limited available resources on compute node 101, scheduler 300 can first use scheduling component 303 to schedule processor core 1 in compute node 101 for the first job, processor core 2 in compute node 101 for the second job, and processor core 5 in compute node 102 for the third job. Scheduler 300 then uses middleware 301 to instruct cache pool 120 to schedule cache areas with lower access latency for the second and third jobs. For the second and third jobs, whose input / output data are dependent or shared, and because they share the same input data, or because temporary data output by the second job serves as input data for the third job, cache pool 120 can first identify multiple candidate cache areas and, for each candidate cache area, estimate the total latency for access to that cache area by processor core 2 in compute node 101 and processor core 5 in compute node 102. Next, cache pool 120 can determine a target cache region from the candidate cache regions based on the sum of the latencies corresponding to the candidate cache regions, and schedule the target cache region to the second and third jobs. This reduces the latency for processor core 2 in compute node 101 and processor core 5 in compute node 102 when executing jobs and reading or writing data shared between the second and third jobs in the target cache region.
[0103] It is worth noting that the various implementation examples described above primarily describe how the scheduler 300 schedules computing resources, computing nodes, and storage resources in the cache pool 120 for a job flow. In actual applications, the resources scheduled by the scheduler 300 for a job flow may also include network resources and storage resources in the storage cluster 200. The specific implementation methods for scheduling network resources and storage resources are well known in the prior art and will not be elaborated upon here. Furthermore, in actual applications, the implementation process of the scheduler 300 scheduling computing resources and storage resources for a job flow based on data flow description information is not limited to the implementation examples described above. For example, the scheduler 300 may schedule resources for a job flow by combining the various implementation examples described above.
[0104] Furthermore, after completing resource scheduling, the data processing system 10 may execute each job in the job flow based on the scheduled computing resources and storage resources.
[0105] In a specific implementation, the job orchestration plug-in 302 can determine the dependencies between multiple jobs in the job flow based on the data flow description information provided by the middleware 301, such as the execution of job 2 depends on the completion of job 1, and feedback the dependencies between the multiple jobs to the middleware 301. The scheduler 300 can also include a dependency management component 304, as shown in Figure 4. In addition, the middleware 301 can provide the dependency management component 304 with the dependency relationship, so that the dependency management component 304 can control the processor cores on the computing nodes to execute each job in the job flow in sequence based on the dependency relationship. During the execution of each job, different jobs with shared input / output data can obtain input data from the same cache area (the input data was pre-loaded from the storage cluster 200 into the cache area), or a second job with shared input / output data can obtain input data from the local cache area 1011 (the input data was loaded from the cache area into the local cache area 1011 by the first job). In this way, the same data can be prevented from being migrated multiple times from the storage cluster 200 to the cache pool 120 and the computing nodes. Moreover, between job 1 and job 2, whose input / output data have a dependency relationship, the data output by job 1 can be shared with job 2 through the cache area in the cache pool 120 or the local cache area in the computing node. In this way, the data output by job 1 can be avoided from being frequently migrated between the cache pool 120, the storage cluster 200 and the computing node.
[0106] For example, a job flow includes a first job, a second job, and a third job, and there is a dependency relationship between the input data / output data of the first job and the second job, and a sharing relationship between the input data / output data of the first job and the third job. Then, the dependency management component 304 can control the processor cores on different computing nodes to execute the first job and the third job in parallel. During this process, multiple processor cores can read the data used as input for the first job and the third job from the same cache area. After the first job is completed, the temporary data output by the first job can be cached in the local cache area in the computing node. Then, the dependency management component 304 can control other processor cores on the computing node where the first job is located to execute the second job, etc. At this time, the processor core reads the temporary data output by the first job from the local cache area and uses it as input data for the second job to execute the second job. In this way, the execution of the job flow can be realized.
[0107] In a further possible implementation, after multiple jobs in a job stream are executed, the cache pool 120 can send the data in the cache area to the storage cluster 200 for persistent storage, such as sending the data "outDATA2" (and the data "tmpData3") in cache area 4 shown in FIG7 to the storage cluster 200 for persistent storage. The area in the storage cluster 200 used to store the data can be scheduled by the scheduler 300. After completing the persistent storage of the data, the scheduler 300 can instruct the cache pool 120 to release the cache area scheduled for the multiple jobs to reclaim the storage resources corresponding to the cache area. In this case, the lifecycle of the cache area scheduled for the multiple jobs is bound to the job stream.
[0108] Cache pool 120 can release all cache areas scheduled for a job stream after the entire job stream has completed execution. Alternatively, during the execution of a job stream, when some jobs have completed execution and the data in the cache areas scheduled for those jobs is no longer used as input data for other jobs, cache pool 120 can write the data in those cache areas to storage cluster 200 and release those cache areas. In this way, cache pool 120 can reclaim resources by prematurely reclaiming those cache areas without affecting the execution of the job stream, thereby improving resource utilization for those cache areas. For example, after reclaiming the cache areas, they can be rescheduled for jobs in other job streams.
[0109] In addition, when each job in the job flow defines input data and output data based on a label, the scheduler 300 can also support third-party components to query the label or the input data and output data defined by the label. For example, before submitting a job, the user can query the labels defined in the scheduler 300 through the third-party component.
[0110] In a specific implementation, the scheduler 300 may also include a tag management component 305, and the scheduler 300 may also be connected to a third-party component 306, as shown in Figure 4. The tag management component 305 is used to record and manage tags defined by users when submitting jobs. For example, a user may customize tags in the data flow description information submitted for Job A. The tag management component 305 can then verify the legality of the user-defined tags sent by the middleware 301 based on all previously recorded tags. Specifically, this can include verifying whether the user-defined tags overlap with all tags already recorded by the tag management component 305, verifying whether the naming format of the user-defined tags complies with specifications, and so on. If the user-defined tags are legal, the tag management component 305 can record the tags and include them in its management. If the user-defined tags are illegal, the tag management component 305 can notify the user of the tag definition anomaly through the middleware 301. Furthermore, the tag management component 305 can also create new tags, such as automatically creating new tags to define the input and output data of a completed job based on the input and output data of the completed job.
[0111] When a user needs to query the tags recorded in the scheduler 300, they can use the third-party component 306 to generate a tag query request and send the tag query request to the tag management component 305. In response to the query request, the tag management component 305 returns all the recorded tags to the third-party component. Alternatively, based on the identifier of the target job carried in the query request, the tag management component 305 returns the tags used to define the input and output data of the target job to the third-party component.
[0112] After querying the label, the user can also use the third-party component 306 to query the dependency relationship between different jobs based on the label. For example, the user can use the third-party component 306 to generate a data dependency query request, which may include the labels of multiple jobs, and the multiple jobs may be jobs included in one or more job streams. Then, the third-party component 306 can send the data dependency query request to the job orchestration plug-in 302. The job orchestration plug-in 302 queries the XJDFG corresponding to the multiple jobs based on the labels of the multiple jobs included in the data dependency query request, thereby determining the dependency relationship between the input data / output data of the multiple jobs based on the XJDFG, and feeding back the queried dependency relationship to the third-party component 306.
[0113] In this embodiment, the scheduler 300 schedules at least two jobs to the same computing node based on the dependency / sharing relationship between the input data / output data of multiple jobs in the job flow, so that the at least two jobs can use the local cache area on the computing node to store the data shared between the jobs, which can effectively avoid unnecessary migration of data shared between jobs between the storage cluster 200, the cache pool 120 and the computing nodes, that is, it can avoid the same data from being migrated multiple times between the storage cluster, the cache pool 120 and the computing nodes, and optimize the scheduling of computing resources and storage resources at the job flow level, thereby effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0114] The embodiment shown in FIG. 3 above uses user-submitted data flow description information as an example. In other possible embodiments, the data processing system 10 may analyze and obtain the dependencies between the input data and output data of multiple jobs in the job flow. This will be described in detail below.
[0115] Referring to Figure 8, a flow chart of another resource scheduling method provided by an embodiment of the present application is shown. As shown in Figure 8, the method may specifically include:
[0116] S801: The scheduler 300 obtains a job flow submitted by a user, wherein the job flow submitted by the user includes multiple jobs, and the multiple jobs include at least a first job and a second job.
[0117] The specific implementation of step S801 can be found in the relevant description of the embodiment shown in FIG3 , and will not be elaborated here.
[0118] In this embodiment, after obtaining the job stream submitted by the user, the scheduler 300 can analyze and determine the dependency between the input data / output data of the multiple jobs by first running the multiple jobs in the job stream. The specific implementation process can be shown in the following steps.
[0119] S802 : The scheduler 300 obtains available resources of each computing node in the computing cluster 100 .
[0120] S803: The scheduler 300 schedules first resources for multiple jobs in the job flow according to available resources of each computing node.
[0121] In specific implementations, the scheduler 300 can schedule resources (including computing resources, storage resources, network resources, etc.) in the computing cluster 100 to the job flow based on the resource requirements of each job in the job flow and the currently available resources of each computing node in the computing cluster 100. Taking the scheduling of computing resources as an example, the scheduler 300 can schedule some of the multiple processor cores currently available in the computing cluster 100 to the job flow, so that the multiple processor cores can be used to execute the jobs in the job flow.
[0122] In addition, the scheduler 300 will also schedule the storage resources and network resources required for executing the multiple jobs respectively.
[0123] For ease of distinction and description, the scheduler 300 preliminarily schedules computing resources, storage resources, and network resources for the job flow, which are collectively referred to as first resources in this embodiment.
[0124] S804: The scheduler 300 collects running information of multiple jobs in the job flow. The multiple jobs are run based on the scheduled first resource. The running information includes input data and output data of each of the multiple jobs.
[0125] In this embodiment, after the scheduler 300 schedules the first resource for the job flow, the data processing system 10 can use the first resource to execute each job in the job flow, so that the scheduler 300 can collect input data and output data of each job during the operation of the job flow.
[0126] Exemplarily, the collected input data may be, for example, an identifier of the input data of each job, and may also include information such as the data type, data length, and data storage location of the input data.
[0127] The collected output data may be, for example, an identifier of the output data of each job, and may also include information such as the data type, data operation, number of openings, and temporary type of the output data.
[0128] S805: The scheduler 300 analyzes the collected operation information to obtain data flow description information, where the data flow description information is used to indicate the dependency or sharing relationship between the input data / output data of multiple jobs in the job flow.
[0129] In specific implementation, the scheduler 300 can determine job A and job B that use the same data as input data by analyzing the input data and output data of each job during execution. Then, the scheduler 300 can determine that there is a sharing relationship between the input data / output data of job A and job B.
[0130] In addition, by analyzing the input data and output data of each job during execution, the scheduler 300 can determine that the data generated and output by executing job C will be used as input data when executing job D. The scheduler 300 can then determine that there is a dependency relationship between the input data / output data of job C and job D.
[0131] In this way, the scheduler 300 can obtain the data flow description information corresponding to the job flow based on the sharing relationship and dependency relationship between the input data / output data of different jobs.
[0132] S806: The scheduler 300 re-schedules a second resource for multiple jobs in the job flow, where the second resource is different from the first resource; or, the scheduler 300 schedules a target resource for multiple jobs in the job flow, where the target resource is used to replace part of the first resources that have been scheduled for multiple jobs.
[0133] In this embodiment, the scheduler 300 can reschedule resources for multiple jobs in the job flow based on the job flow submitted by the user and the data flow description information analyzed by the scheduler 300, namely, the second resource in step S805. The specific implementation of the scheduler 300 scheduling the second resource for multiple jobs based on the data flow description information can be found in the relevant description of step S302 in the previous embodiment and will not be repeated here.
[0134] Alternatively, because scheduler 300 has already scheduled the first resource for the job flow, scheduler 300 can, based on the job flow submitted by the user and the data flow description information analyzed by scheduler 300, schedule resources for multiple jobs in the job flow that may overlap with the already scheduled first resource. Based on this, scheduler 300 can optimize the resource scheduling of the job flow by simply replacing some of the already scheduled first resources.
[0135] In specific implementation, the scheduler 300 can first determine the third resource that needs to be scheduled (resource scheduling has not yet been performed) based on the job flow and data flow description information, and compare the third resource that needs to be scheduled with the scheduled first resource to determine the target resource that is not included in the first resource, and determine the resource A that is included in the first resource but not included in the third resource, so that the scheduler 300 can schedule the target resource to the job flow and instruct the job flow to use the target resource to replace part of the resources in the first resource (that is, replace resource A in the first resource).
[0136] In actual application scenarios, the job flow submitted by the user may be executed multiple times. For example, the user expects the data processing system 10 to periodically execute the job flow to regularly obtain the job execution results he needs. Therefore, after running the job flow once in the data processing system 10, the scheduler 300 can analyze the dependency and sharing relationship between the input data / output data of multiple jobs in the job flow, thereby optimizing the resource scheduling of the job flow based on the dependency and sharing relationship, thereby effectively improving the IO performance of the data processing system 10 and alleviating the problem of resource waste.
[0137] The data processing system and resource scheduling method provided by the present application are described in detail above in conjunction with Figures 1 to 8. The scheduler provided by the present application will be described below in conjunction with Figures 9 to 10.
[0138] FIG9 is a schematic diagram of the structure of a scheduler provided in the present application. The scheduler 900 shown in FIG9 is applied to a data processing system, which also includes a computing cluster and a storage cluster. The computing cluster includes multiple computing nodes, and the computing cluster and the storage cluster are connected via a network.
[0139] As shown in FIG9 , the scheduler 900 includes:
[0140] An acquisition module 901 is configured to acquire data flow description information and a job flow submitted by a user. The job flow includes multiple jobs. The data flow description information indicates the relationship between input data or output data of each job in the multiple jobs.
[0141] Scheduling module 902 is used to schedule a second job related to the input data or output data of the first job and the first job to the first computing node according to data flow description information when the input data or output data of the first job among multiple jobs is stored in the first computing node among multiple computing nodes, and the second job shares the input data or output data of the first job.
[0142] In a possible implementation, the relationship between the input data or output data of each of the multiple jobs includes:
[0143] The dependency between the input data or output data of each job in multiple jobs;
[0144] Alternatively, the sharing relationship between the input data or output data of each job in multiple jobs.
[0145] In one possible implementation, when scheduling the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, the scheduling module 902 is specifically configured to:
[0146] When the available resources of the first computing node meet the resource requirements of the first job and the second job, the first job and the second job are scheduled to the first computing node according to the data flow description information.
[0147] In a possible implementation, when acquiring the data flow description information, the acquisition module 901 is specifically configured to:
[0148] Scheduling a first resource for the plurality of jobs according to the job flow;
[0149] Collecting operation information generated when multiple jobs are run based on the first resource, the operation information including input data and output data of each of the multiple jobs;
[0150] Analyze the operation information and obtain data flow description information.
[0151] In one possible implementation, when scheduling the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, the scheduling module 902 is specifically configured to:
[0152] Generate an inter-job data flow graph XJDFG corresponding to multiple jobs based on the data flow description information. The XJDFG includes job nodes, data nodes, and connecting lines. Job nodes are used to indicate jobs in a job flow, data nodes are used to indicate input data or output data of jobs in a job flow, and connecting lines are used to indicate the relationship between job nodes and data nodes.
[0153] According to XJDFG, a first job and a second job related to input data or output data of the first job are scheduled to a first computing node.
[0154] In a possible implementation, when acquiring the data flow description information, the acquisition module 901 is specifically configured to:
[0155] Obtain a configuration file submitted by a user and written in a domain-specific language (DSL). The configuration file includes multiple configuration items, which are used to configure input data and output data of multiple jobs.
[0156] In a possible implementation, the computing cluster further includes a cache pool, and the multiple jobs further include a third job. Then, the scheduling module 902 is further configured to:
[0157] Based on the data flow description information, a third job related to the input data or output data of the second job is scheduled to the second computing node among the multiple computing nodes, and a target cache area in the cache pool is scheduled to the second job and the third job; wherein the third job shares the input data or output data of the second job through the target cache area.
[0158] In one possible embodiment, the cache pool includes multiple available cache areas, the multiple cache areas include a target cache area, and the sum of the delays of the first computing node and the second computing node accessing the target cache area is less than the sum of the delays of the first computing node and the second computing node accessing other cache areas in the multiple cache areas.
[0159] The scheduler 900 shown in FIG9 corresponds to the scheduler 300 in the embodiments shown in FIG1 to FIG8 . Therefore, the functions and technical effects of the scheduler 900 can be found in the relevant descriptions in the embodiments shown in FIG1 to FIG8 , and will not be elaborated here.
[0160] Figure 10 is a schematic diagram of the structure of a scheduler provided in this application. As shown in Figure 10, the scheduler 1000 includes a processor 1001, a memory 1002, a communication interface 1003 and a bus 1004. The processor 1001, the memory 1002 and the communication interface 1003 communicate through the bus 1004. The bus 1004 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in Figure 10, but it does not mean that there is only one bus or one type of bus. The communication interface 1003 is used to communicate with the outside, such as receiving jobs (and target data) submitted by users through a terminal or client.
[0161] It should be understood that in the embodiment of the present application, the processor 1001 may be a CPU, or may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete device components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.
[0162] The memory 1002 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1001. The memory 1002 may also include a non-volatile random access memory. For example, the memory 1002 may also store device type information.
[0163] The memory 1002 may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).
[0164] The memory 1002 stores executable codes, and the processor 1001 executes the executable codes to perform the method performed by the aforementioned scheduler 300 .
[0165] Specifically, when implementing the embodiments shown in Figures 1 to 8, and when the scheduler 300 described in the embodiments shown in Figures 1 to 8 is implemented by software, the software or program code required to execute the functions of the scheduler 300 in the embodiments shown in Figures 1 to 8 is stored in the memory 1002, and the processor 1001 is used to execute the instructions in the memory 1002 to implement the method executed by the scheduler 300.
[0166] It should be understood that the scheduler 1000 according to the embodiment of the present application may correspond to the scheduler 900 in the embodiment of the present application, and may correspond to the scheduler 300 executing the method shown in Figures 1 to 8 in the embodiment of the present application, and the above-mentioned and other operations and / or functions implemented by the scheduler 1000 are respectively for implementing the corresponding processes in the method embodiment shown in Figures 1 to 8. For the sake of brevity, they will not be repeated here.
[0167] In addition, an embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is run on a scheduler, the scheduler executes the method executed by the scheduler 300 in the above embodiment.
[0168] In addition, embodiments of the present application further provide a computer program product. When the computer program product is executed by a scheduler, the one or more schedulers execute any of the aforementioned job scheduling methods. The computer program product may be a software installation package. When any of the aforementioned job scheduling methods is required, the computer program product may be downloaded and executed on a computer.
[0169] It should also be noted that the above-described scheduler embodiment is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided in this application, the connection relationship between the modules indicates that there is a communication connection between them, which can be specifically implemented as one or more communication buses or signal lines.
[0170] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate and are merely used to describe the manner in which objects with the same attributes are described in the embodiments of this application.
[0171] Through the description of the above embodiments, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course can also be implemented by special hardware including application-specific integrated circuits, special CPUs, special memories, special components, etc. In general, all functions performed by computer programs can be easily implemented with corresponding hardware, and the specific hardware structures used to implement the same function can also be diverse, such as analog circuits, digital circuits or special circuits, etc. However, for the present application, software program implementation is a better implementation method in most cases. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer's floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a number of instructions to enable a computer device (which can be a personal computer, training equipment, or network equipment, etc.) to execute the methods described in each embodiment of the present application.
[0172] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).
[0173] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and such modifications or substitutions should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.
Claims
1. A resource scheduling method, characterized in that: The method is applied to a scheduler in a data processing system, wherein the data processing system further comprises a computing cluster and a storage cluster, wherein the computing cluster comprises a plurality of computing nodes, and the computing cluster and the storage cluster are connected via a network; The method comprises: The scheduler obtains data flow description information and a job flow submitted by a user, wherein the job flow includes a plurality of jobs, and the data flow description information is used to indicate a relationship between input data or output data of each of the plurality of jobs; When the input data or output data of a first job among the multiple jobs is stored in a first computing node among the multiple computing nodes, the scheduler schedules a second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, and the second job shares the input data or output data of the first job.
2. The method according to claim 1, characterized in that The relationship between the input data or output data of each of the multiple jobs includes: A dependency relationship between input data or output data of each of the multiple jobs; Alternatively, the sharing relationship between the input data or output data of each job in the multiple jobs.
3. The method according to claim 1 or 2, characterized in that: The scheduler schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, including: When available resources of the first computing node meet resource requirements of the first job and the second job, the scheduler schedules the first job and the second job to the first computing node according to the data flow description information.
4. The method according to any one of claims 1 to 3, characterized in that: The scheduler obtains data flow description information, including: The scheduler schedules a first resource for the plurality of jobs according to the job flow; The scheduler collects operation information generated when the multiple jobs are run based on the first resource, wherein the operation information includes input data and output data of each of the multiple jobs; The scheduler analyzes the operation information to obtain the data flow description information.
5. The method according to any one of claims 1 to 4, characterized in that: The scheduler schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, including: The scheduler generates an inter-job data flow graph XJDFG corresponding to the multiple jobs according to the data flow description information, wherein the XJDFG includes job nodes, data nodes, and connecting lines, wherein the job nodes are used to indicate jobs in the job flow, the data nodes are used to indicate input data or output data of jobs in the job flow, and the connecting lines are used to indicate the relationship between the job nodes and the data nodes; The scheduler schedules the first job and the second job related to input data or output data of the first job to the first computing node according to the XJDFG.
6. The method according to any one of claims 1 to 4, characterized in that: The scheduler obtains data flow description information, including: The scheduler obtains a configuration file submitted by a user and written based on a domain specific language (DSL). The configuration file includes a plurality of configuration items, and the plurality of configuration items are used to configure input data and output data of the plurality of jobs.
7. The method according to any one of claims 1 to 6, characterized in that: The computing cluster further includes a cache pool, the multiple jobs further include a third job, and the method further includes: The scheduler schedules the third job related to the input data or output data of the second job to a second computing node among the plurality of computing nodes according to the data flow description information, and schedules the target cache area in the cache pool to the second job and the third job; The third job shares the input data or output data of the second job through the target cache area.
8. The method according to claim 7, characterized in that The cache pool includes multiple available cache areas, the multiple cache areas include the target cache area, and the sum of the delays of the first computing node and the second computing node accessing the target cache area is less than the sum of the delays of the first computing node and the second computing node accessing other cache areas in the multiple cache areas.
9. A scheduler, characterized in that: The scheduler is applied to a data processing system, the data processing system further comprising a computing cluster and a storage cluster, the computing cluster comprising a plurality of computing nodes, the computing cluster and the storage cluster being connected via a network; The scheduler comprises: An acquisition module, used to acquire data flow description information and a job flow submitted by a user, wherein the job flow includes a plurality of jobs, and the data flow description information is used to indicate the relationship between input data or output data of each job in the plurality of jobs; A scheduling module is used to schedule a second job related to the input data or output data of the first job among the multiple jobs and the first job to the first computing node according to the data flow description information when the input data or output data of the first job among the multiple jobs is stored in the first computing node among the multiple computing nodes, and the second job shares the input data or output data of the first job.
10. The scheduler according to claim 9, characterized in that: The relationship between the input data or output data of each of the multiple jobs includes: A dependency relationship between input data or output data of each of the multiple jobs; Alternatively, the sharing relationship between the input data or output data of each job in the multiple jobs.
11. The scheduler according to claim 9 or 10, characterized in that: When the scheduling module schedules the second job related to the input data or output data of the first job and the first job to the first computing node according to the data flow description information, the scheduling module is specifically used to: When available resources of the first computing node meet resource requirements of the first job and the second job, the first job and the second job are scheduled to the first computing node according to the data flow description information.
12. The scheduler according to any one of claims 9 to 11, characterized in that: When acquiring the data stream description information, the acquisition module is specifically used to: Scheduling a first resource for the plurality of jobs according to the job flow; Collecting operation information generated when the multiple jobs are run based on the first resource, the operation information including input data and output data of each job in the multiple jobs; The operation information is analyzed to obtain the data flow description information.
13. A scheduler, characterized in that: The scheduler includes a processor and a memory, wherein the memory stores program instructions, and the processor runs the program instructions to execute the method according to any one of claims 1 to 8.
14. A data processing system, characterized in that: The data processing system includes a computing cluster, a storage cluster, and a scheduler, and the scheduler is used to execute the method according to any one of claims 1 to 8.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, which, when executed on a scheduler, enable the scheduler to execute the method according to any one of claims 1 to 8.
Citation Information
Patent Citations
Scheduling method, scheduler, storage medium and system
CN110297699A
Resource scheduling method and device and server
CN115113991A
Allocation of cloud computing resources
US20200379813A1
