A batch scheduling system and method

Abstract

The application discloses a batch scheduling system and method, which can be applied to the field of cloud computing or the field of finance. The system comprises a management cluster service module, a process scheduling service module and a job scheduling service module. The management cluster service module is used for obtaining process information of a product, generating an operation request based on the process information, and sending the operation request to the process scheduling service module. The process scheduling service module is used for responding to the operation request, analyzing a process model of the product, obtaining a job set corresponding to the product process, and sending jobs in the job set to a job scheduling service module bound to the product. The job scheduling service module is used for receiving the jobs sent by the process scheduling service module, placing the jobs in a job queue, and in response to a job consumption request sent by a scheduling client, searching for a job corresponding to the job consumption request in the job queue and feeding back the job to the scheduling client. The system can solve the problem of performance decline existing in the existing batch scheduling system and realize dynamic expansion of scheduling capacity.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a batch scheduling system and method. BACKGROUND

[0002] The existing batch scheduling system concentrates all the scheduling functions of the batch flow and batch job of the product used, such as analysis, execution, monitoring, etc. in the scheduling service architecture. With more and more products using scheduling (usually up to hundreds or even thousands), the product flow will reach tens of thousands, and the number of jobs included will reach millions, plus the simple jobs split by the distributed job during execution, the number of jobs required to be executed by the scheduling system every day will reach tens of millions or even hundreds of millions.

[0003] Under the existing batch scheduling system, the scheduling of all product flows and jobs is concentrated in a scheduling cluster and influences each other. When the data volume is large, the performance of the scheduling system will decrease. When the number of products and product flows reaches a certain level, such scheduling system will not be able to meet the scheduling of millions, tens of millions, or even hundreds of millions of flows and jobs, and the scheduling capacity cannot be expanded horizontally with the increase of the products accessed. SUMMARY

[0004] The embodiments of the present application provide a batch scheduling system and method, which can solve the problem of performance decrease caused by large product flow and job data volume in the existing batch scheduling system, and can realize dynamic expansion of scheduling capacity.

[0005] Therefore, the first aspect of the present application provides a batch scheduling system, which comprises a management cluster service module, a flow scheduling service module, at least one job scheduling service module, and a scheduling client.

[0006] The management cluster service module is configured to obtain flow information of a product, generate an operation request based on the flow information, and send the operation request to the flow scheduling service module.

[0007] The flow scheduling service module is configured to respond to the operation request, analyze a flow model of the product, obtain a job set corresponding to the flow of the product, and send jobs in the job set to the job scheduling service module bound to the product.

[0008] The job scheduling service module is configured to receive the jobs sent by the flow scheduling service module, place the jobs in a job queue, respond to a job consumption request sent by the scheduling client, find the jobs corresponding to the job consumption request in the job queue, and feed back the jobs to the scheduling client, so that a job execution flow associated with the scheduling client executes the jobs.

[0009] Optionally, the job scheduling service module is further configured to:

[0010] after placing the job in the job queue, set a job state corresponding to the job as to-be-executed;

[0011] after feeding back the job to the scheduling client, set the job state corresponding to the job as being executed;

[0012] after receiving the job execution result corresponding to the job returned by the scheduling client, set the job state corresponding to the job as successful; and send the execution success notification information corresponding to the job to the process scheduling service module, so that the process scheduling service module sends the next job in the job set to the job scheduling service module.

[0013] Optionally, the job scheduling service module is further configured to:

[0014] when the job is a distributed job, split the distributed job into a plurality of simple jobs, and place the plurality of simple jobs in the job queue respectively;

[0015] when the job state corresponding to each of the plurality of simple jobs is set as execution success, send the execution success notification information corresponding to the job to the process scheduling service module.

[0016] Optionally, the process scheduling service module is further configured to:

[0017] set a process state corresponding to the process according to the job state corresponding to the job in the job set of the process, and feed back the process state to the management cluster service module.

[0018] Optionally, the system further comprises a timing cluster service module;

[0019] the timing cluster service module is configured to receive the timing information of the product sent by the management service cluster module, and send the operation request to the process scheduling service module according to the timing information.

[0020] The second aspect of the present application provides a batch scheduling method, which is applied to a batch scheduling system, the system comprising a management cluster service module, a process scheduling service module, at least one job scheduling service module, and a scheduling client; the method comprising:

[0021] the management cluster service module acquires process information of a product, generates an operation request based on the process information, and sends the operation request to the process scheduling service module;

[0022] The flow scheduling service module parses the flow model of the product in response to the operation request, obtains a job set corresponding to the flow of the product, and sends a job in the job set to a job scheduling service module bound to the product;

[0023] The job scheduling service module receives the job sent by the flow scheduling service module, places the job in a job queue, and in response to a job consumption request sent by the scheduling client, finds a job corresponding to the job consumption request in the job queue and feeds back the job to the scheduling client, so that a job execution flow associated with the scheduling client executes the job.

[0024] Optionally, the method further comprises:

[0025] After the job scheduling service module places the job in the job queue, the job scheduling service module sets a job state corresponding to the job to be executed;

[0026] After the job scheduling service module feeds back the job to the scheduling client, the job scheduling service module sets the job state corresponding to the job to be executed;

[0027] After the job scheduling service module receives the job execution result corresponding to the job returned by the scheduling client, the job scheduling service module sets the job state corresponding to the job to execution success, and sends execution success notification information corresponding to the job to the flow scheduling service module, so that the flow scheduling service module sends the next job in the job set to the job scheduling service module.

[0028] Optionally, the method further comprises:

[0029] When the job is a distributed job, the job scheduling service module splits the distributed job into a plurality of simple jobs, and places the plurality of simple jobs in the job queue respectively;

[0030] When the job state corresponding to each of the plurality of simple jobs is set to execution success, the job scheduling service module sends execution success notification information corresponding to the job to the flow scheduling service module.

[0031] Optionally, the method further comprises:

[0032] The flow scheduling service module sets a flow state corresponding to the flow according to the job state corresponding to the job in the job set corresponding to the flow, and feeds back the flow state to the management cluster service module.

[0033] Optionally, the batch scheduling system further comprises a timing cluster service module; the method further comprises:

[0034] The timing cluster service module receives the timing information of the product sent by the management service cluster module, and sends the operation request to the flow scheduling service module according to the timing information.

[0035] From the above technical solutions, the embodiments of the present application have the following advantages:

[0036] The embodiments of the present application provide a batch scheduling system, which comprises a management cluster service module, a flow scheduling service module, at least one job scheduling service module and a scheduling client; wherein the management cluster service module is used for obtaining the flow information of a product, generating an operation request based on the flow information, and sending the operation request to the flow scheduling service module; the flow scheduling service module is used for responding to the operation request, analyzing the flow model of the product, obtaining the job set corresponding to the flow of the product, and sending the jobs in the job set to the job scheduling service module bound to the product; the job scheduling service module is used for receiving the jobs sent by the flow scheduling service module, and placing the jobs in a job queue; in response to the job consumption request sent by the scheduling client, the job corresponding to the job consumption request is searched in the job queue, and the job is fed back to the scheduling client, so that the job execution process of the scheduling client associated job executes the job. The embodiments of the present application provide a two-level scheduling system which can realize scheduling product isolation and dynamically expand scheduling capability. The scheduling system is divided into flow scheduling and job scheduling, and the scheduling of product flow and the scheduling of product job are physically isolated. The job scheduling of products is physically isolated according to the difference of the job scheduling service module, so that the job scheduling of products bound to different job scheduling service modules does not affect each other, thereby solving the problem that the scheduling performance of each other and the low performance problem caused by the large amount of data of product flow and product job under the existing scheduling architecture. In addition, the scheduling capability of the system can be dynamically expanded with the increase of the hardware resources of the job scheduling service module, solving the problem that the existing scheduling system cannot be horizontally expanded, realizing the improvement of the scheduling capability and performance of the scheduling system, and making the scheduling system have the ability of flexible expansion. BRIEF DESCRIPTION OF DRAWINGS

[0037] Figure 1 The structure schematic diagram of the batch scheduling system provided by the embodiments of the present application is shown;

[0038] Figure 2 The working principle schematic diagram of the batch scheduling system provided by the embodiments of the present application is shown;

[0039] Figure 3 The flow schematic diagram of the batch scheduling method provided by the embodiments of the present application is shown. DETAILED DESCRIPTION

[0040] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0041] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present application and above drawings, if any, are used for distinguishing between similar objects, and do not necessarily have a particular order or sequence. It should be understood that the data thus used are interchangeable, so that the embodiments of the present application described herein can be practiced in other than the order described herein without departing from the scope of the present application. Furthermore, the terms "comprise" and "have" and any variations thereof are intended to cover a non-exclusive inclusion, for example, a process, method, article, or apparatus that comprises a list of steps or units is not necessarily limited to those steps or units that are clearly listed, but can include other steps or units that are not clearly listed or inherent to such processes, methods, articles, or apparatus.

[0042] It should be noted that the batch scheduling system and method provided by the present application can be used in the field of cloud computing or the field of finance. The above is only an example, and does not limit the application field of the batch scheduling system and method provided by the present application.

[0043] The batch scheduling system provided by the present application will be introduced below through system embodiments.

[0044] Referring to Figure 1 , Figure 1 The structure schematic diagram of the batch scheduling system provided by the present application is shown in the embodiment of the present application. As Figure 1 shown, the batch scheduling system comprises a management cluster service module 101, a process scheduling service module 102, at least one job scheduling service module 103, and a scheduling client 104. Optionally, the batch scheduling system can further comprise a timing cluster service module 105. The management cluster service module can also be referred to as an admin cluster service module, and the timing cluster service module can also be referred to as a timer cluster service module.

[0045] It should be noted that the batch scheduling system usually includes a process scheduling service module 102, and the job scheduling service module 103 can be expanded to multiple according to actual needs (such as product quantity). The process scheduling service module 102 includes a set of process scheduling clusters and a set of process databases, and the process databases are used to store product process information using the batch scheduling system and process scheduling functions such as process start, process model analysis, process execution record query. Each job scheduling service module 103 includes a set of job scheduling clusters and its corresponding job database; the product and the job scheduling service module 103 are in a N:1 relationship, N is an integer greater than or equal to 1; that is, one product can only be bound to one job scheduling service module, and one job scheduling service module can have job information of multiple products, and each set of job scheduling clusters and the corresponding job database is responsible for storing the job information of the product bound thereto and the job scheduling functions such as splitting and merging, execution, skipping, re-picking, result feedback, job execution record query.

[0046] The reason why the two-level scheduling architecture is proposed in the embodiment of the application is that the number of batch processes under one product is usually within 100 (a product usually includes only 10 or fewer batch processes), and the scheduling system faces several hundred products, and the number of processes involved in scheduling is within the 100,000 level. After splitting into a two-level scheduling architecture, the scheduling of jobs and the scheduling of processes are isolated from each other, and interact through the job execution results reported by the job scheduling service module. Based on this, a set of process scheduling clusters is sufficient to meet the scheduling performance requirements of all product processes. A process is usually composed of tens, hundreds or even thousands of jobs, and some of the jobs are distributed jobs, which can be split into thousands of simple jobs in the face of hundreds of libraries and thousands of tables. Therefore, the job data amount of the several hundred products finally reached can reach the level of ten million or even one hundred million, and the several job scheduling service modules owned by the two-level scheduling system can undertake the job of one hundred million level, and through the job data isolation between the job scheduling service modules, the job scheduling of the products of different job scheduling clusters does not affect the performance, and as the number of products using the scheduling system increases, the job scheduling cluster can be dynamically expanded to bind new products, and the scheduling capability of the system can be dynamically expanded with the increase of hardware resources.

[0047] The functions of each module in the batch scheduling system provided by the embodiment of the application will be introduced below.

[0048] Among them, the management cluster service module 101 is used to obtain the process information of the product, and generate an operation request based on the process information, and send the operation request to the process scheduling service module 102.

[0049] In the embodiment of the present application, the management cluster service module 101 provides a front-end page of the scheduling system, on which the process of each product can be imported or exported, or the operation of adding, modifying, deleting, arranging, etc. of the process job can be directly performed on the page. Then the product batch process can be started manually through the page, or the timing process can be set.

[0050] If the management cluster service module 101 detects that the relevant personnel start the product batch process through the page, the management cluster service module 101 can obtain the process information of the product (such as product information, process name, process model, etc.), and generate an operation request according to the obtained process information, and send the operation request to the process scheduling service module 102, so that the process scheduling service module 102 persists the process information carried by the operation request to the process database.

[0051] If the management cluster service module 101 detects that the relevant personnel set the timing process, the timing information of the product (such as the timing execution time and the timing execution period of the process of the product, etc.) can be sent to the timing cluster service module 105, so that the timing cluster service module 105 generates an operation request according to the process information of the product according to the timing information, and sends the operation request to the process scheduling service module 102. More specifically, the timing cluster service module 105 can receive the timing information sent by the management cluster service module 101 and store the timing information, so as to provide the functions of scheduling process timing setting, periodic timing triggering process, etc. according to the timing information, that is, sending an operation request to the process scheduling service module 102 at the process execution time indicated by the timing information.

[0052] Among them, the process scheduling service module 102 is configured to respond to the received operation request (including the operation request sent by the management cluster service module 101 and the operation request sent by the timing cluster service module 105), analyze the process model of the product, obtain the job set corresponding to the process of the product, and send the jobs in the job set to the job scheduling service module 103 bound to the product.

[0053] Specifically, the process scheduling service module 102 receives the operation request sent by the management cluster service module 101 or the timing cluster service module 105, correspondingly stores the process information (including product process definition information and process execution information) carried by the operation request, and analyzes the process model of the product, and obtains the job set corresponding to the process of the product, which includes a plurality of jobs arranged in a process order. The analyzed jobs will be sent to the job scheduling service module 103 bound to the product in the process order.

[0054] Further, the process scheduling service module 102 can also receive the job state fed back by the job scheduling service module 103, which can be understood as the job execution result, and then set the process state corresponding to the process according to the job state corresponding to each job in the job set of the process, and feed back the process state to the management cluster service module 101. Specifically, the job scheduling service module 103 can report the job state corresponding to each job in the job set to the process scheduling service module 102, and accordingly, the process scheduling service module 102 can adjust the process state corresponding to the process according to the job state corresponding to each job, for example, switch among the states of to-be-executed, executing, paused, failed, and succeeded, and at the same time, the process scheduling service module will also feed back the process state to the management cluster service module 101, so that the management cluster service module 101 displays the process state on the front-end page.

[0055] The job scheduling service module 103 is configured to receive the job sent by the process scheduling service module 102, place the job in a job queue, and in response to the job consumption request sent by the scheduling client 104, find the job corresponding to the job consumption request in the job queue, and feed back the found job to the scheduling client 104, so that the job execution process associated with the scheduling client 104 executes the job.

[0056] Specifically, the job scheduling service module 103 can receive the job sent by the process scheduling service module 102, and store the job definition information and job execution information corresponding to the job. In addition, the job scheduling service module 103 can also judge whether the job is a distributed job; if not, the job scheduling service module 103 can directly add the job to the job queue; if it is a distributed job, it can split the job into a corresponding number of simple jobs according to the splitting logic provided by the scheduling client 104, and then add the multiple simple jobs obtained by splitting to the job queue respectively. The jobs placed in the job queue can be consumed by the corresponding scheduling client 104, and receive the job execution result reported by the scheduling client 104, so that the job scheduling service module 103 adjusts the job state corresponding to the job according to the job execution result, for example, switches among the states of to-be-executed, executing, failed, and succeeded, and then feeds back the job state to the process scheduling service module 102.

[0057] More specifically, the job scheduling service module 103 can set the job state in the following ways: after the job scheduling service module 103 places a job in the job queue, the job scheduling service module 103 can set the job state corresponding to the job as pending execution; in response to the job consumption request sent by the scheduling client 104, after the job scheduling service module 103 feeds back the job to the scheduling client 104, the job scheduling service module 103 can set the job state corresponding to the job as executing; after the job scheduling service module 103 receives the job execution result corresponding to the job returned by the scheduling client 104, the job scheduling service module 103 can set the job state corresponding to the job as successful, and send an execution success notification message corresponding to the job to the process scheduling service module 102. After the process scheduling service module 102 receives the execution success notification message corresponding to the job, the process scheduling service module 102 can send the next job in the job set to the job scheduling service module 103, so that the job scheduling service module 103 performs scheduling work corresponding to the job.

[0058] It should be understood that for a distributed job, it will be split by the job scheduling service module 103 into multiple simple jobs, and the multiple simple jobs will be placed in the job queue respectively. Accordingly, when the job state corresponding to each of the multiple simple jobs is set as execution success, it can be considered that the distributed job composed of the multiple simple jobs is successfully executed, at which time the execution success notification information corresponding to the distributed job can be sent to the process scheduling service module 102.

[0059] The scheduling client 104 is a client required by each product batch application, which is used to implement specific distributed splitting logic and determine the number of simple jobs that can be split from the distributed job of the product itself. The scheduling client 104 polls the job scheduling service module 103 to send a job consumption request to pull the batch job of itself from the job scheduling service module 103, and executes the specific business of the job, and reports the job execution result to the job scheduling service module 103 after execution is completed.

[0060] The embodiments of the present application provide a two-level scheduling system capable of realizing scheduling product isolation and dynamically expanding scheduling capability, which is divided into process scheduling and job scheduling, and the scheduling of product processes and the scheduling of product jobs are physically isolated. The job scheduling of products is physically isolated according to different job scheduling service modules, so that the job scheduling of products bound to different job scheduling service modules does not affect each other, thereby solving the problem that the scheduling performance of each other is affected and the performance is low due to the large amount of data of product processes and product jobs under the existing scheduling architecture. In addition, the system can also dynamically expand the scheduling capability with the increase of the hardware resources of the job scheduling service module, solve the problem that the existing scheduling system cannot be horizontally expanded, realize the improvement of the scheduling capability and performance of the scheduling system, and make the scheduling system have the ability of flexible expansion.

[0061] To facilitate understanding of the working principle of the batch scheduling system provided in the embodiments of this application, the following is combined with... Figure 2 The diagram shown illustrates the working principle of the batch scheduling system, and an example is provided to explain the working principle of the batch scheduling system.

[0062] like Figure 2 As shown, assume that products A, B, and C comprise a total of 20 processes and 200,000 jobs (including distributed jobs, which, after being split by the job scheduling service module, amount to over 2 million jobs), and are bound to the job scheduling cluster JOB_CLUSTER_1. Since a job scheduling cluster can only schedule 3 million jobs at most, for the newly added products D, E, and F, which comprise a total of 18 processes and 150,000 jobs (the distributed jobs, after being split into simple jobs, amount to nearly 2 million jobs), the scheduling system adds a job scheduling cluster JOB_CLUSTER_2 to schedule products D, E, and F.

[0063] 1. Products A and D import their respective processes flow_a and flow_d through the admin front end, and set the processes to be executed on a scheduled basis or to be initiated manually.

[0064] 2. When importing a process, the admin frontend sends the process information (product information, process name, process model, etc.) to the process scheduling service module, which then stores the process information. When the admin frontend sets a process to be executed on a scheduled basis, the timer cluster service module stores the scheduled task information (scheduled task name, process name, scheduling period, execution time, etc.).

[0065] 3. When a process is manually initiated, the admin frontend sends an operation request to the process scheduling service module. When a process is initiated on a schedule, a timer is triggered, sending the operation request to the process scheduling service module.

[0066] 4. Upon receiving the operation requests from flow_a and flow_d, the process scheduling service module sets the status of flow_a and flow_d to pending execution, parses the process models of flow_a and flow_d, extracts the corresponding job sets, and sends the information (product name, process name, job name, job type, etc.) of the first job in the job set corresponding to flow_a (job_a_1) to the job scheduling service module JOB_CLUSTER_1, and sends the information of the first job in the job set corresponding to flow_d (job_d_1) to the job scheduling service module JOB_CLUSTER_2, according to the job execution order in the process.

[0067] 5、Job Cluster 1 receives the job scheduling request sent by the flow scheduling service module, saves the job information of job a 1 to the job queue, and sets the job state of job a 1 to be pending. Similarly, Job Cluster 2 performs similar operations on the job of product D, job d 1.

[0068] 6、The scheduling client of product A sends a job consumption request to Job Cluster 1, and Job Cluster 1 returns job a 1 as the request parameter to the scheduling client of product A, and sets the job state of job a 1 to be executing, and sends a request notification to the flow scheduling service module, and the flow scheduling service module sets the flow state of flow a to be executing.

[0069] 7、After the scheduling client of product A pulls job a 1, the scheduling client executes the specific business logic of job a 1, and after the execution is completed, the scheduling client reports the job execution result to Job Cluster 1, and Job Cluster 1 sets the job state of job a 1 to be successful, and sends a notification to the flow scheduling service module, and the flow scheduling service module sends the next job job a 2 to Job Cluster 1 again, and the operation returns to step 5 until the last job of flow a is executed, and the flow scheduling service module sets the flow state of flow a to be successful.

[0070] 8、If job a 1 is a distributed job, Job Cluster 1 needs to split the job into 1000 simple jobs (assuming that the splitting concurrency is 1000), and after the 1000 simple jobs are stored and executed, the job state of the distributed job job a 1 is updated to be successful, and a notification is sent to the flow scheduling cluster service to obtain the next job. (The execution process of the flow job of product D is consistent with that of product A, and the execution is completed in the interaction between the scheduling client of product D and Job Cluster 2 and the flow scheduling service module of Job Cluster 2 and the flow scheduling service module).

[0071] The embodiments of the present application also provide a batch scheduling method, which is described below with reference to Figure 3 , Figure 3 A flowchart of a batch scheduling method provided by the embodiments of the present application is shown in FIG. 1. The method is applied to the batch scheduling system described above. As shown in FIG. 1, the method comprises the following steps. Figure 3 ​

[0072] Step 301: The management cluster service module acquires the flow information of a product, and generates an operation request based on the flow information, and sends the operation request to the flow scheduling service module;

[0073] Step 302: The flow scheduling service module responds to the operation request, parses the flow model of the product, obtains a job set corresponding to the flow of the product, and sends jobs in the job set to a job scheduling service module bound to the product;

[0074] Step 303: The job scheduling service module receives the jobs sent by the flow scheduling service module, and places the jobs in a job queue; in response to a job consumption request sent by the scheduling client, the job scheduling service module searches for a job corresponding to the job consumption request in the job queue, and feeds back the job to the scheduling client, so that a job execution flow associated with the scheduling client executes the job.

[0075] Optionally, the method further comprises:

[0076] After the job scheduling service module places the jobs in the job queue, the job scheduling service module sets a job state corresponding to the jobs as to-be-executed;

[0077] After the job scheduling service module feeds back the jobs to the scheduling client, the job scheduling service module sets the job state corresponding to the jobs as executing;

[0078] After the job scheduling service module receives a job execution result corresponding to the jobs returned by the scheduling client, the job scheduling service module sets the job state corresponding to the jobs as execution success; and the job scheduling service module sends execution success notification information corresponding to the jobs to the flow scheduling service module, so that the flow scheduling service module sends a next job in the job set to the job scheduling service module.

[0079] Optionally, the method further comprises:

[0080] When the job is a distributed job, the job scheduling service module splits the distributed job into a plurality of simple jobs, and places the plurality of simple jobs in the job queue respectively;

[0081] When the job state corresponding to each of the plurality of simple jobs is set as execution success, the job scheduling service module sends execution success notification information corresponding to the jobs to the flow scheduling service module.

[0082] Optionally, the method further comprises:

[0083] The process scheduling service module sets a process state corresponding to the process according to a job state corresponding to a job in a job set corresponding to the process, and feeds back the process state to the management cluster service module.

[0084] Optionally, the batch scheduling system further comprises a timing cluster service module; and the method further comprises:

[0085] The timing cluster service module receives the timing information of the product sent by the management service cluster module, and sends the operation request to the process scheduling service module according to the timing information.

[0086] The embodiments of the present application provide a two-level scheduling method capable of realizing scheduling product isolation and dynamically expanding scheduling capability. The scheduling method is divided into process scheduling and job scheduling, and the scheduling of product processes and the scheduling of product jobs are physically isolated. The job scheduling between products is physically isolated according to different job scheduling service modules, so that the job scheduling of products bound to different job scheduling service modules does not affect each other, thereby solving the problem that the scheduling performance of each other is affected and the performance is low due to the large amount of data of product processes and product jobs under the existing scheduling architecture. In addition, the method can also dynamically expand the scheduling capability with the increase of the hardware resources of the job scheduling service module, solves the problem that the existing scheduling system cannot be horizontally expanded, realizes the improvement of the scheduling capability and performance of the scheduling system, and enables the scheduling system to have the ability of flexible expansion.

[0087] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0088] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, another division mode can be used. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0089] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0090] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0091] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (English full name: Read-Only Memory, English abbreviation: ROM), a random access memory (English full name: Random Access Memory, English abbreviation: RAM), a magnetic disk or an optical disk, and various computer program storage media.

[0092] It should be understood that in the present application, "at least one" means one or more, and "multiple" means two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0093] The above-described embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A batch scheduling system, characterized by, The system comprises a management cluster service module, a process scheduling service module, at least one job scheduling service module, and a scheduling client; The management cluster service module is configured to obtain process information of a product, generate an operation request based on the process information, and send the operation request to the process scheduling service module; The process scheduling service module is configured to respond to the operation request, parse a process model of the product, obtain a job set corresponding to the process of the product, and send a job in the job set to a job scheduling service module bound to the product; The job scheduling service module is configured to receive the job sent by the process scheduling service module, place the job in a job queue, respond to a job consumption request sent by the scheduling client, find a job corresponding to the job consumption request in the job queue, and feed back the job to the scheduling client, so that a job execution process associated with the scheduling client executes the job, the job scheduling service module is physically isolated from the process scheduling service module, and the job scheduling service module can be dynamically expanded according to an increase in the number of products to obtain multiple job scheduling service modules, and job data between the multiple job scheduling service modules are isolated from each other.

2. The system of claim 1, wherein, The job scheduling service module is further configured to: After placing the job in the job queue, set a job state corresponding to the job to be executed; After feeding back the job to the scheduling client, set the job state corresponding to the job to be executed; After receiving a job execution result corresponding to the job returned by the scheduling client, set the job state corresponding to the job to be successful; and send execution success notification information corresponding to the job to the process scheduling service module, so that the process scheduling service module sends a next job in the job set to the job scheduling service module.

3. The system of claim 2, wherein, The job scheduling service module is further configured to: When the job is a distributed job, split the distributed job into multiple simple jobs, and place the multiple simple jobs in the job queue respectively; When the job state corresponding to each of the multiple simple jobs is set to execution success, send execution success notification information corresponding to the job to the process scheduling service module.

4. The system of claim 1 or 2, wherein, The process scheduling service module is further configured to: According to a job state corresponding to a job in the job set corresponding to the process, set a process state corresponding to the process, and feed back the process state to the management cluster service module.

5. The system of claim 1, wherein, The system further comprises a timing cluster service module; The timing cluster service module is configured to receive timing information of the product sent by the management cluster service module, and send the operation request to the process scheduling service module according to the timing information.

6. A method of batch scheduling, the method comprising: The method is applied to a batch scheduling system, the system comprises a management cluster service module, a process scheduling service module, at least one job scheduling service module, and a scheduling client, and the method comprises: The management cluster service module acquires process information of a product, and generates an operation request based on the process information, and sends the operation request to the process scheduling service module; The process scheduling service module responds to the operation request, parses a process model of the product, obtains a job set corresponding to the process of the product, and sends jobs in the job set to a job scheduling service module bound to the product; The job scheduling service module receives the jobs sent by the process scheduling service module, and places the jobs in a job queue; in response to a job consumption request sent by the scheduling client, the job scheduling service module searches the job queue for a job corresponding to the job consumption request, and feeds back the job to the scheduling client, so that a job execution process associated with the scheduling client executes the job. The job scheduling service module is physically isolated from the process scheduling service module. The job scheduling service module can be dynamically expanded according to the increase in the number of products, to obtain multiple job scheduling service modules. The job data between the multiple job scheduling service modules are mutually isolated.

7. The method of claim 6, wherein, The method further comprises: After the job scheduling service module places the job in the job queue, the job scheduling service module sets a job state corresponding to the job to be executed; After the job scheduling service module feeds back the job to the scheduling client, the job scheduling service module sets the job state corresponding to the job to be executed; After the job scheduling service module receives the job execution result corresponding to the job returned by the scheduling client, the job scheduling service module sets the job state corresponding to the job to be executed successfully; and the job scheduling service module sends execution success notification information corresponding to the job to the process scheduling service module, so that the process scheduling service module sends a next job in the job set to the job scheduling service module.

8. The method of claim 7, wherein, The method further comprises: When the job is a distributed job, the job scheduling service module splits the distributed job into multiple simple jobs, and places the multiple simple jobs in the job queue respectively; When the job state corresponding to each of the multiple simple jobs is set to be executed successfully, the job scheduling service module sends execution success notification information corresponding to the job to the process scheduling service module.

9. The method according to claim 6 or 7, characterized in that, The method further comprises: The process scheduling service module sets a process state corresponding to the process according to the job state corresponding to the job in the job set corresponding to the process, and feeds back the process state to the management cluster service module.

10. The method of claim 6, wherein, The batch scheduling system further comprises a timing cluster service module; and the method further comprises: The timing cluster service module receives the timing information of the product sent by the management cluster service module, and sends the operation request to the process scheduling service module according to the timing information.

Images