A job scheduling method and device based on distributed cache

Abstract

The present invention provides a method and device for job scheduling based on distributed cache, which can be used in the financial field. The method includes: generating a job execution plan according to the job configuration of the job to be executed in the cluster; searching for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache; judging whether the job that meets the execution conditions is executable; if so, executing the job, and updating the execution status of the job in the distributed cache to successful execution. The present application realizes the logical and physical isolation of the job scheduling of each service cluster from the job management cluster. The scheduling of the job is triggered by the service cluster itself, there is no single point performance bottleneck, the concurrency of job scheduling is improved, the scheduling dependence of each service cluster on the management cluster is eliminated, and the high availability of job scheduling is guaranteed.

Description

Technical Field

[0001] The present application belongs to the field of distributed technology, and specifically, relates to a job scheduling method and device based on distributed cache. Background Art

[0002] At present, the scheduling of batch jobs in distributed systems is mainly centralized management mode. That is, a centralized management cluster generates a job execution plan based on pre-configured parameters, and then continuously scans the information of the jobs to be executed, and determines whether each job meets the execution requirements based on trigger conditions, dependencies, etc. If so, it sends a scheduling instruction to the job executor of the service cluster. However, this mode has the following disadvantages:

[0003] 1. Due to the single cluster management and scheduling method, the concurrency of job scheduling has a performance bottleneck and cannot meet the requirements of massive job scheduling in distributed systems.

[0004] 2. Job scheduling relies on a centralized management cluster. If the cluster fails, job scheduling of all service clusters cannot be executed, which does not meet the high availability requirements of the distributed system. Summary of the invention

[0005] The present application provides a job scheduling method and device based on distributed cache, so as to at least solve the problem that the scheduling of batch jobs in the current distributed system cannot meet the requirements of massive job scheduling and the job scheduling depends on the centralized management cluster. If the cluster is abnormal, the job scheduling of all service clusters cannot be executed.

[0006] According to a first aspect of the present application, a distributed cache-based job scheduling method is provided, comprising:

[0007] Generate a job execution plan based on the job configuration of the job to be executed in the cluster;

[0008] Find the job that meets the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache;

[0009] Determine whether the job that meets the execution conditions is executable;

[0010] If yes, the job is executed, and the execution status of the job is updated in the distributed cache as successful execution.

[0011] In one embodiment, determining whether a job that meets the execution condition is executable includes:

[0012] Add distributed transaction locks to jobs that meet the execution conditions in the distributed cache;

[0013] If the lock fails, the job is abandoned;

[0014] If the lock is successful, the job is executed.

[0015] In one embodiment, the distributed cache-based job scheduling method further includes:

[0016] If the lock is successful, the execution status of the job in the distributed cache is updated to processing before the job is executed.

[0017] In one embodiment, generating a job execution plan according to the acquired job configuration of the job to be executed in the cluster includes:

[0018] Get the job configuration of the job to be executed in this cluster;

[0019] Get the previous dependent jobs and job execution time of the job according to the job configuration;

[0020] Generate a job execution plan based on the previous dependent jobs and job execution time.

[0021] In one embodiment, the distributed cache-based job scheduling method further includes:

[0022] If the job execution fails, the job execution status in the distributed cache is updated to execution failure, and an alarm is issued.

[0023] According to a second aspect of the present application, a job scheduling device based on distributed cache is also provided, comprising:

[0024] A job execution plan generating unit, used to generate a job execution plan according to the acquired job configuration of the job to be executed in the cluster;

[0025] A search unit, used to search for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache;

[0026] A judgment unit, used to judge whether a job that meets the execution condition is executable;

[0027] The execution unit is used for executing the job if yes, and updating the execution status of the job in the distributed cache as execution success.

[0028] In one embodiment, the determining unit includes:

[0029] A locking module, used to add distributed transaction locks to jobs that meet the execution conditions in the distributed cache;

[0030] The lock failure module is used to abandon the execution of the job if the lock fails;

[0031] The lock success module is used to execute the job if the lock is successful.

[0032] In one embodiment, the distributed cache-based job scheduling device further includes:

[0033] The first status update module is used to update the execution status of the job in the distributed cache to being processed if the lock is successful, and then execute the job.

[0034] In one embodiment, the job execution plan generating unit includes:

[0035] The acquisition module is used to obtain the job configuration of the job to be executed in this cluster;

[0036] A content acquisition module is used to obtain the previous dependent jobs and job execution time of the job according to the job configuration;

[0037] The execution plan generation module is used to generate a job execution plan based on the previous dependent jobs and job execution time.

[0038] In one embodiment, the distributed cache-based job scheduling device further includes:

[0039] The second status update module is used to update the job execution status in the distributed cache to execution failure and issue an alarm if the job execution fails.

[0040] According to the third aspect of the present application, an electronic device is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the program, the steps of the job scheduling method based on distributed cache are implemented.

[0041] According to a fourth aspect of the present application, a computer-readable storage medium is also provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the job scheduling method based on distributed cache are implemented.

[0042] As can be seen from the above technical scheme, the present application provides a method and device for job scheduling based on distributed cache, the method comprising: generating a job execution plan according to the job configuration of the job to be executed in the cluster obtained; searching for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache; judging whether the job that meets the execution conditions is executable; if so, executing the job, and updating the execution status of the job in the distributed cache to successful execution. By using the distributed public cache as an information sharing carrier for job execution, the service cluster determines whether the batch job needs to be started and executed by itself, and judges the triggering conditions such as the execution plan time, whether the previous job has been executed, and whether the current job has been executed by the execution status of the previous and next dependent jobs of the job. If the execution conditions are met, the job execution status in the distributed cache is updated to being executed, and the job is executed, and the execution result updated after the execution is completed is stored in the distributed cache, thereby realizing the logical and physical isolation of the job scheduling of each service cluster from the job management cluster, the scheduling of the job is triggered by the service cluster itself, there is no single point performance bottleneck, the job scheduling concurrency is improved, the scheduling dependence of each service cluster on the management cluster is eliminated, and the high availability of job scheduling is guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A flow chart of a job scheduling method based on distributed cache provided in this application.

[0045] Figure 2 This is a flowchart for determining whether a job that meets the execution conditions is executable in an embodiment of the present application.

[0046] Figure 3 A flowchart for generating a job execution plan in an embodiment of the present application.

[0047] Figure 4 This is a connection diagram of a distributed system in an embodiment of the present application.

[0048] Figure 5 A structural block diagram of a job scheduling device based on distributed cache provided in this application.

[0049] Figure 6 It is a structural block diagram of the judgment unit in the embodiment of the present application.

[0050] Figure 7 This is a structural block diagram of the job execution plan generation unit in the embodiment of the present application.

[0051] Figure 8 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] It should be noted that the job scheduling method and device based on distributed cache disclosed in this application can be applied to the financial field, and can also be used in other fields besides the financial field. This application does not limit the application field of the disclosed job scheduling method and device based on distributed cache.

[0054] At present, the scheduling of batch jobs in distributed systems is mainly centralized management mode. That is, a centralized management cluster generates a job execution plan through pre-configured parameters, and then continuously scans the information of the jobs to be executed, and determines whether each job meets the execution requirements based on trigger conditions, dependencies, etc. If so, a scheduling instruction is initiated to the job executor of the service cluster. However, this mode has disadvantages. For example, due to the single cluster management and scheduling method, the concurrency of job scheduling has a performance bottleneck and cannot meet the requirements of massive job scheduling in distributed systems. In addition, job scheduling depends on the centralized management cluster. If the cluster is abnormal, the job scheduling of all service clusters cannot be executed, which does not meet the high availability requirements of distributed systems.

[0055] Based on this, the present application provides a job scheduling method based on distributed cache, a job scheduling device based on distributed cache, an electronic device and a computer-readable medium, respectively. By using the distributed public cache as an information sharing carrier for job execution, the service cluster determines whether the batch job needs to be started and executed by itself, and determines the triggering conditions such as the execution plan time, whether the previous job has been executed, and whether the current job has been executed through the execution status of the previous and subsequent dependent jobs of the job. If the execution conditions are met, the job execution status in the distributed cache is updated to being executed, and the job is executed, and the updated execution result after the execution is completed is stored in the distributed cache, thereby realizing the logical and physical isolation of the job scheduling of each service cluster from the job management cluster. The job scheduling is triggered by the service cluster itself, and there is no single-point performance bottleneck, which improves the job scheduling concurrency, eliminates the scheduling dependence of each service cluster on the management cluster, and ensures the high availability of job scheduling.

[0056] Based on the above content, the present application also provides a distributed cache-based job scheduling device for implementing the distributed cache-based job scheduling method provided in one or more embodiments of the present application. The device can communicate with the client device by itself or through a third-party server, and return the execution result to the client.

[0057] It is understandable that the client device may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0058] In another practical application scenario, the part of the distributed cache-based job scheduling performed by the aforementioned distributed cache-based job scheduling device can be executed in a server as described above, or all operations can be completed in the client device. The specific selection can be made based on the processing capability of the client device and the limitations of the user's usage scenario. This application does not limit this. If all operations are completed in the client device, the client device may also include a processor for specific processing of distributed cache-based job scheduling.

[0059] The client device may have a communication module (i.e., a communication unit) that can communicate with a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and other implementation scenarios may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or a server cluster consisting of multiple servers, or a server structure of a distributed device.

[0060] The server and the client device may communicate with each other using any suitable network protocol, including network protocols that have not yet been developed on the date of filing this application. The network protocols may include, for example, TCP / IP, UDP / IP, HTTP, HTTPS, etc. Of course, the network protocols may also include, for example, RPC (Remote Procedure Call Protocol) and REST (Representational State Transfer) protocols used on top of the above protocols.

[0061] The details are described in detail through the following embodiments and application examples.

[0062] This application provides a job scheduling method based on distributed cache, such as Figure 1 As shown, including:

[0063] S101: Generate a job execution plan according to the acquired job configuration of the job to be executed in the cluster.

[0064] S102: searching for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache.

[0065] S103: Determine whether the job that meets the execution condition is executable.

[0066] S104: If yes, execute the job, and update the execution status of the job in the distributed cache to successful execution.

[0067] In the batch job execution window, each container in the service cluster scans each job in the execution plan and determines whether it is executable based on information such as system time and the execution status of other jobs in the distributed cache. For executable jobs, the execution status is updated in the distributed cache to "processing" and execution begins. Finally, after the job execution is completed, the execution result is updated to the distributed cache. Figure 4 The figure shows a connection diagram of a distributed system.

[0068] In a specific embodiment, in a distributed application system, the job configuration information of batch jobs of different service clusters, such as names, scheduling conditions, dependencies between predecessors and successors, and execution clusters, is stored in a distributed cache; at the same time, each service cluster can query the job execution status of other clusters on the distributed cache. When the batch job execution cycle starts, each cluster reads the job configuration from the distributed cache and generates a batch execution plan for the service cluster and persists it in a local database or file. The execution plan includes information such as the execution time of each batch job, the predecessor-dependent job, etc.

[0069] In one embodiment, it is determined whether a job that meets the execution condition is executable, such as Figure 2 As shown, including:

[0070] S201: Adding a distributed transaction lock to the job that meets the execution condition in the distributed cache.

[0071] S202: If locking fails, the operation is abandoned.

[0072] S203: If the locking is successful, the operation is executed.

[0073] The containers of each service cluster in the distributed system read the job execution plan in the distributed public cache, select the jobs that need to be executed by this cluster, and obtain the execution status of the jobs that have predecessor and successor dependencies with the jobs of this cluster; determine the trigger conditions such as the execution plan time, whether the predecessor has been executed, and whether the current job has been executed. If the execution conditions are met, the job execution status in the distributed cache is updated to executing, and the job is executed; after the job execution is completed, the execution result is updated to the distributed cache.

[0074] In a specific embodiment, the job configurations such as the batch job name, scheduling conditions, dependencies between previous and next items, and execution clusters are stored in the distributed cache. Each service cluster reads the job configurations to be executed by the cluster in the distributed cache, generates a job execution plan and saves it in the container of the cluster. Each container of the service cluster searches for jobs that meet the execution conditions in the execution plan in combination with information such as system time and the execution status of other jobs in the distributed cache. When a job that meets the execution conditions is found, for executable jobs, a distributed transaction lock is attempted to be added to the job in the distributed cache. If the lock fails, it means that the job has been applied for execution by other containers, and the job is abandoned. If the lock succeeds, the job is executed.

[0075] In one embodiment, the distributed cache-based job scheduling method further includes:

[0076] If the lock is successful, the execution status of the job in the distributed cache is updated to processing before the job is executed.

[0077] In a specific embodiment, when the lock is successfully acquired, the job execution status is first updated in the distributed cache as being in progress, and then the job is started to be executed.

[0078] In one embodiment, a job execution plan is generated according to the obtained job configuration of the job to be executed in the cluster, such as Figure 3 As shown, including:

[0079] S301: Obtain the job configuration of the job to be executed in the cluster.

[0080] S302: Obtain the previous dependent job of the job and the job execution time according to the job configuration.

[0081] S303: Generate a job execution plan according to the previous dependent jobs and job execution time.

[0082] In a specific embodiment, first, in a distributed application system, the job configurations such as the batch job names, scheduling conditions, previous and next item dependencies, and execution clusters of different service clusters are maintained in a distributed cache; at the same time, each service cluster can query the job execution status of other clusters on the distributed cache. Then, when the batch job execution cycle starts, each cluster reads the job configuration from the distributed cache and generates a batch execution plan for the current service cluster and persists it in a local database or file. The execution plan includes the execution time of each batch job, previous item dependencies, and other information.

[0083] In one embodiment, the distributed cache-based job scheduling method further includes:

[0084] If the job execution fails, the job execution status in the distributed cache is updated to execution failure, and an alarm is issued.

[0085] In a specific embodiment, if the job execution ends normally, the job execution status in the distributed cache is updated to successful execution; if the job execution ends abnormally, the job execution status in the distributed cache is first updated to failed execution, and then an alarm is sent to the monitoring platform.

[0086] The method provided in the present application uses a distributed public cache as an information sharing carrier for job execution. The service cluster determines whether a batch job needs to be started and executed by itself. It determines the triggering conditions such as the execution plan time, whether the previous job has been executed, and whether the current job has been executed through the execution status of the previous and subsequent dependent jobs of the job. If the execution conditions are met, the job execution status in the distributed cache is updated to being executed, and the job is executed. The updated execution result after the execution is completed is stored in the distributed cache, thereby achieving the logical and physical isolation of the job scheduling of each service cluster from the job management cluster. The job scheduling is triggered by the service cluster itself, and there is no single-point performance bottleneck, which improves the concurrency of job scheduling, eliminates the scheduling dependence of each service cluster on the management cluster, and ensures the high availability of job scheduling.

[0087] Based on the same inventive concept, the embodiment of the present application also provides a job scheduling device based on distributed cache, which can be used to implement the method described in the above embodiment, as described in the following embodiment. Because the principle of solving the problem by the job scheduling device based on distributed cache is similar to the job scheduling method based on distributed cache. As used below, the term "unit" or "module" can implement a combination of software and / or hardware of a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware is also possible and conceived.

[0088] According to the second aspect of the present application, a job scheduling device based on distributed cache is also provided, such as Figure 5 As shown, including:

[0089] A job execution plan generating unit 501 is used to generate a job execution plan according to the acquired job configuration of the job to be executed in the cluster;

[0090] A search unit 502 is used to search for a job that meets the execution condition in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache;

[0091] A judging unit 503, used to judge whether the job meeting the execution condition is executable;

[0092] The execution unit 504 is configured to execute the job if yes, and update the execution status of the job in the distributed cache to successful execution.

[0093] In the batch job execution window, each container in the service cluster scans each job in the execution plan and determines whether it is executable based on information such as system time and the execution status of other jobs in the distributed cache. For executable jobs, the execution status is updated in the distributed cache to "processing" and execution begins. Finally, after the job execution is completed, the execution result is updated to the distributed cache. Figure 4 The figure shows a connection diagram of a distributed system.

[0094] In a specific embodiment, in a distributed application system, the job configuration information of batch jobs of different service clusters, such as names, scheduling conditions, dependencies between predecessors and successors, and execution clusters, is stored in a distributed cache; at the same time, each service cluster can query the job execution status of other clusters on the distributed cache. When the batch job execution cycle starts, each cluster reads the job configuration from the distributed cache and generates a batch execution plan for the service cluster and persists it in a local database or file. The execution plan includes information such as the execution time of each batch job, the predecessor-dependent job, etc.

[0095] In one embodiment, if Figure 6 As shown, the judging unit 503 includes:

[0096] The locking module 601 is used to add a distributed transaction lock to the job that meets the execution condition in the distributed cache;

[0097] The locking failure module 602 is used to abandon the execution of the job if the locking fails;

[0098] The locking success module 603 is used to execute the operation if the locking is successful.

[0099] The containers of each service cluster in the distributed system read the job execution plan in the distributed public cache, select the jobs that need to be executed by this cluster, and obtain the execution status of the jobs that have predecessor and successor dependencies with the jobs of this cluster; determine the trigger conditions such as the execution plan time, whether the predecessor has been executed, and whether the current job has been executed. If the execution conditions are met, the job execution status in the distributed cache is updated to executing, and the job is executed; after the job execution is completed, the execution result is updated to the distributed cache.

[0100] In a specific embodiment, the job configurations such as the batch job name, scheduling conditions, dependencies between previous and next items, and execution clusters are stored in the distributed cache. Each service cluster reads the job configurations to be executed by the cluster in the distributed cache, generates a job execution plan and saves it in the container of the cluster. Each container of the service cluster searches for jobs that meet the execution conditions in the execution plan in combination with information such as system time and the execution status of other jobs in the distributed cache. When a job that meets the execution conditions is found, for executable jobs, a distributed transaction lock is attempted to be added to the job in the distributed cache. If the lock fails, it means that the job has been applied for execution by other containers, and the job is abandoned. If the lock succeeds, the job is executed.

[0101] In one embodiment, the distributed cache-based job scheduling device further includes:

[0102] The first status update module is used to update the execution status of the job in the distributed cache to being processed if the lock is successful, and then execute the job.

[0103] In a specific embodiment, when the lock is successfully acquired, the job execution status is first updated in the distributed cache as being in progress, and then the job is started to be executed.

[0104] In one embodiment, if Figure 7 As shown, the job execution plan generating unit 501 includes:

[0105] The acquisition module 701 is used to acquire the job configuration of the job to be executed in the cluster;

[0106] Content acquisition module 702, used to acquire the previous dependent job and job execution time of the job according to the job configuration;

[0107] The execution plan generation module 703 is used to generate a job execution plan according to the previous dependent jobs and the job execution time.

[0108] In a specific embodiment, first, in a distributed application system, the job configurations such as the batch job names, scheduling conditions, previous and next item dependencies, and execution clusters of different service clusters are maintained in a distributed cache; at the same time, each service cluster can query the job execution status of other clusters on the distributed cache. Then, when the batch job execution cycle starts, each cluster reads the job configuration from the distributed cache and generates a batch execution plan for the current service cluster and persists it in a local database or file. The execution plan includes the execution time of each batch job, previous item dependencies, and other information.

[0109] In one embodiment, the distributed cache-based job scheduling device further includes:

[0110] The second status update module is used to update the job execution status in the distributed cache to execution failure and issue an alarm if the job execution fails.

[0111] In a specific embodiment, if the job execution ends normally, the job execution status in the distributed cache is updated to successful execution; if the job execution ends abnormally, the job execution status in the distributed cache is first updated to failed execution, and then an alarm is sent to the monitoring platform.

[0112] The present application provides a job scheduling device based on distributed cache. By using the distributed public cache as an information sharing carrier for job execution, the service cluster determines whether a batch job needs to be started and executed by itself. By using the execution status of the previous and next dependent jobs of the job, the trigger conditions such as the execution plan time, whether the previous job has been executed, and whether the current job has been executed are determined. If the execution conditions are met, the job execution status in the distributed cache is updated to being executed, and the job is executed. The updated execution result after the execution is completed is stored in the distributed cache, thereby achieving the logical and physical isolation of the job scheduling of each service cluster from the job management cluster. The job scheduling is triggered by the service cluster itself, and there is no single-point performance bottleneck. The job scheduling concurrency is improved, the scheduling dependence of each service cluster on the management cluster is eliminated, and the high availability of job scheduling is guaranteed.

[0113] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0115] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0116] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0117] The present invention uses specific embodiments to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

[0118] The embodiments of the present application also provide a specific implementation of an electronic device that can implement all the steps in the method in the above embodiments, see Figure 8 , the electronic device specifically includes the following contents:

[0119] Processor 801, memory 802, communication interface 803, bus 804 and non-volatile memory 805;

[0120] The processor 801, memory 802, and communication interface 803 communicate with each other via the bus 804;

[0121] The processor 801 is used to call the computer program in the memory 802 and the non-volatile memory 805. When the processor executes the computer program, all the steps in the method in the above embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0122] S101: Generate a job execution plan according to the acquired job configuration of the job to be executed in the cluster.

[0123] S102: searching for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache.

[0124] S103: Determine whether the job that meets the execution condition is executable.

[0125] S104: If yes, execute the job, and update the execution status of the job in the distributed cache to successful execution.

[0126] In the batch job execution window, each container in the service cluster scans each job in the execution plan and determines whether it is executable based on information such as system time and the execution status of other jobs in the distributed cache. For executable jobs, the execution status is updated in the distributed cache to "processing" and execution begins. Finally, after the job execution is completed, the execution result is updated to the distributed cache.

[0127] In a specific embodiment, in a distributed application system, the job configuration information of batch jobs of different service clusters, such as names, scheduling conditions, dependencies between predecessors and successors, and execution clusters, is stored in a distributed cache; at the same time, each service cluster can query the job execution status of other clusters on the distributed cache. When the batch job execution cycle starts, each cluster reads the job configuration from the distributed cache and generates a batch execution plan for the service cluster and persists it in a local database or file. The execution plan includes information such as the execution time of each batch job, the predecessor-dependent job, etc.

[0128] By judging the execution status of the previous and subsequent dependent jobs of the job to be executed, the trigger conditions such as the execution plan time, whether the previous job has been executed, and whether the current job has been executed are determined. If the execution conditions are met, the job execution status in the distributed cache is updated to being executed, and the job is executed. The updated execution result after the execution is completed is stored in the distributed cache, thereby achieving the logical and physical isolation of the job scheduling of each service cluster from the job management cluster. The job scheduling is triggered by the service cluster itself, and there is no single point performance bottleneck, which improves the concurrency of job scheduling, eliminates the scheduling dependence of each service cluster on the management cluster, and ensures the high availability of job scheduling.

[0129] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the method in the above embodiments. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, all the steps of the method in the above embodiments are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0130] S101: Generate a job execution plan according to the acquired job configuration of the job to be executed in the cluster.

[0131] S102: searching for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache.

[0132] S103: Determine whether the job that meets the execution condition is executable.

[0133] S104: If yes, execute the job, and update the execution status of the job in the distributed cache to successful execution.

[0134] In the batch job execution window, each container in the service cluster scans each job in the execution plan and determines whether it is executable based on information such as system time and the execution status of other jobs in the distributed cache. For executable jobs, the execution status is updated in the distributed cache to "processing" and execution begins. Finally, after the job execution is completed, the execution result is updated to the distributed cache.

[0135] In a specific embodiment, in a distributed application system, the job configuration information of batch jobs of different service clusters, such as names, scheduling conditions, dependencies between predecessors and successors, and execution clusters, is stored in a distributed cache; at the same time, each service cluster can query the job execution status of other clusters on the distributed cache. When the batch job execution cycle starts, each cluster reads the job configuration from the distributed cache and generates a batch execution plan for the service cluster and persists it in a local database or file. The execution plan includes information such as the execution time of each batch job, the predecessor-dependent job, etc.

[0136] By judging the execution status of the previous and subsequent dependent jobs of the job to be executed, the trigger conditions such as the execution plan time, whether the previous job has been executed, and whether the current job has been executed are determined. If the execution conditions are met, the job execution status in the distributed cache is updated to being executed, and the job is executed. The updated execution result after the execution is completed is stored in the distributed cache, thereby achieving the logical and physical isolation of the job scheduling of each service cluster from the job management cluster. The job scheduling is triggered by the service cluster itself, and there is no single point performance bottleneck, which improves the concurrency of job scheduling, eliminates the scheduling dependence of each service cluster on the management cluster, and ensures the high availability of job scheduling.

[0137] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the hardware + program embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. Although the embodiment of this specification provides the method operation steps described in the embodiment or flow chart, more or less operation steps can be included based on conventional or non-creative means. The order of steps listed in the embodiment is only one way of executing the order of many steps, and does not represent the only execution order. When the device or terminal product in practice is executed, it can be executed in sequence or in parallel according to the method shown in the embodiment or the accompanying drawings (for example, a parallel processor or a multi-threaded processing environment, or even a distributed data processing environment). The term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, product or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, product or device. In the absence of more restrictions, it is not excluded that there are other identical or equivalent elements in the process, method, product or device including the elements. For the convenience of description, the above device is described by dividing it into various modules according to its functions. Of course, when implementing the embodiments of this specification, the functions of each module can be implemented in the same one or more software and / or hardware, or the module implementing the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The present invention is described with reference to the flowchart and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the process and / or box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process Figure 1 A process or multiple processes and / or boxes Figure 1A device for specifying functions in a box or multiple boxes. It should be understood by those skilled in the art that the embodiments of this specification may be provided as methods, systems or computer program products. Therefore, the embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware. Moreover, the embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts refer to the partial description of the method embodiment. In the description of this specification, the description of the reference term "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the embodiment of this specification. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction. The above description is only an embodiment of the embodiment of this specification and is not intended to limit the embodiment of this specification. For those skilled in the art, the embodiment of this specification may have various changes and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the embodiment of this specification shall be included in the scope of the claims of the embodiment of this specification.

Claims

1. A job scheduling method based on distributed cache, characterized in that: include: Each service cluster reads the job configuration to be executed by the service cluster in the distributed cache; Generate a job execution plan based on the job configuration of the job to be executed in the service cluster and save it in the container of the service cluster. Each container of the service cluster scans each job in the execution plan. Searching for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache; Determine whether the job that meets the execution conditions is executable; If yes, the job is executed, and the execution status of the job is updated in the distributed cache as successful execution.

2. The distributed cache-based job scheduling method according to claim 1, characterized in that: The step of determining whether a job that meets the execution condition is executable includes: Add distributed transaction locks to jobs that meet the execution conditions in the distributed cache; If the lock fails, the job is abandoned; If the lock is successful, the job is executed.

3. The distributed cache-based job scheduling method according to claim 2, characterized in that: Also includes: If the lock is successful, the execution status of the job in the distributed cache is updated to processing before the job is executed.

4. The distributed cache-based job scheduling method according to claim 1, characterized in that: The generating of a job execution plan according to the obtained job configuration of the job to be executed in the service cluster includes: Get the job configuration of the job to be executed in this service cluster; Obtaining the previous dependent job and job execution time of the job according to the job configuration; A job execution plan is generated according to the preceding dependent job and the job execution time.

5. The distributed cache-based job scheduling method according to claim 3 or 4, characterized in that: Also includes: If the job execution fails, the job execution status in the distributed cache is updated to execution failure, and an alarm is issued.

6. A job scheduling device based on distributed cache, characterized in that: include: A job execution plan generation unit is used for each service cluster to read the job configuration to be executed by the service cluster in the distributed cache; Generate a job execution plan based on the job configuration of the job to be executed in the service cluster and save it in the container of the service cluster. Each container of the service cluster scans each job in the execution plan. A search unit, used to search for jobs that meet the execution conditions in the job execution plan according to the distributed application system time and the execution status of other jobs in the distributed cache; A judgment unit, used to judge whether a job that meets the execution condition is executable; The execution unit is used for executing the job if yes, and updating the execution status of the job in the distributed cache as execution success.

7. The distributed cache-based job scheduling device according to claim 6, characterized in that: The judging unit comprises: A locking module, used to add distributed transaction locks to jobs that meet the execution conditions in the distributed cache; The lock failure module is used to abandon the execution of the job if the lock fails; The lock success module is used to execute the job if the lock is successful.

8. The distributed cache-based job scheduling device according to claim 7, characterized in that: Also includes: The first status update module is used to update the execution status of the job in the distributed cache to being processed if the lock is successful, and then execute the job.

9. The job scheduling device based on distributed cache according to claim 6, characterized in that: The job execution plan generating unit comprises: The acquisition module is used to obtain the job configuration of the job to be executed in this service cluster; A content acquisition module, used for acquiring the previous dependent job and the job execution time of the job according to the job configuration; The execution plan generation module is used to generate a job execution plan according to the preceding dependent job and the job execution time.

10. The distributed cache-based job scheduling device according to claim 8 or 9, characterized in that: Also includes: The second status update module is used to update the job execution status in the distributed cache to execution failure and issue an alarm if the job execution fails.

11. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the distributed cache-based job scheduling method described in any one of claims 1 to 5 are implemented.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the distributed cache-based job scheduling method described in any one of claims 1 to 5 are implemented.

Images