A distributed task scheduling method, system and program product

Abstract

The application relates to the technical field of task scheduling, and discloses a distributed task scheduling method, system and program product. The application is based on a lockless design, introduces grouping isolation through the divide-and-conquer idea, takes an application group as a scheduling unit of a scheduler, each application group runtime is only connected to a certain fixed scheduler, each scheduler only schedules all tasks under the application group associated with the executor that maintains a heartbeat with the scheduler, no lock is needed during scheduling, high availability and high performance can be simultaneously met, and large-scale distributed task scheduling is easy. Further, during scheduling, each scheduler only schedules tasks under the own group, is independent of each other and does not need to wait, and no performance loss caused by additional level locking on a database is caused. Through extension of the scheduler, the overall concurrency and execution efficiency of tasks can be improved, and horizontal expansion is easy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of task scheduling, and particularly relates to a distributed task scheduling method, system and program product. BACKGROUND

[0002] With increasingly complex business, the traditional single-node task scheduling system has been unable to meet the needs of modern enterprises. In processing large-scale, high-concurrency, high-reliability tasks, the single-node task scheduling system will face many problems, such as performance bottleneck, poor fault tolerance, limited scalability, etc. Therefore, distributed task scheduling technology emerges as the times require, which can distribute tasks to multiple nodes for concurrent execution and improve the overall task processing capability. However, the existing distributed task scheduling mostly relies on database row-level locks to ensure scheduling consistency, and when a large number of tasks occur, frequent lock competition will cause performance bottlenecks, and only one main scheduler that obtains the lock is scheduling tasks at the same time, which also brings difficulties in horizontal expansion. SUMMARY

[0003] Therefore, the present application provides a distributed task scheduling method, system and program product to solve the problem that the existing distributed task scheduling mostly relies on database row-level locks to ensure scheduling consistency, and when a large number of tasks occur, frequent lock competition will cause performance bottlenecks, and only one main scheduler that obtains the lock is scheduling tasks at the same time, which also brings difficulties in horizontal expansion.

[0004] In a first aspect, the present application provides a distributed task scheduling method, which is used for a scheduler cluster module, the scheduler cluster module includes a plurality of schedulers, the scheduler cluster module is connected with an executor cluster module, the executor cluster module includes a plurality of application group clusters, each application group cluster includes a plurality of executors; the method includes:

[0005] sending the scheduler address list to the executor cluster module, so that the executor cluster module controls each executor in any application group cluster to send a service discovery request based on the scheduler address list, the service discovery request carrying application group cluster information; receiving the service discovery request sent by the executor cluster module in turn, and performing small-scale distributed master selection based on the received service discovery request by using a preset database table to determine a target scheduler; sending the actual scheduler address of the target scheduler to the executor cluster module, so that the target application group cluster in the executor cluster module is connected with the target scheduler, the target application group cluster including the same application group cluster where the multiple executors sending the service discovery request are located; when the target scheduler is connected with the target application group cluster, obtaining a distributed scheduling task and sending the distributed scheduling task to the target application group cluster after splitting, so that the multiple executors in the target application group cluster execute the split distributed scheduling task and obtain a resource information set; and receiving a heartbeat report signal carrying the resource information set sent by the executor cluster module.

[0006] The distributed task scheduling method provided by the application promotes the executors to send service discovery requests carrying application group cluster information by sending the scheduler address list to the executor cluster module, the scheduler cluster module can perform small-scale distributed master selection according to these requests and a preset database table, and then determine a target scheduler corresponding to a target application group cluster, each application group is connected to a fixed scheduler during running, each scheduler only schedules all tasks under the application group associated with the executors maintaining heartbeat with the scheduler, and the scheduling does not need to be locked, so that high availability and high performance can be simultaneously met, and large-scale distributed task scheduling is easy. Further, the distributed scheduling task is sent to the target application group cluster after being split by the target scheduler, so that the multiple executors in the target application group cluster execute the split distributed scheduling task and obtain a resource information set, in this way, each scheduler only schedules tasks under its own group during scheduling, and is independent and does not need to wait, and there is no performance loss caused by additional database locking, and then the overall task concurrency and execution efficiency can be improved by expanding the scheduler, and horizontal expansion is easy.

[0007] In an optional implementation, performing small-scale distributed master selection based on the received service discovery request by using a preset database table to determine a target scheduler includes:

[0008] Obtaining a preset database table, the preset database table is a table recording information of each application grouping cluster and a corresponding scheduler; judging whether there is record information corresponding to the service discovery request in the preset database table; when there is record information in the preset database table, sending a PING request to a first scheduler corresponding to the record information to detect whether the first scheduler is alive; when the first scheduler is alive, determining the first scheduler as a target scheduler; when the first scheduler is abnormal, determining a scheduler receiving the service discovery request as the target scheduler.

[0009] The distributed task scheduling method provided by the application can record information of each application grouping cluster and a corresponding scheduler through a preset database table, so that a reasonable corresponding relationship between a target scheduler selected subsequently and a corresponding application grouping cluster can be ensured, and the orderedness and continuity of task scheduling at a grouping level can be maintained. Further, when there is record information in the table, a PING request is sent to detect whether a corresponding first scheduler is alive, and the alive scheduler is determined as the target scheduler. Through this detection mechanism, the task can be prevented from being allocated to a scheduler that may have appeared a fault or an abnormality, the application grouping cluster can be connected to a scheduler that works normally and can stably provide scheduling services, and thus the task scheduling process can be ensured not to be interrupted due to a problem of the scheduler itself, and the stability of the entire distributed task scheduling system is improved. Further, when the first scheduler is abnormal, the scheduler receiving the service discovery request can be quickly determined as the target scheduler, the influence of a fault of the scheduler on task execution is minimized, and the fault tolerance capability of the system in response to an abnormal condition at a scheduler level is effectively enhanced. Therefore, by implementing the application, the connection relationship between each application grouping cluster and a scheduler can be reasonably arranged according to actual conditions of the scheduler in real time, the dynamic change of a scheduler resource in a distributed environment can be better adapted, and the efficiency of resource utilization and the smoothness of task scheduling can be maintained.

[0010] In an alternative embodiment, the method further comprises:

[0011] Based on the resource information set, a first same set is determined based on a plurality of executors in the same application grouping cluster; when a heartbeat report signal is not received within a preset number of times or the resource information set does not satisfy a preset condition, a corresponding executor is deleted from the same set to obtain a second same set; and the split distributed scheduling task is sent to the executors in the second same set.

[0012] The distributed task scheduling method provided by the application can preliminarily integrate and classify the executors with the task execution capability under the same group, facilitate subsequent unified management and task allocation, further set a preset number and a preset condition to determine whether the executors meet the requirement of continuing to participate in task allocation, accurately screen out the executors that are currently healthy and in good resource state and can reliably execute tasks, send the split distributed scheduling tasks to the executors in the second same set, effectively avoid allocating tasks to the executors that may have problems or are too high in load and cannot smoothly execute tasks, greatly improve the probability of successful task execution, and guarantee the reliability of the distributed task scheduling.

[0013] In an optional implementation, the method further includes:

[0014] determining an abnormal execution result based on the heartbeat report signal; and performing failover and retry on the abnormal execution result.

[0015] The distributed task scheduling method provided by the application can determine an abnormal execution result based on a heartbeat report signal, can monitor the execution of the executors in real time, avoid the problem that a task has a problem but is not known for a long time, causes the subsequent business process to be blocked, guarantee that the task execution state is within a controllable range, further perform failover and retry on the abnormal execution result, and enhance the reliability of task execution.

[0016] In an optional implementation, the method further includes:

[0017] When the target scheduler is abnormal, competing for the target distributed lock by using other surviving schedulers; taking the first scheduler that obtains the target distributed lock as a new target scheduler, and sending the scheduler address of the new target scheduler to the target application group cluster.

[0018] The distributed task scheduling method provided by the application can quickly start the selection of a new target scheduler when the target scheduler is abnormal, avoid the problem that the task scheduling is stalled for a long time, guarantee the continuity of the task scheduling process, ensure that the tasks of the target application group cluster can be continuously arranged and executed, further take the first scheduler that obtains the target distributed lock as a new target scheduler, and send the scheduler address of the new target scheduler to the target application group cluster, realize seamless switching of the scheduling service, minimize the influence of the scheduler fault on task execution, and maintain the stable operation state of the entire distributed task scheduling system.

[0019] In a second aspect, the present application provides a distributed task scheduling method, which is used for an executor cluster module, the executor cluster module comprises a plurality of application group clusters, each of the application group clusters comprises a plurality of executors, the executor cluster module is connected with a scheduler cluster module, the scheduler cluster module comprises a plurality of schedulers; the method comprises:

[0020] When the scheduler address list sent by the scheduler cluster module is received, based on the scheduler address list, any application group cluster is controlled to send a service discovery request to each of the executors in the application group cluster in turn, so that the scheduler cluster module determines a target scheduler based on the service discovery request, and the service discovery request carries application group cluster information; the actual scheduler address of the target scheduler sent by the scheduler cluster module is received, and based on the actual scheduler address, the target application group cluster is controlled to be connected with the target scheduler, and the target application group cluster is the same application group cluster as the plurality of executors that send the service discovery request; when a plurality of sub-scheduling tasks sent by the target scheduler are received, the plurality of sub-scheduling tasks are executed by using the plurality of executors in the target application group cluster to obtain a resource information set; and a heartbeat report signal carrying the resource information set is sent to the target scheduler in the scheduler cluster module.

[0021] The distributed task scheduling method provided by the present application can control each of the executors in the application group cluster to send a service discovery request carrying application group cluster information after the executor cluster module receives the scheduler address list sent by the scheduler cluster module, so that the executors can actively indicate the group to which the executors belong to the scheduler cluster, thereby guiding the scheduler cluster to accurately determine the target scheduler corresponding to the request. Further, the executor cluster module receives the actual scheduler address of the target scheduler, and controls the corresponding target application group cluster to establish a connection with the target scheduler, so that the executors under each application group cluster can be accurately connected to the scheduler suitable for serving the task of the group of the executors, thereby avoiding the problem of task allocation confusion caused by the blind connection of the executors to the schedulers. Further, the plurality of sub-scheduling tasks are executed by using the plurality of executors in the target application group cluster, thereby guaranteeing the rationality and efficiency of task scheduling and execution, and improving the task execution efficiency.

[0022] In an optional implementation, the method further comprises:

[0023] When the target scheduler exception instruction sent by the scheduler cluster module is received, a new service discovery request is controlled to be sent by the target application group cluster to the scheduler cluster module, so that the scheduler cluster module determines a new target scheduler in other living schedulers based on the service discovery request; and the scheduler address of the new target scheduler sent by the scheduler cluster module is received.

[0024] The distributed task scheduling method provided by the application can quickly respond when the executor cluster module receives the target scheduler exception instruction sent by the scheduler cluster module, control the target application grouping cluster to send a new service discovery request, avoid the task scheduling stagnation caused by the target scheduler failure, ensure the continuous progress of the task scheduling process, and minimize the interruption of the task execution caused by the scheduler exception. Further, by sending a new service discovery request to the scheduler cluster module, the new target scheduler is determined in other surviving schedulers based on the request, and the scheduler address of the new target scheduler is received, so that the executor cluster module can quickly reestablish the connection with the new scheduler for the target application grouping cluster, ensure that the task scheduling process will not be interrupted due to the problem of the scheduler itself, and improve the stability of the entire distributed task scheduling system.

[0025] In an optional embodiment, the method further comprises:

[0026] When the multiple executors in the target application grouping cluster fail to execute the multiple sub-scheduling tasks, the multiple sub-scheduling tasks are re-executed according to a preset custom retry strategy; and when the re-execution of the multiple sub-scheduling tasks fails, a task failure instruction is sent to the scheduler cluster module, so that the scheduler cluster module sends the multiple sub-scheduling tasks to multiple executors in other application grouping clusters.

[0027] The distributed task scheduling method provided by the application re-executes the multiple sub-scheduling tasks by using the preset custom retry strategy, maximizes the use of existing executor resources, effectively improves the success rate of task execution, enhances the reliability of task execution, avoids the additional resource consumption and coordination cost caused by the premature transfer of tasks to other executors, realizes the deep mining and reasonable use of the executor resources in the current grouping cluster, and improves the resource utilization efficiency. Further, when the re-execution of the multiple sub-scheduling tasks fails, a task failure instruction is sent to the scheduler cluster module, so that the scheduler cluster module transfers the tasks to the executors in other application grouping clusters. By reasonably controlling the timing of task transfer, the efficient operation of the entire distributed task scheduling system can be maintained, and the system resources can be more focused on the tasks that really need to be re-allocated.

[0028] In a third aspect, the application provides a distributed task scheduling system, which comprises a scheduler cluster module and an executor cluster module. The scheduler cluster module comprises multiple schedulers, and the executor cluster module comprises multiple application grouping clusters. Each application grouping cluster comprises multiple executors, and the scheduler cluster module and the executor cluster module are connected. The scheduler cluster module is configured to execute the distributed task scheduling method of the first aspect or any of the corresponding embodiments thereof. The executor cluster module is configured to execute the distributed task scheduling method of the second aspect or any of the corresponding embodiments thereof.

[0029] The distributed task scheduling system provided by the application is easy to horizontally expand, supports large-scale distributed task scheduling, meets the requirements of high availability and high performance, can fully utilize cluster resources, and improves the overall performance and throughput of the system.

[0030] In a fourth aspect, the application provides a computer program product, including computer instructions for causing a computer to execute the distributed task scheduling method of the first aspect or any of the corresponding embodiments thereof, or to execute the distributed task scheduling method of the second aspect or any of the corresponding embodiments thereof. BRIEF DESCRIPTION OF DRAWINGS

[0031] In order to more clearly illustrate the specific embodiments of the application or the technical solutions in the prior art, the following will briefly introduce the drawings needed to be used in the specific embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0032] Figure 1 is a structural block diagram of the distributed task scheduling system according to the embodiment of the application;

[0033] Figure 2 is a flowchart of the distributed task scheduling method according to the embodiment of the application;

[0034] Figure 3 is a flowchart of another distributed task scheduling method according to the embodiment of the application;

[0035] Figure 4 is a flowchart of the distributed task scheduling method based on group isolation according to the embodiment of the application;

[0036] Figure 5 is a process diagram of the service discovery of the executor to the scheduler according to the embodiment of the application;

[0037] Figure 6 is a process diagram of the load checking of the executor according to the embodiment of the application;

[0038] Figure 7 is a functional module structure diagram of the distributed task scheduling device based on group isolation according to the embodiment of the application;

[0039] Figure 8 is a hardware structure diagram of the computer device of the embodiment of the application. DETAILED DESCRIPTION

[0040] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0041] At present, the existing distributed task scheduling technical solutions still have some deficiencies:

[0042] 1. The existing technical solutions are essentially dependent on database row-level locks, and when a large number of scheduling tasks occur, there will be serious lock competition, which is prone to database performance bottlenecks and cannot support large-scale distributed task scheduling.

[0043] 2. During scheduling, the scheduler that obtains the database row-level lock schedules the task, only one scheduler can obtain the lock at the same time, and other schedulers must wait, so that the main node has a performance bottleneck under the pressure of a large number of tasks, which affects the overall concurrency and execution efficiency of the task and does not have horizontal scalability.

[0044] 3. If the hardware performance of the executors is too different, it is also impossible to control the task allocation of different nodes to reduce the pressure of some nodes, so that the cluster nodes cannot be fully mobilized, and resources cannot be reasonably utilized.

[0045] According to the embodiments of the present application, a distributed task scheduling method embodiment is provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] In the present embodiment, a distributed task scheduling method is provided, which can be used in a scheduler cluster module 11 as shown in the figure. Figure 1 The scheduler cluster module 11 includes a plurality of schedulers 111.

[0047] Further, the scheduler cluster module 11 is connected with an executor cluster module 12, and the executor cluster module 12 includes a plurality of application group clusters 121, each of which includes a plurality of executors 1211.

[0048] Further, the scheduler 111 represents the scheduling node of the task, which can split and issue the scheduling task of the user, and find the appropriate executor to execute the task.

[0049] The executor 1211 represents an actual execution node of a task, which can receive and execute a task issued by the scheduler and report relevant state information.

[0050] Figure 2 is a flowchart of a distributed task scheduling method according to an embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 2

[0051] In step S201, the scheduler address list is sent to the executor cluster module, so that the executor cluster module controls any application group cluster to send a service discovery request to each executor in the application group cluster based on the scheduler address list.

[0052] The service discovery request carries application group cluster information, indicating the process of the executor requesting the actual connection address from the scheduler, and the executors under the same application group cluster will be connected to the same scheduler.

[0053] Further, the scheduler address list can include address information (such as IP address and port number, etc.) of multiple schedulers 111 in the scheduler cluster module 11.

[0054] In an example, there are three schedulers in the scheduler cluster module 11, namely scheduler A (IP address: 192.168.1.100, port number: 8080), scheduler B (IP address: 192.168.1.101, port number: 8080) and scheduler C (IP address: 192.168.1.102, port number: 8080), and the generated scheduler address list contains the address information entries corresponding to the three schedulers.

[0055] Specifically, the scheduler cluster module 11 can send the corresponding scheduler address list to the executor cluster module 12. Wherein, the communication process can follow the corresponding network protocol (such as TCP / IP protocol, etc.), to ensure that the address list can be accurately transmitted to the executor cluster module 12. For example, the data packet containing the scheduler address list can be packaged, and sent according to the network address of the executor cluster module 12. After the executor cluster module 12 receives the data packet, it performs unpacking operation to obtain the content of the scheduler address list therein.

[0056] Further, after the executor cluster module 12 receives the scheduler address list, it can start the executors 1211 in the same application group cluster 121 according to the content of the scheduler address list, and make each executor 1211 in the application group cluster 121 send a service discovery request to the scheduler cluster module 11.

[0057] ​Further, under the service discovery mechanism, the executors 1211 under the whole application group cluster 121 are finally connected to the same scheduler 111.

[0058] In step S202, the service discovery requests sent by the executor cluster module are received in sequence, and a small-scale distributed master election is performed based on the received service discovery requests using a preset database table to determine a target scheduler.

[0059] The preset database table is an information table recording each application group cluster and the corresponding scheduler; and the target scheduler is the scheduler determined to be connected to each executor 1211 in the application group cluster 121 sending the service discovery request.

[0060] Specifically, after the scheduler 111 in the scheduler cluster module 11 receives the service discovery request, a small-scale distributed master election can be performed using the preset database table, and the corresponding target scheduler is determined through the master election.

[0061] In step S203, the actual scheduler address of the target scheduler is sent to the executor cluster module, so that the target application group cluster in the executor cluster module is connected to the target scheduler.

[0062] The target application group cluster is the same application group cluster 121 in which the multiple executors sending the service discovery request in step S201 are located.

[0063] Specifically, after the target scheduler is determined, the actual scheduler address of the target scheduler can be sent to the executor cluster module 12.

[0064] Further, after the executor cluster module 12 receives the actual scheduler address of the target scheduler, the target application group cluster can be connected to the target scheduler according to the actual scheduler address, i.e., each executor 1211 in the target application group cluster is connected to the target scheduler.

[0065] In step S204, when the target scheduler is connected to the target application group cluster, a distributed scheduling task is obtained and sent to the target application group cluster after being split, so that the multiple executors in the target application group cluster execute the split distributed scheduling task and obtain a resource information set.

[0066] The resource information set can include CPU and memory information, etc., and can reflect the workload and resource utilization of the executors.

[0067] Specifically, after the target scheduler is successfully connected to the target application group cluster, the target scheduler can obtain the corresponding distributed scheduling task.

[0068] Further, the obtained distributed scheduling task can be split and then sent to each executor 1211 in the target application grouping cluster.

[0069] Further, each executor 1211 respectively executes the corresponding scheduling task sent and can form a corresponding resource information set according to the task execution result.

[0070] Step S205, receiving the heartbeat report signal carrying the resource information set sent by the executor cluster module.

[0071] The heartbeat report signal indicates a periodic signal sent by the executor to the scheduler, and can include the executor address and the executor load information.

[0072] Specifically, the executor cluster module 12 can send the heartbeat report signal carrying the resource information set to the scheduler cluster module 11. In this way, the target scheduler in the scheduler cluster module 11 can know the resource utilization of each executor 1211 in the target application grouping cluster in real time, so as to reasonably allocate the scheduling task.

[0073] The distributed task scheduling method provided by the embodiment, by sending the scheduler address list to the executor cluster module, prompting the executor to send the service discovery request carrying the application grouping cluster information, the scheduler cluster module can determine the target scheduler corresponding to the target application grouping cluster according to these requests and the preset database table, and determine the target scheduler corresponding to the target application grouping cluster according to these requests and the preset database table. By taking the application grouping as the scheduling unit of the scheduler, each application grouping will only be connected to a fixed scheduler during running, and each scheduler will only schedule all tasks under the application grouping associated with the executor that maintains the heartbeat with itself. The scheduling does not need to be locked, and can simultaneously meet high availability and high performance, and is easy to large-scale distributed task scheduling. Further, the target scheduler splits and sends the distributed scheduling task to the target application grouping cluster, so that multiple executors in the target application grouping cluster respectively execute the split distributed scheduling task and obtain a resource information set. In this way, each scheduler only schedules the tasks under its own group during scheduling, and is independent of each other and does not need to wait. There is no performance loss caused by the lock of the database, and the overall task concurrency and execution efficiency can be improved by expanding the scheduler, and the horizontal expansion is easy.

[0074] In an optional embodiment, in the step S202, the target scheduler is determined by using the preset database table to perform small-scale distributed master selection based on the received service discovery request, including:

[0075] Step a1, obtaining a preset database table.

[0076] Step a2, judging whether there is record information corresponding to the service discovery request in the preset database table.

[0077] Step a3, when there is record information in the preset database table, sending a PING request to the first scheduler corresponding to the record information to detect whether the first scheduler is alive.

[0078] Step a4, when the first scheduler is alive, determining the first scheduler as the target scheduler.

[0079] Step a5, when the first scheduler is abnormal, determining the scheduler receiving the service discovery request as the target scheduler.

[0080] Specifically, if there is record information corresponding to the currently received service discovery request in the preset database table, it indicates that there is already another executor in the application group cluster sending a request to the scheduler for election, at this time, only a PING request needs to be sent to detect whether the first scheduler in the record is alive. The PING alive detection between schedulers can use HTTP protocol communication, and further can filter out schedulers with high load.

[0081] Further, if the first scheduler in the record is alive, the scheduler of the group is directly returned as the target scheduler, that is, the one scheduler is directly determined as the target scheduler. Through this detection mechanism, the task can be avoided to be assigned to a scheduler that may have faults or abnormalities, so as to ensure that the application group cluster can be connected to a scheduler that can stably provide scheduling services, thereby ensuring that the task scheduling process will not be interrupted due to the problem of the scheduler itself, and the stability of the entire distributed task scheduling system is improved. Otherwise, complete the usurpation, write the information of the scheduler currently receiving the service discovery request into the database table, and become the target scheduler of the application group cluster, which maximally reduces the impact of the scheduler failure on the task execution, and effectively enhances the fault tolerance of the system to abnormal conditions at the scheduler level.

[0082] In an optional embodiment, the distributed task scheduling method of the embodiment further includes: determining a first same set based on a plurality of executors in the same application group cluster based on the resource information set; deleting the corresponding executor from the same set when no heartbeat report signal is received within a preset number of times or the resource information set does not satisfy a preset condition, to obtain a second same set; and sending the split distributed scheduling task to the executors in the second same set.

[0083] Specifically, when receiving the resource information set sent by the executor cluster module 12, the corresponding scheduler can add the executors under the same application group to the same set, i.e., the first same set, according to the dimension of the application group cluster 121. In this way, the executors with task execution capability under the same group are preliminarily integrated and classified, which facilitates subsequent unified management and task allocation.

[0084] Further, if there is no heartbeat report for more than a preset number of times or the CPU and memory of the resource information exceed a preset threshold, the corresponding executor 1211 is deleted from the first same set to obtain a second same set. In this way, those executors that are currently healthy and have good resource status and can reliably execute tasks can be accurately screened out.

[0085] Further, the split distributed scheduling task is continuously sent to the executors in the second same set, which can effectively avoid assigning tasks to executors that may have problems or high loads and thus cannot successfully execute tasks, greatly improving the probability of successful execution of tasks and ensuring the reliability of distributed task scheduling.

[0086] In an optional embodiment, the distributed task scheduling method of the embodiment further includes determining an abnormal execution result based on the heartbeat report signal, and performing failover and retry on the abnormal execution result.

[0087] Specifically, the scheduler can periodically receive heartbeat report signals sent from each executor. These heartbeat report signals contain a wealth of information, such as the address of the executor, the current load information of the executor (such as CPU usage, memory usage, etc.), and key state information related to the task being executed.

[0088] Further, by analyzing the heartbeat report signal, the content related to the task execution situation can be extracted. For example, whether there is an error prompt in task execution, whether the task execution progress has been stalled for a long time, and whether the load of the executor has abnormally increased.

[0089] (1) Task execution error: If the heartbeat report signal explicitly indicates that a specific error code (such as a database connection failure error code, a syntax error prompt during program execution, etc.) occurs when executing a task, or the executor feedbacks that an unsolvable abnormal situation (such as file reading failure, network request timeout, etc.) is encountered during task execution, the scheduler can determine that the task has an abnormal execution result.

[0090] (2) Task progress abnormality: If it is found from the heartbeat report signal that the execution progress of a task has been maintained at a certain stage for a long time, exceeding the progress range that the task should reach within the corresponding time period, it may mean that the task execution has been blocked or has other abnormal situations, and the scheduler will identify it as an abnormal execution result. For example, a data analysis task should normally complete the data reading and preliminary processing stage within 1 minute, but the heartbeat report shows that the executor has been stalled for 20 minutes, which means that the task execution has a problem.

[0091] (3) Executor load abnormality aspect: When the executor load information in the heartbeat report signal shows that the CPU usage of the executor is too high (such as more than 90%) or the memory occupancy is almost full, it may cause the task to be unable to continue to be normally executed, at this time, the scheduler will also judge that the task related to the executor has an abnormal execution result, because the too high load may cause the task to be in a state of waiting for resources or directly fail to execute due to insufficient resources.

[0092] Further, the abnormal execution result can be failover and retry.

[0093] In an example, the failover of the abnormal execution result can be implemented by the following steps:

[0094] (1) Select a suitable alternative executor: Once it is determined that there is an abnormal execution result of the task, the scheduler can first screen other suitable executors to undertake the task in the entire executor cluster. The scheduler will consider multiple factors to select the alternative executor, such as the current load of other executors (preferably select executors with lighter load and more sufficient resources), the application group cluster (try to select executors belonging to the same application group cluster as the original executor to ensure the similarity and compatibility of the task execution environment, and the like), and the past task execution success rate and the like, to ensure that the selected executor has a high probability of successfully executing the task.

[0095] (2) Update the task allocation information: The scheduler can remove the task with abnormality from the task queue of the original executor, and reassign it to the task queue of the selected alternative executor, and update the related task scheduling records, mark that the task has been failovered and the new executor information and the like, so as to track and manage the execution of the task. For example, the task is originally assigned to executor A for execution, after the abnormality, the scheduler transfers the task to executor B, and records the change in the task management module of the scheduling system, to facilitate subsequent monitoring of the execution progress of the task on executor B.

[0096] The retry of the abnormal execution result can be implemented by the following steps:

[0097] (1) Determine the number of retries and the time interval: The scheduler can determine the retry operation of the task with abnormal execution result according to the preset custom retry strategy. The retry strategy usually specifies the number of retries (such as a maximum of 3 times) and the time interval between each retry (for example, 1 minute between each retry). These parameters can be flexibly set according to different task types, importance, and past execution experience and the like, with the purpose of reasonably utilizing system resources and as much as possible improving the probability of successful execution of the task.

[0098] (2) Perform retry operation: according to the determined retry number and time interval, the scheduler can instruct the original executor to re-execute the task that appears abnormal. Before each retry, the executor can perform some necessary preparations, such as cleaning up the temporary files left over from the last execution, resetting the related program state, etc., and then trying to execute the task again, and after each retry, the feedback signal is still reported to the scheduler through the heartbeat, and the scheduler continues to monitor these feedbacks to judge whether the retry is successful.

[0099] (3) Judge the retry result and further process: if the task is successfully executed within the specified number of retries, the scheduler can mark it as successfully completed, and process the subsequent result collection, data integration, etc. according to the normal process. But if the task still fails after multiple retries, the scheduler will perform failover operation again to find other suitable executors to continue trying to execute the task, or take other corresponding measures according to system settings (such as recording detailed failure reasons, notifying relevant operation and maintenance personnel, etc.), to ensure that the task will not be completely shelved due to execution exception, and try to ensure that the task can be successfully executed eventually, maintaining the reliability and stability of the entire distributed task scheduling system.

[0100] By determining the abnormal execution result through the heartbeat feedback signal, the execution of the task by the executor can be monitored in real time, avoiding the problem that the task may be problematic for a long time without being known, causing the subsequent business process to be blocked, ensuring that the task execution status is within the controllable range. Further, the abnormal execution result is subjected to failover and retry, enhancing the reliability of task execution.

[0101] In an optional implementation, the distributed task scheduling method of the embodiment of the application further includes: when the target scheduler is abnormal, using other surviving schedulers to compete for a target distributed lock; taking the first scheduler that obtains the target distributed lock as a new target scheduler, and sending the scheduler address of the new target scheduler to the target application group cluster.

[0102] Specifically, if the target scheduler appears abnormal during the task scheduling execution process, such as sudden network failure leading to disconnection with the executor, hardware failure causing inability to continue processing the task, or software level crash, etc. At this time, other surviving schedulers can be used to compete for a target distributed lock, and the first one that gets the lock will take over, write its own information into the scheduler address information table associated with the application group, become the scheduler of the group, and send the scheduler address of the first scheduler that gets the lock to the corresponding target application group cluster.

[0103] The target distributed lock represents a mechanism for ensuring that only one scheduler can obtain the scheduling right of the target application group cluster at the same time, and can be implemented based on some distributed lock service, such as a common distributed lock based on Redis or a distributed lock based on ZooKeeper.

[0104] Further, the scheduler can try to obtain the target distributed lock through a predetermined rule and algorithm to enter a competition state. For example, assuming that there are three live schedulers, scheduler A, scheduler B, and scheduler C, they simultaneously initiate a lock obtaining request to the distributed lock service, and each tries to obtain the lock through corresponding logic.

[0105] A distributed task scheduling method is provided in the embodiment, which can be used in an executor cluster module 12 as shown in the figure. Figure 1 The executor cluster module 12 includes a plurality of application group clusters 121, and each application group cluster includes a plurality of executors 1211.

[0106] Further, the executor cluster module 12 is connected with a scheduler cluster module 11, and the scheduler cluster module 11 includes a plurality of schedulers 111.

[0107] Figure 3 A flowchart of the distributed task scheduling method according to the embodiment of the application is shown in the figure. Figure 3 The flowchart includes the following steps:

[0108] In step S301, when receiving the scheduler address list sent by the scheduler cluster module, based on the scheduler address list, the control controls each executor in any application group cluster to send a service discovery request in turn, so that the scheduler cluster module determines a target scheduler based on the service discovery request.

[0109] Specifically, when receiving the scheduler address list sent by the scheduler cluster module 11, the executors 1211 in the same application group cluster 121 can be started according to the content of the scheduler address list, and then each executor 1211 in the same application group cluster 121 sends a service discovery request to the scheduler cluster module 11 in turn.

[0110] Further, the scheduler cluster module 11 can determine a target scheduler connected with each executor 1211 in the application group cluster 121 according to the received service discovery request. The specific process can refer to the specific process of the target scheduler in the distributed task scheduling method for the scheduler cluster module 11 described above, which will not be described here.

[0111] Step S302, receiving the actual scheduler address of the target scheduler sent by the scheduler cluster module, and based on the actual scheduler address, controlling the target application group cluster to connect with the target scheduler.

[0112] The target application group cluster is the same application group cluster 121 where the plurality of executors that send the service discovery request in step S301 are located.

[0113] Specifically, after receiving the actual scheduler address of the target scheduler sent by the scheduler cluster module 11, each executor 1211 in the target application group cluster can be connected with the target scheduler according to the actual scheduler address.

[0114] Step S303, when receiving a plurality of sub-scheduling tasks sent by the target scheduler, executing the plurality of sub-scheduling tasks by using a plurality of executors in the target application group cluster to obtain a resource information set.

[0115] Specifically, when receiving a plurality of sub-scheduling tasks sent by the target scheduler, each executor 1211 in the target application group cluster can execute the corresponding scheduling task issued respectively and form a corresponding resource information set according to the task execution result.

[0116] Step S304, sending a heartbeat report signal carrying the resource information set to the target scheduler in the scheduler cluster module.

[0117] Specifically, after the plurality of sub-scheduling tasks are executed, a heartbeat report signal carrying the resource information set can be sent to the target scheduler in the scheduler cluster module 11. In this way, the target scheduler in the scheduler cluster module 11 can learn the resource utilization of each executor 1211 in the target application group cluster in real time, so as to reasonably allocate scheduling tasks.

[0118] The distributed task scheduling method provided in the embodiment, after the executor cluster module receives the scheduler address list sent by the scheduler cluster module, each executor in the application group cluster will send a service discovery request carrying application group cluster information, so that the executors can actively indicate to the scheduler cluster the group to which they belong, so that the scheduler cluster can accurately determine the target scheduler corresponding to the request based on these requests. Further, the executor cluster module receives the actual scheduler address of the target scheduler, and controls the corresponding target application group cluster to establish a connection with the target scheduler, so that the executors under each application group cluster can accurately connect to the scheduler suitable for serving the task of their own group, thereby avoiding the problem of task allocation confusion caused by the blind connection of the executors to the scheduler. Further, by using a plurality of executors in the target application group cluster to execute a plurality of sub-scheduling tasks, the rationality and efficiency of task scheduling and execution are guaranteed, and the task execution efficiency is improved.

[0119] In an optional embodiment, the distributed task scheduling method of the embodiment of the present application further comprises: when the target application group cluster receives the target scheduler exception instruction sent by the scheduler cluster module, sending a new service discovery request to the scheduler cluster module, so that the scheduler cluster module determines a new target scheduler among the other surviving schedulers based on the service discovery request; and receiving the scheduler address of the new target scheduler sent by the scheduler cluster module.

[0120] Specifically, when the scheduler cluster module monitors that the target scheduler has an abnormal situation (for example, the target scheduler cannot normally perform the task scheduling duty due to network failure, hardware failure, software crash, etc.), the target scheduler exception instruction is sent to the target application group cluster associated therewith, avoiding the situation that the task scheduling is stalled due to the failure of the target scheduler, ensuring the continuous progress of the task scheduling process, and minimizing the interruption influence on the task execution caused by the abnormality of the scheduler.

[0121] Further, after receiving the target scheduler exception instruction, the target application group cluster sends a new service discovery request to the scheduler cluster module, which requests the other surviving schedulers, that is, under the control of the target scheduler exception instruction, the target application group cluster controls each executor 1211 to send a new service discovery request to the scheduler cluster module 11, ensuring that the task scheduling process will not be interrupted due to the problem of the scheduler itself, and improving the stability of the entire distributed task scheduling system.

[0122] Further, according to the received new service discovery request, a new target scheduler connected to each executor 1211 in the target application group cluster can be determined among the other surviving schedulers in the scheduler cluster module 11. The specific determination process can refer to the determination process in the distributed task scheduling method for the scheduler cluster module 11 described above, which will not be described here.

[0123] Further, the scheduler cluster module 11 can continue to send the scheduler address of the new target scheduler to the target application group cluster.

[0124] In an optional embodiment, the distributed task scheduling method of the embodiment of the present application further comprises: when the target application group cluster receives the target scheduler exception instruction sent by the scheduler cluster module, sending a new service discovery request to the scheduler cluster module, so that the scheduler cluster module determines a new target scheduler among the other surviving schedulers based on the service discovery request; and receiving the scheduler address of the new target scheduler sent by the scheduler cluster module.

[0125] Specifically, after the multiple executors in the target application group cluster receive the multiple sub-scheduling tasks, the multiple sub-scheduling tasks are executed locally.

[0126] Further, if the local execution process fails, such as a program error (for example, a syntax error occurs during code execution, an external interface called returns an error prompt, etc.), resource exhaustion causes the task to be unable to continue (such as memory overflow, insufficient disk space affecting data storage, etc.), the task execution progress stagnates for a long time (exceeds the time range corresponding to the stage that should be reached under normal circumstances), etc., the multiple sub-scheduling tasks can be re-executed according to a preset custom retry strategy, which maximizes the use of existing executor resources, effectively improves the success rate of task execution, enhances the reliability of task execution, avoids the additional resource consumption and coordination costs that may be generated by prematurely transferring tasks to other executors, realizes deep mining and reasonable use of executor resources within the local group cluster, and improves resource utilization efficiency.

[0127] The preset custom retry strategy can be set in advance according to factors such as the importance of the task, past execution experience, and task type characteristics. For example, the strategy can specify that for some critical business computing tasks, the number of retries is set to 3, and the time interval for each retry is 1 minute; for general data processing tasks, the number of retries is set to 2, and the interval between each retry is 30 seconds, etc. If the current execution failure condition meets the triggering condition of the retry strategy (such as not reaching the upper limit of the number of retries, etc.), the corresponding executor will be instructed to prepare for a retry operation.

[0128] Further, if the retry fails, a task failure instruction can be sent to the scheduler cluster module, so that the scheduler cluster module sends the multiple sub-scheduling tasks to multiple executors in other application group clusters for execution. By reasonably controlling the timing of task transfer, the efficient operation of the entire distributed task scheduling system can be maintained, and system resources can be more focused on tasks that truly need to be redistributed for execution.

[0129] In this embodiment, a distributed task scheduling system is provided, as shown in Figure 1 The distributed task scheduling system 1 includes a scheduler cluster module 11 and an executor cluster module 12, and the scheduler cluster module 11 and the executor cluster module 12 are connected.

[0130] Specifically, the scheduler cluster module 11 includes multiple schedulers 111; the executor cluster module 12 includes multiple application group clusters 121, and each application group cluster includes multiple executors 1211.

[0131] Further, the scheduler cluster module 11 is configured to execute the following steps. Figure 2The illustrated distributed task scheduling method; executor cluster module 12 is used for executing as Figure 3 The illustrated distributed task scheduling method.

[0132] The distributed task scheduling system provided by the embodiment is easy to horizontally expand, supports large-scale distributed task scheduling, meets the requirements of high availability and high performance, can fully utilize cluster resources, and improves the overall performance and throughput of the system.

[0133] In an example, a distributed task scheduling method based on group isolation is provided, as shown in Figure 4 The illustrated distributed task scheduling method includes the following steps:

[0134] Step 1, deploy the scheduler cluster, wait for the executor to register and report the heartbeat, and the scheduling server locally maintains its own load information. Further, in step 1, the scheduler refers to the scheduling node of the task, splits and issues the scheduling task of the user, and finds a suitable executor to execute the task; the executor refers to the actual execution node of the task, receives and executes the task issued by the scheduler and reports the relevant state information; the load information includes node resource information (CPU and memory, etc.).

[0135] Step 2, fill in the scheduler address list, start the executors under the same application group, and the executors periodically service discovery, request any scheduler, and obtain the actual scheduler address corresponding to the application group. Further, in step 2, service discovery refers to the process in which the executor requests the actual connection address from the scheduler, and the executors under the same application group will be connected to the same scheduler. Further, in step 2, the executors will bring the application group information when performing service discovery.

[0136] Step 3, after receiving the service discovery request of the executor, the scheduler will perform small-scale distributed master election: there is a record in the database table for each group and the corresponding scheduler information table, if the scheduler finds a record in the table, it means that there is another executor in the application group that has previously requested the scheduler to perform election, at this time only the PING request is sent to detect whether the scheduler in the record is alive. If the scheduler in the record is alive, it will be directly returned as the scheduler of the group; otherwise, complete the usurpation, write its own information into the database table, become the scheduler of the group and return. Further, in step 3, the PING alive detection between schedulers uses HTTP protocol communication, which will filter out schedulers with high load.

[0137] Step 4, under the service discovery mechanism, the entire application group under the executor is finally connected to the same scheduler, and the subsequent executors send heartbeat reports to the scheduler, including the executor address and the executor load information. If the scheduler node is abnormal during this period, all executors under the application group will request other surviving schedulers, and the surviving schedulers will compete for a distributed lock. The first one to get the lock will take over and write its own information to the application group associated scheduler address information table, becoming the scheduler of the group and returning. After that, the executors are connected to the same scheduler.

[0138] Step 5, the scheduler maintains the application group executor cluster node information it holds, and periodically pulls the task information associated with all applications from the database for scheduling processing. The tasks will be sent to the executors with normal load and heartbeat. Further, in step 5, the method for the scheduler to check the normal load of the executors includes: the executors periodically report node resource information (CPU and memory, etc.) with the application group; the scheduler adds the executors under the same application group to the same set according to the application group dimension; if there is no heartbeat report for more than a preset number of times, it is removed from the set, and if the CPU and memory exceed the preset threshold, it is also removed from the set.

[0139] Step 6, after the executors receive the tasks from the scheduler, they execute them locally and report the execution results; the scheduler will perform failover and retry on abnormal execution results. Further, in step 6, after the executors fail to execute locally, they will retry according to the self-defined retry policy. If the retry still fails, the scheduler will assign the task to other executors for processing.

[0140] Among them, in combination with the above steps, the process of the executors discovering services to the scheduler is as shown in Figure 5 ; the process of checking the load of the executors is as shown in Figure 6 .

[0141] Further, a distributed task scheduling device based on group isolation is provided, as shown in Figure 7 , which includes the following modules:

[0142] The scheduler module is used to receive user scheduling tasks, maintain node state information, and split and schedule tasks to the executor cluster. Further, the scheduler includes a task submodule, a node submodule, and a scheduling submodule. The task submodule is used to create and view scheduling tasks, manage the execution status of the tasks, and perform failover and retry on the tasks that fail to execute. The node submodule maintains the state information of all executor cluster nodes under all application groups and periodically cleans up the executor nodes with expired heartbeats. The scheduling submodule pulls the task information associated with all applications from the database, splits and schedules the tasks, and reasonably allocates resources according to the task target and the load of each executor node.

[0143] an executor module, configured to actually execute a task, periodically perform service discovery on the dispatcher, periodically report executor state information to the dispatcher, and receive a task issued by the dispatcher and report execution result data to the dispatcher after local execution. Further, the executor comprises a registration submodule, a heartbeat submodule, and an execution submodule. The registration submodule periodically performs service discovery and obtains an actually connected dispatcher, and executors under the same application group are connected to the same dispatcher. The heartbeat submodule periodically sends heartbeat reports, including an executor address and executor load information (CPU and memory, etc.). The execution submodule executes an actual task, and after local execution fails, retries according to a self-defined retry strategy and reports task state information to the dispatcher.

[0144] a database module, configured to store a dispatcher address information table associated with an application group, a task information table associated with an application group, and an application group self-information table, etc. The dispatcher address information table associated with an application group is used for service discovery, the task information table associated with an application group is used for associated task execution state, and the application group self-information table is used to display an application list.

[0145] The grouping-isolation-based distributed task scheduling method and device provided in the example have the following beneficial effects:

[0146] (1) Compared with the prior art relying on database row-level locks, the present application is based on a lock-free design, introduces grouping isolation through the divide-and-conquer idea, uses an application group as a scheduling unit of a dispatcher, each application group runtime is only connected to a certain fixed dispatcher, and each dispatcher only schedules all tasks under the application group associated with the dispatcher that maintains a heartbeat with the dispatcher. No lock is needed during scheduling, and high availability and high performance can be simultaneously satisfied, and large-scale distributed task scheduling is easy.

[0147] (2) During scheduling, each dispatcher only schedules tasks under its own group, is independent of each other and does not need to wait, and there is no performance loss caused by additional level locking on the database. By expanding the dispatcher, the overall concurrency and execution efficiency of the task can be improved, and horizontal expansion is easy.

[0148] (3) The executor reports load information through a heartbeat, the dispatcher calculates a health degree according to the load information of the executor, and automatically selects an executor with optimal load to execute a task, thereby fully utilizing cluster resources. In addition, the dispatcher also maintains its own load information and provides overload protection, and executors under an application group are only connected to a dispatcher with normal load, thereby ensuring that the pressure on each dispatcher is not too large.

[0149] In an optional embodiment, taking an executor under an application group A connected to the same dispatcher B as an example, the implementation process of the above grouping-isolation-based distributed task scheduling method specifically comprises:

[0150] Step 1, deploy 3 scheduler node clusters, wait for executor registration and heartbeat report, the scheduler address list is: scheduler A: scheduler B: scheduler C, the scheduling server locally maintains its own load information. Further, in step 1, the scheduler refers to the task scheduling node, which splits and issues the user's scheduling task, and finds the appropriate executor to execute the task; the executor refers to the actual execution node of the task, which receives and executes the task issued by the scheduler and reports the relevant state information; the load information includes node resource information (CPU and memory, etc.).

[0151] Step 2, fill in the scheduler address list in step 1, start the executors under application group A, and the executors will perform service discovery and start a background thread for regular service discovery after startup, request any scheduler, and get the actual scheduler address corresponding to the application group. Here it is assumed that executor A requests scheduler B address for service discovery, and executor B requests scheduler C address for service discovery, and both requests are made with application group A.

[0152] Step 3, after receiving the executor service discovery request, the scheduler will perform a small distributed master election: there is a database table that records the information of each group and the corresponding scheduler, if the scheduler finds a record in the table, it means that there is already another executor in the application group that has previously requested the scheduler for election, at this time only a PING request is sent to detect whether the scheduler in the record is alive. If the scheduler in the record is alive, it will be directly returned as the scheduler for the group; otherwise, it will complete the usurpation and write its own information into the database table to become the scheduler for the group and return. Here it is assumed that scheduler B receives the request from executor A first, writes its own address and application group A information into the database table, and returns scheduler B to executor A; subsequently, scheduler C finds that application group A already has a scheduler B record, and returns scheduler B to executor B. Further, in step 3, the PING alive detection between schedulers uses HTTP protocol communication, which filters out schedulers with high load.

[0153] Step 4, under the service discovery mechanism, all executors under the entire application group A will eventually connect to the same scheduler, and the subsequent executors will send heartbeat reports to this scheduler, including the executor address and the executor load information. If this scheduler node is abnormal during the period, all executors under the application group will request other surviving schedulers, and the surviving schedulers will compete for a distributed lock, the first one to get the lock will usurp, write its own information into the application group associated scheduler address information table, become the scheduler for the group and return, and then the executors connect to the same scheduler.

[0154] Step 5: The scheduler maintains the cluster node information of the application group executors it holds, and periodically pulls task information associated with all applications from the database for scheduling. Tasks are sent to executors with normal load and heartbeat. Further, in Step 5, the scheduler checks the executor load by: the executor periodically reporting node resource information (CPU and memory, etc.) along with the application group; the scheduler adding executors under the same application group to the same set according to the application group dimension; removing executors from the set if no heartbeat is reported more than a preset number of times, and removing them from the set if CPU and memory exceed preset thresholds.

[0155] Step 6: After receiving the task from the scheduler, the executor executes it locally and reports the execution result. The scheduler will perform failover and retry for abnormal execution results. Further, in step 6, if the executor fails to execute locally, it will retry according to a custom retry strategy. If the retry still fails, the scheduler will assign the task to another executor for processing.

[0156] This invention also provides a computer device for performing the above-described... Figure 2 and Figure 3 The distributed task scheduling method shown is illustrated.

[0157] Please see Figure 8 , Figure 8 This is a schematic diagram of the structure of a computer device provided in an optional embodiment of the present invention, such as... Figure 8 As shown, the computer device includes one or more processors 10, memory 20, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components communicate with each other via different buses and can be mounted on a common motherboard or otherwise installed as needed. The processors can process instructions executed within the computer device, including instructions stored in or on memory to display graphical information of a GUI on external input / output devices (such as display devices coupled to the interfaces). In some alternative implementations, multiple processors and / or multiple buses can be used with multiple memories, if desired. Similarly, multiple computer devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 8 Take a processor 10 as an example.

[0158] Processor 10 may be a central processing unit, a network processor, or a combination thereof. Processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The programmable logic device may be a complex programmable logic device (CAMP), a field-programmable gate array (FPGA), a general-purpose array logic (GDA), or any combination thereof.

[0159] The memory 20 stores instructions executable by the at least one processor 10 to cause the at least one processor 10 to perform the methods illustrated by the above embodiments.

[0160] The memory 20 can include a program storage area and a data storage area. The program storage area can store an operating system, application programs required by at least one function, and the like. The data storage area can store data created according to the use of the computer device, and the like. In addition, the memory 20 can include a high-speed random access memory, and can further include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some alternative embodiments, the memory 20 can optionally include a memory disposed remotely from the processor 10, which can be connected to the computer device through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0161] The memory 20 can include a volatile memory such as a random access memory, and can further include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk, and a combination of the above-mentioned kinds of memories.

[0162] The computer device further includes a communication interface 30 for communication of the computer device with other devices or communication networks.

[0163] The embodiments of the present application also provide a computer readable storage medium. The above-mentioned methods according to the embodiments of the present application can be implemented in hardware, firmware, or recorded in a storage medium, or stored in a remote storage medium or a non-transitory machine readable storage medium and downloaded to a local storage medium through network downloading of computer code, so that the methods described herein can be processed by such software on a storage medium using a general purpose computer, a special purpose processor, or programmable or special purpose hardware. The storage medium can be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid state disk, and the like. Further, the storage medium can also include a combination of the above-mentioned kinds of memories. It can be understood that the computer, the processor, the microprocessor controller, or the programmable hardware includes a storage component that can store or receive software or computer code, when the software or computer code is accessed and executed by the computer, the processor, or the hardware, the methods illustrated by the above embodiments are implemented.

[0164] Part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, can invoke or provide the method and / or technical solutions according to the present application. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source files, executable files, installation package files and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0165] Although the embodiments of the present application are described in conjunction with the drawings, various modifications and changes can be made by those skilled in the art without departing from the spirit and scope of the present application, and such modifications and changes fall within the scope defined by the appended claims.

Claims

1. A method for distributed task scheduling, the method comprising: The application relates to a method for a scheduler cluster module, the scheduler cluster module comprising a plurality of schedulers, the scheduler cluster module being connected with an executor cluster module, the executor cluster module comprising a plurality of application group clusters, each application group cluster comprising a plurality of executors; the method comprising: sending a scheduler address list to the executor cluster module, so that the executor cluster module controls each executor in any application group cluster to send a service discovery request based on the scheduler address list, the service discovery request carrying application group cluster information; receiving the service discovery request sent by the executor cluster module in sequence, and determining a target scheduler by using a preset database table based on the received service discovery request, the preset database table being an information table recording each application group cluster and a corresponding scheduler; sending an actual scheduler address of the target scheduler to the executor cluster module, so that a target application group cluster in the executor cluster module is connected with the target scheduler, the target application group cluster being the same application group cluster as the plurality of executors sending the service discovery request; when the target scheduler is connected with the target application group cluster, acquiring a distributed scheduling task, and sending the distributed scheduling task to the target application group cluster after splitting, so that the plurality of executors in the target application group cluster respectively execute the split distributed scheduling task and obtain a resource information set; receiving a heartbeat report signal carrying the resource information set sent by the executor cluster module.

2. The method of claim 1, wherein, Based on the received service discovery request, a preset database table is used to perform small-scale distributed master selection to determine a target scheduler, comprising: acquiring the preset database table; judging whether there is record information corresponding to the service discovery request in the preset database table; when the record information exists in the preset database table, sending a PING request to a first scheduler corresponding to the record information to detect whether the first scheduler is alive; when the first scheduler is alive, the first scheduler is determined as the target scheduler; when the first scheduler is abnormal, a scheduler receiving the service discovery request is determined as the target scheduler.

3. The method of claim 1, wherein, The method further comprises: based on the resource information set, a first same set is determined based on a plurality of executors in a same application group cluster; when the heartbeat report signal is not received within a preset number of times or the resource information set does not meet a preset condition, a corresponding executor is deleted from the same set to obtain a second same set; the split distributed scheduling task is sent to an executor in the second same set.

4. The method of claim 1, wherein, The method further comprises: an abnormal execution result is determined based on the heartbeat report signal; the abnormal execution result is failovered and retried.

5. The method of claim 1, wherein, The method further comprises: when the target scheduler is abnormal, other alive schedulers are used to compete for a target distributed lock; The first obtained scheduler of the target distributed lock is taken as a new target scheduler, and a scheduler address of the new target scheduler is sent to the target application group cluster.

6. A method for distributed task scheduling, the method comprising: The method comprises the following steps: When receiving a scheduler address list sent by the scheduler cluster module, based on the scheduler address list, control each executor in any application group cluster to send a service discovery request in turn, so that the scheduler cluster module determines a target scheduler based on the service discovery request, and the service discovery request carries application group cluster information; Receive the actual scheduler address of the target scheduler sent by the scheduler cluster module, and based on the actual scheduler address, control the target application group cluster to connect with the target scheduler, the target application group cluster being the same application group cluster as the multiple executors sending the service discovery request; When receiving multiple sub-scheduling tasks sent by the target scheduler, execute the multiple sub-scheduling tasks by using the multiple executors in the target application group cluster to obtain resource information set; Send a heartbeat report signal carrying the resource information set to the target scheduler in the scheduler cluster module.

7. The method of claim 6, wherein, The method further comprises: When receiving a target scheduler exception instruction sent by the scheduler cluster module, control the target application group cluster to send a new service discovery request to the scheduler cluster module, so that the scheduler cluster module determines a new target scheduler among other surviving schedulers based on the service discovery request; Receive the scheduler address of the new target scheduler sent by the scheduler cluster module.

8. The method of claim 6, wherein, The method further comprises: When the multiple executors in the target application group cluster fail to execute the multiple sub-scheduling tasks, re-execute the multiple sub-scheduling tasks according to a pre-defined custom retry strategy; When re-executing the multiple sub-scheduling tasks fails, send a task failure instruction to the scheduler cluster module, so that the scheduler cluster module sends the multiple sub-scheduling tasks to the multiple executors in other application group clusters.

9. A distributed task scheduling system, characterized by, The system comprises a scheduler cluster module and an executor cluster module, the scheduler cluster module comprises multiple schedulers, the executor cluster module comprises multiple application group clusters, each application group cluster comprises multiple executors, and the scheduler cluster module and the executor cluster module are connected; The scheduler cluster module is configured to perform the distributed task scheduling method in any one of claims 1 to 5; The executor cluster module is configured to perform the distributed task scheduling method in any one of claims 6 to 8.

10. A computer program product, characterised in that, The computer program product comprises computer instructions for causing a computer to perform the distributed task scheduling method in any one of claims 1 to 8.

Images