Distributed task scheduling method, system and program product

Abstract

The invention relates to the technical field of task scheduling, and discloses a distributed task scheduling method and system and a program product. Based on lock-free design, grouping isolation is introduced through a divide-and-conquer thought, application groups are used as scheduling units of schedulers, each application group is only connected to a certain fixed scheduler during operation, and the scheduling efficiency is improved. Each scheduler only schedules all tasks under the application group associated with the executor which keeps heartbeat with the scheduler at present, locking is not needed during scheduling, high availability and high performance can be met at the same time, and large-scale distributed task scheduling is easy. Furthermore, during scheduling, each scheduler only schedules the tasks in the group of the scheduler, the tasks are mutually independent and do not need to wait, performance loss caused by additional level locking on the database is avoided, and by expanding the scheduler, the overall concurrency and execution efficiency of the tasks can be improved, and horizontal capacity expansion is easy.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to a distributed task scheduling method, system and program product. Background Art

[0002] With the increasing complexity of business, traditional single-node task scheduling systems can no longer meet the needs of modern enterprises. When dealing with large-scale, high-concurrency, and high-reliability tasks, single-node task scheduling systems face many problems, such as performance bottlenecks, poor fault tolerance, and limited scalability. Therefore, distributed task scheduling technology came into being, which can assign tasks to multiple nodes for concurrent execution and improve the overall task processing capabilities. However, most existing distributed task scheduling relies on database row-level locks to ensure scheduling consistency. When a large number of tasks appear, frequent lock competition will cause performance bottlenecks, and at the same time, only one master scheduler that has obtained the lock is scheduling tasks, which will also bring difficulties in horizontal expansion. Summary of the invention

[0003] In view of this, the present invention provides a distributed task scheduling method, system and program product to solve the problem that most existing distributed task scheduling relies on database row-level locks to ensure scheduling consistency. When a large number of tasks appear, frequent lock competition will cause performance bottlenecks, and there is only one main scheduler that obtains the lock to schedule tasks at the same time, which will also bring difficulties in horizontal expansion.

[0004] In a first aspect, the present invention provides a distributed task scheduling method for a scheduler cluster module, the scheduler cluster module includes multiple schedulers, the scheduler cluster module is connected to an executor cluster module, the executor cluster module includes multiple application group clusters, each application group cluster includes multiple executors; the method includes:

[0005] Send the scheduler address list to the executor cluster module, so that the executor cluster module controls each executor in any application group cluster to send a service discovery request based on the scheduler address list, and the service discovery request carries the application group cluster information; receive the service discovery requests sent by the executor cluster module in turn, and use the preset database table to perform small distributed master election based on the received service discovery requests to determine the target scheduler; send the actual scheduler address of the target scheduler to the executor cluster module, so that the target application group cluster in the executor cluster module is connected to the target scheduler, and the target application group cluster includes the same application group cluster where multiple executors sending service discovery requests are located; when the target scheduler is connected to the target application group cluster, obtain the distributed scheduling task, and split the distributed scheduling task and send it to the target application group cluster, so that multiple executors in the target application group cluster respectively execute the split distributed scheduling tasks and obtain the resource information set; receive the heartbeat report signal carrying the resource information set sent by the executor cluster module.

[0006] The distributed task scheduling method provided by the present invention sends a scheduler address list to the executor cluster module, prompting the executor to send a service discovery request carrying application group cluster information. The scheduler cluster module can perform small distributed master selection based on these requests and the preset database table, and then determine the target scheduler corresponding to the target application group cluster. By using the application group as the scheduling unit of the scheduler, each application group will only be connected to a fixed scheduler when running, and each scheduler will only schedule all tasks under the application group associated with the executor that currently maintains a heartbeat with itself. There is no need to lock during scheduling, which can meet high availability and high performance at the same time, and is easy to schedule large-scale distributed tasks. Further, the target scheduler is used to split the distributed scheduling task and send it to the target application group cluster, so that multiple executors in the target application group cluster respectively execute the split distributed scheduling tasks and obtain a resource information set. In this way, during the scheduling period, each scheduler only schedules the tasks under its own group, which are independent of each other and do not need to wait, and there is no performance loss caused by additional level locks on the database. Then, by extending the scheduler, the overall concurrency and execution efficiency of the task can be improved, and horizontal expansion is easy.

[0007] In an optional implementation, based on the received service discovery request, a small distributed master election is performed using a preset database table to determine a target scheduler, including:

[0008] Obtain a preset database table, which is an information table that records each application group cluster and the corresponding scheduler; determine whether there is record information corresponding to the service discovery request in the preset database table; when there is record information in the preset database table, send a PING request to the first scheduler corresponding to the record information to detect whether the first scheduler is alive; when the first scheduler is alive, determine the first scheduler as the target scheduler; when the first scheduler is abnormal, determine the scheduler that receives the service discovery request as the target scheduler.

[0009] The distributed task scheduling method provided by the present invention records the information of each application group cluster and the corresponding scheduler by presetting a database table, which can ensure that there is a reasonable correspondence between the target scheduler selected subsequently and the corresponding application group cluster, which helps to maintain the orderliness and consistency of task scheduling at the group level. Further, when there is record information in the table, a PING request is sent to detect whether the corresponding first scheduler is alive, and the surviving scheduler is determined as the target scheduler. Through this detection mechanism, it is possible to avoid assigning tasks to schedulers that may have failed or abnormal, ensuring that the application group cluster can be connected to a scheduler that works normally and can stably provide scheduling services, thereby ensuring that the task scheduling process will not be interrupted due to problems with the scheduler itself, and improving the stability of the entire distributed task scheduling system. Further, when the first scheduler is abnormal, the scheduler that receives the service discovery request can be quickly determined as the target scheduler, which minimizes the impact of the scheduler failure on the task execution, and effectively enhances the system's fault tolerance for abnormal situations at the scheduler level. Therefore, by implementing the present invention, it is possible to reasonably arrange the connection relationship between each application group cluster and the scheduler in real time according to the actual situation of the scheduler, better adapt to the dynamic changes of scheduler resources in a distributed environment, and maintain the efficiency of resource utilization and the smoothness of task scheduling.

[0010] In an optional embodiment, the method further includes:

[0011] Based on the resource information set, a first identical set is determined based on multiple executors in the cluster grouped based on the same application; when no heartbeat report signal is received within a preset number of times or the resource information set does not meet the preset conditions, the corresponding executor is deleted from the identical set to obtain a second identical set; the split distributed scheduling task is sent to the executor in the second identical set.

[0012] The distributed task scheduling method provided by the present invention determines the first identical set through multiple executors in the same application group cluster, and can preliminarily integrate and classify the executors with the ability to execute tasks under the same group, so as to facilitate subsequent unified management and task allocation. Furthermore, by setting a preset number of times and preset conditions to determine whether the executor meets the requirements for continuing to participate in task allocation, those executors that are currently healthy and have good resource status and can reliably execute tasks can be accurately screened out. Sending the split distributed scheduling tasks to the executors in the second identical set can effectively avoid assigning tasks to executors that may have problems or are too loaded to execute tasks smoothly, greatly improving the probability of successful execution of tasks and ensuring the reliability of distributed task scheduling.

[0013] In an optional embodiment, the method further includes:

[0014] Determine abnormal execution results based on heartbeat reporting signals; perform failover and retry on abnormal execution results.

[0015] The distributed task scheduling method provided by the present invention determines abnormal execution results based on heartbeat reporting signals, and can monitor the execution of tasks by the executor in real time, thereby avoiding the problem that problems with tasks are not known for a long time, resulting in the obstruction of subsequent business processes, and ensuring that the task execution status is within the monitorable range. Furthermore, abnormal execution results are fault-transferred and retried, thereby enhancing the reliability of task execution.

[0016] In an optional embodiment, the method further includes:

[0017] When the target scheduler is abnormal, other surviving schedulers are used to compete for the target distributed lock; the first scheduler that obtains the target distributed lock is used as the new target scheduler, and the scheduler address of the new target scheduler is sent to the target application group cluster.

[0018] The distributed task scheduling method provided by the present invention can quickly start the selection of a new target scheduler by using other surviving schedulers to compete for the target distributed lock when the target scheduler is abnormal, thereby avoiding the problem of long-term stagnation of task scheduling, ensuring the continuity of the task scheduling process, and ensuring that the tasks of the target application group cluster can be continuously arranged and executed. Furthermore, the first scheduler that obtains the target distributed lock is used as the new target scheduler, and the scheduler address of the new target scheduler is sent to the target application group cluster, thereby realizing seamless switching of scheduling services, minimizing the impact of scheduler failure on task execution, and maintaining the stable operation of the entire distributed task scheduling system.

[0019] In a second aspect, the present invention provides a distributed task scheduling method for an executor cluster module, the executor cluster module includes multiple application group clusters, each application group cluster includes multiple executors, the executor cluster module is connected to a scheduler cluster module, and the scheduler cluster module includes multiple schedulers; the method includes:

[0020] When receiving the scheduler address list sent by the scheduler cluster module, based on the scheduler address list, control each executor in any application group cluster to send a service discovery request in turn, so that the scheduler cluster module determines the target scheduler based on the service discovery request, and the service discovery request carries the application group cluster information; receive the actual scheduler address of the target scheduler sent by the scheduler cluster module, and based on the actual scheduler address, control the target application group cluster to connect with the target scheduler, and the target application group cluster is the same application group cluster where multiple executors sending service discovery requests are located; when receiving multiple sub-scheduling tasks sent by the target scheduler, use multiple executors in the target application group cluster to execute multiple sub-scheduling tasks to obtain a resource information set; send a heartbeat report signal carrying the resource information set to the target scheduler in the scheduler cluster module.

[0021] In the distributed task scheduling method provided by the present invention, after the executor cluster module receives the scheduler address list sent by the scheduler cluster module, it will control each executor in the application group cluster to send a service discovery request carrying the application group cluster information, so that the executor can actively indicate to the scheduler cluster which group it belongs to, thereby guiding the scheduler cluster to accurately determine the corresponding target scheduler based on these requests. Furthermore, the executor cluster module receives the actual scheduler address of the target scheduler, and controls the corresponding target application group cluster to establish a connection with the target scheduler, so that the executor under each application group cluster can be accurately connected to the scheduler suitable for serving its own group task, thereby avoiding the problem of task allocation confusion that may be caused by the executor blindly connecting to the scheduler. Furthermore, multiple executors in the target application group cluster are used to execute multiple sub-scheduling tasks, which ensures the rationality and efficiency of task scheduling and execution, and improves the efficiency of task execution.

[0022] In an optional embodiment, the method further includes:

[0023] When receiving a target scheduler exception instruction sent by the scheduler cluster module, control the target application group cluster to send a new service discovery request to the scheduler cluster module, so that the scheduler cluster module determines a new target scheduler among other surviving schedulers based on the service discovery request; receive the scheduler address of the new target scheduler sent by the scheduler cluster module.

[0024] The distributed task scheduling method provided by the present invention can respond quickly after the executor cluster module receives the target scheduler exception instruction sent by the scheduler cluster module, and control the target application group cluster to send a new service discovery request, thereby avoiding the task scheduling stagnation state due to the target scheduler failure, ensuring the continuous advancement of the task scheduling process, and minimizing the interruption effect on task execution caused by the scheduler exception. Furthermore, by sending a new service discovery request to the scheduler cluster module, prompting it to determine a new target scheduler in other surviving schedulers based on the request, and receiving the scheduler address of the new target scheduler, the executor cluster module can quickly re-establish the connection with the new scheduler for the target application group cluster, ensuring that the task scheduling process will not be interrupted due to problems with the scheduler itself, and improving the stability of the entire distributed task scheduling system.

[0025] In an optional embodiment, the method further includes:

[0026] When multiple executors in the target application group cluster fail to execute multiple sub-scheduling tasks, the multiple sub-scheduling tasks are re-executed according to the preset custom retry strategy; when the re-execution of multiple sub-scheduling tasks fails, a task failure instruction is sent to the scheduler cluster module, so that the scheduler cluster module sends the multiple sub-scheduling tasks to multiple executors in other application group clusters.

[0027] The distributed task scheduling method provided by the present invention re-executes multiple sub-scheduling tasks by presetting a custom retry strategy, maximizes the use of existing executor resources, effectively improves the success rate of task execution, enhances the reliability of task execution, avoids the extra resource consumption and coordination costs that may be generated by prematurely transferring tasks to other executors, and realizes the deep mining and reasonable use of executor resources within the group cluster, thereby improving resource utilization efficiency. Furthermore, when re-executing multiple sub-scheduling tasks fails, a task failure instruction is sent to the scheduler cluster module, prompting it to transfer the task to the executor within other application group clusters. By reasonably controlling the timing of task transfer, it helps to maintain the efficient operation of the entire distributed task scheduling system, so that system resources can be more focused on tasks that really need to reallocate execution resources.

[0028] In a third aspect, the present invention provides a distributed task scheduling system, which includes: a scheduler cluster module and an executor cluster module, the scheduler cluster module includes multiple schedulers, the executor cluster module includes multiple application group clusters, each application group cluster includes multiple executors, and the scheduler cluster module and the executor cluster module are connected; the scheduler cluster module is used to execute the distributed task scheduling method of the first aspect or any corresponding embodiment thereof; the executor cluster module is used to execute the distributed task scheduling method of the second aspect or any corresponding embodiment thereof.

[0029] The distributed task scheduling system provided by the present invention is easy to expand horizontally by executing the corresponding distributed task scheduling methods in the scheduler cluster module and the executor cluster module respectively, supports large-scale distributed task scheduling, meets the requirements of high availability and high performance, and can make full use of cluster resources, thereby improving the overall performance and throughput of the system.

[0030] In a fourth aspect, the present invention provides a computer program product, comprising computer instructions, which are used to enable a computer to execute the distributed task scheduling method of the above-mentioned first aspect or any corresponding embodiment, or to execute the distributed task scheduling method of the above-mentioned second aspect or any corresponding embodiment. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 is a structural block diagram of a distributed task scheduling system according to an embodiment of the present invention;

[0033] Figure 2 is a flow chart of a distributed task scheduling method according to an embodiment of the present invention;

[0034] Figure 3 is a flow chart of another distributed task scheduling method according to an embodiment of the present invention;

[0035] Figure 4 is a flow chart of a distributed task scheduling method based on group isolation according to an embodiment of the present invention;

[0036] Figure 5 is a schematic diagram of a process in which an executor performs service discovery on a scheduler according to an embodiment of the present invention;

[0037] Figure 6 is a schematic diagram of a process of performing load inspection on an actuator according to an embodiment of the present invention;

[0038] Figure 7 is a functional module structure diagram of a distributed task scheduling device based on group isolation according to an embodiment of the present invention;

[0039] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0041] At present, the existing distributed task scheduling technology solutions still have some shortcomings:

[0042] 1. Existing technical solutions essentially rely on database row-level locks. When a large number of scheduling tasks appear, serious lock competition will occur, which is prone to database performance bottlenecks and cannot support large-scale distributed task scheduling.

[0043] 2. During the scheduling period, the task is scheduled by the scheduler that obtains the database row-level lock. Only one scheduler can obtain the lock at the same time, and other schedulers must wait. The main node is under great pressure and will have performance bottlenecks, affecting the overall concurrency and execution efficiency of the task, and has no horizontal scalability.

[0044] 3. If the hardware performance of the executors differs greatly, it is impossible to control the task allocation of different nodes to reduce the pressure on certain nodes, the cluster nodes cannot be fully mobilized, and resources cannot be used reasonably.

[0045] According to an embodiment of the present invention, an embodiment of a distributed task scheduling method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0046] In this embodiment, a distributed task scheduling method is provided, which can be used for example Figure 1 The scheduler cluster module 11 shown includes a plurality of schedulers 111 .

[0047] Furthermore, the scheduler cluster module 11 is connected to the executor cluster module 12 , and the executor cluster module 12 includes a plurality of application group clusters 121 , and each application group cluster includes a plurality of executors 1211 .

[0048] Furthermore, the scheduler 111 represents a scheduling node for tasks, which can split and distribute the user's scheduling tasks and find a suitable executor to execute the tasks.

[0049] The executor 1211 represents the actual execution node of the task, which can receive and execute the task issued by the scheduler and report relevant status information.

[0050] Figure 2 is a flow chart of a distributed task scheduling method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0051] Step S201 : sending a scheduler address list to an executor cluster module, so that the executor cluster module sequentially controls each executor in any application group cluster to send a service discovery request based on the scheduler address list.

[0052] The service discovery request carries the application group cluster information, which indicates the process of the executor requesting the scheduler for the actual connection address. The executors under the same application group cluster will be connected to the same scheduler.

[0053] Furthermore, the scheduler address list may include address information (such as IP addresses and port numbers, etc.) of multiple schedulers 111 in the scheduler cluster module 11 .

[0054] In one instance, there are three schedulers in the scheduler cluster module 11, namely scheduler A (IP address: 192.168.1.100, port number: 8080), scheduler B (IP address: 192.168.1.101, port number: 8080) and scheduler C (IP address: 192.168.1.102, port number: 8080). The generated scheduler address list contains address information entries corresponding to these three schedulers.

[0055] Specifically, the scheduler cluster module 11 can send the corresponding scheduler address list to the executor cluster module 12. The communication process can follow the corresponding network protocol (such as TCP / IP protocol, etc.) to ensure that the address list can be accurately transmitted to the executor cluster module 12. For example, the data packet containing the scheduler address list can be encapsulated and sent according to the network address of the executor cluster module 12. After receiving the data packet, the executor cluster module 12 performs an unpacking operation to obtain the contents of the scheduler address list.

[0056] Furthermore, after receiving the scheduler address list, the executor cluster module 12 can start the executor 1211 in the same application group cluster 121 according to the content of the scheduler address list, and enable each executor 1211 in the application group cluster 121 to send a service discovery request to the scheduler cluster module 11.

[0057] Furthermore, with the support of the service discovery mechanism, the executors 1211 under the entire application group cluster 121 will eventually be connected to the same scheduler 111.

[0058] Step S202, receiving service discovery requests sent by the executor cluster modules in sequence, and performing small distributed master election based on the received service discovery requests using a preset database table to determine a target scheduler.

[0059] The preset database table is an information table recording each application group cluster and the corresponding scheduler; the target scheduler is a determined scheduler connected to each executor 1211 in the application group cluster 121 that sends the service discovery request.

[0060] Specifically, after receiving the service discovery request, the scheduler 111 in the scheduler cluster module 11 may use a preset database table to perform a small distributed leader election, and determine the corresponding target scheduler through the leader election.

[0061] Step S203: Send the actual scheduler address of the target scheduler to the executor cluster module, so that the target application group cluster in the executor cluster module is connected to the target scheduler.

[0062] The target application group cluster is the same application group cluster 121 where the multiple executors sending the service discovery request in step S201 are located.

[0063] Specifically, after the target scheduler is determined, the actual scheduler address of the target scheduler may be sent to the executor cluster module 12 .

[0064] Furthermore, after receiving the actual scheduler address of the target scheduler, the executor cluster module 12 can connect the target application group cluster with the target scheduler according to the actual scheduler address, that is, each executor 1211 in the target application group cluster is connected to the target scheduler.

[0065] Step S204, when the target scheduler is connected to the target application group cluster, it obtains the distributed scheduling task, splits the distributed scheduling task and sends it to the target application group cluster, so that multiple executors in the target application group cluster respectively execute the split distributed scheduling task and obtain a resource information set.

[0066] The resource information set may include CPU and memory information, etc., which may reflect the workload and resource utilization of the executor.

[0067] Specifically, when the target scheduler is successfully connected to the target application group cluster, the target scheduler can obtain the corresponding distributed scheduling task.

[0068] Furthermore, the obtained distributed scheduling tasks may be split and sent to each executor 1211 in the target application group cluster.

[0069] Furthermore, each executor 1211 executes the corresponding dispatched task respectively and can form a corresponding resource information set according to the task execution result.

[0070] Step S205: receiving a heartbeat report signal carrying a resource information set sent by the executor cluster module.

[0071] The heartbeat report signal refers to a periodic signal sent by the executor to the scheduler, and may include the executor address and executor load information.

[0072] Specifically, the executor cluster module 12 can send a heartbeat report signal carrying a resource information set to the scheduler cluster module 11. In this way, the target scheduler in the scheduler cluster module 11 can understand the resource utilization of each executor 1211 in the target application group cluster in real time, thereby reasonably allocating scheduling tasks.

[0073] The distributed task scheduling method provided in this embodiment sends a list of scheduler addresses to the executor cluster module, prompting the executor to send a service discovery request carrying application group cluster information. The scheduler cluster module can perform small distributed master selection based on these requests and the preset database table, and then determine the target scheduler corresponding to the target application group cluster. By using the application group as the scheduling unit of the scheduler, each application group will only be connected to a fixed scheduler when running, and each scheduler will only schedule all tasks under the application group associated with the executor that currently maintains a heartbeat with itself. There is no need to lock during scheduling, which can meet high availability and high performance at the same time, and is easy to schedule large-scale distributed tasks. Further, the target scheduler is used to split the distributed scheduling task and send it to the target application group cluster, so that multiple executors in the target application group cluster respectively execute the split distributed scheduling tasks and obtain a resource information set. In this way, during the scheduling period, each scheduler only schedules the tasks under its own group, which are independent of each other and do not need to wait, and there is no performance loss caused by additional level locks on the database. Then, by extending the scheduler, the overall concurrency and execution efficiency of the task can be improved, and horizontal expansion is easy.

[0074] In an optional implementation, in the above step S202, based on the received service discovery request, a small distributed master election is performed using a preset database table to determine a target scheduler, including:

[0075] Step a1, obtaining a preset database table.

[0076] Step a2: determine whether there is record information corresponding to the service discovery request in the preset database table.

[0077] Step a3: When there is record information in the preset database table, a PING request is sent to the first scheduler corresponding to the record information to detect whether the first scheduler is alive.

[0078] Step a4: when the first scheduler survives, the first scheduler is determined as the target scheduler.

[0079] Step a5: When the first scheduler is abnormal, the scheduler that receives the service discovery request is determined as the target scheduler.

[0080] Specifically, if the preset database table contains record information corresponding to the currently received service discovery request, it means that other executors in the application group cluster that sent the service discovery request have previously requested the scheduler to elect. At this time, it is only necessary to send a PING request to detect whether the first scheduler in the record is alive. Among them, the PING survival detection between schedulers can use HTTP protocol communication, and further filter out schedulers with higher loads.

[0081] Furthermore, if the first scheduler in the record survives, it will directly return to the scheduler of the group, that is, it will directly determine the scheduler as the target scheduler. Through this detection mechanism, it is possible to avoid assigning tasks to schedulers that may have failed or abnormal, ensuring that the application group cluster can connect to a scheduler that is working normally and can provide scheduling services stably, thereby ensuring that the task scheduling process will not be interrupted due to problems with the scheduler itself, and improving the stability of the entire distributed task scheduling system. Otherwise, the usurpation is completed, and the information of the scheduler that currently receives the service discovery request is written into the database table, becoming the target scheduler of the application group cluster, minimizing the impact of scheduler failures on task execution, and effectively enhancing the system's fault tolerance for abnormal situations at the scheduler level.

[0082] In an optional implementation, the distributed task scheduling method of an embodiment of the present invention further includes: based on a resource information set, determining a first identical set based on multiple executors in the same application grouping cluster; when no heartbeat report signal is received within a preset number of times or the resource information set does not meet a preset condition, deleting the corresponding executor from the identical set to obtain a second identical set; and sending the split distributed scheduling task to the executor in the second identical set.

[0083] Specifically, after receiving the resource information set sent by the executor cluster module 12, the corresponding scheduler can add the executors under the same application group to the same set, i.e., the first same set, according to the dimension of the application group cluster 121. In this way, the executors with the ability to execute tasks under the same group can be preliminarily integrated and classified, which is convenient for subsequent unified management and task allocation.

[0084] Furthermore, if there is no heartbeat report for more than a preset number of times or the CPU and memory in the resource information set exceed the preset threshold, the corresponding executor 1211 will be deleted from the first identical set to obtain a second identical set. In this way, those executors that are currently healthy and in good resource status and can reliably execute tasks can be accurately screened out.

[0085] Furthermore, continuing to send the split distributed scheduling tasks to the executors in the second identical set can effectively avoid allocating tasks to executors that may have problems or are too loaded to execute tasks smoothly, greatly improving the probability of successful execution of tasks and ensuring the reliability of distributed task scheduling.

[0086] In an optional implementation, the distributed task scheduling method of the embodiment of the present invention further includes: determining an abnormal execution result based on a heartbeat report signal; and performing a failover and retry on the abnormal execution result.

[0087] Specifically, the scheduler can periodically receive heartbeat report signals from each executor. These heartbeat report signals contain rich information, such as the address of the executor, the current load information of the executor (such as CPU usage, memory usage, etc.), and key status information related to the task being executed.

[0088] Furthermore, by analyzing the heartbeat report signal, we can extract content related to the task execution status. For example, we can check whether there are any error prompts in the task execution, whether the task execution progress is stagnant for a long time, whether the executor load has an abnormal surge, etc. These situations can further determine the abnormal execution results:

[0089] (1) Task execution error reporting: If the heartbeat report signal clearly indicates that a specific error code occurred when executing a task (such as an error code for a database connection failure, a syntax error prompt when the program is running, etc.), or the executor feedbacks that an unresolvable abnormal situation was encountered during the task execution (such as a file read failure, a network request timeout, etc.), the scheduler can determine that the task has an abnormal execution result based on this.

[0090] (2) Abnormal task progress: If the heartbeat report signal shows that the execution progress of a task remains unchanged at a certain stage for a long time, exceeding the normal progress range that the task should reach within the corresponding time period, this may also mean that the task execution is blocked or other abnormal conditions occur, and the scheduler will identify it as an abnormal execution result. For example, a data analysis task should normally complete the data reading and preliminary processing stage within 1 minute, but the heartbeat report shows that the executor has been stagnant for 20 minutes at this stage, then it is likely that there is a problem with the task execution.

[0091] (3) Abnormal executor load: When the executor load information in the heartbeat report signal shows that the executor's CPU usage is too high (for example, exceeding 90%) or the memory usage is almost full, the task may not be able to continue to execute normally. At this time, the scheduler will also determine that the task related to the executor has an abnormal execution result, because the excessive load may cause the task to wait for resources or directly fail to execute due to insufficient resources.

[0092] Furthermore, abnormal execution results can be failed over and retried.

[0093] In one example, failover for abnormal execution results can be achieved through the following steps:

[0094] (1) Selecting a suitable replacement executor: Once a task with an abnormal execution result is identified, the scheduler can first select other suitable executors from the entire executor cluster to take over the task. The scheduler will consider multiple factors to select replacement executors, such as the current load of other executors (executors with lighter loads and more abundant resources are preferred), the application group cluster they are in (try to select executors that belong to the same application group cluster as the original executor to ensure the similarity and compatibility of the task execution environment, etc.), and past task execution success rates, etc., to ensure that the selected executor has a high probability of successfully executing the task.

[0095] (2) Update task assignment information: The scheduler can remove the abnormal task from the task queue of the original executor and reallocate it to the task queue of the selected replacement executor. At the same time, it updates the relevant task scheduling records, marking the task as having undergone a failover and the new executor information, etc., to facilitate subsequent tracking and management of task execution. For example, a task was originally assigned to executor A for execution. After an abnormality occurs, the scheduler transfers the task to executor B and records this change in the task management module of the scheduling system, which facilitates subsequent monitoring of the execution progress of the task on executor B.

[0096] Retrying abnormal execution results can be achieved through the following steps:

[0097] (1) Determine the number of retries and the time interval: The scheduler can decide to retry tasks with abnormal execution results based on a preset custom retry strategy. The retry strategy usually specifies the number of retries (for example, a maximum of 3 retries) and the time interval between each retry (for example, 1 minute between each retry). These parameters can be flexibly set based on factors such as different task types, importance, and past execution experience. The purpose is to maximize the probability of successful task execution while making reasonable use of system resources.

[0098] (2) Execute retry operations: According to the determined number of retries and time intervals, the scheduler can instruct the original executor to re-execute the abnormal task. Before each retry, the executor may perform some necessary preparations, such as cleaning up temporary files left over from the last execution, resetting related program states, etc., and then try to execute the task again. After each retry, it will still feedback the execution status to the scheduler through the heartbeat report signal. The scheduler continues to monitor these feedbacks to determine whether the retry is successful.

[0099] (3) Determine the retry result and further process: If the task is successfully executed within the specified number of retries, the scheduler can mark it as successfully completed and handle subsequent result collection, data integration, etc. according to the normal process. However, if the task still fails after multiple retries, the scheduler will perform a failover operation again, find other suitable executors to continue to try to execute the task, or take other corresponding measures according to the system settings (such as recording detailed reasons for failure, notifying relevant operation and maintenance personnel, etc.) to ensure that the task will not be completely shelved due to execution abnormalities, and try to ensure that the task can be successfully executed in the end, maintaining the reliability and stability of the entire distributed task scheduling system.

[0100] By determining abnormal execution results through heartbeat reporting signals, the execution of tasks by the executor can be monitored in real time, avoiding the problem that problems with tasks are not known for a long time, resulting in the obstruction of subsequent business processes, and ensuring that the task execution status is within the monitorable range. Furthermore, the abnormal execution results are failed over and retried, which enhances the reliability of task execution.

[0101] In an optional implementation, the distributed task scheduling method of an embodiment of the present invention also includes: when the target scheduler is abnormal, using other surviving schedulers to compete for the target distributed lock; using the first scheduler that obtains the target distributed lock as the new target scheduler, and sending the scheduler address of the new target scheduler to the target application group cluster.

[0102] Specifically, if the target scheduler has an exception during the task scheduling process, such as a sudden network failure that causes the loss of connection with the executor, a hardware failure that makes it unable to continue processing tasks, or a crash at the software level, etc. At this time, other surviving schedulers can be used to compete for a target distributed lock. The first one to get the lock will usurp the throne and write its own information into the scheduler address information table associated with the application group, becoming the scheduler of the group, and sending the address of the first scheduler to get the lock to the corresponding target application group cluster.

[0103] Among them, the target distributed lock represents a mechanism for ensuring that only one scheduler can obtain the scheduling right for the target application group cluster at the same time, which can usually be implemented based on some distributed lock services (such as the common distributed lock based on Redis or the distributed lock based on ZooKeeper, etc.).

[0104] Furthermore, the scheduler can try to acquire the target distributed lock through established rules and algorithms, entering a competitive state. For example, suppose there are three surviving schedulers, scheduler A, scheduler B, and scheduler C. They will simultaneously initiate a request to acquire the lock from the distributed lock service, and each will try to obtain the lock through corresponding logic.

[0105] In this embodiment, a distributed task scheduling method is provided, which can be used for example Figure 1 The executor cluster module 12 shown includes a plurality of application group clusters 121 , and each application group cluster includes a plurality of executors 1211 .

[0106] Furthermore, the executor cluster module 12 is connected to the scheduler cluster module 11 , and the scheduler cluster module 11 includes a plurality of schedulers 111 .

[0107] Figure 3 is a flow chart of a distributed task scheduling method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0108] Step S301, when receiving the scheduler address list sent by the scheduler cluster module, based on the scheduler address list, control each executor in any application group cluster to send a service discovery request in turn, so that the scheduler cluster module determines the target scheduler based on the service discovery request.

[0109] Specifically, after receiving the scheduler address list sent by the scheduler cluster module 11, the executor 1211 in the same application group cluster 121 can be started according to the content of the scheduler address list, and then each executor 1211 in the same application group cluster 121 sends a service discovery request to the scheduler cluster module 11 in turn.

[0110] Further, the scheduler cluster module 11 can determine the target scheduler connected to each executor 1211 in the application group cluster 121 according to the received service discovery request. The specific process can refer to the specific process of the target scheduler in the distributed task scheduling method for the scheduler cluster module 11, which will not be repeated here.

[0111] Step S302: receiving the actual scheduler address of the target scheduler sent by the scheduler cluster module, and controlling the target application group cluster to connect with the target scheduler based on the actual scheduler address.

[0112] The target application group cluster is the same application group cluster 121 where the multiple executors sending the service discovery request in step S301 are located.

[0113] Specifically, after receiving the actual scheduler address of the target scheduler sent by the scheduler cluster module 11, each executor 1211 in the target application group cluster can be connected to the target scheduler according to the actual scheduler address.

[0114] Step S303, when multiple sub-scheduling tasks sent by the target scheduler are received, multiple executors in the target application grouping cluster are used to execute the multiple sub-scheduling tasks to obtain a resource information set.

[0115] Specifically, when multiple sub-scheduling tasks sent by the target scheduler are received, each executor 1211 in the target application grouping cluster can be used to execute the corresponding dispatched scheduling tasks respectively and form a corresponding resource information set according to the task execution results.

[0116] Step S304: Send a heartbeat report signal carrying a resource information set to a target scheduler in the scheduler cluster module.

[0117] Specifically, when multiple sub-scheduling tasks are completed, a heartbeat report signal carrying a resource information set can be sent to the target scheduler in the scheduler cluster module 11. In this way, the target scheduler in the scheduler cluster module 11 can understand the resource utilization of each executor 1211 in the target application group cluster in real time, thereby reasonably allocating scheduling tasks.

[0118] In the distributed task scheduling method provided in this embodiment, after the executor cluster module receives the scheduler address list sent by the scheduler cluster module, it will control each executor in the application group cluster to send a service discovery request carrying the application group cluster information, so that the executor can actively indicate to the scheduler cluster which group it belongs to, thereby guiding the scheduler cluster to accurately determine the corresponding target scheduler based on these requests. Furthermore, the executor cluster module receives the actual scheduler address of the target scheduler, and controls the corresponding target application group cluster to establish a connection with the target scheduler, so that the executor under each application group cluster can be accurately connected to the scheduler suitable for serving its own group task, thereby avoiding the problem of task allocation confusion that may be caused by the executor blindly connecting to the scheduler. Furthermore, multiple executors in the target application group cluster are used to execute multiple sub-scheduling tasks, which ensures the rationality and efficiency of task scheduling and execution, and improves the efficiency of task execution.

[0119] In an optional embodiment, the distributed task scheduling method of an embodiment of the present invention also includes: when receiving a target scheduler exception instruction sent by the scheduler cluster module, controlling the target application group cluster to send a new service discovery request to the scheduler cluster module, so that the scheduler cluster module determines a new target scheduler from other surviving schedulers based on the service discovery request; and receiving the scheduler address of the new target scheduler sent by the scheduler cluster module.

[0120] Specifically, when the scheduler cluster module detects an abnormal situation in the target scheduler (for example, the target scheduler cannot perform task scheduling duties normally due to network failure, hardware failure or software crash), it will send a target scheduler exception instruction to the target application group cluster associated with it, avoiding the task scheduling stagnation state due to the target scheduler failure, ensuring the continuous advancement of the task scheduling process, and minimizing the interruption impact of the scheduler exception on task execution.

[0121] Furthermore, after receiving the target scheduler exception instruction, the target application group cluster sends a new service discovery request to the scheduler cluster module to request other surviving schedulers, that is, under the control of the target scheduler exception instruction, each executor 1211 in the target application group cluster is controlled to send a new service discovery request to the scheduler cluster module 11, ensuring that the task scheduling process will not be interrupted due to problems with the scheduler itself, thereby improving the stability of the entire distributed task scheduling system.

[0122] Further, according to the received new service discovery request, a new target scheduler connected to each executor 1211 in the target application group cluster can be re-determined from other surviving schedulers in the scheduler cluster module 11. The specific determination process can refer to the determination process in the distributed task scheduling method for the scheduler cluster module 11, which will not be repeated here.

[0123] Furthermore, the scheduler cluster module 11 may continue to send the scheduler address of the new target scheduler to the target application group cluster.

[0124] In an optional embodiment, the distributed task scheduling method of an embodiment of the present invention also includes: when multiple executors within the target application group cluster fail to execute multiple sub-scheduling tasks, re-executing the multiple sub-scheduling tasks according to a preset custom retry strategy; when re-executing multiple sub-scheduling tasks fails, sending a task failure instruction to the scheduler cluster module, so that the scheduler cluster module sends the multiple sub-scheduling tasks to multiple executors within other application group clusters.

[0125] Specifically, after receiving the multiple sub-scheduling tasks, the multiple executors in the target application group cluster execute the multiple sub-scheduling tasks locally.

[0126] Furthermore, if the local execution process fails, such as program errors (for example, syntax errors occur when the code is running, the external interface called returns an error prompt, etc.), resource exhaustion causes the task to be unable to continue (such as memory overflow, insufficient disk space affects data storage, etc.), and the task execution progress stagnates for a long time (exceeds the time range corresponding to the stage that should be reached under normal circumstances), then multiple sub-scheduling tasks can be re-executed according to the preset custom retry strategy, maximizing the use of existing executor resources, effectively improving the success rate of task execution, enhancing the reliability of task execution, avoiding the additional resource consumption and coordination costs that may be incurred by transferring tasks to other executors too early, and realizing in-depth mining and rational use of executor resources within this group cluster, thereby improving resource utilization efficiency.

[0127] Among them, the preset custom retry strategy can be pre-set according to the importance of the task, past execution experience, task type characteristics and other factors. For example, the strategy may stipulate that for some key business computing tasks, the number of retries is set to 3 times, and the time interval between each retry is 1 minute; for general data processing tasks, the number of retries is set to 2 times, and the interval between each retry is 30 seconds. If the current execution failure meets the triggering conditions of the retry strategy (for example, the retry limit has not been reached, etc.), then the corresponding executor will be instructed to prepare for the retry operation.

[0128] Furthermore, if the retry still fails, a task failure instruction can be sent to the scheduler cluster module, so that the scheduler cluster module sends multiple sub-scheduling tasks to multiple executors in other application group clusters for execution. By reasonably controlling the timing of task transfer, it helps to maintain the efficient operation of the entire distributed task scheduling system, so that system resources can be more focused on tasks that really need to reallocate execution resources.

[0129] In this embodiment, a distributed task scheduling system is provided. Figure 1 As shown, the distributed task scheduling system 1 includes a scheduler cluster module 11 and an executor cluster module 12, and the scheduler cluster module 11 and the executor cluster module 12 are connected.

[0130] Specifically, the scheduler cluster module 11 includes a plurality of schedulers 111 ; the executor cluster module 12 includes a plurality of application group clusters 121 , and each application group cluster includes a plurality of executors 1211 .

[0131] Further, the scheduler cluster module 11 is used to execute Figure 2The distributed task scheduling method shown; the executor cluster module 12 is used to execute Figure 3 The distributed task scheduling method shown.

[0132] The distributed task scheduling system provided in this embodiment is easy to expand horizontally by executing the corresponding distributed task scheduling methods in the scheduler cluster module and the executor cluster module respectively, supports large-scale distributed task scheduling, meets the requirements of high availability and high performance, and can make full use of cluster resources, thereby improving the overall performance and throughput of the system.

[0133] In one example, a distributed task scheduling method based on group isolation is provided, such as Figure 4 As shown, the following steps are included:

[0134] Step 1: deploy the scheduler cluster, wait for the executor to register and report the heartbeat, and the scheduler server maintains its own load information locally. Furthermore, in step 1, the scheduler refers to the scheduling node of the task, which splits and issues the user's scheduling task and finds a suitable executor to execute the task; the executor refers to the actual execution node of the task, which receives and executes the task issued by the scheduler and reports the relevant status information; the load information includes the node resource information (CPU and memory, etc.).

[0135] Step 2, fill in the list of scheduler addresses, start the executors under the same application group, and the executors periodically discover services, request any scheduler, and obtain the actual scheduler address corresponding to the application group. Further, in step 2, service discovery refers to the process in which the executor requests the actual connection address from the scheduler, and the executors under the same application group will connect to the same scheduler. Further, in step 2, the executor will bring the application group information when performing service discovery.

[0136] Step 3: After the scheduler receives the service discovery request from the executor, it will conduct a small distributed election: the database table contains an information table that records each group and the corresponding scheduler. If the scheduler finds that there is a record in the table, it means that another executor in the application group has requested the scheduler for election in advance. At this time, it only needs to send a PING request to detect whether the scheduler in the record is alive. If the scheduler in the record is alive, it will directly return as the scheduler of the group; otherwise, it will complete the usurpation, write its own information into the database table, become the scheduler of the group and return. Furthermore, in step 3, the PING survival detection between schedulers uses HTTP protocol communication, which will filter out schedulers with higher loads.

[0137] Step 4: With the help of the service discovery mechanism, the executors under the entire application group will eventually connect to the same scheduler. The subsequent executors will send heartbeat reports to this scheduler, including the executor address and executor load information. If the scheduler node is abnormal during this period, all executors under the application group will request other surviving schedulers. The surviving schedulers will compete for a distributed lock. The first one to get the lock will usurp the position and write its own information into the scheduler address information table associated with the application group, becoming the scheduler of the group and returning. After that, the executors will connect to the same scheduler again.

[0138] In step 5, the scheduler maintains the information of the cluster nodes of the application group executors it holds, and periodically pulls the task information associated with all applications from the database for scheduling. The tasks will be sent to the executors with normal load and heartbeat. Furthermore, in step 5, the method for the scheduler to check the normal load of the executors includes: the executors regularly report node resource information (CPU and memory, etc.) with the application group; the scheduler adds the executors under the same application group to the same set according to the application group dimension; if there is no heartbeat report for more than a preset number of times, it will be removed from the set, and if the CPU and memory exceed the preset threshold, it will also be removed from the set.

[0139] In step 6, after receiving the task from the scheduler, the executor executes it locally and reports the execution result. The scheduler will failover and retry the abnormal execution result. Furthermore, in step 6, after the local execution of the executor fails, it will retry according to the custom retry strategy. If the retry still fails, the scheduler will assign the task to other executors for processing.

[0140] Among them, combined with the above steps, the process of the executor discovering services to the scheduler is as follows: Figure 5 The process of load checking the actuator is as follows: Figure 6 shown.

[0141] Furthermore, a distributed task scheduling device based on group isolation is provided, such as Figure 7 As shown, it includes the following modules:

[0142] The scheduler module is used to receive user scheduling tasks, maintain node status information, split tasks and schedule them to the executor cluster. Furthermore, the scheduler includes a task submodule, a node submodule, and a scheduling submodule. Among them, the task submodule is used to create and view scheduled tasks, manage the execution status of tasks, and failover and retry failed tasks; the node submodule maintains the status information of all executor cluster nodes under all application groups, and regularly cleans up executor nodes with heartbeat timeouts; the scheduling submodule pulls task information associated with all applications from the database, splits and schedules tasks, and reasonably allocates resources according to task goals and the load of each executor node.

[0143] The executor module is used to actually execute tasks, perform service discovery to the scheduler regularly, report executor status information to the scheduler regularly, receive tasks issued by the scheduler, and report execution result data to the scheduler after local execution. Furthermore, the executor includes a registration submodule, a heartbeat submodule, and an execution submodule. Among them, the registration submodule regularly performs service discovery to obtain the actually connected scheduler. The executors under the same application group are all connected to the same scheduler; the heartbeat submodule regularly sends heartbeat reports, including the executor address and executor load information (CPU and memory, etc.); the execution submodule executes the actual task. After the local execution fails, it will retry according to the custom retry strategy and report the task status information to the scheduler.

[0144] The database module is used to store the scheduler address information table associated with the application group, the task information table associated with the application group, the application group's own information table, etc., wherein the scheduler address information table associated with the application group is used for service discovery, the task information table associated with the application group is used to associate the task execution status, and the application group's own information table is used to display the list of connected applications.

[0145] The distributed task scheduling method and device based on group isolation provided in this example have the following beneficial effects:

[0146] (1) Compared with the prior art that relies on database row-level locks, the present invention is based on a lock-free design and introduces group isolation through the divide-and-conquer concept. The application group is used as the scheduling unit of the scheduler. Each application group will only be connected to a fixed scheduler when running. Each scheduler will only schedule all tasks under the application group currently associated with the executor that maintains a heartbeat with itself. There is no need to lock during scheduling, which can meet both high availability and high performance, and is easy to schedule large-scale distributed tasks.

[0147] (2) During scheduling, each scheduler only schedules the tasks in its own group. They are independent of each other and do not need to wait. There is no performance loss caused by additional level locks on the database. By extending the scheduler, the overall concurrency and execution efficiency of tasks can be improved, making horizontal expansion easier.

[0148] (3) The executor reports its load information through heartbeats. The scheduler calculates the health of the executor based on the executor's load information and automatically selects the executor with the optimal load to execute the task, making full use of cluster resources. In addition, the scheduler also maintains its own load information and provides overload protection. The executors in the application group will only connect to schedulers with normal loads, ensuring that each scheduler is not overloaded.

[0149] In an optional embodiment, taking the example of the executors under the application group A being connected to the same scheduler B, the implementation process of the above-mentioned distributed task scheduling method based on group isolation specifically includes:

[0150] Step 1, deploy 3 scheduler node clusters, wait for executor registration and heartbeat report, the scheduler address list is: scheduler A: scheduler B: scheduler C, the scheduling server locally maintains its own load information. Further, in step 1, the scheduler refers to the scheduling node of the task, splits and issues the user's scheduling task, and finds a suitable executor to execute the task; the executor refers to the actual execution node of the task, receives and executes the task issued by the scheduler and reports the relevant status information; the load information includes node resource information (CPU and memory, etc.).

[0151] Step 2: Fill in the list of scheduler addresses in step 1 and start the executor under application group A. After the executor is started, it will perform a service discovery and start the background thread for regular service discovery. It will request any scheduler to obtain the actual scheduler address corresponding to the application group. Here, it is assumed that executor A requests the address of scheduler B for service discovery, and executor B requests the address of scheduler C for service discovery, and both requests bring application group A.

[0152] Step 3: After the scheduler receives the service discovery request from the executor, it will conduct a small distributed election: the database table contains an information table that records each group and the corresponding scheduler. If the scheduler finds that there is a record in the table, it means that another executor in the application group has requested the scheduler to elect in advance. At this time, it only needs to send a PING request to detect whether the scheduler in the record is alive. If the scheduler in the record is alive, it will be directly returned as the scheduler of the group; otherwise, it will complete the usurpation, write its own information into the database table, become the scheduler of the group and return. Here, it is assumed that scheduler B receives the request from executor A first, first writes its own address and application group A information into the database table and returns scheduler B to executor A; subsequently, scheduler C checks the table and finds that there is a record of scheduler B in application group A, and also returns scheduler B to executor B. Furthermore, in step 3, the PING survival detection between schedulers uses HTTP protocol communication, which will filter out schedulers with higher loads.

[0153] Step 4: With the help of the service discovery mechanism, the executors under the entire application group A will eventually connect to the same scheduler, and the subsequent executors will send heartbeat reports to this scheduler, including the executor address and executor load information. If the scheduler node is abnormal during this period, all executors under the application group will request other surviving schedulers, and the surviving schedulers will compete for a distributed lock. The first one to get the lock will usurp the position and write its own information into the scheduler address information table associated with the application group, becoming the scheduler of the group and returning. After that, the executors will connect to the same scheduler again.

[0154] In step 5, the scheduler maintains the information of the cluster nodes of the application group executors it holds, and periodically pulls the task information associated with all applications from the database for scheduling. The tasks will be sent to the executors with normal load and heartbeat. Furthermore, in step 5, the method for the scheduler to check the normal load of the executors includes: the executors regularly report node resource information (CPU and memory, etc.) with the application group; the scheduler adds the executors under the same application group to the same set according to the application group dimension; if there is no heartbeat report for more than a preset number of times, it will be removed from the set, and if the CPU and memory exceed the preset threshold, it will also be removed from the set.

[0155] In step 6, after receiving the task from the scheduler, the executor executes it locally and reports the execution result. The scheduler will failover and retry the abnormal execution result. Furthermore, in step 6, after the local execution of the executor fails, it will retry according to the custom retry strategy. If the retry still fails, the scheduler will assign the task to other executors for processing.

[0156] The embodiment of the present invention also provides a computer device for executing the above Figure 2 and Figure 3 The distributed task scheduling method shown.

[0157] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0158] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0159] The memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the method shown in the above embodiment.

[0160] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0162] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0163] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0164] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0165] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A distributed task scheduling method, characterized in that: For a scheduler cluster module, the scheduler cluster module includes a plurality of schedulers, the scheduler cluster module is connected to an executor cluster module, the executor cluster module includes a plurality of application group clusters, each application group cluster includes a plurality of executors; the method includes: Sending a scheduler address list to the executor cluster module, so that the executor cluster module sequentially controls each executor in any application group cluster to send a service discovery request based on the scheduler address list, wherein the service discovery request carries the application group cluster information; receiving the service discovery request sent by the executor cluster module in sequence, and performing small distributed master election based on the received service discovery request in sequence using a preset database table to determine a target scheduler, wherein the preset database table is an information table recording each application group cluster and the corresponding scheduler; Sending the actual scheduler address of the target scheduler to the executor cluster module, so that the target application group cluster in the executor cluster module is connected to the target scheduler, and the target application group cluster is the same application group cluster where the multiple executors sending the service discovery request are located; When the target scheduler is connected to the target application group cluster, a distributed scheduling task is obtained, and the distributed scheduling task is split and sent to the target application group cluster, so that the multiple executors in the target application group cluster respectively execute the split distributed scheduling task and obtain a resource information set; Receive a heartbeat report signal carrying the resource information set sent by the executor cluster module.

2. The method according to claim 1, characterized in that Based on the received service discovery request, a small distributed master election is performed using a preset database table to determine a target scheduler, including: Obtaining the preset database table; Determine whether there is record information corresponding to the service discovery request in the preset database table; When the record information exists in the preset database table, sending a PING request to the first scheduler corresponding to the record information to detect whether the first scheduler is alive; When the first scheduler survives, determining the first scheduler as the target scheduler; When the first scheduler is abnormal, the scheduler that receives the service discovery request is determined as the target scheduler.

3. The method according to claim 1, characterized in that The method further comprises: Based on the resource information set, determining a first identical set based on a plurality of executors in the same application grouping cluster; When the heartbeat report signal is not received within a preset number of times or the resource information set does not meet a preset condition, the corresponding executor is deleted from the same set to obtain a second same set; The split distributed scheduling tasks are sent to the executors in the second identical set.

4. The method according to claim 1, characterized in that The method further comprises: Determine the abnormal execution result based on the heartbeat report signal; Failover and retry are performed on the abnormal execution result.

5. The method according to claim 1, characterized in that The method further comprises: When the target scheduler is abnormal, other surviving schedulers are used to compete for the target distributed lock; The scheduler that first obtains the target distributed lock is used as the new target scheduler, and the scheduler address of the new target scheduler is sent to the target application group cluster.

6. A distributed task scheduling method, characterized in that: For an executor cluster module, the executor cluster module includes a plurality of application group clusters, each application group cluster includes a plurality of executors, the executor cluster module is connected to a scheduler cluster module, the scheduler cluster module includes a plurality of schedulers; the method includes: When receiving the scheduler address list sent by the scheduler cluster module, based on the scheduler address list, controlling each executor in any application group cluster to send a service discovery request in turn, so that the scheduler cluster module determines the target scheduler based on the service discovery request, and the service discovery request carries the application group cluster information; Receive the actual scheduler address of the target scheduler sent by the scheduler cluster module, and control the target application group cluster to connect with the target scheduler based on the actual scheduler address, wherein the target application group cluster is the same application group cluster where the multiple executors sending the service discovery request are located; When receiving a plurality of sub-scheduling tasks sent by the target scheduler, using the plurality of executors in the target application group cluster to execute the plurality of sub-scheduling tasks to obtain a resource information set; Sending a heartbeat report signal carrying the resource information set to the target scheduler in the scheduler cluster module.

7. The method according to claim 6, characterized in that The method further comprises: When receiving a target scheduler exception instruction sent by the scheduler cluster module, controlling the target application group cluster to send a new service discovery request to the scheduler cluster module, so that the scheduler cluster module determines a new target scheduler from other surviving schedulers based on the service discovery request; Receive the new scheduler address of the target scheduler sent by the scheduler cluster module.

8. The method according to claim 6, characterized in that The method further comprises: When the multiple executors in the target application group cluster fail to execute the multiple sub-scheduling tasks, re-execute the multiple sub-scheduling tasks according to a preset custom retry strategy; When re-executing the multiple sub-scheduling tasks fails, a task failure instruction is sent to the scheduler cluster module, so that the scheduler cluster module sends the multiple sub-scheduling tasks to the multiple executors in other application group clusters.

9. A distributed task scheduling system, characterized in that: The system comprises: a scheduler cluster module and an executor cluster module, wherein the scheduler cluster module comprises a plurality of schedulers, the executor cluster module comprises a plurality of application group clusters, each application group cluster comprises a plurality of executors, and the scheduler cluster module is connected to the executor cluster module; The scheduler cluster module is used to execute the distributed task scheduling method according to any one of claims 1 to 5; The executor cluster module is used to execute the distributed task scheduling method described in any one of claims 6 to 8.

10. A computer program product, characterized in that The method comprises computer instructions, wherein the computer instructions are used to enable a computer to execute the distributed task scheduling method according to any one of claims 1 to 8.

Images