Task scheduling method and system

Abstract

The embodiment of the invention provides a task scheduling method and system, and relates to the technical field of computers. According to the method, the task is subjected to partition management, the corresponding relation between the partition and the cluster is maintained by using the routing rule, and the routing rule is dynamically updated according to the cluster capacity expansion and contraction scene, so that the timeliness of the corresponding relation between the partition and the cluster is ensured. When a to-be-processed task request is received, determining a target cluster according to a partition to which a to-be-processed task belongs and a routing rule, searching whether the to-be-processed task has an effective task node in the target cluster, and if so, determining the effective task node as a target task node of the to-be-processed task, otherwise, a new task node is allocated to the to-be-processed task, so that the task consistency is ensured, and the rapid growth of services is better protected.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a task scheduling method and system. Background Art

[0002] Currently, multi-cluster task scheduling usually uses a hash algorithm to map tasks to corresponding clusters. Each cluster manages a portion of tasks, and each task is assigned to the same cluster for processing, thereby ensuring task consistency. However, in the scenario of cluster expansion and contraction, the same task may be assigned to different clusters before and after expansion and contraction, resulting in the problem that task consistency cannot be guaranteed. Summary of the invention

[0003] In view of this, an object of the present invention is to provide a task scheduling method and system, which can ensure task consistency.

[0004] In order to achieve the above purpose, the technical solution adopted by the embodiment of the present invention is as follows:

[0005] In a first aspect, the present invention provides a task scheduling method, which is applied to a task scheduling system, wherein the task scheduling system includes a gateway, at least one cluster and a plurality of task nodes; the method includes:

[0006] The gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, and sends the task to be processed to the target cluster; the routing rule is used to record the correspondence between the partition and the cluster, and is updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks;

[0007] The target cluster determines the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or uses the newly allocated task node as the target task node of the task to be processed; the target task node is the task node for scheduling the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been allocated to the task to be processed.

[0008] In an optional implementation manner, the gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, including:

[0009] When the gateway only records the new routing rule, the gateway determines the target cluster according to the partition to which the task to be processed belongs and the new routing rule.

[0010] In an optional implementation manner, the target cluster includes a first cluster and a second cluster; the gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, including:

[0011] When the gateway records the old routing rules and the new routing rules, the gateway determines the first cluster according to the partition to which the task to be processed belongs and the old routing rules, and determines the second cluster according to the partition to which the task to be processed belongs and the new routing rules.

[0012] In an optional implementation, the target cluster determines the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or uses the newly allocated task node as the target task node of the task to be processed, including:

[0013] The first cluster determines, according to the pending tasks sent by the gateway, a first partition corresponding to the pending tasks;

[0014] If there is a valid task node corresponding to the task to be processed in the status data of the first partition, the first cluster determines the valid task node as the target task node; otherwise, the first cluster notifies the gateway that there is no valid task node corresponding to the task to be processed; the status data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes the task and the task node that schedules the task;

[0015] The gateway sends the to-be-processed task to the second cluster;

[0016] The second cluster determines, according to the pending tasks sent by the gateway, a second partition corresponding to the pending tasks;

[0017] If there is a valid task node corresponding to the task to be processed in the status data of the second partition, the second cluster determines the valid task node as the target task node; otherwise, the second cluster uses the newly assigned task node as the target task node for the task to be processed and adds the mapping relationship between the task to be processed and the newly assigned task node to the status data of the second partition.

[0018] In an optional implementation manner, before the step of the gateway determining the target cluster according to the partition to which the task to be processed belongs and the routing rule, the method further includes:

[0019] The gateway receives a task allocation request; the task allocation request includes a task to be processed;

[0020] The gateway determines the partition to which the task belongs according to the hash value of the task to be processed.

[0021] In an optional implementation, after the step of using the newly allocated task node as the target task node of the task to be processed, the method further includes:

[0022] When a task node is newly allocated for the task to be processed, the target cluster sends the task to be processed to the target task node so that the target task node schedules the task to be processed.

[0023] In an optional implementation manner, the routing rule includes a new routing rule; and the method further includes:

[0024] Each of the task nodes sends a heartbeat message to the gateway regularly according to an update cycle; the heartbeat message includes the task node and at least one task to be updated scheduled on the task node;

[0025] For each of the tasks to be updated, the gateway determines the cluster to be updated according to the partition to which the task to be updated belongs and the new routing rule, and sends the task to be updated and the corresponding task node to the cluster to be updated;

[0026] The cluster to be updated determines whether there is a mapping relationship of the task to be updated in the status data of the partition to which it belongs; the status data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes the task, the task node that schedules the task and the latest running time;

[0027] If so, the cluster to be updated updates the latest running time corresponding to the task to be updated according to the current system time;

[0028] If it does not exist, the cluster to be updated uses the current system time as the latest running time, generates a mapping relationship corresponding to the task to be updated according to the task to be updated, the task node and the latest running time, and saves the mapping relationship corresponding to the task to be updated to the status data of the partition to which it belongs.

[0029] In an optional implementation, the routing rule includes an old routing rule, and the method further includes:

[0030] For each of the clusters, the cluster traverses and queries the status data of each partition; the status data is used to record the mapping relationship of existing tasks in the same partition of the same cluster; each of the mapping relationships includes the task, the task node that schedules the task, and the latest running time;

[0031] The cluster determines the task node corresponding to the failure time threshold value as an invalid task node when the difference between the current system time and the latest running time is greater than the failure time threshold value;

[0032] The cluster deletes the mapping relationship corresponding to the invalid task node from the state data;

[0033] When the state data of the partition is empty and the partition is not a partition managed by the cluster, the cluster deletes the state data and notifies the gateway to delete the corresponding relationship between the partition and the cluster from the old routing rule;

[0034] When all the changed corresponding relationships are deleted from the old routing rules, the gateway deletes the old routing rules.

[0035] In an optional embodiment, the method further comprises:

[0036] When there are multiple task nodes for the same task, each of the clusters retains one task node from the multiple task nodes and deletes the remaining task nodes according to a preset rule.

[0037] In a second aspect, the present invention provides a task scheduling system, the task scheduling system comprising a gateway, at least one cluster and a plurality of task nodes;

[0038] The gateway is used to determine the target cluster according to the partition to which the task to be processed belongs and the routing rule, and send the task to be processed to the target cluster; the routing rule is used to record the correspondence between the partition and the cluster, and is updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks;

[0039] The target cluster is used to determine the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or to use the newly allocated task node as the target task node of the task to be processed; the target task node is the task node that schedules the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been allocated to the task to be processed.

[0040] Compared with the prior art, the task scheduling method and system provided by the embodiments of the present invention are applied to the task scheduling system, the gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rules, and sends the task to be processed to the target cluster; the routing rules are used to record the correspondence between the partitions and clusters, and are updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks; the target cluster determines the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or uses the newly assigned task node as the target task node of the task to be processed; the target task node is the task node for scheduling the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been assigned to the task to be processed.

[0041] It can be seen that the embodiment of the present invention manages tasks by partitions, maintains the correspondence between partitions and clusters by using routing rules, and dynamically updates routing rules according to cluster expansion and contraction scenarios, thereby ensuring the timeliness of the correspondence between partitions and clusters. When a request for a pending task is received, the target cluster is determined according to the partition to which the pending task belongs and the routing rules, and a search is performed in the target cluster to determine whether there is a valid task node for the pending task. If so, the valid task node is determined as the target task node for the pending task, otherwise a new task node is assigned to the pending task, thereby ensuring task consistency and better safeguarding the rapid growth of the business.

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A structural schematic diagram of a task scheduling system provided by an embodiment of the present invention is shown.

[0045] Figure 2 A schematic diagram of a process flow of a task scheduling method provided by an embodiment of the present invention is shown.

[0046] Figure 3 Shows Figure 2 A flowchart of the sub-steps of step S40.

[0047] Figure 4 Another flowchart of the task scheduling method provided by an embodiment of the present invention is shown.

[0048] Figure 5 Another flowchart of the task scheduling method provided by an embodiment of the present invention is shown.

[0049] Icon: 10-task scheduling system; 100-gateway; 200-cluster; 300-task node. DETAILED DESCRIPTION

[0050] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0051] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

[0052] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0053] In the prior art, users can encapsulate business logic into tasks and send the tasks to the task scheduling system through the user terminal for scheduling. The simplest task scheduling system is a single cluster scheduling system composed of several server nodes. However, there is an upper limit to the carrying capacity of a single cluster. When the task volume is large, a single cluster will not be able to carry the service.

[0054] The inventors have found that when a single cluster cannot carry services, it is usually considered to expand the task scheduling system into multiple clusters, which is horizontal expansion. Adding a cluster or reducing a cluster is also a process of horizontal expansion. The multi-cluster task scheduling system uses a hash algorithm to map the task hash to the corresponding cluster. When the cluster does not expand or shrink, each task will be hashed to the same cluster, and each cluster records the tasks it manages.

[0055] When the multi-cluster task scheduling system is scaled up or down, the number of clusters in the multi-cluster changes. When the clusters are mapped using the hash algorithm, the clusters corresponding to the same task may be different before and after the scale-up or down, which will make it impossible to ensure task consistency. Task consistency means that when the same task generates multiple concurrent allocation requests in the same time period, multiple allocation requests for the same task are guaranteed to be allocated to the same task node.

[0056] In order to solve the above problems, the core improvement of the present invention is to manage tasks by partitions, maintain the correspondence between partitions and clusters by using routing rules, and dynamically update routing rules according to cluster expansion and contraction scenarios, thereby ensuring the timeliness of the correspondence between partitions and clusters. When a request for a pending task is received, the target cluster is determined according to the partition to which the pending task belongs and the routing rules, and a search is performed in the target cluster to find out whether there is a valid task node for the pending task. If so, the valid task node is determined as the target task node for the pending task, otherwise a new task node is assigned to the pending task, thereby ensuring task consistency and better safeguarding the rapid growth of the business.

[0057] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0058] Figure 1 FIG. 1 is a schematic diagram showing the structure of a task scheduling system 10 provided in an embodiment of the present invention, which provides an implementation method of a task scheduling system. Figure 1 The task scheduling system 10 includes a gateway 100, at least one cluster 200 and a plurality of task nodes 300, and the cluster 200 is respectively communicated with the gateway 100 and the task nodes 300.

[0059] The gateway 100 is used to receive a task assignment request sent by a user terminal, and forward the task assignment request to the corresponding cluster 200 according to the locally maintained routing rules. The cluster 200 is used to assign a task node 300 to a new task, send the task to be processed to the newly assigned task node 300, and maintain the mapping relationship between existing tasks and task nodes.

[0060] It should be noted that the server in the cluster 200 may be a cloud server, a network server, a database server, etc.; the task node 300 may be an electronic device such as a server or a computer, etc. The present invention is not limited to this.

[0061] Please refer to Figure 2 , Figure 2 A flow chart of a task scheduling method provided by an embodiment of the present invention is shown. The method is applied to a task scheduling system, which includes a gateway, at least one cluster and multiple task nodes. The method includes the following steps:

[0062] Step S30: The gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, and sends the task to be processed to the target cluster.

[0063] The routing rules are used to record the correspondence between partitions and clusters and are updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks.

[0064] In an embodiment of the present invention, the number of partitions can be preset according to the number of tasks in the application scenario. For example, the number of partitions is initialized to 32, and all tasks that do not overlap with each other are assigned to 32 partitions for management, and each partition manages multiple tasks assigned to it.

[0065] Under the same routing rules, all partitions are assigned to multiple clusters for management without overlap, and each cluster manages multiple partitions assigned to it. The gateway determines the target cluster based on the corresponding relationship in the routing rules and the partition to which the pending task belongs, and sends the pending task to the target cluster for processing. Among them, the pending task can be a transcoding task, a peer-to-peer (P2P) streaming task, and other tasks triggered by user requests.

[0066] Step S40: The target cluster determines the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or determines the newly allocated task node as the target task node of the task to be processed.

[0067] The target task node is the task node that schedules the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been assigned to the task to be processed.

[0068] In an embodiment of the present invention, when a valid task node corresponding to a task to be processed is recorded in the target cluster, the target cluster uses the valid task node corresponding to the task to be processed as the target task node for the task to be processed, and sends the access address of the target task node to the gateway, which then returns the access address of the target node to the user.

[0069] When no valid task node corresponding to the task to be processed is recorded in the target cluster, the target cluster newly allocates a task node for the task to be processed, and uses the newly allocated task node as the target task node of the task to be processed, so as to schedule the task to be processed.

[0070] In summary, the task scheduling method provided by the embodiment of the present invention is applied to the task scheduling system, the gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rules, and sends the task to be processed to the target cluster; the routing rules are used to record the correspondence between the partitions and clusters, and are updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks; the target cluster determines the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or uses the newly assigned task node as the target task node of the task to be processed; the target task node is the task node for scheduling the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been assigned to the task to be processed.

[0071] It can be seen that the embodiment of the present invention manages tasks by partitions, maintains the correspondence between partitions and clusters by using routing rules, and dynamically updates routing rules according to cluster expansion and contraction scenarios, thereby ensuring the timeliness of the correspondence between partitions and clusters. When a request for a pending task is received, the target cluster is determined according to the partition to which the pending task belongs and the routing rules, and a search is performed in the target cluster to determine whether there is a valid task node for the pending task. If so, the valid task node is determined as the target task node for the pending task, otherwise a new task node is assigned to the pending task, thereby ensuring task consistency and better safeguarding the rapid growth of the business.

[0072] Optionally, when the multi-cluster is not expanded or reduced or the expansion or reduction has been completed, the gateway uses the new routing rule to record the latest correspondence between the partition and the cluster. Figure 2 In step S30, the gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, which sub-steps may include:

[0073] When the gateway only records new routing rules, the gateway determines the target cluster based on the partition to which the task to be processed belongs and the new routing rules.

[0074] In the embodiment of the present invention, the staff allocates the partitions to each cluster for management according to the number of clusters in the multi-cluster and the business load of each cluster, and generates routing rules according to the corresponding relationship between the clusters and the partitions and sends them to the gateway and each cluster.

[0075] As a possible implementation, assume that the task scheduling system includes a cluster a and two partitions. The two partitions are partition 1 and partition 2, and partition 1 and partition 2 are managed by cluster a. Then, the new routing rules recorded in the gateway are shown in Table 1. Assuming that the partition to which the task to be processed belongs is partition 1, cluster a is determined as the target cluster according to the routing rules in Table 1.

[0076] Table 1

[0077]

[0078]

[0079] It should be noted that the specific format of the routing rules can be set according to actual application conditions, and the present invention is not limited to this.

[0080] Optionally, during the process of multi-cluster expansion and contraction, the gateway needs both the old routing rules and the new routing rules to ensure task consistency. The target cluster includes the first cluster and the second cluster. Figure 2 In step S30, the gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, which sub-steps may include:

[0081] When the gateway records the old routing rules and the new routing rules, the gateway determines the first cluster according to the partition to which the task to be processed belongs and the old routing rules, and determines the second cluster according to the partition to which the task to be processed belongs and the new routing rules.

[0082] In the embodiment of the present invention, when the multi-cluster is expanded or reduced, the staff regenerates the routing rules according to the latest number of clusters in the multi-cluster and the business load of each cluster, and sends the new routing rules to the gateway and each cluster. The gateway uses the original routing rules as the old routing rules and the received routing rules as the new routing rules. Each cluster determines the management scope of the distinction according to the received routing rules.

[0083] As a possible implementation method, before the multi-cluster expansion, assume that the routing rules recorded on the gateway are as shown in Table 1. Expand the multi-cluster and add cluster b. Partition 2 is still managed by cluster a, and partition 1 is assigned to cluster b for management. The old routing rules recorded in the gateway are shown in Table 1, and the new routing rules are shown in Table 2. Assuming that the partition to which the task to be processed belongs is partition 1, cluster a is determined as the first cluster according to the routing rules in Table 1, and cluster b is determined as the second cluster according to the routing rules in Table 2.

[0084] Table 2

[0085] Partition Cluster Partition 1 Cluster b Partition 2 Cluster a

[0086] It can be seen that the process of multi-cluster expansion and contraction is the process of changing the corresponding relationship between partitions and clusters. In the embodiment of the present invention, the gateway and each cluster determine the target cluster according to the routing rules before and after the change, which can effectively improve the probability of task consistency.

[0087] Optionally, during the multi-cluster scaling process, the target task node can be determined by combining the target cluster before scaling and the target cluster after scaling to ensure task consistency. Figure 3 , Figure 2 The sub-steps of step S40 may include:

[0088] Step S401: The first cluster determines a first partition corresponding to the task to be processed according to the task to be processed sent by the gateway.

[0089] In the embodiment of the present invention, the first cluster is the target cluster corresponding to the partition to which the pending task belongs before expansion and contraction, and the second cluster is the target cluster corresponding to the partition to which the pending task belongs after expansion and contraction. First, the gateway sends the pending task to the first cluster, and the first cluster determines the partition to which the pending task belongs, i.e., the first partition, according to the hash value of the pending task.

[0090] Step S402: The first cluster determines whether there is a valid task node corresponding to the task to be processed in the state data of the first partition. If yes, step S403 is executed; otherwise, step S404 is executed.

[0091] The state data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes a task and a task node for scheduling the task.

[0092] Step S403: The first cluster determines the valid task node as the target task node.

[0093] Step S404: The first cluster notifies the gateway that there is no valid task node corresponding to the task to be processed.

[0094] In the embodiment of the present invention, the cluster records the existing tasks in the managed partitions through the state data, and the partitions correspond to the state data one by one. During the expansion and contraction process, the old routing rules and the new routing rules in the gateway are Table 1 and Table 2 respectively. Assume that before the expansion and contraction, there is already a task task1-1 in partition 1, and task1-1 is assigned to task node n1 for scheduling. Under the action of the task synchronization mechanism, the mapping relationship between task task1-1 and task node n1 is recorded in the state data of partition 1 in cluster a.

[0095] During the expansion process, the gateway receives an allocation request for the pending task task1-1, and determines cluster a as the first cluster and cluster b as the second cluster based on task1-1. Assuming that the difference between the current system time and the latest running time corresponding to task task1-1 in cluster a is not greater than the fault time threshold, cluster a determines the task node corresponding to task task1-1 as the valid task node, and determines the valid task node as the target task node to avoid re-assigning a new task node to task task1-1 when cluster b does not receive the heartbeat information of task task1-1, thereby ensuring task consistency.

[0096] Assuming that there is no mapping relationship between task task1-1, task node n1 and the latest running time in the status data of partition 1 in cluster a, or when the difference between the current system time and the latest running time corresponding to task task1-1 in cluster a is greater than the failure time threshold, the cluster a gateway sends a message that there is no valid task node corresponding to the pending task task1-1.

[0097] Step S405: The gateway sends the task to be processed to the second cluster.

[0098] Step S406: The second cluster determines a second partition corresponding to the task to be processed according to the task to be processed sent by the gateway.

[0099] In an embodiment of the present invention, after the gateway receives a message from the first cluster that there is no valid task node corresponding to the pending task, the gateway sends the pending task to the second cluster. The second cluster determines the partition to which the pending task belongs, that is, the second partition, based on the received pending task. For example, the first partition determined by the first cluster (cluster a) based on task task1-1 is partition 1, and the second partition determined by the second cluster (cluster b) based on task task1-1 is also partition 1, thereby ensuring task consistency.

[0100] Step S407: The second cluster determines whether there is a valid task node corresponding to the task to be processed in the status data of the second partition. If yes, execute step S408; otherwise, execute step S409.

[0101] Step S408: The second cluster determines the valid task node as the target task node.

[0102] Step S409: The second cluster uses the newly allocated task node as the target task node of the task to be processed, and adds the mapping relationship between the task to be processed and the newly allocated task node to the status data of the second partition.

[0103] In an embodiment of the present invention, when there is a valid task node corresponding to the pending task in the status data of the second partition, the second cluster uses the valid task node as the target task node. When there is no valid task node corresponding to the pending task in the status data of the second partition, the second cluster newly allocates a task node for the pending task, and uses the newly allocated task node as the target task node, and adds the mapping relationship between the pending task and the newly allocated task node to the status data of the second partition.

[0104] It can be seen that the embodiment of the present invention combines the routing rules before and after the cluster is expanded or reduced to obtain the target task node corresponding to the pending task. If there is a valid task node on the cluster corresponding to the old routing rule, the valid task node is directly used as the target task node, which can ensure the effective use of the task node assigned to the pending task to continue scheduling the pending task. If there is no valid task node on the cluster corresponding to the old routing rule, the target task node is determined according to the cluster corresponding to the new routing rule, thereby ensuring task consistency.

[0105] Optionally, in actual applications, a hash algorithm is used to obtain the partition to which the task belongs, thereby ensuring the consistency of the partition management task range. Figure 4 ,exist Figure 2 Before step S30, the method further includes the following steps:

[0106] Step S10: The gateway receives a task allocation request.

[0107] The task allocation request includes pending tasks.

[0108] Step S20: The gateway determines the partition to which the task to be processed belongs according to the hash value of the task to be processed.

[0109] In the embodiment of the present invention, the user sends a task allocation request to the gateway through the user terminal, and the user terminal is connected to the gateway in communication. The gateway calculates the hash value of the task to be processed according to a preset hash algorithm, and determines the partition to which the task to be processed belongs according to the hash value.

[0110] It should be noted that the gateway and each cluster calculate the hash value of the task according to the same hash algorithm, so as to ensure that the same task belongs to the same partition, thereby ensuring task consistency.

[0111] Optionally, in actual applications, the pending tasks are sent to the target task node for scheduling in a timely manner. Figure 4 ,exist Figure 2 After step S40, the method further includes the following steps:

[0112] Step S50, when a task node is newly allocated for a task to be processed, the target cluster sends the task to be processed to the target task node so that the target task node schedules the task to be processed.

[0113] In an embodiment of the present invention, when there is no valid task node corresponding to the pending task, a new task node needs to be allocated for the pending task as the target task node. The target cluster sends the pending task to the target task node, and the target task node allocates resources to schedule the pending task.

[0114] Optionally, in actual applications, the routing rules include new routing rules, and the task node maintains communication with the cluster corresponding to the partition to which the task belongs by sending heartbeat information, so as to timely and effectively migrate the mapping relationship of existing tasks, thereby ensuring task consistency. Figure 5 , the task scheduling method also includes the following steps:

[0115] Step S60: Each task node sends a heartbeat message to the gateway regularly according to the update cycle.

[0116] The heartbeat message includes a task node and at least one task to be updated scheduled on the task node.

[0117] In an embodiment of the present invention, each task node executing a task sends heartbeat information to the network manager regularly according to the update cycle, and updates the latest running time of the task node through the heartbeat information, thereby ensuring the communication connection between the task node and the cluster corresponding to the partition to which the task being executed belongs.

[0118] Step S70: For each task to be updated, the gateway determines the cluster to be updated according to the partition to which the task to be updated belongs and the new routing rule, and sends the task to be updated and the corresponding task node to the cluster to be updated.

[0119] In an embodiment of the present invention, each task to be updated is processed in turn according to the heartbeat information, and the gateway determines the partition to which it belongs according to the hash value of the task to be updated. Since the new routing rule maintains the correspondence between the latest partition and the cluster, the cluster to be updated corresponding to each task to be updated is determined according to the partition to which it belongs and the new routing relationship, and the task to be updated and the corresponding task node are sent to the cluster to be updated.

[0120] Step S80: The cluster to be updated determines whether there is a mapping relationship of the task to be updated in the status data of the partition to which it belongs.

[0121] The status data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes the task, the task node of the scheduled task, and the latest running time.

[0122] Step S90: If it exists, the cluster to be updated updates the latest running time corresponding to the task to be updated according to the current system time.

[0123] Step S100, if it does not exist, the cluster to be updated uses the current system time as the latest running time, generates a mapping relationship corresponding to the task to be updated according to the task to be updated, the task node and the latest running time, and saves the mapping relationship corresponding to the task to be updated to the status data of the partition to which it belongs.

[0124] As a possible implementation method, continuing to take Table 1 and Table 2 as an example, before cluster expansion, cluster a manages partition 1 and partition 2. After cluster expansion, cluster b manages partition 1, and cluster a manages partition 2. The heartbeat information of the old task task1-1 on task node n1 is forwarded to cluster b to be updated through the gateway. When cluster b receives the heartbeat information of task task1-1 for the first time, the mapping relationship of task task1-1 does not exist in the status data of the partition to which task task1-1 belongs. Cluster b needs to generate the mapping relationship of task task1-1 according to task task1-1, task node n1 and the current time of the system, and save the newly generated mapping relationship to the status data of the partition to which it belongs.

[0125] When cluster b receives the heartbeat information of task task1-1 for a non-first time, the state data of the partition to which task task1-1 belongs already contains a mapping relationship of task task1-1, and the latest running time in the mapping relationship is updated to the current system time.

[0126] It can be seen that the embodiment of the present invention ensures timely and effective communication connection between the task node and the corresponding cluster by having the task node periodically send heartbeat information to the cluster corresponding to the partition to which the task being executed belongs, effectively improving the probability of tasks being scheduled by the same task node, thereby ensuring task consistency and realizing the automation of task migration and routing switching.

[0127] Optionally, in practical applications, the routing rules include old routing rules, and the old routing rules are updated according to the latest running time of the existing tasks, effectively improving the efficiency of task allocation. The task scheduling method also includes the following steps:

[0128] For each cluster, the cluster traverses and queries the status data of each partition; the status data is used to record the mapping relationship of existing tasks in the same partition of the same cluster; each mapping relationship includes the task, the task node of the scheduled task and the latest running time; the cluster determines the task node corresponding to the difference between the current system time and the latest running time greater than the failure time threshold as an invalid task node.

[0129] The cluster deletes the mapping relationship corresponding to the invalid task node from the status data; when the status data of the partition is empty and the partition is not managed by the cluster, the cluster deletes the status data and notifies the gateway to delete the corresponding relationship between the partition and the cluster from the old routing rules; when all the changed corresponding relationships are deleted from the old routing rules, the gateway deletes the old routing rules.

[0130] In the embodiment of the present invention, each cluster periodically traverses and queries the status data of each partition managed by it, and when there is a mapping relationship in which the difference between the current system time and the latest running time is greater than the fault time threshold, the task node in the mapping relationship is determined as an invalid task node. In other words, a valid task node is a task node corresponding to which the difference between the current system time and the latest running time is not greater than the fault time threshold.

[0131] Since the latest running time is updated by the task node periodically sending heartbeat information to the cluster corresponding to the partition to which the task belongs, the status data corresponding to the partition that was in the management scope before the cluster was expanded or reduced and is no longer in the management scope after the expansion or reduction will no longer be updated.

[0132] Continuing with Table 1 and Table 2 as examples, before cluster expansion, cluster a manages partition 1 and partition 2. After cluster expansion, cluster b manages partition 1, and cluster a manages partition 2. After cluster expansion and contraction, the state data of partition 1 in cluster a contains a mapping relationship between task task1-1, task node n1, and the latest running time, and the state data of partition 1 in cluster b contains a mapping relationship between task task1-1, task node n1, and the latest running time.

[0133] Among them, cluster a cannot receive the heartbeat information of task task1-1, and the latest running time corresponding to task task1-1 in cluster a is no longer updated. Cluster b can receive the heartbeat information of task task1-1, and the latest running time corresponding to task task1-1 in cluster b can be updated regularly.

[0134] When the difference between the current system time and the latest running time corresponding to task task1-1 in cluster a is greater than the fault time threshold, the mapping relationship of task task1-1 is deleted from the status data of partition 1 in cluster a. Since there is only the mapping relationship of task task1-1 in the status data of partition 1 in cluster a, after deleting the mapping relationship of task task1-1, the status data of partition 1 in cluster a is empty. At the same time, since partition 1 is not a partition managed by cluster a, cluster a deletes the status data of partition 1 and notifies the gateway to delete the corresponding relationship between partition 1 and cluster a from the old routing rules.

[0135] Since the old routing rules only record one changed correspondence (i.e., the correspondence between partition 1 and cluster a), the gateway will delete the correspondence between partition 1 and cluster a from the old routing rules and then delete the old routing rules. After that, since the task switching and routing update caused by cluster expansion and contraction are completed, the task allocation requests under partition 1 will be directly sent to cluster b using the new routing rules, ensuring task consistency in horizontal expansion and contraction scenarios.

[0136] It can be seen that the embodiment of the present invention continuously updates the old routing rules by deleting the mapping relationships corresponding to invalid task nodes, so that the gateway deletes the old routing rules after all the changed corresponding relationships are deleted from the old routing rules, thereby ensuring that the gateway allocates tasks according to the new routing rules, while ensuring task consistency and improving the efficiency of task allocation.

[0137] Optionally, in actual applications, when the task node network is unstable, the same task may be assigned to multiple task nodes for scheduling. To ensure task consistency, it is necessary to delete redundant task nodes and only retain one task node for scheduling the task. The task scheduling method also includes the following steps:

[0138] When there are multiple task nodes for the same task, each cluster retains one task node from the multiple task nodes and deletes the remaining task nodes according to preset rules.

[0139] As a possible implementation method, continuing to take Table 1 and Table 2 as examples, before cluster expansion, cluster a manages partition 1 and partition 2. After cluster expansion, cluster b manages partition 1, and cluster a manages partition 2. During cluster expansion, assuming that the gateway receives an allocation request for task task1-2, determines that the partition to which task1-2 belongs is partition 1 according to the hash value of task task1-2, and sends task task1-2 to cluster a according to the old routing rule. The state data of partition 1 in cluster a does not record the mapping relationship of task task1-2, then cluster a notifies the gateway that there is no valid task node corresponding to task task1-2.

[0140] The gateway sends task task1-2 to cluster b according to the new routing rule. The status data of partition 1 in cluster b does not record the mapping relationship of task task1-2. Therefore, task node n2 is assigned to task task1-2, and the mapping relationship between task task1-2, task node n2 and the latest running time is saved in the status data of partition 1.

[0141] Assume that the task node n2 network fails and cannot send heartbeat information to the gateway according to the update cycle, resulting in cluster b being unable to update the latest running time of task task1-2 in time, and the mapping relationship of task task1-2 is deleted when the difference between the current system time and the latest running time is greater than the failure time threshold.

[0142] After the mapping relationship of task task1-2 is deleted, if the gateway receives a task task1-2 allocation request, the gateway sends the task task1-2 allocation request to cluster b. Cluster b does not find the mapping relationship of task task1-2 in the status data of partition 1, and re-assigns task node n3 to task task1-2, and saves the mapping relationship between task task1-2, task node n3 and the latest running time to the status data of partition 1.

[0143] Assume that after cluster B adds the mapping relationship between task task1-2, task node n3 and the latest running time in the status data of partition 1, task node n2 resumes heartbeat communication with cluster B. Cluster B will save the mapping relationship between task task1-2, task node n2 and the latest running time to the status data of partition 1.

[0144] At this time, there are two mapping relationships of task task1-2 in the state data of partition 1 of cluster b, namely, the mapping relationship of task task1-2, task node n3 and the latest running time, and the mapping relationship of task task1-2, task node n2 and the latest running time. In order to ensure task consistency, one of the task nodes is retained according to the preset rules, and the remaining task nodes are deleted. For example, the preset rules are set to retain the earliest task node, retain the task node with the longest communication time, etc.

[0145] It can be seen that the embodiment of the present invention monitors the number of task nodes scheduling the same task in real time. When there are multiple task nodes scheduling the same task, the redundant task nodes are deleted in time to ensure that each task can only be scheduled by one task node, thereby ensuring task consistency.

[0146] Based on the same inventive concept, the basic principle of the task scheduling system provided by the embodiment of the present invention and the technical effects produced are the same as those of the above embodiment. For the sake of brief description, for parts not mentioned in this embodiment, reference may be made to the corresponding contents in the above embodiment.

[0147] Please continue to refer to Figure 1 , wherein the gateway 100 is used to determine the target cluster 200 according to the partition to which the task to be processed belongs and the routing rules, and send the task to be processed to the target cluster 200; the routing rules are used to record the correspondence between the partitions and clusters, and are updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks.

[0148] The target cluster 200 is used to determine the valid task node corresponding to the task to be processed as the target task node 300 of the task to be processed, or to use the newly assigned task node as the target task node 300 of the task to be processed; the target task node is the task node for scheduling the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been assigned to the task to be processed.

[0149] In summary, the task scheduling system provided by the embodiment of the present invention manages tasks by partitions, maintains the correspondence between partitions and clusters by using routing rules, and dynamically updates routing rules according to cluster expansion and contraction scenarios, thereby ensuring the timeliness of the correspondence between partitions and clusters. When a request for a pending task is received, the target cluster is determined according to the partition to which the pending task belongs and the routing rules, and a search is performed in the target cluster to determine whether there is a valid task node for the pending task. If so, the valid task node is determined as the target task node for the pending task, otherwise a new task node is assigned to the pending task, thereby ensuring task consistency and better safeguarding the rapid growth of the business.

[0150] Optionally, when the gateway 100 only records a new routing rule, the gateway 100 is used to determine the target cluster 200 according to the partition to which the task to be processed belongs and the new routing rule.

[0151] Optionally, the target cluster includes a first cluster and a second cluster. When the gateway 100 records old routing rules and new routing rules, the gateway 100 is used to determine the first cluster 200 according to the partition to which the task to be processed belongs and the old routing rules, and determine the second cluster 200 according to the partition to which the task to be processed belongs and the new routing rules.

[0152] Optionally, the first cluster 200 is used to determine the first partition corresponding to the pending task based on the pending task sent by the gateway 100; if there is a valid task node corresponding to the pending task in the status data of the first partition, the first cluster 200 is used to determine the valid task node as the target task node 300; otherwise, the first cluster 200 notifies the gateway 100 that there is no valid task node corresponding to the pending task; the status data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes the task node of the task and the scheduled task.

[0153] The gateway 100 is used to send the pending task to the second cluster 200; the second cluster 200 is used to determine the second partition corresponding to the pending task based on the pending task sent by the gateway 100; if there is a valid task node corresponding to the pending task in the status data of the second partition, the second cluster 200 is used to determine the valid task node as the target task node 300, otherwise the second cluster is used to use the newly assigned task node as the target task node 300 of the pending task, and add the mapping relationship between the pending task and the newly assigned task node to the status data of the second partition.

[0154] Optionally, the gateway 100 is used to receive a task allocation request; the task allocation request includes a task to be processed; the gateway 100 is used to determine the partition to which the task belongs according to a hash value of the task to be processed.

[0155] Optionally, when a task node is newly allocated for a task to be processed, the target cluster 200 is used to send the task to be processed to the target task node 300 so that the target task node 300 schedules the task to be processed.

[0156] Optionally, the routing rules include new routing rules, and each task node 300 is used to send heartbeat messages to the gateway 100 regularly according to the update cycle; the heartbeat message includes the task node and at least one task to be updated scheduled on the task node; for each task to be updated, the gateway 100 is used to determine the cluster to be updated 200 based on the partition to which the task to be updated belongs and the new routing rules, and send the task to be updated and the corresponding task node to the cluster to be updated 200.

[0157] The cluster to be updated 200 is used to determine whether there is a mapping relationship of the task to be updated in the status data of the partition to which it belongs; the status data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes a task, a task node of the scheduled task and the latest running time.

[0158] If it exists, the cluster to be updated 200 is used to update the latest running time corresponding to the task to be updated according to the current time of the system; if it does not exist, the cluster to be updated 200 is used to use the current time of the system as the latest running time, generate a mapping relationship corresponding to the task to be updated according to the task to be updated, the task node and the latest running time, and save the mapping relationship corresponding to the task to be updated to the status data of the partition to which it belongs.

[0159] Optionally, the routing rules include old routing rules. For each cluster 200, cluster 200 is used to traverse and query the status data of each partition; the status data is used to record the mapping relationship of existing tasks in the same partition of the same cluster; each mapping relationship includes the task, the task node of the scheduled task and the latest running time; cluster 200 is used to determine the task node corresponding to the fault time threshold whose difference between the current system time and the latest running time is greater than that as an invalid task node.

[0160] Cluster 200 is used to delete the mapping relationship corresponding to the invalid task node from the status data; when the status data of the partition is empty and the partition is not a partition managed by the cluster, cluster 200 is used to delete the status data and notify gateway 100 to delete the corresponding relationship between the partition and the cluster from the old routing rules; when all the changed corresponding relationships are deleted from the old routing rules, gateway 100 is used to delete the old routing rules.

[0161] Optionally, when there are multiple task nodes for the same task, each cluster is used to retain one task node among the multiple task nodes according to a preset rule and delete the remaining task nodes.

[0162] In several embodiments provided by the present invention, it should be understood that the disclosed system and method can also be implemented in other ways. The system embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] In addition, the functional modules in the various embodiments of the present invention may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0164] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0165] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A task scheduling method, It is characterized in that Applied to a task scheduling system, the task scheduling system includes a gateway, at least one cluster and multiple task nodes; the method includes: The gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, and sends the task to be processed to the target cluster; the routing rule is used to record the correspondence between the partition and the cluster, and is updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks; The target cluster determines the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or uses the newly allocated task node as the target task node of the task to be processed; the target task node is the task node for scheduling the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been allocated to the task to be processed.

2. The task scheduling method according to claim 1, It is characterized in that The gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rules, including: When the gateway only records the new routing rule, the gateway determines the target cluster according to the partition to which the task to be processed belongs and the new routing rule.

3. The task scheduling method according to claim 1, It is characterized in that The target cluster includes a first cluster and a second cluster; The gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rules, including: When the gateway records the old routing rules and the new routing rules, the gateway determines the first cluster according to the partition to which the task to be processed belongs and the old routing rules, and determines the second cluster according to the partition to which the task to be processed belongs and the new routing rules.

4. The task scheduling method according to claim 3, It is characterized in that The target cluster determines the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or determines the newly allocated task node as the target task node of the task to be processed, including: The first cluster determines, according to the pending tasks sent by the gateway, a first partition corresponding to the pending tasks; If there is a valid task node corresponding to the task to be processed in the status data of the first partition, the first cluster determines the valid task node as the target task node; otherwise, the first cluster notifies the gateway that there is no valid task node corresponding to the task to be processed; the status data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes the task and the task node that schedules the task; The gateway sends the to-be-processed task to the second cluster; The second cluster determines, according to the pending tasks sent by the gateway, a second partition corresponding to the pending tasks; If there is a valid task node corresponding to the task to be processed in the status data of the second partition, the second cluster determines the valid task node as the target task node; otherwise, the second cluster uses the newly assigned task node as the target task node for the task to be processed and adds the mapping relationship between the task to be processed and the newly assigned task node to the status data of the second partition.

5. The task scheduling method according to any one of claims 1 to 4, It is characterized in that Before the gateway determines the target cluster according to the partition to which the task to be processed belongs and the routing rule, the method further includes: The gateway receives a task allocation request; the task allocation request includes a task to be processed; The gateway determines the partition to which the task belongs according to the hash value of the task to be processed.

6. The task scheduling method according to claim 1, It is characterized in that After the step of using the newly allocated task node as the target task node of the task to be processed, the method further includes: When a task node is newly allocated for the task to be processed, the target cluster sends the task to be processed to the target task node so that the target task node schedules the task to be processed.

7. The task scheduling method according to any one of claims 1 to 4, It is characterized in that The routing rule includes a new routing rule; and the method further includes: Each of the task nodes sends a heartbeat message to the gateway regularly according to an update cycle; the heartbeat message includes the task node and at least one task to be updated scheduled on the task node; For each of the tasks to be updated, the gateway determines the cluster to be updated according to the partition to which the task to be updated belongs and the new routing rule, and sends the task to be updated and the corresponding task node to the cluster to be updated; The cluster to be updated determines whether there is a mapping relationship of the task to be updated in the status data of the partition to which it belongs; the status data is used to record the mapping relationship of existing tasks in the same cluster and the same partition; each mapping relationship includes the task, the task node that schedules the task and the latest running time; If so, the cluster to be updated updates the latest running time corresponding to the task to be updated according to the current system time; If it does not exist, the cluster to be updated uses the current system time as the latest running time, generates a mapping relationship corresponding to the task to be updated according to the task to be updated, the task node and the latest running time, and saves the mapping relationship corresponding to the task to be updated to the status data of the partition to which it belongs.

8. The task scheduling method according to any one of claims 1 to 4, It is characterized in that The routing rule includes an old routing rule, and the method further includes: For each of the clusters, the cluster traverses and queries the status data of each partition; the status data is used to record the mapping relationship of existing tasks in the same partition of the same cluster; each of the mapping relationships includes the task, the task node that schedules the task, and the latest running time; The cluster determines the task node corresponding to the failure time threshold value as an invalid task node when the difference between the current system time and the latest running time is greater than the failure time threshold value; The cluster deletes the mapping relationship corresponding to the invalid task node from the state data; When the state data of the partition is empty and the partition is not a partition managed by the cluster, the cluster deletes the state data and notifies the gateway to delete the corresponding relationship between the partition and the cluster from the old routing rule; When all the changed corresponding relationships are deleted from the old routing rules, the gateway deletes the old routing rules.

9. The task scheduling method according to any one of claims 1 to 4, It is characterized in that The method further comprises: When there are multiple task nodes for the same task, each of the clusters retains one task node from the multiple task nodes and deletes the remaining task nodes according to a preset rule.

10. A task scheduling system, It is characterized in that The task scheduling system includes a gateway, at least one cluster and multiple task nodes; The gateway is used to determine the target cluster according to the partition to which the task to be processed belongs and the routing rule, and send the task to be processed to the target cluster; The routing rules are used to record the corresponding relationship between partitions and clusters, and are updated when the number of clusters changes; each cluster manages at least one partition, and each partition manages multiple tasks; The target cluster is used to determine the valid task node corresponding to the task to be processed as the target task node of the task to be processed, or to use the newly allocated task node as the target task node of the task to be processed; the target task node is the task node that schedules the task to be processed; the valid task node represents the task node that maintains communication with the target cluster and has been allocated to the task to be processed.

Images