Task scheduling based on distributed cluster

Abstract

Provided in the present disclosure are a task scheduling method based on a distributed cluster, and a related device. The method is applied to any target node among a plurality of nodes comprised in a distributed cluster, wherein a target database is deployed in the distributed cluster, and a task allocation result for a plurality of tasks to be executed is maintained in the target database. The method comprises: on the basis of a preset election strategy, electing a scheduling node; in response to a target node being elected as the scheduling node, accessing a target database, and on the basis of a task allocation result for a plurality of tasks that is maintained in the target database, determining whether there are several tasks among the plurality of tasks that can be allocated; and if there are several tasks among the plurality of tasks that can be allocated, allocating to the several tasks corresponding nodes from among a plurality of nodes, and updating a task allocation result for the several tasks in the target database, such that the plurality of nodes determine, on the basis of the task allocation result for the plurality of tasks that is maintained in the target database, whether there are tasks among the plurality of tasks that are allocated to the plurality of nodes themselves, and if so, executing the allocated tasks.

Description

Task scheduling based on distributed clusters Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to task scheduling based on distributed clusters. Background Technology

[0002] In distributed clusters, a distributed scheduling framework is often used to manage and execute a series of tasks. To ensure the correctness and consistency of task execution, when the distributed scheduling framework needs to schedule the execution of a task, it first allows each node in the cluster to interact with the database to compete for the database lock corresponding to the task (such as row-level locks, table-level locks, or other distributed locks), thereby attempting to gain exclusive execution rights for the current task. Specifically, after a node successfully acquires the database lock corresponding to the task, a row in the database containing information such as the specific content and execution strategy of the task (taking a row-level lock as an example) is locked. Other nodes will not be able to access this row, and therefore cannot know the specific content and execution strategy of the task. In this way, only the node that successfully acquires the database lock can execute the task, while other nodes must wait for the lock to be released before they can continue to try, thus preventing the same task from being executed concurrently on different nodes.

[0003] However, if there are a large number of tasks to be executed, or if a task has a very short execution cycle, such as being executed every 5 seconds, it will frequently trigger various nodes to repeatedly compete for database locks, resulting in serious waste of resources and reduced performance of the distributed cluster. Summary of the Invention

[0004] In view of this, one or more embodiments of this disclosure provide a task scheduling method and related equipment based on a distributed cluster.

[0005] In a first aspect, this disclosure provides a task scheduling method based on a distributed cluster, applied to any target node among multiple nodes included in the distributed cluster; a target database is deployed in the distributed cluster, and the target database maintains the task allocation results of multiple tasks to be executed; the method includes: electing a scheduling node among the multiple nodes according to a preset election strategy; in response to the target node being elected as the scheduling node, accessing the target database, and determining whether there are several allocatable tasks among the multiple tasks according to the task allocation results of the multiple tasks maintained in the target database; if so, allocating corresponding nodes for the several tasks from the multiple nodes, and updating the task allocation results of the several tasks to the target database, so that the multiple nodes can determine whether there are tasks assigned to them among the multiple tasks according to the task allocation results of the multiple tasks maintained in the target database, and if so, executing the assigned tasks.

[0006] Secondly, this disclosure provides a task scheduling device based on a distributed cluster, applied to any target node among multiple nodes included in the distributed cluster; a target database is deployed in the distributed cluster, and the target database maintains the task allocation results of multiple tasks to be executed; the device includes: an election unit, used to elect a scheduling node among the multiple nodes according to a preset election strategy; a determination unit, used to, in response to the target node being elected as the scheduling node, access the target database and determine whether there are several allocatable tasks among the multiple tasks according to the task allocation results of the multiple tasks maintained in the target database; and a task allocation unit, used to, if so, allocate corresponding nodes to the several tasks from the multiple nodes and update the task allocation results of the several tasks to the target database, so that the multiple nodes can determine whether there are tasks assigned to them among the multiple tasks according to the task allocation results of the multiple tasks maintained in the target database, and if so, execute the assigned tasks.

[0007] Accordingly, this disclosure also provides a computing device, including: a memory and a processor; the memory stores a computer program / instruction executable by the processor; when the processor executes the computer program / instruction, it performs the task scheduling method based on a distributed cluster described in the first aspect above.

[0008] Accordingly, this disclosure also provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, performs the task scheduling method based on a distributed cluster as described in the first aspect above.

[0009] Accordingly, this disclosure also provides a computer program product, which includes a computer program / instruction, and when the computer program / instruction is executed by a processor, it performs the task scheduling method based on a distributed cluster as described in the first aspect above.

[0010] In summary, this application can be applied to any target node in a distributed cluster comprising multiple nodes. A target database is deployed in this distributed cluster, maintaining the task allocation results for multiple tasks to be executed. First, the target node can elect a scheduling node from among the multiple nodes according to a preset election strategy. Upon successful election, the target node can access the target database and, based on the task allocation results maintained in the target database, determine whether there are several allocatable tasks among the multiple tasks. If several allocatable tasks exist, the target node can allocate corresponding nodes for these tasks from among the multiple nodes and update the task allocation results to the target database. Correspondingly, each node in the cluster can access the target database and, based on the task allocation results maintained in the target database, determine whether there is a task assigned to it among the multiple tasks; if so, it executes the assigned task. Thus, this application introduces an election mechanism and a task allocation mechanism into the existing distributed scheduling framework, enabling multiple nodes in the distributed cluster to compete for the scheduling node position. The elected scheduling node then uniformly allocates the tasks to be executed to each node in the cluster, allowing each node to directly execute the assigned tasks without having to compete for tasks by preempting database locks. This not only avoids the concurrent execution of the same task on different nodes but also greatly improves the task execution efficiency of the distributed cluster. Attached Figure Description

[0011] Figure 1 is a schematic diagram of a system architecture provided in an exemplary embodiment;

[0012] Figure 2 is a flowchart illustrating a task scheduling method based on a distributed cluster, provided in an exemplary embodiment.

[0013] Figure 3 is a schematic diagram of the structure of a task scheduling device based on a distributed cluster provided in an exemplary embodiment;

[0014] Figure 4 is a schematic diagram of the structure of a computing device provided in an exemplary embodiment. Detailed Implementation

[0015] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this disclosure as detailed in the appended claims.

[0016] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this disclosure in other embodiments. In some other embodiments, the methods may include more or fewer steps than those described in this disclosure. Furthermore, a single step described in this disclosure may be broken down into multiple steps in other embodiments; and multiple steps described in this disclosure may be combined into a single step in other embodiments.

[0017] It should be noted that "multiple" as used in this application refers to two or more.

[0018] In distributed clusters, distributed scheduling frameworks are often used to manage and execute a series of tasks. These tasks can be one-time tasks or tasks that need to be executed periodically according to a preset duration. To ensure the correctness and consistency of task execution, when the distributed scheduling framework needs to schedule the execution of a task, it first allows each node in the distributed cluster to interact with the database to compete for database locks (such as row-level locks, table-level locks, and other distributed locks). This attempts to obtain exclusive execution rights for the current task. That is, only the node that successfully acquires the database lock can execute the task, while other nodes must wait for the lock to be released before they can continue to try, thus preventing the same task from being executed concurrently on different nodes.

[0019] However, if a large number of tasks need to be executed, or if a task has a very short execution cycle, such as executing every 5 seconds, frequent database lock contention among nodes will occur. It should be understood that each attempt by a node to acquire a database lock may involve network communication, database queries, and updates. Even if the acquisition fails, these operations consume the node's CPU, memory, and network bandwidth. Therefore, when task execution cycles are short and lock contention is intense, it leads to severe resource waste and reduces the performance of the distributed cluster. Furthermore, frequent lock requests also increase the burden on the database. The database needs to handle a large number of lock requests, lock conflicts, deadlock detection and resolution operations, which severely consumes the database server's CPU and memory resources and may increase disk I / O pressure, among other things.

[0020] Based on this, this disclosure provides a technical solution in which a scheduling node elected in the cluster distributes tasks to each node in the cluster, so that each node can directly execute the assigned tasks without having to compete for database locks.

[0021] In implementation, this application can be applied to any target node among multiple nodes in a distributed cluster. A target database is deployed in this distributed cluster, maintaining the task allocation results for multiple tasks to be executed. First, the target node can elect a scheduling node from among the multiple nodes according to a preset election strategy. Upon successful election, the target node can access the target database and, based on the task allocation results maintained in the target database, determine whether there are several allocatable tasks among the multiple tasks. If several allocatable tasks exist, the target node can allocate corresponding nodes for these tasks from among the multiple nodes and update the task allocation results to the target database. Correspondingly, each node in the cluster can access the target database and, based on the task allocation results maintained in the target database, determine whether there is a task assigned to it among the multiple tasks; if so, it executes the assigned task.

[0022] In the above technical solutions, this application introduces an election mechanism and a task allocation mechanism into the existing distributed scheduling framework, which enables multiple nodes in the distributed cluster to compete for the scheduling node, and the elected scheduling node uniformly allocates the tasks to be executed to each node in the cluster. This allows each node to directly execute the assigned tasks without having to compete for database locks, thereby greatly improving the task execution efficiency of the distributed cluster.

[0023] Please refer to Figure 1, which is a schematic diagram of a system architecture provided by an exemplary embodiment. As shown in Figure 1, one or more embodiments provided in this disclosure can be specifically implemented in the system architecture shown in Figure 1 or a similar system architecture. As shown in Figure 1, the system can be a distributed cluster, including nodes 100a, 100b, 100c, 100d, etc. In one illustrated embodiment, nodes 100a, 100b, 100c, and 100d can establish communication connections with each other in any possible way.

[0024] In one illustrated embodiment, a distributed scheduling framework for managing and executing tasks is deployed on the distributed cluster. The distributed scheduling framework introduces an election mechanism for scheduling nodes and a task allocation mechanism for scheduling nodes. Based on this, nodes 100a, 100b, 100c, and 100d in the cluster can participate in the election of scheduling nodes, and the elected scheduling nodes will allocate tasks to each node for execution. For details, please refer to the following embodiments, which will not be elaborated here.

[0025] As shown in Figure 1, a target database is deployed in the distributed cluster. In one illustrated embodiment, the target database can be centrally deployed on any node in the distributed cluster (e.g., node 100a); alternatively, the target database can be distributed across multiple nodes in the distributed cluster (e.g., nodes 100a, 100b, and 100c, etc.), and this disclosure does not impose any specific limitations on this. In one illustrated embodiment, each node in the distributed cluster can access the target database to perform operations such as querying, updating, writing, or deleting data in the target database.

[0026] The target database can maintain task allocation results for multiple tasks to be executed in the distributed cluster. These results can include whether a task has been assigned, and if so, which node it was assigned to. It should be noted that this application does not specifically limit the specific type and content of the tasks. For example, these tasks can be tasks that need to be executed periodically, such as compaction operations on the target database, asynchronous query tasks, etc., and this disclosure does not specifically limit them.

[0027] In one illustrated embodiment, the target database can maintain the task allocation results in the form of a data table. For example, the target database can maintain a task allocation table, which may include a task identifier field (or task identifier column) and a task allocation field (or task allocation column). The task identifier field can be used to record the task ID of each task, and the task allocation field can be used to record the task allocation result of each task, such as whether the task has been allocated and which node it has been allocated to, etc., which are not specifically limited in this disclosure. In some possible embodiments, the task allocation table may also include any possible fields for recording other information, which are not specifically limited in this disclosure.

[0028] Furthermore, in one illustrated implementation, each node can use a heartbeat mechanism to inform other nodes of its online status. For example, each node can initiate a heartbeat every 5 seconds. Specifically, this can involve sending a heartbeat message containing its own status information to other nodes in the cluster and updating the database with the time information (e.g., a timestamp) of the most recent heartbeat. In other words, each node can update the time information in the database every 5 seconds. For example, if a node does not initiate a heartbeat for an extended period, i.e., does not update its heartbeat time information in the database on time, it can be considered offline. Conversely, if a node initiates a heartbeat on time, i.e., updates its heartbeat time information in the database on time, it can be considered online. It should be understood that offline nodes typically cannot perform tasks successfully.

[0029] In one illustrated embodiment, the target database may also maintain a node information table, which may include a node identifier field (or node identifier column), a heartbeat information field (or heartbeat information column), and a scheduling node indication field (or scheduling node indication column). The node identifier field may be used to record the node identifier of each node (e.g., the node's IP address), the heartbeat information field may be used to record the timestamp of the most recent heartbeat initiated by each node, and the scheduling node indication field may be used to record indication information of whether each node is a scheduling node, etc., and this disclosure does not specifically limit these aspects.

[0030] In one illustrated implementation, in response to the completion of initialization of the distributed cluster, there is no scheduling node yet; or, in response to the original election of the scheduling node in the distributed cluster being offline, multiple nodes in the distributed cluster can elect a scheduling node from among multiple nodes according to a preset election strategy.

[0031] For example, if the distributed cluster has just completed initialization and there is no scheduling node yet, nodes 100a, 100b, 100c, and 100d in the cluster can start electing a scheduling node.

[0032] It should be noted that this application does not impose any specific limitations on the strategy for electing and scheduling nodes. In one illustrated implementation, nodes 100a, 100b, 100c, and 100d can elect a scheduling node based on their respective IP addresses. Specifically, each node can query the node information table in the target database and obtain multiple IP addresses recorded in the node identifier field. If its own IP address is the largest among these multiple IP addresses, it can qualify to become a scheduling node; otherwise, it can abstain from the election. In other words, the scheduling node can be the node with the largest IP address among the multiple participating nodes.

[0033] For example, node 100a has an IP address of 192.168.1.156, node 100b has an IP address of 192.168.1.139, node 100c has an IP address of 192.168.1.100, and node 100d has an IP address of 192.168.1.33. These IP addresses can be converted to their corresponding integer values. Since the integer value corresponding to node 100a's IP address (1921681156) is the largest among them, node 100a can be successfully elected as the scheduling node.

[0034] In one illustrated embodiment, this application may also elect a scheduling node by referring to the CPU, memory, network bandwidth, and other resources of each node, and this disclosure does not specifically limit this. For example, the scheduling node may be the node with the most CPUs, or the scheduling node may be the node with the most memory, etc., and this disclosure does not specifically limit this.

[0035] Furthermore, after node 100a is elected as the scheduling node, node 100a can access the target database to write indication information indicating that node 100a is the scheduling node in the scheduling node indication field corresponding to node 100a in the node information table, thus completing the election of the scheduling node.

[0036] In one of the illustrated embodiments, in addition to the above-described scheme of recording the information of the election and scheduling node in the target database, this application can also implement the election and scheduling node through a related distributed consensus algorithm, and this disclosure does not specifically limit it.

[0037] For example, if the originally elected scheduling node (node ​​100a) goes offline after a period of time, then nodes 100b, 100c, and 100d can begin to elect a new scheduling node.

[0038] For example, let's take node 100b with IP address 192.168.1.139, node 100c with IP address 192.168.1.100, and node 100d with IP address 192.168.1.33 as examples. We can first convert these multiple IP addresses to their corresponding integer values. Since the integer value corresponding to node 100b's IP address (1921681139) is the largest among them, node 100b can be successfully elected as the new scheduling node.

[0039] Furthermore, after node 100b is elected as the scheduling node, node 100a's original scheduling node qualification can be revoked. Specifically, node 100b can access the target database to write an indication in the scheduling node indication field corresponding to node 100a in the node information table, indicating that node 100a is not a scheduling node. Additionally, node 100b can write an indication in the scheduling node indication field corresponding to node 100b in the node information table, indicating that node 100b is a scheduling node. This completes the election of a new scheduling node.

[0040] The following will use node 100b as the scheduling node and nodes 100a, 100c, and 100d as ordinary nodes in the cluster as an example to illustrate the task scheduling method provided in this application.

[0041] First, node 100b can access the target database and, based on the task allocation results of multiple tasks maintained in the target database, determine whether there are any assignable tasks among the multiple tasks. In one illustrated embodiment, the assignable tasks may include: tasks without assigned nodes, or tasks whose assigned nodes are offline. Specifically, node 100b can query the task allocation table maintained in the target database. For each queried task, if the task allocation field corresponding to the task does not record a node identifier, or if the node corresponding to the node identifier recorded in the task allocation field corresponding to the task is offline, then the task can be determined to be an assignable task.

[0042] In one illustrated implementation, if node 100b determines that there are several allocatable tasks among multiple tasks, node 100b can allocate corresponding nodes for these tasks from multiple nodes in the cluster and update the task allocation results of the tasks to the target database. Specifically, node 100b can access the target database to write the node identifier of the allocated node to the task allocation field corresponding to these tasks in the task allocation table maintained in the target database.

[0043] Correspondingly, multiple nodes in the cluster (including scheduling nodes and ordinary nodes) can access the target database and determine whether there is a task assigned to them among the multiple tasks maintained in the target database. If so, the assigned task is executed. Specifically, multiple nodes can query the task allocation table maintained in the target database to determine whether there is a task among the multiple tasks whose task allocation field records their node identifier. For details, please refer to the description of the embodiment corresponding to Figure 2 below, which will not be elaborated here.

[0044] It should be noted that, as mentioned above, in addition to the function of assigning tasks, the scheduling node also has the functions of a regular node, that is, the scheduling node can also assign tasks to itself and execute the assigned tasks.

[0045] Thus, this application introduces an election mechanism and a task allocation mechanism into the existing distributed scheduling framework, enabling multiple nodes in the distributed cluster to compete for the scheduling node position. The elected scheduling node then uniformly allocates the tasks to be executed to each node in the cluster, allowing each node to directly execute the assigned tasks without having to compete for database locks, thereby greatly improving the task execution efficiency of the distributed cluster.

[0046] In one illustrated embodiment, the scheduling node in this application can also be regarded as the master node among multiple nodes, and other nodes that are not elected as the scheduling node can be regarded as slave nodes among multiple nodes.

[0047] Please refer to Figure 2, which is a flowchart illustrating a task scheduling method based on a distributed cluster according to an exemplary embodiment. This method can be applied to the system architecture shown in Figure 1, specifically to any target node in the distributed cluster shown in Figure 1 (e.g., any one of node 100a, node 100b, node 100c, and node 100d). A target database is deployed in the distributed cluster, maintaining the task allocation results for multiple tasks to be executed. As shown in Figure 2, the method may specifically include the following steps S201-S203.

[0048] Step S201: According to the preset election strategy, elect a scheduling node among the multiple nodes included in the distributed cluster; a target database is deployed in the distributed cluster, and the target database maintains the task allocation results of multiple tasks to be executed.

[0049] As described above, the distributed cluster includes multiple nodes. In response to the completion of the initialization of the distributed cluster, there is no scheduling node yet; or, in response to the original election of the scheduling node in the distributed cluster being offline, the multiple nodes in the distributed cluster can elect the scheduling node among the multiple nodes according to the preset election strategy. The specific election implementation method can be referred to the description of the embodiment corresponding to Figure 1 above, and will not be repeated here.

[0050] In one illustrated embodiment, a target database is deployed in a distributed cluster, which maintains node information for multiple nodes in the cluster. In another illustrated embodiment, the target database may maintain this node information in the form of data tables.

[0051] In one illustrated embodiment, the node information table may include a node identifier field, a heartbeat information field, and a scheduling node indication field. The node identifier field can be used to record the node identifier of each node (e.g., the node's IP address), the heartbeat information field can be used to record the timestamp of the most recent heartbeat initiated by each node, and the scheduling node indication field can be used to record indication information indicating whether each node is a scheduling node.

[0052] For example, the node information table can be shown in Table 1 below.

[0053] The node identifier field can be the ip_addr field shown in Table 1. The ip_addr field of node 1 records the IP address of node 1, which is 114.10.10.1; the ip_addr field of node 2 records the IP address of node 2, which is 10.10.10.2; the ip_addr field of node 3 records the IP address of node 3, which is 10.10.10.3; and the ip_addr field of node 4 records the IP address of node 4, which is 10.10.10.233.

[0054] The heartbeat information field can be the last_active_time field shown in Table 1. The last_active_time field of node 1 records the timestamp of the most recent heartbeat initiated by node 1, which is 10000000000; the last_active_time field of node 2 records the timestamp of the most recent heartbeat initiated by node 2, which is 1000000000; the last_active_time field of node 3 records the timestamp of the most recent heartbeat initiated by node 3, which is 1000000000; and the last_active_time field of node 4 records the timestamp of the most recent heartbeat initiated by node 4, which is 987654321.

[0055] The scheduling node indicator field can be the is_master field shown in Table 1. The is_master field of node 1 records "true", which indicates that node 1 is a scheduling node; the is_master field of node 2 records "false", which indicates that node 2 is not a scheduling node; the is_master field of node 3 records "false", which indicates that node 3 is not a scheduling node; and the is_master field of node 4 records "false", which indicates that node 4 is not a scheduling node.

[0056] In one illustrated embodiment, the node information table may include any other possible fields besides those shown in Table 1. Alternatively, the node information table may directly record information about the scheduling node, such as "114.10.10.1 is master", without recording relevant indication information in the is_master field of each node, etc. This disclosure does not specifically limit this.

[0057] As shown in Table 1, Node 4 has the largest IP address, followed by Node 1. Node 4 might have been the original scheduling node, but since the timestamp recorded in the last_active_time field of Node 4 is less than the preset threshold, it indicates that Node 4 has not initiated a heartbeat for a long time, meaning that Node 4 is offline. Therefore, Node 1 was elected as the new scheduling node. Table 1

[0058] In one of the illustrated implementations, each node in the cluster can access the target database and query the node information table shown in Table 1 above to obtain information such as the IP address, heartbeat information, and whether it is a scheduling node, thereby detecting whether the currently elected scheduling node is offline, so as to trigger the election of a new scheduling node.

[0059] Step S202: In response to the election of the target node as the scheduling node, access the target database and determine whether there are several tasks that can be allocated among the multiple tasks based on the task allocation results of the multiple tasks maintained in the target database.

[0060] In one illustrated embodiment, in response to any target node in the cluster (e.g., node 1 shown in Table 1, or node 100b shown in Figure 1) being elected as the scheduling node, the target node can access the aforementioned target database and, based on the task allocation results of multiple tasks maintained in the target database, determine whether there are any assignable tasks among the multiple tasks. In one illustrated embodiment, the assignable tasks may include: tasks on unassigned nodes, or tasks assigned to nodes that are offline.

[0061] In one illustrated embodiment, the target database can maintain the task allocation results in the form of a data table. The task allocation table may include a task identifier field and a task allocation field. The task identifier field can be used to record the task ID of each task, and the task allocation field can be used to record the task allocation result of each task, including, for example, whether the task has been allocated and which node it has been allocated to.

[0062] For example, the task allocation table can be shown in Table 2 below.

[0063] The task identifier field can be the id field shown in Table 2. The multiple tasks to be executed can include task 40001, task 40002, task 40003, and task 40004.

[0064] The task allocation field can be the locked_by_ip field shown in Table 2. The locked_by_ip field of task 40001 records the IP address of the node to which task 40001 is assigned, which is 10.10.10.2, i.e., node 2 in Table 1 above. The locked_by_ip field of task 40002 records the empty character "null", indicating that task 40002 has not been assigned any node. The locked_by_ip field of task 40003 records the IP address of the node to which task 40003 is assigned, which is 10.10.10.3, i.e., node 3 in Table 1 above. The locked_by_ip field of task 40004 records the IP address of the node to which task 40004 is assigned, which is 10.10.10.233, i.e., node 4 in Table 1 above.

[0065] In one illustrated embodiment, the task assignment table may also include any other possible fields besides those shown in Table 2, such as fields for recording task content and fields for recording task execution plans, etc., which are not specifically limited in this disclosure.

[0066] As shown in Table 2, task 40002 has not been assigned to any node. Furthermore, combining this with Table 1 above, we know that node 4, to which task 40004 was assigned, is offline. Therefore, the target node can determine that tasks 40002 and 4004 are assignable tasks. In one illustrated implementation, the node's IP address can be bound to its online status. Table 2

[0067] In one of the illustrated embodiments, the target node elected as the scheduling node can periodically query the task allocation table maintained by the target database to see if there are several tasks available for allocation, so as to promptly allocate suitable nodes for tasks that have not been allocated nodes or whose nodes are offline, thereby ensuring the efficiency of task execution.

[0068] Step S203: If yes, then assign corresponding nodes to the plurality of tasks from the plurality of nodes, and update the task allocation results of the plurality of tasks to the target database, so that the plurality of nodes can determine whether there is a task assigned to them among the plurality of tasks based on the task allocation results of the plurality of tasks maintained in the target database. If yes, then execute the assigned task.

[0069] Furthermore, if the target node determines that there are several allocatable tasks among multiple tasks, the target node can allocate corresponding nodes for these tasks from multiple nodes in the cluster and update the task allocation results of these tasks to the target database.

[0070] In one illustrated implementation, the target node can first identify several online nodes from a pool of nodes, and then assign corresponding nodes to the aforementioned tasks from among these online nodes. Specifically, the target node can access the target database and query the heartbeat information field in the node information table; if the timestamp recorded in the heartbeat information field corresponding to the queryed nodes is not less than a preset threshold, then these nodes can be determined to be online; otherwise, these nodes are determined to be offline.

[0071] In one illustrated embodiment, the target node may also save the list of online nodes locally, etc., without being specifically limited in this disclosure.

[0072] In one illustrated embodiment, the target node can access the target database to write the node identifier of the assigned node into the task assignment field corresponding to the tasks in the task assignment table maintained in the target database, thereby updating the task assignment results to the target database.

[0073] It should be noted that this application does not impose any specific restrictions on the task allocation strategy.

[0074] In one of the illustrated embodiments, the available tasks can be assigned to corresponding nodes in sequence according to the preset order of the online nodes determined above. In other words, the available tasks can be assigned to online nodes in sequence by polling. For example, if nodes 1 and 2 have been assigned tasks, then it is node 3's turn to be assigned tasks next, and so on. This disclosure does not make any specific limitations in this regard.

[0075] In one of the illustrated embodiments, the complexity of the task, execution time, and other information can be further considered, along with the computing resources and CPU utilization of the online nodes, to allocate several allocatable tasks to suitable online nodes. This disclosure does not impose any specific limitations on this.

[0076] For example, using Tables 1 and 2 above, the target node can determine that tasks 40002 and 40004 are allocable tasks, and that nodes 1, 2, and 3 are online. Further, the target node can determine to assign task 40002 to node 1 and task 40004 to node 2. After completing the task allocation in memory, the target node can write the task allocation results back to the target database all at once to reduce I / O interactions. Specifically, the target node can write the IP address of node 1 (114.10.10.1) to the `locked_by_ip` field corresponding to task 40002 in the task allocation table, and the IP address of node 2 (10.10.10.2) to the `locked_by_ip` field corresponding to task 40004.

[0077] In one illustrated implementation, multiple nodes in the cluster (including scheduling nodes and ordinary nodes) can access the target database to synchronize the task allocation results recorded in the task allocation table. Specifically, they can query the task allocation table to see if there is a task in the task allocation field that identifies their own node, i.e., to determine if there is a task assigned to them. If there is a task assigned to them, they execute the assigned task.

[0078] In one illustrated implementation, multiple nodes in the cluster can batch cache tasks whose task assignment field is their own node identifier into a local thread pool and execute the tasks according to the task execution plan (e.g., start time, number of repetitions, etc.).

[0079] In one of the illustrated embodiments, multiple nodes in the cluster can periodically query the task allocation table maintained by the target database to see if there are any tasks assigned to them, and maintain the number of tasks currently assigned to them, so as to execute the tasks assigned to them in a timely manner.

[0080] Furthermore, once any target task among multiple tasks has been completed, the node identifier recorded in the task allocation field corresponding to that target task in the task allocation table can be deleted. This target task can be a periodic task that needs to be executed at regular intervals. Accordingly, the completion of the target task can mean that it has been completed within the current period, or it can mean that the target task has been terminated (e.g., reaching a preset number of repetitions) and does not need to be executed again; this disclosure does not specifically limit this.

[0081] In one illustrated embodiment, after the target task is completed, the assigned node can access the target database to delete the node identifier recorded in the task allocation field corresponding to the target task in the task allocation table. Alternatively, in one illustrated embodiment, after the target task is completed, the node to which the target task was assigned can also send a message to the scheduling node indicating that the target task has been completed. The scheduling node can then access the target database based on the received message and delete the node identifier recorded in the task allocation field corresponding to the target task in the task allocation table, etc. This disclosure does not specifically limit the scope of the embodiments.

[0082] In one illustrated implementation, in response to any target node in the cluster (e.g., node 2 shown in Table 1, or node 100c shown in Figure 1) not being elected as the scheduling node, as described above, the target node can access the target database and determine whether there is a task assigned to the target node among the multiple tasks maintained in the target database based on the task allocation results of the multiple tasks; if so, the assigned task is executed.

[0083] In summary, this application, referencing the master-slave mechanism, introduces an election and task allocation mechanism into the existing distributed scheduling framework. This allows multiple nodes in the distributed cluster to compete for the position of scheduler. The elected scheduler then uniformly allocates tasks to be executed to each node in the cluster and writes the task allocation results to the database. Each node can then directly execute its assigned task by reading the task allocation results from the database. This achieves "one-write-many-read" task allocation, eliminates lock contention, and effectively avoids the concurrent execution of the same task on different nodes, significantly improving the task execution efficiency and performance of the distributed cluster.

[0084] Corresponding to the above-described method and process, embodiments of this disclosure also provide a task scheduling device based on a distributed cluster. Please refer to Figure 3, which is a schematic diagram of the structure of a task scheduling device based on a distributed cluster provided in an exemplary embodiment. This device 30 can be applied to any target node in the distributed cluster shown in Figure 1 (e.g., any one of node 100a, node 100b, node 100c, and node 100d). A target database is deployed in the distributed cluster, which maintains the task allocation results of multiple tasks to be executed.

[0085] As shown in Figure 3, the device 30 includes: an election unit 301, configured to elect a scheduling node among the plurality of nodes according to a preset election strategy; a determination unit 302, configured to, in response to the election of a target node as the scheduling node among the plurality of nodes, access the target database and determine whether there are several tasks that can be assigned among the plurality of tasks based on the task allocation results of the plurality of tasks maintained in the target database; and a task allocation unit 303, configured to, if so, allocate corresponding nodes to the several tasks from the plurality of nodes and update the task allocation results of the several tasks to the target database, so that the plurality of nodes can access the target database and determine whether there are tasks assigned to them among the plurality of tasks based on the task allocation results of the plurality of tasks maintained in the target database, and if so, execute the assigned tasks.

[0086] In one illustrated embodiment, the apparatus 30 further includes a task execution unit 304, configured to: in response to the target node not being elected as the scheduling node among the plurality of nodes, access the target database, determine whether there is a task among the plurality of tasks assigned to the target node based on the task allocation results of the plurality of tasks maintained in the target database; if so, execute the assigned task.

[0087] In one illustrated implementation, the assignable tasks include: tasks on unassigned nodes, or tasks assigned to nodes that are offline.

[0088] In one illustrated embodiment, the target database maintains a task allocation table, which includes a task allocation field for recording the task allocation results, including the node identifier of the assigned node.

[0089] In one illustrated embodiment, the task allocation unit 303 is specifically configured to: access the target database, and write the node identifier of the allocated node into the task allocation field corresponding to the plurality of tasks in the task allocation table maintained by the target database, so that the plurality of nodes access the target database and query the task allocation field in the task allocation table to determine whether there is a task among the plurality of tasks whose corresponding task allocation field records its own node identifier.

[0090] In one illustrated embodiment, the apparatus 30 further includes a node identifier deletion unit 305, configured to: in response to the completion of execution of any target task among the plurality of tasks by the node to which it is assigned, the node accesses the target database and deletes the node identifier recorded in the task allocation field corresponding to the target task in the task allocation table.

[0091] In one illustrated implementation, the node identifier of a node includes the node's IP address.

[0092] In one illustrated embodiment, the target database also maintains a node information table, which includes a scheduling node indication field. The scheduling node indication field is used to record indication information indicating whether a node is a scheduling node. After the target node is elected as the scheduling node among the plurality of nodes, the device 30 further includes an indication information writing unit 306, which is used to: access the target database and write indication information indicating that the target node is a scheduling node into the scheduling node indication field corresponding to the target node in the node information table maintained by the target database.

[0093] In one illustrated embodiment, the task allocation unit 303 is specifically configured to: determine a plurality of online nodes from the plurality of nodes, and allocate corresponding nodes to the plurality of tasks from the plurality of online nodes.

[0094] In one illustrated embodiment, the node information table further includes a heartbeat information field, which is used to record the timestamp of the node's most recent heartbeat. The task allocation unit 303 is specifically used to: access the target database and query the heartbeat information field in the node information table; in response to the timestamp recorded in the heartbeat information field corresponding to several nodes not being less than a preset threshold, determine that the several nodes are online; otherwise, determine that the several nodes are offline.

[0095] In one illustrated embodiment, the task allocation unit 303 is specifically configured to: sequentially allocate corresponding nodes to the plurality of tasks according to a preset order of the plurality of online nodes.

[0096] In one illustrated embodiment, the election unit 301 is specifically used to: elect a scheduling node among the plurality of nodes based on the IP addresses of the plurality of nodes; wherein the IP address of the scheduling node is the largest IP address among the plurality of nodes.

[0097] In one illustrated embodiment, the election unit 301 is specifically configured to: in response to the completion of initialization of the distributed cluster, or the original scheduling node being offline, elect a scheduling node from among the plurality of nodes according to a preset election strategy.

[0098] The specific implementation process of the functions and roles of each unit in the aforementioned device 30 is detailed in the description of the above embodiments and will not be repeated here. It should be understood that the aforementioned device 30 can be implemented through software, hardware, or a combination of both. Taking software implementation as an example, as a logical device, it is formed by the processor (CPU) of the device loading the corresponding computer program instructions into memory for execution. From a hardware perspective, in addition to the CPU and memory, the device typically includes other hardware such as chips for wireless signal transmission and reception, and / or other hardware such as boards for implementing network communication functions.

[0099] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the units or modules can be selected to achieve the purpose of this disclosure according to actual needs. Those skilled in the art can understand and implement this without any inventive effort.

[0100] The devices, units, and modules described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, in-vehicle computer, or any combination of these devices.

[0101] Corresponding to the above method embodiments, the embodiments of this disclosure also provide a computing device. Please refer to FIG4, which is a schematic diagram of the structure of a computing device provided in an exemplary embodiment. The computing device can be any target node in the distributed cluster shown in FIG1 (e.g., any one of node 100a, node 100b, node 100c, and node 100d). A target database is deployed in the distributed cluster, which maintains the task allocation results of multiple tasks to be executed. As shown in FIG4, the computing device includes a processor 1001 and a memory 1002, and may further include an input device 1004 (e.g., a keyboard) and an output device 1005 (e.g., a display). The processor 1001, memory 1002, input device 1004, and output device 1005 can be connected via a bus or other means. As shown in FIG4, the memory 1002 includes a computer-readable storage medium 1003, which stores a computer program executable by the processor 1001. The processor 1001 can be a CPU, a microprocessor, or an integrated circuit used to control the execution of the above method embodiments. When the processor 1001 runs the stored computer program, it can execute various steps of the task scheduling method based on a distributed cluster in this embodiment of the present disclosure, including: applying to any target node among multiple nodes included in the distributed cluster; a target database is deployed in the distributed cluster, and the target database maintains the task allocation results of multiple tasks to be executed; the method includes: electing a scheduling node among the multiple nodes according to a preset election strategy; in response to the target node being elected as the scheduling node among the multiple nodes, accessing the target database, determining whether there are several allocatable tasks among the multiple tasks according to the task allocation results of the multiple tasks maintained in the target database; if so, allocating corresponding nodes for the several tasks from the multiple nodes, and updating the task allocation results of the several tasks to the target database, so that the multiple nodes can access the target database, determine whether there are tasks assigned to them among the multiple tasks according to the task allocation results of the multiple tasks maintained in the target database, and if so, executing the assigned tasks, etc.

[0102] For a detailed description of each step of the above-described task scheduling method based on distributed clusters, please refer to the previous content, which will not be repeated here.

[0103] Corresponding to the above method embodiments, embodiments of this disclosure also provide a computer-readable storage medium storing computer programs. When executed by a processor, these computer programs perform the various steps of the task scheduling method based on a distributed cluster in the embodiments of this disclosure. Please refer to the descriptions of the above embodiments for details, which will not be repeated here.

[0104] The above description is merely a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

[0105] In a typical configuration, a terminal device includes one or more CPUs, input / output interfaces, network interfaces, and memory.

[0106] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0107] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data.

[0108] Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0109] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0110] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, embodiments of this disclosure can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this disclosure can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

Claims

1. A task scheduling method based on a distributed cluster, applied to any target node among multiple nodes included in the distributed cluster; A target database is deployed in the distributed cluster, and the target database maintains the task allocation results of multiple tasks to be executed; the method includes: According to the preset election strategy, a scheduling node is elected from among the multiple nodes; In response to the election of the target node as the scheduling node, the target database is accessed, and based on the task allocation results of the multiple tasks maintained in the target database, it is determined whether there are several tasks that can be allocated among the multiple tasks. If so, then a corresponding node is assigned to the plurality of tasks from the plurality of nodes, and the task assignment results of the plurality of tasks are updated to the target database, so that the plurality of nodes can determine whether there is a task assigned to them among the plurality of tasks based on the task assignment results of the plurality of tasks maintained in the target database, and if so, the assigned task is executed.

2. The method according to claim 1, further comprising: In response to the fact that the target node is not elected as the scheduling node, the target database is accessed, and based on the task allocation results of the multiple tasks maintained in the target database, it is determined whether there is a task among the multiple tasks that is assigned to the target node; If so, then execute the assigned task.

3. The method according to claim 2, wherein, The available tasks include: tasks on unassigned nodes, or tasks assigned to nodes that are offline.

4. The method according to claim 3, wherein, The target database maintains a task allocation table, which includes a task allocation field. The task allocation field is used to record the task allocation results, and the task allocation results include the node identifier of the assigned node.

5. The method according to claim 4, wherein, The step of updating the task allocation results of the plurality of tasks to the target database, so that the plurality of nodes determine whether there is a task assigned to them among the plurality of tasks based on the task allocation results of the plurality of tasks maintained in the target database, includes: Access the target database, and write the node identifier of the assigned node into the task allocation field corresponding to the several tasks in the task allocation table maintained by the target database, so that the multiple nodes can access the target database and query the task allocation field in the task allocation table to determine whether there is a task among the multiple tasks whose corresponding task allocation field records its own node identifier.

6. The method according to claim 5, further comprising: In response to the completion of any target task among the plurality of tasks by the node to which it was assigned, the node accesses the target database and deletes the node identifier recorded in the task assignment field corresponding to the target task in the task assignment table.

7. The method according to claim 4, wherein, The node identifier includes the node's IP address.

8. The method according to claim 1, wherein, The target database also maintains a node information table, which includes a scheduling node indication field. The scheduling node indication field is used to record indication information as to whether a node is a scheduling node. After the target node is elected as the scheduling node, the method further includes: Access the target database, and write indication information to indicate that the target node is a scheduling node in the scheduling node indication field corresponding to the target node in the node information table maintained by the target database.

9. The method according to claim 8, wherein, The process of assigning corresponding nodes to the plurality of tasks from the plurality of nodes includes: Several online nodes are determined from the plurality of nodes, and corresponding nodes are assigned to the several tasks from the several online nodes.

10. The method according to claim 9, wherein, The node information table also includes a heartbeat information field, which is used to record the timestamp of the node's most recent heartbeat. The step of determining a plurality of online nodes from the plurality of nodes includes: Access the target database and query the heartbeat information field in the node information table; If the timestamp recorded in the heartbeat information field corresponding to the nodes is not less than a preset threshold, the nodes are determined to be online; otherwise, the nodes are determined to be offline.

11. The method according to claim 9, wherein, The process of assigning corresponding nodes to the tasks from the online nodes includes: According to the preset order of the online nodes, the corresponding nodes are assigned to the tasks in sequence.

12. The method according to claim 7, wherein, The step of electing a scheduling node from the plurality of nodes according to a preset election strategy includes: Based on the IP addresses of the plurality of nodes, a scheduling node is elected from among the plurality of nodes; wherein the IP address of the scheduling node is the largest IP address among the plurality of nodes.

13. The method according to any one of claims 1-12, wherein, The step of electing a scheduling node from the plurality of nodes according to a preset election strategy includes: In response to the completion of the initialization of the distributed cluster, or the original scheduling node being offline, a scheduling node is elected from among the multiple nodes according to a preset election strategy.

14. A task scheduling device based on a distributed cluster, applied to any target node among multiple nodes included in the distributed cluster; A target database is deployed in the distributed cluster, and the target database maintains the task allocation results of multiple tasks to be executed; The device includes: An election unit is used to elect a scheduling node from the plurality of nodes according to a preset election strategy. The determining unit is configured to, in response to the election of the target node as the scheduling node, access the target database and determine whether there are several assignable tasks among the multiple tasks based on the task allocation results of the multiple tasks maintained in the target database. The task allocation unit is configured to, if so, allocate corresponding nodes to the plurality of tasks from the plurality of nodes, and update the task allocation results of the plurality of tasks to the target database, so that the plurality of nodes can determine whether there is a task assigned to them among the plurality of tasks based on the task allocation results of the plurality of tasks maintained in the target database, and if so, execute the assigned task.

15. A computing device, comprising: A memory and a processor; wherein the memory stores a computer program / instructions executable by the processor; and when the processor executes the computer program / instructions, it performs the method as described in any one of claims 1-13.

16. A computer-readable storage medium having a computer program / instructions stored thereon, wherein, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-13.

17. A computer program product comprising a computer program / instructions, wherein, When the computer program / instructions are executed by the processor, they implement the method as described in any one of claims 1-13.

Images