A task scheduling method, device, system and storage medium

Abstract

Embodiments of the present application provide a task scheduling method, device, system and storage medium. In the embodiments of the present application, a task group corresponds to a message topic, a lightweight task scheduling function can be deployed on a task processing node without introducing additional dependent resources, and the task processing node is used to map tasks in the task group to a plurality of topic partitions included in the message topic corresponding to the task group. On this basis, the task processing cluster can be used as a consumption cluster of the target message topic, and the topic partitions are allocated to each task processing node by using the message processing node where the target message topic is located. That is, the message queue technology is reused in the allocation link of the topic partitions. In this way, the task processing node can balance the distribution of the tasks in the task group to each task processing node in the task processing cluster by using the message queue technology and taking the topic partitions as an intermediate medium, so that the task scheduling can be implemented without relying on a data source.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of cloud computing, and particularly relates to a task scheduling method, device, system and storage medium. BACKGROUND

[0002] A streaming process is a continuous processing mode for unbounded data or events. At present, in a streaming process, a data source usually undertakes task scheduling work. The data source needs to have load balancing capability and push to a suitable task processing node in a push mode after load balancing of data processing tasks.

[0003] However, as the types of data sources increase, many types of data sources do not have load balancing capability, which brings difficulties to the streaming process scenario. SUMMARY

[0004] Aspects of the present application provide a task scheduling method, device, system and storage medium to realize task scheduling without relying on a data source.

[0005] The present application provides a task scheduling method, applied to a task processing node, and the method comprises the following steps.

[0006] If it is determined that the current task processing node itself is a master scheduling end of a target task group, a mapping relationship between a current task in the target task group and a plurality of topic partitions contained in a target message topic configured for the target task group is established;

[0007] The mapping relationship is transmitted to other task processing nodes in a task processing cluster corresponding to the target task group;

[0008] The task processing cluster is used as a consumption cluster of the target message topic, and the plurality of topic partitions are distributed to each task processing node in the task processing cluster by using a message processing node where the target message topic is located;

[0009] Based on the mapping relationship, a target task mapped by a topic partition allocated to the current task processing node itself is determined and the target task is started.

[0010] The present application also provides a task processing node, comprising a memory, a processor and a communication component.

[0011] The memory is used to store one or more computer instructions;

[0012] The processor is coupled with the memory and the communication component, and is used to execute the one or more computer instructions, so as to:

[0013] If it is determined that the current task processing node is a master scheduling end of the target task group, a mapping relationship between a current task in the target task group and a plurality of topic partitions included in a target message topic configured for the target task group is established;

[0014] The mapping relationship is transmitted to other task processing nodes in a task processing cluster corresponding to the target task group through the communication component;

[0015] The task processing cluster is used as a consumption cluster of the target message topic, and the plurality of topic partitions are allocated to each task processing node in the task processing cluster by a message processing node where the target message topic is located;

[0016] Based on the mapping relationship, a target task to which a topic partition allocated to the current task processing node is mapped is determined, and the target task is started.

[0017] Embodiments of the present application also provide a data processing system, comprising: a task processing cluster and a message queue cluster, the task processing cluster being a consumption cluster of a target message topic in the message queue cluster, a plurality of topic partitions being included in the target message topic, the target message topic corresponding to a target task group;

[0018] A master scheduling end of the target task group in the task processing cluster is configured to establish a mapping relationship between a current task in the target task group and a plurality of topic partitions included in the target message topic, and transmit the mapping relationship to other task processing nodes in a task processing cluster corresponding to the target task group;

[0019] A message processing node where the target message topic is located in the message queue cluster is configured to allocate the plurality of topic partitions to each task processing node in the task processing cluster, so that each task processing node determines a target task to which a topic partition allocated to the task processing node is mapped based on the mapping relationship, and starts the target task.

[0020] Embodiments of the present application also provide a computer readable storage medium storing computer instructions, when the computer instructions are executed by one or more processors, causing the one or more processors to perform the task scheduling method.

[0021] In the embodiments of the present application, a task group corresponds to a message topic, a lightweight task scheduling function can be deployed on the task processing node without introducing additional dependent resources, and the task processing node is used to map the tasks in the task group to a plurality of topic partitions included in the message topic corresponding to the task group. On this basis, the task processing cluster can be used as a consumption cluster of the target message topic, and the message processing node where the target message topic is located is used to distribute the topic partitions to each task processing node. That is, the message queue technology can be reused in the distribution of the topic partitions. In this way, the task processing node can use the message queue technology to evenly distribute the tasks in the task group to each task processing node in the task processing cluster by taking the topic partitions as an intermediate medium, so that the task scheduling can be implemented without relying on the data source. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate embodiments of the present application and the description thereof serve to explain the present application, but do not constitute improper limitations on the present application. In the drawings:

[0023] Figure 1 A structural schematic diagram of a data processing system is provided for an exemplary embodiment of the present application;

[0024] Figure 2 A flowchart of a task scheduling method is provided for another exemplary embodiment of the present application;

[0025] Figure 3 A structural schematic diagram of a task processing node is provided for still another exemplary embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application will be described clearly and completely in combination with the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0027] Currently, with the increasing variety of data sources, many lack load balancing capabilities, posing a challenge to streaming processing scenarios. To address this, some embodiments of this application propose: a task group corresponds to a message topic. A lightweight task scheduling function can be deployed on the task processing nodes without introducing additional dependent resources. The task processing nodes then map tasks within the task group to multiple topic partitions contained under the corresponding message topic. Based on this, the task processing cluster can serve as a consumer cluster for the target message topic, utilizing the message processing nodes where the target message topic resides to distribute topic partitions to various task processing nodes. That is, message queue technology is reused in the topic partition allocation process. In this way, task processing nodes can reuse message queue technology, using topic partitions as an intermediary, to evenly distribute tasks within a task group to various task processing nodes in the task processing cluster, thereby achieving task scheduling without relying on a data source.

[0028] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0029] Figure 1 This is a schematic diagram of the structure of a data processing system provided for an exemplary embodiment of this application. For example... Figure 1 As shown, the system includes a task processing cluster and a message queue cluster. The task processing cluster can contain multiple task processing nodes, and the message queue cluster can contain multiple message processing nodes. A single message processing node can host one or more message topics. A message topic is the top-level container for message transmission and storage in the message queue, used to identify messages of the same logical type. A single message topic can further include multiple topic partitions. A topic partition is a physical concept and can be understood as multiple message queue carriers contained under a message topic; the actual message queues are carried within topic partitions. In this embodiment, a task group corresponds to one task processing cluster, and a message topic is configured for the task group within the message queue cluster.

[0030] In this embodiment, the task scheduling scheme will be described from the perspective of a task group, namely the target task group (hereinafter referred to as the target task group). The message topic corresponding to the target task group is described as the target message topic. A task group can contain multiple tasks. In this embodiment, the task group division mechanism can be set according to actual needs, and this embodiment does not limit it. For example, a job can be considered as a task group. A job can be a complex topology, and one or more nodes in the topology constitute a task. Thus, the task group corresponding to a job can contain one or more tasks. Furthermore, in this embodiment, the source of the task group is not limited, such as... Figure 1As shown, the task group can be sourced from a source data source. In this embodiment, the data source can be diverse, and the data source in this embodiment no longer needs to have load balancing capability. Therefore, the data source in this embodiment can be a DB (DoggaByte, Dao byte) database, an ES (Elasticsearch, Elastic search) distributed search engine, etc. Of course, it can also be other sources, for example, the task group can be sourced from a virtual collection task group, which is hung with a plurality of collection tasks. In this embodiment, a plurality of tasks in the task group can be concurrent, and this embodiment does not limit this.

[0031] For ease of understanding, Figure 1 In this embodiment, only one task processing cluster is shown, that is, the task processing cluster corresponding to the target task group, but it should be understood that the data processing system in this embodiment can also include other task processing clusters. Different task processing clusters can be used to serve different task groups.

[0032] In terms of implementation, the task processing node in this embodiment can be a physical machine, a virtual machine, an instance, etc. The implementation form of the task processing node is not limited in this embodiment. In different application scenarios, the task processing node can be deployed as needed in a suitable location. For example, in an edge computing scenario, the task processing node in this embodiment can be deployed on an edge node.

[0033] In this embodiment, a lightweight task scheduling function can be deployed on the task processing node. Lightweight here means that the task scheduling function in this embodiment does not need to introduce any additional dependent resources. This has obvious lightweight characteristics compared to existing heavy data processing engines such as flink. Even 1 core, 1 g, 1 jar package can be started, which makes the data processing system provided in this embodiment can be deployed more quickly and more conveniently. In addition, since the data processing system in this embodiment can implement task scheduling without relying on the data source, it can be compatible with more scene architectures and can be connected to task groups of various sources. The data processing system provided in this embodiment can also be compatible with the connector in the existing flink architecture to optimize and reduce the weight of the existing flink architecture.

[0034] Reference Figure 1In this embodiment, a task scheduling client can be run on each task processing node in the task processing cluster corresponding to the target task group. The implementation form of the task scheduling client can be an instance, a process, a service, etc., which is not limited herein. Based on the task scheduling client, the task processing node can support the task scheduling logic proposed in this embodiment. On this basis, a main scheduling end can be selected for the target task group in the task processing cluster. Alternatively, each task processing node in the task processing cluster can autonomously determine whether it can serve as the main scheduling end. In an exemplary scheme, each task processing node can respectively perform hypothetical consumption on a plurality of topic partitions included in the target message topic to determine whether it will consume the last topic partition under the target message topic. If so, it can be determined that the task processing node serves as the main scheduling end of the target task group. Of course, this is only exemplary. In this embodiment, other exemplary schemes can also be used to autonomously determine whether each task processing node can serve as the main scheduling end. Alternatively, the main scheduling end for the target task group can be specified by a control node in the task processing cluster instead of being autonomously determined by each task processing node. This embodiment does not limit the implementation manner of determining the main scheduling end of the target task group.

[0035] After the main scheduling end corresponding to the target task group is determined, the main scheduling end can listen to the tasks in the target task group in turns. Alternatively, the tasks in the task group can be listened to in turns in a timing manner. For example, a round of task inquiry can be performed every 10 minutes. If the current round of task is not empty, subsequent task scheduling work can be started.

[0036] Reference Figure 1 In this embodiment, the main scheduling end can establish a mapping relationship between the current round of tasks in the target task group and a plurality of topic partitions included in the target message topic configured for the target task group. The mapping relationship is one-to-one between the tasks and the topic partitions, that is, one task is mapped to one topic partition, and one topic partition is bound to one task. In the process of establishing the mapping relationship, the mapping principle of the main scheduling end is to map the current round of tasks as dispersedly as possible to the plurality of topic partitions under the target message topic. An exemplary mapping strategy can be a vibration strategy:

[0037] The first task is mapped to the first topic partition;

[0038] The second task is mapped to the last topic partition;

[0039] The third task is mapped to the middle topic partition;

[0040] The fourth task is mapped to the middle partition in the second half;

[0041] The fifth task is mapped to the middle partition in the first half;

[0042] By analogy, the current task can be mapped to the corresponding topic partition.

[0043] Of course, the mapping strategy described above is only exemplary, and the present embodiment is not limited thereto. Other mapping strategies can also be used to complete the operation of establishing a mapping relationship, which will not be enumerated here.

[0044] On this basis, the main scheduling end can pass the mapping relationship to other task processing nodes in the task processing cluster corresponding to the target task group. It should be understood that the other task processing nodes refer to the remaining task processing nodes in the task processing cluster except the current task processing node (i.e. the main scheduling end). In an optional implementation manner, the task processing cluster can be used as the consumer cluster of the target message topic, and the main scheduling end can publish the mapping relationship under the target message topic to pass the mapping relationship to other task processing nodes in the task processing cluster. In this way, each task processing node in the task processing cluster will have the mapping relationship corresponding to the target task group. In this implementation manner, since the task processing cluster subscribes to the target message topic, after the main scheduling end publishes the mapping relationship under the target message topic, the mapping relationship can be automatically passed to each task processing node in the task processing cluster, thereby achieving the transmission of the mapping relationship. Of course, in the present embodiment, other implementation manners can also be used to transmit the mapping relationship, including but not limited to intra-cluster broadcasting, etc., which will not be limited herein.

[0045] In the present embodiment, the same cluster identifier (corresponding to the group ID in the message queue technology) can be used to start a consumer client (consumer) on each task processing node in the task processing cluster corresponding to the target task group. The implementation form of the consumer client can be a process, a service, etc., which will not be limited herein, which makes each task processing node in the task processing cluster become a consumer of the target message topic, and the task processing cluster becomes the consumer group of the target message topic.

[0046] In addition, in the embodiment, a special message topic can be additionally maintained for the target task group in the message queue cluster. After the mapping relationship is delivered by the main dispatching end corresponding to the target task group, the other task processing nodes (i.e., non-main dispatching ends) in the task processing cluster corresponding to the target task group can send a confirmation message to the special message topic. An exemplary confirmation message can be an ACK (Acknowledge) message, which can include the dispatching identifier and / or the timestamp corresponding to the mapping relationship, to represent the confirmation for which mapping relationship. The main dispatching end can deliver a mapping relationship taking effect message to the other task processing nodes in the task processing cluster to take effect the mapping relationship on each task processing node in the task processing cluster, if the confirmation message received in the special message topic meets a preset condition. An exemplary mapping relationship taking effect message can be a commit message. The preset condition can include, but is not limited to, that all the other task processing nodes have sent the confirmation message for the mapping relationship or that the remaining task processing nodes except the faulty task processing node have sent the confirmation message for the mapping relationship. It should be understood that there is a premise that the main dispatching end only judges whether the confirmation message for the mapping relationship of the current task meets the preset condition, and the expired confirmation message (i.e., the confirmation message not for the mapping relationship of the current task) can be directly ignored. In addition, an exemplary discovery scheme for the faulty task processing node in the task processing cluster corresponding to the target task group can be that if the main dispatching end confirms that the special message topic does not receive the confirmation message for the mapping relationship of the current task from all the other task processing nodes (i.e., non-main dispatching ends) within a preset time, the faulty task processing node in the task processing cluster corresponding to the target task group can be discovered by detecting a heartbeat or the like.

[0047] Based on this, referring to Figure 1The message processing node where the target message topic is located can assign the plurality of topic partitions included in the target message topic to the task processing cluster as a consumption cluster of the target message topic. That is, each task processing node in the task processing cluster can consume the plurality of topic partitions included in the target message topic. The message processing node can assign the topic partitions according to a consumption strategy corresponding to the message cluster. That is, each consumer in the same consumption cluster is evenly allocated to consume the topic partitions. For example, if the target message topic includes 100 topic partitions, 20 consumers in the consumption cluster are each allocated to 5 topic partitions. Therefore, in the technical concept provided in the embodiment, as long as the current task is mapped to the plurality of topic partitions included in the target message topic as much as possible in the mapping relationship establishment step, the balanced distribution of the current task to each task processing node in the task processing cluster can be ensured by relying on the balanced allocation strategy of the topic partitions in the message queue technology.

[0048] After the assignment of the topic partitions is completed, each task processing node in the task processing cluster corresponding to the target task group is allocated to a topic partition. Based on this, for a single task processing node, the target task mapped to the topic partition allocated to the task processing node can be determined based on the mapping relationship, and the target task is started.

[0049] Optionally, for each non-primary scheduling end corresponding to the target task group, whether the primary scheduling end fails can also be monitored. If it is monitored that the primary scheduling end fails, the non-primary scheduling end can determine whether it becomes a new primary scheduling end. If so, on the basis of completing the currently running task, the mapping relationship establishment operation, the mapping relationship transmission operation, and the operation of assigning the topic partitions by the message processing node can be performed by the primary scheduling end. Among them, on the basis of completing the currently running task, the identity of the primary scheduling end is implemented, which can avoid task omission. In this way, the new primary scheduling end only needs to focus on the scheduling of the next task obtained by it, and does not need to trace the historical task scheduled by the previous primary scheduling end. However, the new primary scheduling end also needs to perform the operation of assigning the topic partitions by the message processing node, because the failure of the previous primary scheduling end can be considered as the reduction of the task processing nodes in the task processing cluster. In the case of changes in the task processing cluster, the topic partitions can be re-assigned to ensure the balance of the assignment.

[0050] Accordingly, in the embodiment, the task group corresponds to a message topic, a lightweight task scheduling function can be deployed on the task processing node without introducing additional dependent resources, and the task processing node maps the tasks in the task group to the multiple topic partitions included in the message topic corresponding to the task group; on this basis, the message queue technology can be reused to allocate the topic partitions to each task processing node. In this way, the task processing node can evenly distribute the tasks in the task group to each task processing node in the task processing cluster by reusing the message queue technology, so that the task scheduling can be implemented without relying on the data source.

[0051] In the above or the following embodiment, in the scheduling process of the current round of tasks, the main scheduling end can use the message processing node to listen to whether the task processing cluster has changed the task processing node; if so, the message processing node can trigger the multiple topic partitions to be allocated to each task processing node in the task processing cluster. That is, the change of the task processing node in the task processing cluster corresponding to the target task group will cause the change of the topic partitions allocated to each task processing node.

[0052] It is worth noting that in the above case, the mapping relationship corresponding to the current round of tasks in the target task group has not changed, but since the topic partitions allocated to each task processing node have changed, the tasks allocated to each task processing node based on the mapping relationship may change. Therefore, each task processing node can check whether the newly allocated topic partition has changed, and if so, the task start and stop can be performed as needed. Specifically, the task mapped by the newly allocated topic partition can be started, and other tasks currently running on the task processing node can be closed. That is, each task processing node needs to switch to only running the task mapped by the newly allocated topic partition. For example, for a task processing node, the tasks distributed on the task processing node are task A and task B before the task processing cluster changes, but after the task processing cluster changes, the task mapped by the newly allocated topic partition of the task processing node becomes task A and task C through the re-allocation of the message processing node, so the task processing node needs to close the originally running task B, start the newly allocated task C, and continue to run task A.

[0053] To ensure the robustness of the data processing system, in the embodiment, an implementation manner of exclusive lock can be adopted to ensure that one task is processed on one task processing node at the same time. In the implementation manner: for the task processing node that has task change, the message processing node can add an exclusive lock to the first partition in the topic partition to which the current task processing node is reassigned; if the addition is successful, the task mapped by the first partition is started; if the addition is not successful, the message processing node waits for the exclusive lock on the first partition to be released, and then performs the operation of adding an exclusive lock to the first partition again, so as to start the task on the first partition on the current task processing node under the condition that it is confirmed that the task on the first partition is not run by other task processing nodes; wherein the first partition is any one of the topic partitions to which the current task processing node is reassigned. It should be understood that the first partition can be originally exclusively occupied by other task processing nodes, and after the topic partition is reassigned, the task processing node that originally exclusively occupies the first partition will release the exclusive lock, and only after the exclusive lock is released by the task processing node, the task processing node that is currently assigned to the first partition can successfully add the exclusive lock to the first partition. In this way, by adding an exclusive lock to the topic partition, the same task can be avoided from being repeatedly processed by multiple task processing nodes.

[0054] It should be understood that the mapping relationship established for the current round of tasks is generally maintained unchanged during the scheduling of the current round of tasks. The main scheduling end can trigger the mapping relationship establishment operation and the mapping relationship transmission operation for the next round of tasks in the case that the next round of tasks in the target task group is not empty, so that each task processing node in the task processing cluster determines the task to which it is assigned in the next round of tasks based on the mapping relationship corresponding to the next round of tasks. The technical details of the mapping relationship establishment operation and the mapping relationship transmission operation can be referred to the foregoing, which will not be repeated here. It should be noted that during the scheduling of the next round of tasks, if the task processing cluster corresponding to the target task group does not change, the foregoing topic partition allocation operation does not need to be performed, and the topic partition to which each task processing node in the task processing cluster is assigned can remain unchanged, and each task processing node only needs to determine the task to which it is assigned in the next round of tasks based on the mapping relationship corresponding to the next round of tasks. Of course, if the task processing cluster corresponding to the target task group changes during the scheduling of the next round of tasks, the message processing node can perform the allocation operation of the topic partition again, each task processing node in the task processing cluster needs to determine the task to which it is reassigned in the next round of tasks based on the mapping relationship corresponding to the next round of tasks, and if the start and stop of the task are involved, the processing scheme provided in the foregoing can be referred to.

[0055] Accordingly, in the embodiment, the operation of establishing the mapping relationship and the operation of allocating the topic partitions can be triggered at appropriate time to dynamically adapt to the task processing cluster variation in the data processing system and the task variation in the target task group.

[0056] Figure 2 A flowchart of a task scheduling method according to another exemplary embodiment of the present application is provided. The method can include:

[0057] In step 200, if it is determined that the current task processing node is the master scheduling end of the target task group, a mapping relationship between the current round of tasks in the target task group and a plurality of topic partitions included in a target message topic configured for the target task group is established.

[0058] In step 201, the mapping relationship is transmitted to other task processing nodes in the task processing cluster corresponding to the target task group.

[0059] In step 202, the task processing cluster is used as a consumption cluster of the target message topic, and the plurality of topic partitions are allocated to each task processing node in the task processing cluster by using the message processing node where the target message topic is located.

[0060] In step 203, based on the mapping relationship, the target task to which the topic partition allocated to the current task processing node itself is mapped is determined and the target task is started.

[0061] In an optional embodiment, the step of transmitting the mapping relationship to other task processing nodes in the task processing cluster corresponding to the target task group can include:

[0062] The mapping relationship is published under the target message topic to transmit the mapping relationship to other task processing nodes in the task processing cluster.

[0063] In an optional embodiment, the method can further include:

[0064] A special message topic is maintained for the target task group to receive the confirmation message returned by other task processing nodes in the task processing cluster after the mapping relationship is acquired.

[0065] In a case where it is determined that the confirmation message received under the special message topic meets a preset condition, a mapping relationship taking effect message is transmitted to other task processing nodes in the task processing cluster to take effect on each task processing node in the task processing cluster.

[0066] In an optional embodiment, the preset condition includes that all other task processing nodes have sent the mapping relationship confirmation message or that, except for the faulty task processing node, the remaining task processing nodes among the other task processing nodes have sent the mapping relationship confirmation message.

[0067] In an optional embodiment, the method can further include:

[0068] listening, by the message processing node, to the task processing cluster for a change of the task processing node;

[0069] if a change occurs, triggering the message processing node to re-allocate the plurality of topic partitions to each of the task processing nodes in the task processing cluster.

[0070] In an optional embodiment, the method can further include:

[0071] if the topic partitions to which the current task processing node is re-allocated change, starting the task mapped by the re-allocated topic partitions;

[0072] stopping other tasks currently running in the current task processing node itself.

[0073] In an optional embodiment, the step of starting the task mapped by the re-allocated topic partitions includes:

[0074] adding an exclusive lock to a first partition in the topic partitions to which the current task processing node is re-allocated by the message processing node;

[0075] if the addition is successful, starting the task mapped by the first partition;

[0076] if the addition is not successful, waiting for the exclusive lock on the first partition to be released, and then performing the operation of adding an exclusive lock again to the first partition to start the task on the first partition on the current task processing node in a case where it is confirmed that the task on the first partition is not running on another task processing node;

[0077] wherein the first partition is any one of the topic partitions to which the current task processing node is re-allocated.

[0078] In an optional embodiment, the method can further include:

[0079] if the next round of tasks in the target task group is not empty, triggering the mapping relationship establishment operation and the mapping relationship transmission operation for the next round of tasks, so that each of the task processing nodes in the task processing cluster determines the task allocated to it in the next round of tasks based on the mapping relationship corresponding to the next round of tasks.

[0080] In an optional embodiment, the method can further include:

[0081] if it is determined that the current task processing node itself is a non-primary scheduling end of the target task group, obtaining the mapping relationship provided by the primary scheduling end;

[0082] consuming, by the message processing node, a plurality of topic partitions included in the target message topic;

[0083] Based on the mapping relationship, a target task mapped by the topic partition to which the current task processing node is allocated is determined, and the target task is started.

[0084] In an optional embodiment, the method can further include:

[0085] If the primary scheduling end is determined to have failed, it is determined whether the current task processing node itself becomes a new primary scheduling end;

[0086] If so, on the basis of completing the task running on the current task processing node, the mapping relationship establishment operation, the mapping relationship transmission operation, and the operation of allocating the topic partitions by the message processing node are performed in the identity of the primary scheduling end.

[0087] It should be noted that the technical details in the above embodiments of the task scheduling method can refer to the related descriptions of the task processing node in the foregoing system embodiments, which will not be described here again, but this should not cause any loss of the protection scope of the present application.

[0088] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a specific order are included, but it should be clearly understood that these operations can be executed or performed in parallel without following the order in which they appear in this text. The serial numbers of the operations, such as 801, 802, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel.

[0089] Figure 3 A structural schematic diagram of a task processing node is provided for another exemplary embodiment of the present application. As shown in the figure, the task processing node can include a memory 30, a processor 31, and a communication component 32. Figure 3

[0090] The processor 31 is coupled to the memory 30 and the communication component 32, and is configured to execute one or more computer instructions in the memory 30, so as to:

[0091] If it is determined that the current task processing node itself is the primary scheduling end of the target task group, a mapping relationship between the current round of tasks in the target task group and a plurality of topic partitions included in the target message topic configured for the target task group is established;

[0092] The mapping relationship is transmitted to other task processing nodes in the task processing cluster corresponding to the target task group through the communication component 32;

[0093] The task processing cluster is used as a consumption cluster of the target message topic, and the plurality of topic partitions are allocated to each task processing node in the task processing cluster by using the message processing node where the target message topic is located.​

[0094] Based on the mapping relationship, the target task mapped by the topic partition to which the current task processing node is allocated is determined, and the target task is started.

[0095] In an optional embodiment, in the process of delivering the mapping relationship to other task processing nodes in the task processing cluster corresponding to the target task group, the processor 31 can be configured to:

[0096] publish the mapping relationship under the target message topic to deliver the mapping relationship to other task processing nodes in the task processing cluster.

[0097] In an optional embodiment, the processor 31 can be further configured to:

[0098] maintain a special message topic for the target task group, so as to receive the confirmation message returned by other task processing nodes in the task processing cluster after the mapping relationship is obtained by the special message topic;

[0099] in a case where the confirmation message received under the special message topic meets a preset condition, deliver a mapping relationship taking effect message to other task processing nodes in the task processing cluster, so as to take effect the mapping relationship on each task processing node in the task processing cluster.

[0100] In an optional embodiment, the preset condition includes that all other task processing nodes have sent the confirmation message for the mapping relationship or that the remaining task processing nodes except the faulty task processing node among other task processing nodes have sent the confirmation message for the mapping relationship.

[0101] In an optional embodiment, the processor 31 can be further configured to:

[0102] monitor whether a change of task processing node occurs in the task processing cluster by using the message processing node;

[0103] if a change occurs, trigger the message processing node to re-allocate the plurality of topic partitions to each task processing node in the task processing cluster.

[0104] In an optional embodiment, the method can further include:

[0105] if the topic partition to which the current task processing node is re-allocated changes, start the task mapped by the re-allocated topic partition;

[0106] close other tasks currently running in the current task processing node itself.

[0107] In an optional embodiment, in the process of starting the task mapped by the re-allocated topic partition, the processor 31 can be configured to:

[0108] adding an exclusive lock to a first partition in the topic partition to which the current task processing node is reassigned by the message processing node;

[0109] if the adding is successful, starting a task mapped by the first partition;

[0110] if the adding is not successful, waiting for the exclusive lock on the first partition to be released, and then performing the adding of the exclusive lock again to start the task on the first partition on the current task processing node in a case that the task on the first partition is not run by other task processing nodes;

[0111] wherein the first partition is any one of the topic partitions to which the current task processing node is reassigned.

[0112] In an optional embodiment, the processor 31 can be further configured to:

[0113] if the next round of tasks in the target task group is not empty, triggering the mapping relationship establishing operation and the mapping relationship transferring operation for the next round of tasks, so that each task processing node in the task processing cluster determines the task allocated to itself in the next round of tasks based on the mapping relationship corresponding to the next round of tasks.

[0114] In an optional embodiment, the processor 31 can be further configured to:

[0115] if it is determined that the current task processing node itself is not the master scheduling end of the target task group, obtaining the mapping relationship provided by the master scheduling end;

[0116] consuming a plurality of topic partitions included in the target message topic by the message processing node;

[0117] determining the target task mapped by the topic partition allocated to the current task processing node itself based on the mapping relationship and starting the target task.

[0118] In an optional embodiment, the processor 31 can be further configured to:

[0119] if the master scheduling end is monitored to fail, determining whether the current task processing node itself becomes a new master scheduling end;

[0120] if yes, performing the mapping relationship establishing operation, the mapping relationship transferring operation, and the operation of allocating the topic partition by the message processing node as the master scheduling end on the basis of completing the task running on the current task processing node.

[0121] Further, as shown in Figure 3 the task processing node further includes other components such as a power supply component 33. Figure 3 only some components are shown schematically, and it does not mean that the task processing node only includes Figure 3The illustrated components. As mentioned in the foregoing system embodiments, the implementation form of the task processing node can be a physical machine, a virtual machine, and an instance, etc., and for the virtualization type of implementation form such as a virtual machine and an instance, the memory 30, the processor 31, the communication component 32, and the power supply component 33 contained in the task processing node can all be virtual components, that is, virtual components generated by relying on actual physical resources through virtualization technology.

[0122] It is worth noting that the technical details in the foregoing embodiments of the task processing node can refer to the related descriptions of the task processing node in the foregoing system embodiments, and will not be repeated here to save space, but this should not cause the loss of the protection scope of the present application.

[0123] Correspondingly, the embodiments of the present application also provide a computer readable storage medium storing a computer program, which can implement each step that can be executed by the task processing node in the method embodiments when the computer program is executed.

[0124] The memory 30 in the foregoing Figure 3 is used to store computer programs and can be configured to store other various data to support operations on the computing platform. Examples of these data include instructions for any application or method operating on the computing platform, contact data, phonebook data, messages, pictures, videos, and the like. The memory 30 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0125] The communication component 32 in the foregoing Figure 3 is configured to facilitate wired or wireless communication between the device where the communication component 32 is located and other devices. The device where the communication component 32 is located can access a wireless network based on a communication standard, such as a WiFi, 2G, 3G, 4G / LTE, 5G, or the like mobile communication network, or a combination thereof. In an example embodiment, the communication component 32 can receive a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 32 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0126] The communication component 32 in the foregoing Figure 3The power component 33 in the electronic device 100 provides power to various components of the electronic device 100 in which the power component 33 is located. The power component 33 can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the electronic device in which the power component is located.

[0127] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In one example, the present application can be implemented using PC-based software, hardware-based software, or a combination of software and hardware. In one embodiment, the present application can be implemented using a combination of software and hardware.

[0128] The present application is described in reference to the flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0129] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams. Figure 1 one or more functions specified in the flowchart illustrations and / or block diagrams.

[0131] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0132] Memory can include non-persistent memory, such as volatile random access memory (RAM) and / or non-volatile memory, such as read only memory (ROM), electrically erasable read only memory (EEPROM), flash memory, or the like. Memory is an example of computer readable media.

[0133] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer readable media does not include transitory media, such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "comprising", "containing", or any other similar term are intended to encompass non-exclusive inclusions such that a process, method, article, or apparatus that comprises a list of elements does not include those elements solely, but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0135] The above description is merely illustrative of the embodiments of the present application and is not intended to limit the scope of the application. Any modifications, equivalent replacements, improvements, and the like that are made within the spirit and principle of the present application should be included in the scope of the present application.

Claims

1. A task scheduling method, characterized by, The method applied to a task processing node comprises: If it is determined that the current task processing node is a master scheduling end of a target task group, a mapping relationship between a current task in the target task group and a plurality of topic partitions contained in a target message topic configured for the target task group is established; The mapping relationship is transmitted to other task processing nodes in a task processing cluster corresponding to the target task group; The task processing cluster is used as a consumption cluster of the target message topic, and the plurality of topic partitions are allocated to each task processing node in the task processing cluster by using a message processing node where the target message topic is located; Based on the mapping relationship, a task mapped by a topic partition allocated to the current task processing node is determined as a target task, and the target task is started.

2. The method of claim 1, wherein, The mapping relationship is transmitted to other task processing nodes in the task processing cluster, comprising: The mapping relationship is published to the target message topic to transmit the mapping relationship to other task processing nodes in the task processing cluster.

3. The method according to any of claims 1 or 2, characterized in that, Further comprising: A special message topic is maintained for the target task group to receive confirmation messages returned by other task processing nodes in the task processing cluster after the mapping relationship is acquired; In a case where it is determined that the confirmation messages received in the special message topic meet preset conditions, a mapping relationship effective message is transmitted to other task processing nodes in the task processing cluster to make the mapping relationship effective on each task processing node in the task processing cluster.

4. The method of claim 3, wherein, The preset conditions comprise: All other task processing nodes have sent the mapping relationship confirmation message; or Except for a faulty task processing node, the remaining task processing nodes in the other task processing nodes have sent the mapping relationship confirmation message.

5. The method of claim 1, wherein, Further comprising: Whether a change of a task processing node in the task processing cluster is listened to by using the message processing node; If a change occurs, the message processing node is triggered to re-allocate the plurality of topic partitions to each task processing node in the task processing cluster.

6. The method of claim 5, wherein, Further comprising: If a topic partition to which the current task processing node is re-allocated changes, a task mapped by the re-allocated topic partition is started; Other tasks currently running in the current task processing node are closed.

7. The method of claim 6, wherein, The task mapped by the re-allocated topic partition is started, comprising: An exclusive lock is added to a first partition in the topic partition to which the current task processing node is re-allocated by using the message processing node; If the addition is successful, the task mapped by the first partition is started; If the addition is not successful, the first partition is waited to be released, and the exclusive lock adding operation is performed again to start the task on the first partition on the current task processing node in a case where it is confirmed that the task on the first partition is not running on other task processing nodes; The first partition is any one of the re-allocated topic partitions.

8. The method of claim 1, wherein, Further comprising: If the next round of tasks in the target task group is not empty, a mapping relationship establishment operation and a mapping relationship transmission operation for the next round of tasks are triggered, so that each task processing node in the task processing cluster determines the task allocated to it in the next round of tasks based on the mapping relationship corresponding to the next round of tasks.

9. The method of claim 1, wherein, Further comprising: If it is determined that the current task processing node itself is a non-primary scheduling end of the target task group, the mapping relationship provided by the primary scheduling end is obtained; Through the message processing node, a plurality of topic partitions contained in the target message topic are consumed; Based on the mapping relationship, the task allocated to the current task processing node itself is determined as a target task, and the target task is started.

10. The method of claim 9, wherein, Further comprising: If the primary scheduling end is found to have failed, it is determined whether the current task processing node itself becomes a new primary scheduling end; If so, the mapping relationship establishment operation, the mapping relationship transmission operation, and the operation of allocating topic partitions by the message processing node are performed as the primary scheduling end on the basis of completing the task running on the current task processing node.

11. A task processing node, characterized by Comprise a memory, a processor and a communication component; The memory is used to store one or more computer instructions; The processor is coupled with the memory and the communication component, and is used to execute the one or more computer instructions, so as to: If it is determined that the current task processing node itself is a primary scheduling end of the target task group, a mapping relationship between the current round of tasks in the target task group and a plurality of topic partitions contained in a target message topic configured for the target task group is established; The mapping relationship is transmitted to other task processing nodes in the task processing cluster corresponding to the target task group through the communication component; The task processing cluster is used as a consumption cluster of the target message topic, and the plurality of topic partitions are allocated to each task processing node in the task processing cluster by using the message processing node where the target message topic is located; Based on the mapping relationship, the task allocated to the current task processing node itself is determined as a target task, and the target task is started.

12. A data processing system, characterized by Comprise: A task processing cluster and a message queue cluster, the task processing cluster is a consumption cluster of a target message topic in the message queue cluster, the target message topic contains a plurality of topic partitions, and the target message topic corresponds to a target task group; The primary scheduling end of the target task group in the task processing cluster is used to establish a mapping relationship between the current round of tasks in the target task group and a plurality of topic partitions contained in the target message topic; the mapping relationship is transmitted to other task processing nodes in the task processing cluster corresponding to the target task group; The message processing node where the target message topic in the message queue cluster is located is used to allocate the plurality of topic partitions to each task processing node in the task processing cluster, so that each task processing node determines the current round of tasks mapped by the topic partition allocated to it based on the mapping relationship, as a target task, and starts the target task.

13. A computer readable storage medium storing computer instructions, wherein, When the computer instructions are executed by one or more processors, the one or more processors are caused to perform the task scheduling method of any one of claims 1-10.

Images