A high-efficiency and reliable distributed task scheduling method and system for the signal creation field

Abstract

The application discloses a kind of high-efficiency reliable distributed task scheduling methods and systems for signal creation field.The system includes the cluster of multiple master-slave electors, and the task executor cluster of multiple task executors;Among them, the task executor cluster is used to elect the task executor in the task executor, and the rest of the task executor is elected as a scheduling executor;The scheduling executor is used to obtain the task list from the business system regularly, then carries out task splitting, task issuing, and monitors the state of the working executor;The working executor is used to execute the corresponding business logic processing in the business system according to the received task, and returns the processing result to the scheduling executor.The application has complete guarantee mechanism, high reliability, high-efficiency task fragmentation, high-efficiency task parallel execution, and simple and easy-to-access business system interface.

Description

Technical Field

[0001] The present invention belongs to the technical field of task scheduling in distributed cluster scenarios, and in particular relates to a simple, efficient, and reliable distributed task scheduling method and system. Background Art

[0002] With the increasing demand for distributed solutions, specifically in the information and communication technology (ICT) sector, distributed clusters are required to quickly, efficiently, and reliably process a large number of tasks and enable a variety of business scenarios. Existing scheduling solutions are either complex, expensive, or unreliable. Therefore, it is necessary to develop a simple, efficient, and reliable distributed task scheduling method and system for the ICT sector. Summary of the Invention

[0003] To overcome the shortcomings of existing scheduling methods and systems, such as low reliability, low efficiency, and insufficient accessibility, the present invention provides a simple, efficient, and reliable task scheduling method and further provides a task scheduling system solution that is easily accessible.

[0004] To achieve the above object, the present invention provides the following technical solutions:

[0005] An efficient and reliable distributed task scheduling method for the information technology field, comprising the following steps:

[0006] 1) Set up multiple task executors to form a task executor cluster and set up multiple master-slave election machines;

[0007] 2) Using each of the master-slave electors to elect task executors in the task executor cluster, one task executor is selected as the scheduling executor, and the remaining task executors are selected as work executors; the scheduling executor is used to regularly obtain task lists, split tasks, issue tasks, and monitor the status of work executors; the work executors are used to execute received tasks;

[0008] 3) The scheduling executor obtains the task list from the business system at regular intervals, and then splits the tasks into pieces through a sharding algorithm and sends the tasks to the work executor. The work executor performs the corresponding business logic processing in the business system according to the received tasks, and returns the processing results to the scheduling executor; when the scheduling executor fails, a new scheduling executor is re-elected by the master-slave electors; when the work executor fails, the corresponding strategy is executed according to the task sending method to ensure that the task is not lost: a) If the task sending method is a consistency protocol algorithm, each work executor is provided with a storage area by the master-slave elector. When the work executor fails, the scheduling executor recovers the tasks of the failed work executor from the storage area corresponding to the failed work executor and redistributes the tasks; b) If the task sending method is a message queue, the task is redistributed according to the timeout retry mechanism of the unconsumed queue provided by the message queue.

[0009] Furthermore, the task executor includes a task setting data model, which includes the minimum number of tasks assigned to the work executor, the task slicing method, the number of work executor task execution threads, the master-slave selector access address and the task distribution method.

[0010] Furthermore, the task slicing method includes a horizontal slicing method, a cyclic slicing method and an overall slicing method.

[0011] Furthermore, the horizontal sharding method is: first determine the number of groups according to the number of task executors, then continuously assign tasks to a group according to the number of groups and the minimum number of tasks in each group, and then assign tasks to the next group until the tasks are completed.

[0012] Furthermore, the cyclic slicing method is: firstly, the number of groups is determined according to the number of task executors, and then the tasks are cyclically distributed among the groups according to the number of groups and the minimum number of tasks per group.

[0013] Furthermore, the overall sharding method is: putting all tasks into one group.

[0014] Furthermore, the scheduling executor is provided with an initJob interface and an afterSharding interface; wherein the initJob interface includes an entry parameter nodeNumber for setting the number of job executors; and the afterSharding interface is used to process the business status of successfully sharded tasks.

[0015] Furthermore, a single task processing interface and a batch task interface are provided in the work executor; when the work executor calls the single task processing interface, the task ID is used as the entry parameter of the single task processing interface, and then the multi-thread processing logic is encapsulated according to the number of execution task threads; when the work executor calls the batch task processing interface, the task list is used as the entry parameter of the batch task processing interface, and then the business processing logic is implemented synchronously by single thread or asynchronously by multi-thread.

[0016] An efficient and reliable distributed task scheduling system for the information technology field, characterized by comprising a voter cluster consisting of multiple master-slave voter clusters and a task executor cluster consisting of multiple task executors; wherein,

[0017] The elector cluster is used to elect the task executors in the task executor cluster, and select one task executor as the scheduling executor and the remaining task executors as the working executors;

[0018] The scheduling executor is used to obtain the task list from the business system at regular intervals, then split and distribute the tasks, and monitor the status of the work executor;

[0019] The work executor is used to execute corresponding business logic processing in the business system according to the received task, and return the processing result to the scheduling executor.

[0020] In a first aspect, the present invention proposes a method for dividing basic components of distributed task scheduling, including:

[0021] JobExecutor: A service node for task scheduling and execution. It is divided into a scheduling executor (master node) and a job executor (slave node).

[0022] Schedule Executor: The master node is exclusive and there is only one of them. It is used to obtain the task list regularly, split tasks, issue tasks, monitor the status of work executors, etc. In particular, when there is only one executor, it can be both a schedule executor and a work executor.

[0023] WorkerExecutor: handles specific business, such as parallel computing, image conversion, etc., and can have multiple workers.

[0024] Master-slave elector (Ballot): elects a scheduling executor (master node) for the task executor cluster, and the others are all worker executors (slave nodes). Work executors with the same task name are automatically grouped into a group of executor clusters. The election method is well known and provided by specific electors such as ZooKeeper, ETCD, etc. Zookeepr, abbreviated as zk, is a distributed, open source distributed application coordination service. Etcd: It is a highly available Key / Value storage system, mainly used for shared configuration and service discovery, and is functionally similar to Zookeeper. mq: MQ (MessageQueue) message queue is a "first-in-first-out" data structure in the basic data structure. It is generally used to solve problems such as application decoupling, asynchronous messaging, and traffic peak shaving, to achieve high performance, high availability, scalability, and eventual consistency architecture.

[0025] By clearly dividing the components, it is conducive to system modularization and improves system cohesion.

[0026] In the second aspect, the present invention proposes an abstract design of a task executor object for distributed task scheduling, including a constructor interface and a task setting data model:

[0027] The constructor interface is executed in the task executor when the task executor is initialized:

[0028] public JobExecutor(String taskName,Job job,String cron)

[0029] taskName: parameter name, grouped by task name during selection. job: task setting data model. cron: task scheduling execution cycle Cron expression (i.e., scheduled task expression).

[0030] Task settings (Job) data model:

[0031]

[0032] In a third aspect, the present invention proposes three task slicing methods, including:

[0033] Horizontal sharding: First, determine the number of groups based on the number of task executors. Then, assign tasks to each group sequentially, based on the number of groups and the minimum number of tasks per group. Once the current group is assigned, assign tasks to the next group, and continue until all tasks are assigned.

[0034] For example, the result of splitting the array [0,1,2,3,4,5,6,7,8,9] into two pieces using the horizontal slicing algorithm is:

[0035] [0,1,2,3,4]

[0036] [5,6,7,8,9]

[0037] Round-robin sharding: First, determine the number of groups based on the number of task executors. Then, distribute tasks cyclically among the groups based on the number of groups and the minimum number of tasks per group. After assigning a task to the current group, immediately assign a task to the next unassigned group, and repeat this cycle until all tasks are assigned.

[0038] For example, the result of dividing the array [0,1,2,3,4,5,6,7,8,9] into two pieces using the cyclic slicing algorithm is:

[0039] [0,2,4,6,8]

[0040] [1,3,5,7,9]

[0041] Overall sharding method: put all tasks into one group.

[0042] In a fourth aspect, the present invention proposes four abstract interfaces for custom implementation by business systems, including:

[0043] Task initialization interface:

[0044] abstract List <string>initJob(int nodeNumber)

[0045] nodeNumber: The number of worker executors required for this task

[0046] This interface is implemented by the business system, where the entry parameter nodeNumber is the number of work executors. The business system can implement it according to the specific business execution characteristics.

[0047] For example, for a file conversion task, each worker executor can only perform one file conversion at a time, and the average conversion time is 10 minutes. There are currently two worker executors. During implementation, two files can be returned at a time. After the execution is completed, the task list can be obtained.

[0048] Processing interface after sharding is completed:

[0049] abstract void afterSharding(List <string>successShardingJobList)

[0050] successShardingJobList: Sharded task list

[0051] This interface is implemented by the business system. After sharding is complete, the scheduler executor calls the interface implemented by the business system. The business system can process the task status of "successfully sharded tasks," such as marking them as "status: processing." The business system can also record the execution time of tasks in "processing" to determine whether task processing has timed out.

[0052] Single task processing interface:

[0053] abstract boolean doJobById(String jobId)

[0054] jobId: single task ID

[0055] Single-task processing interface. The task executor takes the task ID as the entry parameter. The task executor encapsulates the multi-threaded processing logic according to the number of threads executing the task. The number of threads can be set to implement parallel processing of tasks, which improves the processing capability and simplifies business implementation.

[0056] Batch task interface:

[0057] abstract boolean doBatchJob(List <string>jobList)

[0058] jobList: Task ID list

[0059] Batch task processing interface, the task executor takes the task list as the input parameter, and the business system implements the business processing logic in synchronous single thread or asynchronous multi-thread according to the actual situation; the number of threads can be set to realize parallel processing of tasks.

[0060] The business system only needs to implement these four interfaces to quickly access the scheduling system.

[0061] In the fifth aspect, the present invention proposes a distributed task scheduling method: after the task executor is built, one and only one task executor is selected as the scheduling executor through the master-slave selector. The scheduling executor executes the business system's customized initJob interface regularly according to the task Cron execution cycle to obtain the task list, and then divides the task into pieces through the sharding algorithm and distributes it to the work executor. The work executor executes the business system's customized task processing interface to complete the logical processing of the task.

[0062] Furthermore, the following safeguard mechanism is proposed:

[0063] 1. When the master node fails, the master-slave elector (Ballot) re-elects a new scheduling executor (master node) from other work executors (slave nodes).

[0064] 2. When the work executor fails, there are corresponding strategies to ensure that the task is not lost according to different task delivery methods. The strategies include:

[0065] ①Message queue: implemented based on the timeout retry mechanism of the unconsumed queue provided by the queue.

[0066] ② Use the consistency protocol algorithm: Each work executor is provided with a node storage area by the elector. When a work executor (slave node) fails, the scheduling executor (master node) will go to the node storage area to recycle the tasks of the failed work executor and redistribute the tasks.

[0067] 3. Business System Compensation: For tasks that remain in the "status: processing" state for an extended period, the business system will reissue them based on the timeout. The business system determines the processing time and, if the timeout has expired, resubmits the task.

[0068] In particular, through the above mechanism, this method can ensure that the task is executed at least once. Therefore, the business system needs to ensure idempotence in the task execution link to ensure that the task is not executed repeatedly.

[0069] The beneficial effects of the present invention are:

[0070] The architecture of the present invention is simple and easy to implement, with a complete guarantee mechanism and high reliability, efficient task slicing, high efficiency of task parallel execution, and a simple and easy-to-access business system interface. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Figure 1 It is an architectural diagram of the present invention.

[0072] Figure 2 It is a timing diagram of the present invention. DETAILED DESCRIPTION

[0073] The present invention is described in further detail below with reference to the accompanying drawings.

[0074] The main process of the present invention is as follows Figure 2 As shown, the steps include:

[0075] 1. Executor construction: Use the executor construction method to set the corresponding parameters and construct the task executor.

[0076] 2. Master Node Election: The task executor registers with the master-slave elector to elect the scheduling executor (master node). Task executors can be manually deployed and started, or dynamically launched with containerized deployment. Based on the "task name" in the constructor, they automatically join the corresponding executor cluster.

[0077] 3. Execution task acquisition: The scheduling executor is configured according to the Cron expression of the scheduling execution cycle, and regularly calls the task system custom task initialization interface initJob to obtain a list of tasks to be executed.

[0078] 4. Task Splitting: The scheduling executor completes task slicing based on the number of work executors and task splitting settings. There are horizontal, circular, and non-slicing options.

[0079] After sharding is completed, the scheduler executor will call the afterSharding interface. The business system can process the business status of the "successfully sharded task", such as marking it as "status: processing".

[0080] 5. Task Scheduling: The scheduler distributes the split tasks to the worker executors. Distribution methods include message queues (such as Kafka and RocketMQ) and consensus protocol algorithms (such as ZooKeeper).

[0081] 6. Task execution: The work executor calls the task execution single task interface doJobById or the batch task interface doBatchJob implemented by the business system to complete business processing.

[0082] Experimental data:

[0083] Horizontal task splitting algorithm:

[0084] 10,000 task data, divided into 30 parts, executed 1 million times, taking 5.5 seconds

[0085] 100,000 task data, divided into 300 parts, executed 100,000 times, taking 5.7 seconds

[0086] Cyclic task splitting algorithm:

[0087] 10,000 task data, divided into 30 parts, executed 1 million times, taking 20 seconds

[0088] 100,000 task data, divided into 300 parts, executed 100,000 times, taking 20 seconds

[0089] Other implementations:

[0090] Other implementations may change the master-slave election component to ETCD, and the processing logic of the master node election may have slight differences depending on business needs.

[0091] The specific embodiments and drawings of the present invention disclosed above are intended to facilitate understanding of the present invention and its implementation. Those skilled in the art will appreciate that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the embodiments and drawings disclosed in this specification. The scope of protection of the present invention shall be determined by the scope defined in the claims.< / string> < / string> < / string>

Claims

1. An efficient and reliable distributed task scheduling method for the information technology field, comprising the following steps: 1) Set up multiple task executors to form a task executor cluster and set up multiple master-slave election machines; 2) using each of the master-slave electors to elect task executors in the task executor cluster, and electing one task executor as the scheduling executor, and the remaining task executors as the working executors; The scheduling executor is used to regularly obtain task lists, split tasks, issue tasks, and monitor the status of work executors; the work executor is used to execute received tasks; the task executor includes a task setting data model, which includes the minimum number of tasks assigned to the work executor, the task sharding method, the number of threads for executing tasks on the work executor, the access address of the master-slave selector, and the task issuance method; a) The horizontal sharding method is: first, determine the number of groups based on the number of task executors, and then assign tasks to a group continuously according to the number of groups and the minimum number of tasks in each group, and then assign tasks to the next group until all tasks are assigned; b) The cyclic sharding method is: First, the number of groups is determined based on the number of task executors, and then tasks are cyclically distributed among the groups according to the number of groups and the minimum number of tasks per group. c) The overall sharding method is: all tasks are placed in one group; 3) The scheduling executor periodically obtains the task list from the business system, and then splits the task into pieces using a sharding algorithm and sends the task to the work executor. The work executor performs the corresponding business logic processing in the business system according to the received task and returns the processing result to the scheduling executor; When the scheduling executor fails, a new scheduling executor is re-elected through the master-slave electors; When a work executor fails, the corresponding strategy is executed according to the task delivery method to ensure that the task is not lost: a) If the task delivery method is the consistency protocol algorithm, each work executor is provided with a storage area by the master-slave elector. When the work executor fails, the scheduling executor recovers the tasks of the failed work executor from the storage area corresponding to the failed work executor and redistributes the tasks; b) If the task delivery method is the message queue, the task is redistributed according to the timeout retry mechanism of the unconsumed queue provided by the message queue.

2. The method according to claim 1, wherein The scheduling executor is provided with an initJob interface and an afterSharding interface; wherein the initJob interface includes an entry parameter nodeNumber for setting the number of work executors; the afterSharding interface is used to process the business status of the successfully sharded tasks.

3. The method according to claim 1, wherein The work executor is provided with a single task processing interface and a batch task interface; when the work executor calls the single task processing interface, the task ID is used as the entry parameter of the single task processing interface, and then the multi-thread processing logic is encapsulated according to the number of execution task threads; when the work executor calls the batch task processing interface, the task list is used as the entry parameter of the batch task processing interface, and then the business processing logic is implemented synchronously in a single thread or asynchronously in a multi-threaded manner.

4. An efficient and reliable distributed task scheduling system for the information technology field, characterized by: It includes a voter cluster consisting of multiple master-slave voter clusters and a task executor cluster consisting of multiple task executors; The elector cluster is used to elect the task executors in the task executor cluster, and select one task executor as the scheduling executor and the remaining task executors as the working executors; The scheduling executor is used to obtain the task list from the business system at regular intervals, then split and distribute the tasks, and monitor the status of the work executor; The work executor is used to execute the corresponding business logic processing in the business system according to the received task, and return the processing result to the scheduling executor; The task executor includes a task setting data model, which includes the minimum number of tasks assigned to the work executor, the task slicing method, the number of threads for executing tasks on the work executor, the access address of the master-slave selector, and the task dispatching method. a) The horizontal slicing method is: first determine the number of groups based on the number of task executors, then continuously assign tasks to a group based on the number of groups and the minimum number of tasks per group, and then assign tasks to the next group until all tasks are assigned. b) The cyclic slicing method is: first determine the number of groups based on the number of task executors, then cyclically assign tasks between groups based on the number of groups and the minimum number of tasks per group. c) The overall slicing method is: place all tasks in one group. When the scheduling executor fails, a new scheduling executor is re-elected through the master-slave elector; when the work executor fails, the corresponding strategy is executed according to the task distribution method to ensure that the task is not lost: a) If the task distribution method is the consistency protocol algorithm, each work executor is provided with a storage area by the master-slave elector. When the work executor fails, the scheduling executor recovers the tasks of the faulty work executor from the storage area corresponding to the faulty work executor and redistributes the tasks; b) If the task distribution method is the message queue, the task is redistributed according to the timeout retry mechanism of the unconsumed queue provided by the message queue.

Images