Method for executing distributed tasks and related apparatuses and devices

Abstract

The application discloses a distributed task execution method and related devices and equipment. The distributed task execution method comprises the following steps: in response to the starting of a timer, starting a new scheduler, and determining whether a current task is executed by an original scheduler through the new scheduler; if the current task is not executed by the original scheduler, the current task is scheduled by the new scheduler and the original scheduler; if the current task is executed by the original scheduler, a new task is determined from a plurality of distributed tasks by the original scheduler using a distributed lock, and the new task is executed by an executor. The above scheme can improve the execution efficiency of the distributed task.

Description

Technical Field

[0001] This application relates to the technical field of distributed tasks, and in particular to methods for executing distributed tasks and related apparatus and equipment. Background Technology

[0002] A distributed task can consist of multiple independent or interdependent subtasks. The completion status of a distributed task depends entirely on the completion status of its subtasks. A distributed task is considered complete only when all its subtasks are completed.

[0003] In distributed tasks, most tasks are executed in a centralized manner. That is, distributed tasks are scheduled through a centralized scheduling platform and are generally executed by running scripts.

[0004] The above solution cannot be combined with distributed services, resulting in a waste of resources. At the same time, it creates a complex structure of multiple deployment schemes for the service, which reduces the execution efficiency of distributed tasks. Summary of the Invention

[0005] This application provides a method for executing distributed tasks, as well as related apparatus and equipment, to solve the problem of low execution efficiency of distributed tasks in the prior art.

[0006] This application provides a method for executing distributed tasks, which includes: in response to the start of a timer, starting a new scheduler and determining whether the current task has been completed by the original scheduler through the new scheduler; if the current task has not been completed by the original scheduler, scheduling the current task together with the original scheduler through the new scheduler; if the current task has been completed by the original scheduler, determining a new task from multiple distributed tasks through the original scheduler using a distributed lock, and executing the new task through an executor.

[0007] The step of scheduling the current task together with the original scheduler through the new scheduler also includes: in response to the number of schedulers exceeding the preset number, adding a new container service.

[0008] The distributed task execution method also includes: in response to the completion of the current task before the preset time has been reached, shutting down one scheduler and determining whether the number of remaining schedulers exceeds the preset number; if the number of remaining schedulers does not exceed the preset number, then reducing one container service.

[0009] The steps of scheduling the current task together with the new scheduler and the original scheduler include: scheduling the corresponding executors to execute the current task by the new scheduler and the original scheduler respectively.

[0010] The steps of starting a new scheduler in response to the timer's start, and determining whether the current task has been completed by the original scheduler through the new scheduler, include: starting a new scheduler in response to the timer's start, and determining whether there are any subtasks to be executed in the task queue of the current task based on the scheduling rules; if there are no subtasks to be executed in the task queue of the current task, it is determined that the current task has been completed by the original scheduler; if there are subtasks to be executed in the task queue of the current task, it is determined that the current task has not been completed by the original scheduler.

[0011] The steps for determining a new task from multiple distributed tasks using a distributed lock via the original scheduler include: determining the task with creation permission from multiple distributed tasks using a Redis distributed lock via the original scheduler; and identifying the task with creation permission as the new task.

[0012] The steps of determining a new task from multiple distributed tasks using a distributed lock through the original scheduler and executing the new task through the executor also include: writing each subtask corresponding to the new task into the task queue; retrieving each subtask from the task queue through the original scheduler and calling the executor corresponding to the original scheduler to execute each subtask until there are no subtasks to be executed in the task queue.

[0013] The steps of writing each subtask corresponding to the new task into the task queue include: initializing the new task according to preset rules to obtain each subtask corresponding to the new task; writing each subtask into the task queue of the Redis distributed lock's list structure; and calling the Event to execute the task start event in response to all subtasks being written into the task queue.

[0014] The steps of retrieving each subtask from the task queue through the original scheduler and calling the corresponding executor of the original scheduler to execute each subtask until there are no more subtasks to be executed in the task queue include: retrieving each subtask from the task queue sequentially through the original scheduler using the FIFO rule, calling the executor to execute each subtask, and calling the Event to execute the task execution event.

[0015] The process of retrieving subtasks from the task queue using the original scheduler and executing them using the corresponding executor until no more subtasks remain in the queue includes: in response to the original scheduler retrieving a subtask from the task queue, using the Redis distributed lock llen to determine if any subtasks still exist in the queue; if so, the process continues until no more subtasks remain; if no subtasks remain, after executing the currently retrieved subtask, the task completion event is invoked using the Event function.

[0016] This application also provides a distributed task execution device, comprising: a judgment module, configured to start a new scheduler in response to the start of a timer, and determine whether the current task has been completed by the original scheduler through the new scheduler; a scheduling module, configured to schedule the current task jointly by the new scheduler and the original scheduler if the current task has not been completed by the original scheduler; and an execution module, configured to determine a new task from multiple distributed tasks through the original scheduler using a distributed lock, and execute the new task through the executor if the current task has been completed by the original scheduler.

[0017] This application also provides an electronic device, including a memory and a processor coupled to each other, wherein the processor is used to execute program instructions stored in the memory to implement the above-described method for executing any of the distributed tasks.

[0018] This application also provides a computer-readable storage medium storing program instructions thereon, which, when executed by a processor, implement the execution method of any of the above-described distributed tasks.

[0019] In the above-described scheme, when the timer starts, a new scheduler is launched, and the new scheduler determines whether the current task has been completed by the original scheduler. If the current task has not been completed by the original scheduler, the new scheduler and the original scheduler jointly schedule the current task, which can improve the execution efficiency and speed of the current task by adding a scheduler. If the current task has been completed by the original scheduler, the original scheduler uses a distributed lock to determine a new task from multiple distributed tasks, and the executor executes the new task. Thus, the creation and execution of new tasks can be determined by using a distributed lock. The creation and execution of tasks do not depend on any distributed microservice architecture, making the creation and execution of tasks simpler, thereby further improving the execution efficiency of distributed tasks. Attached Figure Description

[0020] Figure 1This is a flowchart illustrating an embodiment of the distributed task execution method of this application;

[0021] Figure 2 This is a flowchart illustrating another embodiment of the distributed task execution method of this application;

[0022] Figure 3 yes Figure 2 A schematic diagram of the framework of one implementation of the distributed system in the example;

[0023] Figure 4 This is a schematic diagram of the framework of an embodiment of the distributed task execution device of this application;

[0024] Figure 5 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0025] Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0026] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0027] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0028] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes a relationship between related objects, which can exist in three ways. For example, A and / or B can exist in three ways: A alone, A and B simultaneously, or B alone. Additionally, the character " / " generally indicates an "or" relationship between the objects before and after it. Furthermore, "more than" in this paper refers to two or more objects.

[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the distributed task execution method of this application.

[0030] Step S11: In response to the timer starting, start a new scheduler and use the new scheduler to determine whether the current task has been completed by the original scheduler.

[0031] In response to the start of a timer, a new dispatcher is launched, and this new dispatcher determines whether the current task has been completed by the original dispatcher. The current task is the distributed task currently being executed by the original dispatcher; at any given time, only one distributed task can be executed. The number of original dispatchers executing the current task can be at least one, specifically one, two, five, etc., and is not limited here.

[0032] The Scheduler can be configured with its own start cycle or specific start time. The scheduler will cycle through the scheduler's start cycles or specific start times. For example, a scheduler can be configured to start every 10 seconds, every 20 seconds, or at 9:00 AM on August 18th. Specific settings can be configured based on actual needs and are not limited here.

[0033] The timer in this embodiment can be Spring's @Scheduled timer, a Java timer, or a timer using other technologies.

[0034] The dispatcher can include task scheduling rules, such as task initialization methods, task content, the number of subtasks corresponding to the task, and task execution methods. It can determine the scheduling of tasks, task queues, executors, and events.

[0035] Step S12: If the current task has not been completed by the original scheduler, the current task will be scheduled jointly by the new scheduler and the original scheduler.

[0036] If the new scheduler determines that the current task has not been completed by the original scheduler, the new scheduler and the original scheduler corresponding to the current task will jointly schedule the current task. This ensures that if the current task has not been completed, a new scheduler will be added to schedule it.

[0037] In a specific application scenario, when a new scheduler and the original scheduler jointly schedule the current task, the current task can be divided into multiple subtasks by modulo partitioning. These subtasks are then assigned to the new scheduler and the original scheduler according to the FIFO rule. This allows the new scheduler and the original scheduler to schedule the corresponding executors to execute the assigned subtasks, thus jointly scheduling the current task.

[0038] In a specific application scenario, when a new scheduler and the original scheduler jointly schedule the current task, the current task can be divided into multiple subtasks based on the key data in the corresponding program data. These subtasks are then assigned to the new and original schedulers according to a FIFO (First-In, First-Out) rule, allowing each scheduler to execute its assigned subtask using its corresponding executor, thus jointly scheduling the current task. The key data in the program data can be the letter "a", the punctuation mark ";", or the word "in," etc. The specific key data can be set based on the actual situation and is not limited here.

[0039] In a specific application scenario, when a new scheduler and the original scheduler jointly schedule the current task, the new scheduler and the original scheduler can each repeatedly execute the current task, and the scheduler with the fastest execution speed is selected as the current task's scheduler. After selection, other schedulers are stopped. Thus, by selecting the scheduler with the highest execution efficiency from among multiple schedulers, the execution efficiency of the task can be improved.

[0040] In one specific application scenario, when a task is currently being executed by one scheduler, and a timer starts, a new scheduler starts and determines that the task is not yet complete. In this case, the new scheduler and the original scheduler jointly schedule the task; that is, two schedulers are used to schedule the task. In another specific application scenario, when a task is currently being executed by two schedulers, and a timer starts, a new scheduler starts and determines that the task is not yet complete. In this case, the new scheduler and the original two schedulers jointly schedule the task; that is, three schedulers are used to schedule the task.

[0041] In a specific application scenario, when the scheduler schedules the current task, it can obtain the current task through the scheduler and then send it to the executor so that the current task can be scheduled by the executor executing the current task.

[0042] In a specific application scenario, when a new scheduler is added to schedule the current task, the executors required during the scheduling process can also be added at the same time. This allows the original executors and the newly added executors to jointly execute the current task based on the scheduling of their respective schedulers, thereby improving the execution efficiency and speed of the current task.

[0043] In a specific application scenario, when a new scheduler is added to schedule the current task, the execution efficiency and speed of the current task can be improved by adding a new set of scheduling logic to schedule the current task together with the original scheduler's scheduling logic.

[0044] Step S13: If the current task has been completed by the original scheduler, a new task is determined from multiple distributed tasks by the original scheduler using a distributed lock, and the new task is executed by the executor.

[0045] When the timer starts, a new scheduler starts. Once the new scheduler determines that the current task has been completed, it terminates. Then, using a distributed lock, it uses the original scheduler corresponding to the completed current task to determine a new task from among multiple distributed tasks, and executes the new task through the executor corresponding to the original scheduler. The distributed lock is a way to control synchronized access to shared resources between distributed systems, preventing interference and ensuring consistency. It is used to ensure that only one task can be executed at a time in a distributed application cluster.

[0046] In this way, when the original scheduler uses distributed locks to determine a new task from multiple distributed tasks, it can avoid setting up a centralized scheduling platform to determine the new task. This makes the creation and execution of tasks independent of any distributed microservice architecture, making the creation and execution of tasks simpler.

[0047] Through the above method, the distributed task execution method of this embodiment starts a new scheduler when the timer starts. The new scheduler determines whether the current task has been completed by the original scheduler. If the current task has not been completed by the original scheduler, the new scheduler and the original scheduler jointly schedule the current task. The addition of a scheduler can improve the execution efficiency and speed of the current task. If the current task has been completed by the original scheduler, the original scheduler uses a distributed lock to determine a new task from multiple distributed tasks and executes the new task through an executor. The creation and execution of the new task can be determined by using a distributed lock. The creation and execution of the task do not depend on any distributed microservice architecture, making the creation and execution of the task simpler and further improving the execution efficiency of the distributed task.

[0048] Please see Figure 2-3 , Figure 2 This is a flowchart illustrating another embodiment of the distributed task execution method of this application. Figure 3 yes Figure 2 The example illustrates the framework of one implementation of the distributed system. In this embodiment, the distributed system is described using two schedulers and their corresponding executors. When there are multiple schedulers or only one scheduler, the framework is similar to this embodiment and will not be repeated here.

[0049] The distributed system 30 in this embodiment is used to implement the distributed task execution method of any embodiment, including a timer 31, an executor 32, a dispatcher 36, an event 35, a task queue 33, and a task 34.

[0050] The timer 31 can be configured with a timed start cycle or a specific time for the scheduler 36. The timer 31 starts cyclically based on the aforementioned timed rules to activate the scheduler 36. The scheduler 36 is configured with scheduling rules for the task 34, such as the task initialization method, task content, the number of subtasks corresponding to the task, and the task execution method. The scheduler 36 includes the original scheduler 361 and the new scheduler 362. The original scheduler 361 and the new scheduler 362 can each be independently configured with all the scheduling rules of the scheduler 36, enabling all the functions of the scheduler 36. Task 34, each task 34 corresponds to a task queue 33; the task queue 33 can include the specific content of the corresponding task 34, and each value of the task queue 33 is a subtask of task 34; executor 32 includes the original executor 321 and the new executor 322, where the original executor 321 corresponds to the original scheduler 361, and the new executor 322 corresponds to the new scheduler 362. Executor 32 is used to specifically execute the subtasks corresponding to task 34; event 35 is used for events such as task 34 creation, task 34 execution, and task 34 end.

[0051] Specifically, timer 31 is used to start and launch scheduler 36 based on timing rules. Scheduler 36 is used to initialize task queue 33, retrieve task 34, and retrieve subtasks in task queue 33. Task 34 is used to generate task queue 33 and its subtasks. Scheduler 36 is also used to receive the start information of timer 31 and notify executor 32 to execute task 34 by executing subtasks in task queue 33. Scheduler 36 is also used to notify event 35 of task execution event.

[0052] The specific execution steps of distributed system 30 are as follows:

[0053] Step S21: In response to the start of the timer, start a new scheduler and use the new scheduler to determine whether the current task has been completed by the original scheduler.

[0054] Timer 31 starts cyclically according to the configured timing rules. When Timer 31 starts, a new scheduler 362 is started, and the new scheduler 362 executes the corresponding scheduling rules. The scheduling rules include the logic for task creation and execution.

[0055] In a specific application scenario, in response to the start of timer 31, a new scheduler 362 is launched. The new scheduler 362 can determine whether the current task has been completed by the original scheduler 361 based on the scheduling rules. The current task is the task currently being executed by the distributed system.

[0056] In a specific application scenario, in response to the start of timer 31, a new scheduler 362 is started. Based on the scheduling rules, the new scheduler 362 can determine whether there are any subtasks that need to be executed in the task queue 33 of the current task 34. If there are no subtasks that need to be executed in the task queue 33 of the current task 34, it is determined that the current task 34 has been completed by the original scheduler 361. If there are subtasks that need to be executed in the task queue 33 of the current task 34, it is determined that the current task 34 has not been completed by the original scheduler 361.

[0057] Step S22: If the current task has not been completed by the original scheduler, the current task will be scheduled together with the original scheduler through a new scheduler. If the number of schedulers exceeds the preset number, a new container service will be added.

[0058] When the new scheduler 362 determines that the current task 34 has not been completed by the original scheduler 361, the new scheduler 362 and the original scheduler 361 jointly schedule the current task 34. In a specific application scenario, when the new scheduler 362 and the original scheduler 361 jointly schedule the current task 34, a new executor 322 corresponding to the new scheduler 362 can be added simultaneously. This allows the original executor 321 and the new executor 322 to jointly execute the current task 34, thereby improving the execution efficiency and speed of the current task 34.

[0059] In a specific application scenario, when an original scheduler 361 is currently scheduling an original executor 321 to execute the current task 34, and timer 31 starts, a new scheduler 362 starts and determines that the current task 34 has not been completed by the original scheduler 361. Then, the new scheduler 362 and the original scheduler 361 jointly schedule the current task 34. In other words, at this time, the new scheduler 362 and the original scheduler 361 respectively schedule the new executor 322 and the original executor 321 to jointly execute the current task 34.

[0060] When multiple schedulers simultaneously schedule the current task, each scheduler can independently schedule its corresponding executor to execute the task. In a specific application scenario, each scheduler can retrieve subtasks of the current task from the task queue and schedule the corresponding executor to execute them. The allocation rules for each scheduler retrieving subtasks from the task queue can be based on the execution status of the executor corresponding to each scheduler, the space required for subtask execution, and / or the order of the subtasks. For specific subtask allocation rules, please refer to step S25.

[0061] In response to the addition of scheduler 36, it is determined whether the number of schedulers 36 exceeds a preset limit. If the number of schedulers 36 exceeds the preset limit, a new container service is added. The preset limit can include 3, 5, 6, etc., and can be set based on actual needs; no specific limitation is made here. The container service can be used to host schedulers 36. When the number of schedulers 36 on a single container service exceeds the preset limit, a new container service is added to increase the startup limit of schedulers 36, thereby further improving task execution efficiency by increasing the number of schedulers 36.

[0062] Step S23: In response to the completion of the current task before the preset time has been reached, shut down one scheduler and determine whether the number of remaining schedulers exceeds the preset number. If the number of remaining schedulers does not exceed the preset number, reduce one container service.

[0063] When the current task is completed and the preset time has not been reached, one scheduler is shut down, and it is checked whether the number of remaining schedulers exceeds the preset number. If the number of schedulers does not exceed the preset number, one container service is reduced to avoid resource waste. The preset time can include (3 / 4, (n-1) / n) minutes, or other time periods, and can be set based on actual conditions; no specific limitation is made here. Here, n is any positive integer.

[0064] In a specific application scenario, when the executor 32 finishes executing the subtask currently retrieved from the task queue 33, the scheduler 36 can determine whether the current task 34 has been completed at a preset frequency. If the current task 34 has been completed, it is determined whether the interval between the completion time of the current task 34 and the current time is less than a preset time. If it is less, one scheduler 36 is shut down. The preset frequency can be once every 1 second, once every 3 seconds, or once every 6 seconds, etc., and can be set according to the actual situation, without limitation here.

[0065] In a specific application scenario, it can be determined whether the interval between the completion time of the current task 34 and the current time is less than 3 / 4 to (n-1) / n of the start time interval of the timer 31. If it is less, a scheduler 36 is shut down. The scheduler 36 to be shut down can be selected according to the start time of each scheduler 36. For example, the scheduler 36 with the longest start time is shut down.

[0066] Step S24: If the current task has been completed, the task with creation permission is determined from multiple distributed tasks by using the Redis distributed lock through the original scheduler, and the task with creation permission is determined as the new task.

[0067] When the new scheduler 362 determines that the current task 34 has been completed, the new scheduler 362 ends and uses the Redis distributed lock from multiple distributed tasks through the original scheduler 361 to determine the task with creation permission, and the task with creation permission is determined as the new task.

[0068] In a specific application scenario, when the new scheduler 362 determines that there are no subtasks to be executed in the task queue 33 of the current task 34, it can further check whether there are new tasks to be created and executed. If no new task has reached its creation time, the new scheduler 362 and timer 31 will end the current startup. If at least one new task has reached its creation time, the new scheduler 362 and the original scheduler 361 can jointly create and schedule the new task.

[0069] Specifically, when creating a new task, multiple distributed tasks request the permission to create the task. The task that obtains the creation permission will be created or executed. In a specific application scenario, multiple distributed tasks can compete for and acquire the lock using the `set` or `setnx` methods, or they can use the locking functionality provided by third-party plugins such as Jedis, Redisson, and Lettuce to compete for and acquire the Redis distributed lock. When a distributed task wins the Redis distributed lock, it acquires the creation permission, and that distributed task becomes the new task that needs to be created.

[0070] Distributed tasks that fail to acquire the lock wait for timer 31 to restart, in order to repeatedly execute step S21.

[0071] Step S25: Write each subtask corresponding to the new task into the task queue, retrieve each subtask from the task queue through the original scheduler, and call the executor corresponding to the original scheduler to execute each subtask until there are no subtasks to be executed in the task queue.

[0072] Once a new task is determined, its corresponding subtasks can be initialized using preset rules. Initialization involves dividing the new task into its multiple subtasks. Preset rules can include modulo-based task partitioning or using fixed subtasks (1=c, 2=b, 3=e), etc., but are not specifically limited here.

[0073] In a specific application scenario, the fixed subtask method can include dividing the current task into multiple subtasks based on the key data in the corresponding program data. In another specific application scenario, the modulo-based task partitioning method can include taking the modulo of the corresponding program data. When the modulo value is a fixed value or follows a certain pattern, the corresponding position of the modulo value in the program data is used to partition the current task into multiple subtasks. This pattern of modulo values ​​can include "1, 2, 3, 4, 5...9" or other patterns.

[0074] After initialization, each subtask is written to the task queue 33 of the Redis distributed lock's list structure. Here, list is a storage structure using Redis technology. In response to all subtasks being written to task queue 33, the scheduler 36 invokes event 35 to execute the task start event. Subtasks are pushed onto the stack sequentially using Redis distributed lock's Rpush method, and popped from the stack using Redis distributed lock's Lpop method. The Redis distributed lock's Rpush command is used to insert one or more subtasks into the tail of task queue 33. The Redis distributed lock's Lpop command is used to retrieve and return the first subtask from task queue 33.

[0075] The scheduler 36 retrieves subtasks from the task queue 33 and schedules the executor 32 to execute them until there are no more subtasks to be executed in the task queue 33. In a specific application scenario, the scheduler 36 can retrieve subtasks from the task queue 33 based on a FIFO rule, the executor 32 can execute the subtasks, and event 35 can be invoked to execute the task execution event. The FIFO rule is First In First Out.

[0076] In a specific application scenario, when the new scheduler 362 and the original scheduler 361 jointly schedule the current task, the task queue 33 has already been written with the subtasks corresponding to the current task. At this time, the task queue 33 allocates the subtasks to the new scheduler 362 or the original scheduler 361 that comes to retrieve them according to the FIFO rule. The timing of the new scheduler 362 or the original scheduler 361 retrieving the subtasks from the task queue 33 depends on the time when the new scheduler 362 or the original scheduler 361 executed the previous subtask and the start time.

[0077] Specifically, when the scheduler starts, it retrieves a subtask from the task queue. The task queue assigns a subtask to the scheduler according to the FIFO rule. The scheduler retrieves and schedules the executor to execute the subtask. After the subtask is completed, the scheduler retrieves another subtask from the task queue, and so on. Each scheduler retrieves a subtask from the task queue according to the time it took to execute the previous subtask and its start time.

[0078] In a specific application scenario, when there are subtasks that need to be executed in task queue 33, each scheduler obtains the subtasks in task queue 33 through the Lpop method of Redis distributed lock. If no subtasks can be obtained, the scheduler checks whether there are any new tasks to be created and executed, i.e., step S24.

[0079] During the execution of a subtask, in response to obtaining a subtask from task queue 33, the Llen function of the Redis distributed lock is used to determine whether there is a subtask to be executed in task queue 33. If there is a subtask to be executed in task queue 33, the process continues by having executor 32 obtain and execute each subtask from task queue 33 in turn until there are no subtasks to be executed in task queue 33. If there are no subtasks to be executed in task queue 33, after executing the currently obtained subtask, event 35 is called to execute the task end event.

[0080] The Redis `Llen` command returns the length of task queue 33. If the list key does not exist, it is interpreted as an empty list, returning 0. In a specific application scenario, in response to retrieving a subtask from task queue 33, the Redis distributed lock's `Llen` command checks if a subtask exists in task queue 33 that needs to be executed. If `Llen` returns 0, it means that the subtask retrieved is the last one in task queue 33 that needs to be executed. When this subtask completes, event 35 (Tasker termination event) is invoked. If the value returned by `Llen` is greater than 0, the subtask is retrieved from task queue 33 using Redis's `Lpop` method and then executed by executor 32.

[0081] Once all subtasks have been completed, it means that the new task has been completed. Then, wait for the timer to start again to repeat step S21.

[0082] Through the above steps, the distributed task execution method of this embodiment, by scheduling the current task together with the original scheduler when the timer starts but the current task has not been completed, can improve the execution efficiency and speed of the current task by adding a new scheduler. Furthermore, when the number of schedulers exceeds a preset number, container services are added to increase the scheduler startup limit, thereby further improving the execution efficiency of each executor. When the current task is completed and before the timer restart time is reached, one scheduler is shut down. And when the number of schedulers does not exceed a preset number, container services are reduced. This allows for scaling based on the task execution status, strengthening the execution capabilities of the executors, accelerating task execution speed, and avoiding situations where task execution efficiency is low and execution time is long during peak periods, as well as resource waste. Moreover, by using distributed locks to determine the creation and execution of new tasks, the creation and execution of tasks do not depend on any distributed microservice architecture, making task creation and execution simpler and further improving the execution efficiency of distributed tasks. This embodiment also reduces the waste of service deployment resources by avoiding the use of scripts to execute distributed tasks.

[0083] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the distributed task execution device of this application. The distributed task execution device 40 includes a judgment module 41, a scheduling module 42, and an execution module 43. The judgment module 41 is used to start a new scheduler in response to the start of a timer, and to determine whether the current task has been completed by the original scheduler through the new scheduler; the scheduling module 42 is used to schedule the current task together with the original scheduler if the current task has not been completed by the original scheduler; the execution module 43 is used to determine a new task from multiple distributed tasks through the original scheduler using a distributed lock if the current task has been completed by the original scheduler, and to execute the new task through the executor.

[0084] The scheduling module 42 is also used to add a new container service in response to the number of schedulers exceeding a preset number.

[0085] The scheduling module 42 is also used to shut down a scheduler in response to the completion of the current task before the preset time has been reached, and to determine whether the number of remaining schedulers exceeds the preset number; if the number of remaining schedulers does not exceed the preset number, then reduce one container service.

[0086] The scheduling module 42 is also used to schedule the corresponding executors to execute the current task through the new scheduler and the original scheduler respectively.

[0087] The judgment module 41 is also used to start a new scheduler in response to the start of the timer, and to determine whether there are subtasks that need to be executed in the task queue of the current task based on the scheduling rules; if there are no subtasks that need to be executed in the task queue of the current task, it is determined that the current task has been completed by the original scheduler; if there are subtasks that need to be executed in the task queue of the current task, it is determined that the current task has not been completed by the original scheduler.

[0088] Execution module 43 is also used to determine the task with creation permission from multiple distributed tasks by using Redis distributed locks through the original scheduler; and to determine the task with creation permission as the new task.

[0089] The execution module 43 is also used to write each subtask corresponding to the new task into the task queue; retrieve each subtask from the task queue through the original scheduler, and call the executor corresponding to the original scheduler to execute each subtask until there are no subtasks to be executed in the task queue.

[0090] Execution module 43 is also used to initialize the new task according to preset rules, obtain the subtasks corresponding to the new task; write each subtask into the task queue of the Redis distributed lock list structure; and in response to all subtasks being written into the task queue, call the Event to execute the task start event.

[0091] The execution module 43 is also used to obtain each subtask from the task queue in sequence using the FIFO rule through the original scheduler, call the executor to execute each subtask, and call the Event to execute the task execution event.

[0092] The execution module 43 is also used to respond to the original scheduler obtaining a subtask from the task queue, and to determine whether there are still subtasks to be executed in the task queue through the Redis distributed lock llen; if there are still subtasks to be executed in the task queue, then continue to obtain each subtask from the task queue through the original scheduler and call the executor corresponding to the original scheduler to execute each subtask, until there are no subtasks to be executed in the task queue; if there are no subtasks to be executed in the task queue, then after executing the currently obtained subtask, call the Event to execute the task end event.

[0093] The above solution can improve the execution efficiency of distributed tasks.

[0094] Please see Figure 5 , Figure 5This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 50 includes a memory 51 and a processor 52 coupled to each other. The processor 52 is used to execute program instructions stored in the memory 51 to implement the steps of the distributed task execution method of any of the above embodiments. In a specific implementation scenario, the electronic device 50 may include, but is not limited to, a microcomputer, a server, etc. In addition, the electronic device 50 may also include mobile devices such as laptops and tablets, which are not limited here.

[0095] Specifically, processor 52 controls itself and memory 51 to implement the steps of any of the above-described distributed task execution method embodiments. Processor 52 may also be referred to as a CPU (Central Processing Unit). Processor 52 may be an integrated circuit chip with signal processing capabilities. Processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 52 may be implemented using integrated circuit chips.

[0096] The above solution can improve the execution efficiency of distributed tasks.

[0097] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 60 stores program instructions 601 that can be executed by a processor. The program instructions 601 are used to implement the steps of the distributed task execution method of any of the above embodiments.

[0098] The above solution can improve the execution efficiency of distributed tasks.

[0099] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection of devices or units may be electrical, mechanical, or other forms.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0101] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0102] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A method for distributed execution of tasks, characterized in that, The distributed task execution method comprises: in response to the timer starting, starting a new scheduler, and determining whether the current task is executed by the original scheduler through the new scheduler; wherein the timer is configured to start periodically or at a specific time; if the current task is not executed by the original scheduler, the current task is scheduled by the new scheduler and the original scheduler; wherein, in response to the number of schedulers exceeding a preset number, a container service is added; if the current task is executed by the original scheduler, a new task is determined from a plurality of distributed tasks by the original scheduler using a distributed lock, and the new task is executed by the new scheduler or the original scheduler using a corresponding executor; wherein, it is determined whether a new task is to be created and executed, if there is no new task to arrive at its creation time, the new scheduler and the timer end this start, if there is at least one new task to arrive at its creation time, the new scheduler and the original scheduler can create and schedule the new task together; in response to the current task being executed and the preset time not being reached, a scheduler is closed, and if it is determined that the number of remaining schedulers does not exceed the preset number, a container service is reduced; after all subtasks are executed, waiting for the timer to start again to cycle the steps of starting the timer and starting the new scheduler.

2. The method of claim 1, wherein, The step of scheduling the current task by the new scheduler and the original scheduler together comprises: the new scheduler and the original scheduler schedule corresponding executors to execute the current task respectively.

3. The method of claim 1, wherein, The step of starting a new scheduler in response to the timer starting, and determining whether the current task is executed by the original scheduler through the new scheduler comprises: in response to the timer starting, starting the new scheduler, and determining whether there is a subtask to be executed in the task queue of the current task through the new scheduler based on a scheduling rule; if there is no subtask to be executed in the task queue of the current task, it is determined that the current task is executed by the original scheduler; if there is a subtask to be executed in the task queue of the current task, it is determined that the current task is not executed by the original scheduler.

4. The method of claim 1, wherein, The step of determining a new task from a plurality of distributed tasks by the original scheduler using a distributed lock comprises: determining a task with creation authority from a plurality of distributed tasks by the original scheduler using a Redis distributed lock; the task with creation authority is determined as the new task.

5. The method of claim 1, wherein, The step of determining a new task from a plurality of distributed tasks by the original scheduler using a distributed lock, and executing the new task by an executor further comprises: writing each subtask corresponding to the new task into a task queue; each subtask is obtained from the task queue by the original scheduler, and each subtask is executed by the corresponding executor of the original scheduler until there is no subtask to be executed in the task queue.

6. The method of claim 5, wherein, The step of writing the sub-tasks corresponding to the new task into the task queue comprises: initializing the new task by a preset rule to obtain the sub-tasks corresponding to the new task; writing the sub-tasks into the task queue in the list structure of the Redis distributed lock; in response to all the sub-tasks being written into the task queue, calling an Event to execute a task start event.

7. The method of claim 5, wherein, The step of obtaining the sub-tasks from the task queue by the original scheduler and executing the sub-tasks by the executor corresponding to the original scheduler until there is no sub-task to be executed in the task queue comprises: obtaining the sub-tasks from the task queue by the original scheduler in turn by the FIFO rule, executing the sub-tasks by the executor, and calling an Event to execute a task execution event.

8. The method of claim 5, wherein, The step of obtaining the sub-tasks from the task queue by the original scheduler and executing the sub-tasks by the executor corresponding to the original scheduler until there is no sub-task to be executed in the task queue further comprises: in response to the original scheduler obtaining the sub-tasks from the task queue, determining whether there is still a sub-task to be executed in the task queue by Redis distributed lock llen; if there is still a sub-task to be executed in the task queue, continuing to execute the step of obtaining the sub-tasks from the task queue by the original scheduler and executing the sub-tasks by the executor corresponding to the original scheduler until there is no sub-task to be executed in the task queue; if there is no sub-task to be executed in the task queue, after executing the sub-task obtained currently, calling an Event to execute a task end event.

9. A distributed task execution device, characterized in that, The distributed task execution device comprises: a judgment module configured to start a new scheduler in response to a timer starting, and determine whether a current task is executed by an original scheduler by the new scheduler; wherein the timer is configured to start at a preset time interval or a preset specific time; a scheduling module configured to schedule the current task by the new scheduler and the original scheduler if the current task is not executed by the original scheduler; wherein a new container service is added in response to the number of schedulers exceeding a preset number; an execution module configured to determine a new task from a plurality of distributed tasks by a distributed lock by the original scheduler if the current task is executed by the original scheduler, and execute the new task by a corresponding executor by the new scheduler or the original scheduler; wherein it is determined whether there is a new task to be created and executed, and if there is no new task to arrive at its creation time, the new scheduler and the timer end this start, and if there is at least one new task to arrive at its creation time, the new scheduler and the original scheduler can create and schedule the new task together; in response to the current task being executed and a preset time not being reached, a scheduler is closed, and if it is determined that the number of remaining schedulers does not exceed the preset number, a container service is reduced. When all sub-tasks are executed, waiting for the timer to be started again to loop the timer starting and starting a new dispatcher.

10. An electronic device, comprising: A computer program product comprising a memory and a processor coupled to each other, the processor being configured to execute program instructions stored in the memory to implement the method of claim 1 to 8.

11. A computer readable storage medium having stored thereon program instructions, wherein, The program instructions, when executed by the processor, implement the method of claim 1 to 8.

Images