Distributed task scheduling method and device

Abstract

The invention discloses a distributed task scheduling method and device, and the method comprises the steps: enabling a task executor to be automatically registered to a scheduling center according to a configured address of the scheduling center; the scheduling center starts a scheduling thread and determines a task type to which the task belongs through the scheduling thread; the scheduling center triggers the task according to the task type and issues the task to a task executor; and the task executor executes the task issued by the scheduling center and reports an execution result to the scheduling center. Therefore, the task type to which the task belongs is determined through the scheduling thread, and task execution is triggered according to the task type, so that the task scheduling strategy is expanded, the task can be executed in parallel through cluster deployment of the scheduling center and the executor, and the task execution efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of task scheduling, and in particular to a distributed task scheduling method and device. Background Art

[0002] At present, Internet services are mainly multi-task and high-concurrency scenarios, and the complexity of a single task and the number of concurrent tasks are increasing. In the prior art, a scheduled task thread pool can be automatically started in the background and tasks are executed according to the configured time rules, wherein a single thread will execute all scheduled tasks. If multiple tasks need to be executed at the same time, they will be executed sequentially, which makes the task scheduling strategy single and reduces the efficiency of task execution. Summary of the invention

[0003] The present invention provides a distributed task scheduling method and device to solve the technical problems of single task scheduling strategy and low task execution efficiency in the above-mentioned prior art.

[0004] To this end, the present invention proposes a distributed task scheduling method, which can determine the task type to which the task belongs through the scheduling thread, and trigger the task execution according to the task type, thereby expanding the task scheduling strategy. At the same time, through the cluster deployment of the scheduling center and the executor, the tasks can be executed in parallel, thereby improving the task execution efficiency.

[0005] Another object of the present invention is to provide a distributed task scheduling device.

[0006] To achieve the above object, the present invention provides a distributed task scheduling method, which includes:

[0007] The task executor automatically registers with the dispatch center according to the address of the configured dispatch center;

[0008] The scheduling center starts a scheduling thread, and determines the task type to which the task belongs through the scheduling thread;

[0009] The dispatch center triggers the task according to the task type and sends the task to the task executor;

[0010] The task executor executes the task issued by the scheduling center and reports the execution result to the scheduling center.

[0011] The distributed task scheduling method of the embodiment of the present invention may also have the following additional technical features:

[0012] In one embodiment of the present invention, determining the task type to which the task belongs through the scheduling thread includes:

[0013] Obtaining a task execution table, wherein the task execution table includes task execution time;

[0014] Based on the task execution time and the current time, a task type to which the task belongs is determined.

[0015] In one embodiment of the present invention, determining the task type to which the task belongs based on the task execution time and the current time includes:

[0016] If the current time exceeds the task execution time, and the exceeding time is greater than or equal to the pre-reading time, then determining that the task type to which the task belongs is a serious timeout task;

[0017] If the current time exceeds the task execution time, and the exceeding time is less than the pre-reading time, then determining that the task type to which the task belongs is a timeout task;

[0018] If the current time does not exceed the task execution time, it is determined that the task type to which the task belongs is an unexpired task.

[0019] In one embodiment of the present invention, the dispatch center triggers the task according to the task type, including:

[0020] If the task type is a serious timeout task, triggering the task according to the scheduling expiration policy;

[0021] If the task type is a timeout task, trigger the task immediately;

[0022] If the task type is an unexpired task, the task is inserted into the time wheel and the task is triggered when the execution time arrives.

[0023] In one embodiment of the present invention, before determining the task type to which the task belongs through the scheduling thread, the method further includes:

[0024] The scheduling thread sends a distributed lock request;

[0025] If the distributed lock is successfully acquired, the scheduling thread determines the task type to which the task belongs.

[0026] In one embodiment of the present invention, the method further comprises:

[0027] The dispatch center receives the execution result reported by the task executor;

[0028] The scheduling thread releases the distributed lock.

[0029] In one embodiment of the present invention, the method further comprises:

[0030] Determining a trigger time for the task;

[0031] If the triggering time of the task exceeds the first preset time, the slow times of the task are incremented by one;

[0032] If the number of slow times triggered by the task within the second preset time period exceeds a preset number, the scheduling center starts the scheduling thread in the slow thread pool.

[0033] To achieve the above object, the present invention provides a distributed task scheduling device on the other hand, the device comprising:

[0034] The registration module is used for the task executor to automatically register with the dispatch center according to the address of the configured dispatch center;

[0035] A determination module, used for the scheduling center to start a scheduling thread and determine the task type to which the task belongs through the scheduling thread;

[0036] A trigger module, used by the dispatch center to trigger the task according to the task type and send the task to the task executor;

[0037] The execution module is used for the task executor to execute the task issued by the scheduling center and report the execution result to the scheduling center.

[0038] Another object of the present invention is to provide an electronic device, comprising:

[0039] at least one processor; and

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the methods described in the above aspect.

[0042] Another object of the present invention is to provide a computer storage medium, wherein the computer storage medium stores computer executable instructions; after the computer executable instructions are executed by a processor, the computer executes any one of the methods described in the above aspect.

[0043] In the distributed task scheduling method and device of the embodiment of the present invention, the task executor automatically registers with the scheduling center according to the address of the configured scheduling center; the scheduling center starts the scheduling thread and determines the task type to which the task belongs through the scheduling thread; the scheduling center triggers the task according to the task type and sends the task to the task executor; the task executor executes the task sent by the scheduling center and reports the execution result to the scheduling center. Thus, the task type to which the task belongs is determined through the scheduling thread, and the task execution is triggered according to the task type, thereby expanding the task scheduling strategy, and at the same time, the tasks can be executed in parallel through the cluster deployment of the scheduling center and the executor, thereby improving the task execution efficiency.

[0044] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0046] Figure 1 is a flow chart of a distributed task scheduling method according to an embodiment of the present invention;

[0047] Figure 2 is a schematic diagram of a task executor and a scheduling center according to an embodiment of the present invention;

[0048] Figure 3 4 is a structural diagram of a distributed task scheduling device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. The present invention will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0050] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0051] The following describes a distributed task scheduling method and device according to an embodiment of the present invention with reference to the accompanying drawings.

[0052] Figure 1 It is a flow chart of a distributed task scheduling method according to an embodiment of the present invention.

[0053] like Figure 1 As shown, the method includes:

[0054] S1, the task executor automatically registers with the dispatch center according to the address of the configured dispatch center;

[0055] In one embodiment of the present invention, the above-mentioned task executor can be deployed in a cluster, and when the executor is started, the corresponding registration thread will be started, and the registration thread will register the current information of the executor with the scheduling center, where the executor information can include the name of the executor (that is, the set appname), and the IP and port of the machine where the executor is located. Based on this, the scheduling center can access the Http interface provided by the executor.

[0056] In one embodiment of the present invention, the above-mentioned scheduling center can also be deployed in a cluster, and the scheduling center can be a separate Web service to trigger task execution. In addition, in one embodiment of the present invention, the interface of the scheduling center can be operated through the front-end page, which greatly simplifies the maintenance and scheduling of the timing task triggering logic.

[0057] Among them, in one embodiment of the present invention, Figure 2 Schematic diagram of a task executor and a scheduling center proposed in an embodiment of the present invention. Figure 2 As shown, the instances of each scheduling center can achieve data consistency and synchronization by sharing the same database resources to ensure that all instances can access the latest task information, thereby ensuring high data availability and consistency and simplifying the management complexity within the cluster.

[0058] In one embodiment of the present invention, the above-mentioned calling center can control the triggering of scheduled tasks, and the executor can execute the tasks issued by the scheduling center. Based on this, the separation of the above-mentioned task execution and task triggering logic can make the tasks more flexible, which can be called at any time and can also be triggered by different scheduling rules.

[0059] S2, the scheduling center starts the scheduling thread and determines the task type to which the task belongs through the scheduling thread;

[0060] In one embodiment of the present invention, the scheduling center may start a scheduling thread and determine the task type to which the task belongs through the scheduling thread, so that the corresponding task can be triggered according to the task type. In one embodiment of the present invention, the above task may be a dual recording task.

[0061] Specifically, in one embodiment of the present invention, the method for determining the task type to which a task belongs by scheduling a thread may include: obtaining a task execution table, wherein the task execution table may include the task execution time, and determining the task type to which the task belongs based on the task execution time and the current time. The task execution table may also include basic information of the task.

[0062] In one embodiment of the present invention, the method for determining the task type to which a task belongs based on the task execution time and the current time may include the following steps:

[0063] S21, if the current time exceeds the task execution time, and the exceeding time is greater than or equal to the pre-reading time, then determining that the task type to which the task belongs is a serious timeout task;

[0064] S22, if the current time exceeds the task execution time, and the exceeding time is less than the pre-reading time, then determining that the task type to which the task belongs is a timeout task;

[0065] S23: If the current time does not exceed the task execution time, determine that the task type to which the task belongs is an unexpired task.

[0066] In one embodiment of the present invention, the pre-reading time may be the time to read the task in advance to ensure that the task can be triggered on time, and the pre-reading time may be set as needed. For example, the pre-reading time may be 10 seconds.

[0067] Furthermore, in one embodiment of the present invention, before determining the task type to which the task belongs by the scheduling thread, the method may further include: the scheduling thread sends a distributed lock request, and if the distributed lock is successfully acquired, the scheduling thread determines the task type to which the task belongs.

[0068] Specifically, in one embodiment of the present invention, the scheduling thread may send a distributed lock request to the database, wherein the distributed lock request may include the following SQL statement:

[0069] select*from jobs_lock where lock_name='schedule_lock'for update

[0070] Among them, jobs_info is the task execution table.

[0071] And, if the above distributed lock request is executed successfully, it means that the current scheduling center successfully obtains the distributed lock, so that the subsequent scheduling thread can determine the task type to which the task belongs. In one embodiment of the present invention, a distributed lock request needs to be sent before each task is executed, so that it can be ensured that only one scheduling center in the scheduling center cluster executes the scheduling task at the same time.

[0072] S3, the dispatch center triggers task execution according to the task type and sends the task to the task executor;

[0073] In one embodiment of the present invention, after obtaining the task type to which the task belongs through the above steps, the scheduling center can trigger the task execution according to the task type and send the task to the task executor.

[0074] Specifically, in one embodiment of the present invention, the method for the scheduling center to trigger task execution according to the task type may include the following steps:

[0075] S31, if the task type is a serious timeout task, triggering task execution according to the scheduling expiration policy;

[0076] S32, if the task type is a timeout task, immediately trigger the task execution;

[0077] S33, if the task type is an unexpired task, insert the task into the time wheel and wait until the execution time triggers the task execution.

[0078] In one embodiment of the present invention, if the task type is a serious timeout task, the task execution can be triggered according to the scheduling expiration policy configured on the front-end page. The scheduling expiration policy can include ignoring the task or executing it once immediately.

[0079] S4, the task executor executes the task issued by the dispatch center and reports the execution result to the dispatch center.

[0080] In one embodiment of the present invention, the task executor can execute the tasks issued by the dispatching center in parallel, thereby improving the task execution efficiency.

[0081] In one embodiment of the present invention, after the task executor executes the task issued by the scheduling center, the execution result can be reported to the scheduling center. After the scheduling center receives the execution result reported by the task executor, the corresponding scheduling thread can release the distributed lock.

[0082] And, in one embodiment of the present invention, after executing the task through the above steps, the task may be executed according to the task execution time in the task execution table, based on which the scheduling thread may be triggered through the corresponding thread pool according to the trigger time of the task. In one embodiment of the present invention, the above thread pool may include a fast thread pool and a slow thread pool.

[0083] Specifically, in one embodiment of the present invention, the above method may further include the following steps:

[0084] S5, determine the triggering time of the task;

[0085] S6, if the triggering time of the task exceeds the first preset time, the slow times of the task are incremented by one;

[0086] S7: If the number of slow times triggered by the task within the second preset time period exceeds a preset number, the scheduling center starts the scheduling thread in the slow thread pool.

[0087] Among them, in one embodiment of the present invention, when the call center calls the executor's HTTP interface to trigger the execution of the task, it will record the trigger time consumed by the triggering of each task. And, in one embodiment of the present invention, if the triggering time of the task exceeds the first preset time, the slow number of times the task is triggered can be incremented by one. If the slow number of times the task is triggered within the second preset time period exceeds the preset number, the scheduling center starts the scheduling thread in the slow thread pool, thereby avoiding the situation where tasks that are frequently triggered and each triggering time is very long block the triggering of other tasks, thereby improving the scheduling performance. In one embodiment of the present invention, the above-mentioned first preset time, second preset time and preset number can be set as needed. For example, the above-mentioned first preset time is 500ms (milliseconds), the second preset time is 1min (minutes), and the preset number is 10 times.

[0088] In the distributed task scheduling method proposed by the present invention, the task executor automatically registers with the scheduling center according to the address of the configured scheduling center; the scheduling center starts the scheduling thread and determines the task type to which the task belongs through the scheduling thread; the scheduling center triggers the task according to the task type and sends the task to the task executor; the task executor executes the task sent by the scheduling center and reports the execution result to the scheduling center. Thus, the task type to which the task belongs is determined by the scheduling thread, and the task execution is triggered according to the task type, thereby expanding the task scheduling strategy, and at the same time, the tasks can be executed in parallel through the cluster deployment of the scheduling center and the executor, thereby improving the task execution efficiency.

[0089] In order to implement the above embodiment, Figure 3 As shown, this embodiment also provides a distributed task scheduling device 10, which includes a registration module 301, a determination module 302, a trigger module 303 and an execution module 304;

[0090] Registration module 301, used for the task executor to automatically register with the dispatch center according to the address of the configured dispatch center;

[0091] Determination module 302, used for the scheduling center to start the scheduling thread, and determine the task type to which the task belongs through the scheduling thread;

[0092] The trigger module 303 is used for the dispatch center to trigger the task according to the task type and send the task to the task executor;

[0093] The execution module 304 is used for the task executor to execute the task issued by the scheduling center and report the execution result to the scheduling center.

[0094] Furthermore, the determination module 302 is specifically configured to:

[0095] Obtaining a task execution table, wherein the task execution table includes task execution time;

[0096] Based on the task execution time and the current time, determine the task type to which the task belongs.

[0097] Furthermore, the determination module 302 is further configured to:

[0098] If the current time exceeds the task execution time, and the exceeding time is greater than or equal to the pre-reading time, the task type to which the task belongs is determined to be a serious timeout task;

[0099] If the current time exceeds the task execution time, and the exceeding time is less than the pre-reading time, the task type to which the task belongs is determined to be a timeout task;

[0100] If the current time does not exceed the task execution time, it is determined that the task type to which the task belongs is an unexpired task.

[0101] Furthermore, the trigger module 303 is specifically used for:

[0102] If the task type is a serious timeout task, the task is triggered according to the scheduling expiration policy;

[0103] If the task type is a timeout task, the task is triggered immediately;

[0104] If the task type is an unexpired task, insert the task into the time wheel and wait until the execution time triggers the task.

[0105] Furthermore, the above device is also used for:

[0106] The scheduling thread sends a distributed lock request;

[0107] If the distributed lock is successfully acquired, the scheduling thread determines the task type to which the task belongs.

[0108] Furthermore, the above device is also used for:

[0109] The dispatch center receives the execution results reported by the task executor;

[0110] The scheduling thread releases the distributed lock.

[0111] Furthermore, the above device is also used for:

[0112] Determine the trigger time of the task;

[0113] If the trigger time of the task exceeds the first preset time, the slow times of the task will be increased by one;

[0114] If the number of slow times triggered by the task within the second preset time period exceeds a preset number, the scheduling center starts the scheduling thread in the slow thread pool.

[0115] In the distributed task scheduling device proposed by the present invention, the task executor automatically registers with the scheduling center according to the address of the configured scheduling center; the scheduling center starts the scheduling thread and determines the task type to which the task belongs through the scheduling thread; the scheduling center triggers the task according to the task type and sends the task to the task executor; the task executor executes the task sent by the scheduling center and reports the execution result to the scheduling center. Thus, the task scheduling strategy is expanded by determining the task type to which the task belongs through the scheduling thread and triggering the task execution according to the task type. At the same time, the tasks can be executed in parallel through the cluster deployment of the scheduling center and the executor, thereby improving the task execution efficiency.

[0116] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0117] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

Claims

1. A distributed task scheduling method, characterized in that: The method comprises: The task executor automatically registers with the dispatch center according to the address of the configured dispatch center; The scheduling center starts a scheduling thread, and determines the task type to which the task belongs through the scheduling thread; The dispatch center triggers the task according to the task type and sends the task to the task executor; The task executor executes the task issued by the scheduling center and reports the execution result to the scheduling center.

2. The method according to claim 1, characterized in that The determining the task type to which the task belongs through the scheduling thread includes: Obtaining a task execution table, wherein the task execution table includes task execution time; Based on the task execution time and the current time, a task type to which the task belongs is determined.

3. The method according to claim 1, characterized in that The determining the task type to which the task belongs based on the task execution time and the current time includes: If the current time exceeds the task execution time, and the exceeding time is greater than or equal to the pre-reading time, then determining that the task type to which the task belongs is a serious timeout task; If the current time exceeds the task execution time, and the exceeding time is less than the pre-reading time, then determining that the task type to which the task belongs is a timeout task; If the current time does not exceed the task execution time, it is determined that the task type to which the task belongs is an unexpired task.

4. The method according to claim 3, characterized in that The dispatch center triggers the task according to the task type, including: If the task type is a serious timeout task, triggering the task according to the scheduling expiration policy; If the task type is a timeout task, trigger the task immediately; If the task type is an unexpired task, the task is inserted into the time wheel and the task is triggered when the execution time arrives.

5. The method according to claim 1, characterized in that Before determining the task type to which the task belongs through the scheduling thread, the method further includes: The scheduling thread sends a distributed lock request; If the distributed lock is successfully acquired, the scheduling thread determines the task type to which the task belongs.

6. The method according to claim 5, characterized in that The method further comprises: The dispatch center receives the execution result reported by the task executor; The scheduling thread releases the distributed lock.

7. The method according to claim 1, characterized in that The method further comprises: Determining a trigger time for the task; If the triggering time of the task exceeds the first preset time, the slow times of the task are incremented by one; If the number of slow times triggered by the task within the second preset time period exceeds a preset number, the scheduling center starts the scheduling thread in the slow thread pool.

8. A distributed task scheduling device, characterized in that: The device comprises: The registration module is used for the task executor to automatically register with the dispatch center according to the address of the configured dispatch center; A determination module, used for the scheduling center to start a scheduling thread and determine the task type to which the task belongs through the scheduling thread; A trigger module, used by the dispatch center to trigger the task according to the task type and send the task to the task executor; The execution module is used for the task executor to execute the task issued by the scheduling center and report the execution result to the scheduling center.

9. An electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer storage medium, wherein: The computer storage medium stores computer executable instructions; after the computer executable instructions are executed by the processor, the method according to any one of claims 1 to 7 can be implemented.

Images