Task scheduling method and device
By preconfiguring the task set with resources and duration and unified management of tasks, the problems of low resource utilization and chaotic management in the task scheduling system are solved, and efficient resource utilization and linear task execution are achieved.
Patent Information
- Application Number
- CN202111356319.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-11-16
- Publication Date
- 2024-11-19
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
In the task scheduling system, the task execution time varies, resulting in insufficient resource allocation, low resource utilization, chaotic management, and serious resource waste.
By obtaining the resource requirements and duration of the task to be executed, comparing it with the remaining configuration resources and resource occupation time of the task set, if the conditions are met, add the task to the task set and execute the task within the resources and duration pre-configured by the task set to achieve the task unified management and resource scheduling of the set.
It improves resource utilization, avoids unnecessary resource waste, and realizes unified management and efficient execution of tasks.
Smart Images

Figure CN114048015B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of big data technology, and in particular to a task scheduling method and device. Background Art
[0002] This section is intended to provide a background or context to the embodiments of the invention recited in the claims. No admission is made that the description herein is prior art by inclusion in this section.
[0003] At present, in the task scheduling system, all tasks can basically be defined as scheduled tasks or one-time non-periodic tasks. Since the execution time of tasks varies, task scheduling mainly consumes a lot of manpower costs, counts the execution time, time start and time end of the task, and connects all tasks in series through task dependency conditions, which makes the management of tasks particularly chaotic and the resource utilization rate is low. Summary of the invention
[0004] The embodiment of the present invention provides a task scheduling method for strengthening the unified management of tasks and improving resource utilization. The method includes:
[0005] Get the tasks to be executed;
[0006] When the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the duration of executing the task to be executed is less than or equal to the resource occupancy duration of the task set, the task to be executed is added to the task set, and the task set is pre-configured with the configured resources and resource occupancy duration;
[0007] The tasks to be executed in the task set are executed within an execution time period corresponding to the resource occupation time of the task set.
[0008] The embodiment of the present invention further provides a task scheduling device for strengthening the unified management of tasks and improving resource utilization, the device comprising:
[0009] The acquisition module is used to obtain tasks to be executed;
[0010] A task adding module, used for adding the to-be-executed task to the task set when the resources required to execute the to-be-executed task are less than or equal to the remaining configured resources of the task set and the execution time of the to-be-executed task is less than or equal to the resource occupation time of the task set, wherein the task set is pre-configured with the configured resources and the resource occupation time;
[0011] The task execution module is used to execute the tasks to be executed in the task set within the execution time period corresponding to the resource occupation time of the task set.
[0012] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned task scheduling method when executing the computer program.
[0013] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned task scheduling method is implemented.
[0014] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the task scheduling method described above is implemented.
[0015] In an embodiment of the present invention, a task to be executed is obtained; when the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the execution time of the task to be executed is less than or equal to the resource occupancy time of the task set, the task to be executed is added to the task set, and the task set is pre-configured with the configured resources and the resource occupancy time; within the execution time period corresponding to the resource occupancy time of the task set, the task to be executed in the task set is executed. The present invention adds the task to be executed to the task set, configures fixed resources and resource occupancy time for the task set, and performs resource scheduling with the task set as the smallest atom, which can strengthen the unified management of tasks; and, linearly executes the tasks in the task set in a specified time period, thereby improving resource utilization and avoiding unnecessary waste of resources. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:
[0017] Figure 1 A flowchart of a task scheduling method provided in an embodiment of the present invention;
[0018] Figure 2 A flowchart of another task scheduling method provided in an embodiment of the present invention;
[0019] Figure 3 A flowchart of another task scheduling method provided in an embodiment of the present invention;
[0020] Figure 4 A schematic diagram of a task scheduling device provided in an embodiment of the present invention;
[0021] Figure 5A schematic diagram of another task scheduling device provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0022] To make the purpose, technical solution and advantages of the embodiments of the present invention more clear, the embodiments of the present invention are further described in detail below in conjunction with the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0023] In the description of this specification, the terms "include", "including", "have", "contain", etc. are all open terms, which mean including but not limited to. The descriptions with reference to the terms "one embodiment", "a specific embodiment", "some embodiments", "for example", etc. mean that the specific features, structures or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures or characteristics described may be combined in a suitable manner in any one or more embodiments or examples. The order of steps involved in each embodiment is used to schematically illustrate the implementation of the present application, and the order of steps is not limited and can be appropriately adjusted as needed.
[0024] Research has found that in the current big data development environment, most tasks are run using cluster resources. As the number of tasks increases, resource usage will accumulate, which will lead to more and more tasks to be managed, and cluster resources will become increasingly scarce. However, the complexity of cluster tasks and the scarcity of resources are mostly caused by insufficient resource allocation. Therefore, the embodiment of the present invention provides a task scheduling method to strengthen the unified management of tasks and improve resource utilization.
[0025] like Figure 1 As shown, a task scheduling method provided by an embodiment of the present invention includes:
[0026] Step 101, obtaining tasks to be executed;
[0027] Step 102, when the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the execution time of the task to be executed is less than or equal to the resource occupation time of the task set, the task to be executed is added to the task set, and the task set is pre-configured with the configured resources and the resource occupation time;
[0028] Step 103: Execute the to-be-executed tasks in the task set within the execution time period corresponding to the resource occupation time of the task set.
[0029] In an embodiment of the present invention, a task to be executed is obtained; when the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the execution time of the task to be executed is less than or equal to the resource occupancy time of the task set, the task to be executed is added to the task set, and the task set is pre-configured with the configured resources and the resource occupancy time; within the execution time period corresponding to the resource occupancy time of the task set, the task to be executed in the task set is executed. The present invention adds the task to be executed to the task set, configures fixed resources and resource occupancy time for the task set, and performs resource scheduling with the task set as the smallest atom, which can strengthen the unified management of tasks; and, linearly executes the tasks in the task set in a specified time period, thereby improving resource utilization and avoiding unnecessary waste of resources.
[0030] In the above step 101, the tasks to be executed may be different tasks submitted by the user, which may be scheduled tasks, short-term tasks that can be executed once, or long-term tasks that are started once and run all the time. Different tasks to be executed may also require different resources and durations.
[0031] For example, Task A requires 2 GB of resources and takes 0.5 hours to execute. Alternatively, Task B is a scheduled startup task that is executed between 10 and 11 a.m. every day. The execution time of Task B is 1 hour and the required resources are 1 GB.
[0032] In the above step 102, the resources and duration required to execute the task to be executed are compared with the remaining configured resources and resource occupancy duration of the task set. If the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the duration to execute the task to be executed is less than or equal to the resource occupancy duration of the task set, the task to be executed is added to the task set.
[0033] Among them, the scheduling platform can pre-configure resources and resource occupancy time for the task set. The configured resources are the fixed resources of the task set, and the resource occupancy time is the specified execution time of the task set. It can be configured according to specific needs, and is not based on the required resources and execution time of a certain task or multiple tasks.
[0034] The above-mentioned scheduling platform refers to a task scheduling platform similar to Elasitc-Job, Easy Scheduler, DolphinScheduler, etc.
[0035] For example, the configured resources of a task set are 4 GB, and the resource usage time is 4 hours, but the remaining configured resources of the task set are only 3 GB. Then, tasks whose resources required for execution are less than or equal to 3 GB and whose execution time is less than or equal to 4 hours can be added to the task set.
[0036] In the above step 103, the to-be-executed tasks in the task set are executed within the execution time period corresponding to the resource occupation time of the task set.
[0037] In one embodiment of the present invention, the resource occupation time of the task set may correspond to a specified execution time period, for example, the resource occupation time of the task set is 4 hours, and the specified execution time period is from 7 to 11. Alternatively, the resource occupation time of the task set may be divided into multiple execution time periods.
[0038] like Figure 2 As shown, in one embodiment of the present invention, after the above step 102, it also includes:
[0039] Step 201 : divide the resource occupation duration of the task set into a plurality of execution time periods, wherein the total duration of the plurality of execution time periods is equal to the resource occupation duration of the task set.
[0040] In specific implementation, the resource occupation time of the task set can be divided into multiple execution time periods of arbitrary lengths. The resource occupation time of the task set can be divided into 2 execution time periods, or 3 or 5 execution time periods; the multiple execution time periods can be of the same length or of different lengths, as long as the total length of the multiple execution time periods is equal to the resource occupation time of the task set. Therefore, the specific division of the resource occupation time of the task set is not limited here.
[0041] For example, if the resource occupancy time of a task set is 4 hours, its specified execution time period can be 7:00 to 9:00, 11:00 to 12:00, 14:00 to 15:00; or it can be 7:00 to 8:00, 10:00 to 11:00, 13:00 to 14:00, 18:00 to 19:00, etc.
[0042] After the resource occupancy time of the task set is divided into multiple execution time periods in the above step 201, if the task to be executed is not completed in the first execution time period, it can continue to be executed in the second execution time period, and so on, until the task to be executed in the task set is completed.
[0043] like Figure 3 As shown, in one embodiment of the present invention, the above step 103 further includes:
[0044] Step 301, executing a task to be executed in a task set in a first execution time period of multiple execution time periods;
[0045] Step 302: If the tasks to be executed in the task set are not completed in the first execution time period, the tasks are continued to be executed in the second execution time period of the multiple execution time periods, and so on, until the tasks to be executed in the task set are completed.
[0046] In this way, through the flexible configuration of the task set, fragmented time and fragmented resources can be more reasonably and fully utilized, thus avoiding resource waste and improving resource utilization.
[0047] In the embodiment of the present invention, when the start time of the execution time period corresponding to the resource occupation time of the task set is reached, when the to-be-executed tasks in the task set are executed in the above step 103, specifically including:
[0048] During the execution time period corresponding to the resource occupation time of the task set, the tasks to be executed are executed in sequence according to the priority order of the tasks.
[0049] In specific implementation, the task priority can be set by the user according to actual needs, or it can be automatically set by preset rules in the task set, for example, according to the time sequence of joining the task set, or according to the order of task names, etc., which is not limited here.
[0050] In the embodiment of the present invention, the tasks in the task set may also be executed in parallel. When executing the tasks to be executed in the task set in the above step 103, specifically, the following is further included:
[0051] During an execution time period corresponding to the resource occupation time of the task set, multiple tasks to be executed in the task set are executed in parallel.
[0052] In specific implementation, the number of pending tasks executed in parallel is not limited here, and the resources required for a single task executed in parallel are not limited here, but the total resources required for multiple pending tasks executed in parallel are less than or equal to the configured resources of the task set.
[0053] For example, if the configuration resource of the task set is 4G, then two tasks requiring 2G memory can be executed simultaneously; or four tasks requiring 1G memory can be executed simultaneously; or one task requiring 3G memory and one task requiring 1G memory can be executed simultaneously, and so on.
[0054] It should be noted that in the embodiment of the present invention, the scheduling platform does not have the concept of tasks at the scheduling level. Everything is scheduled with the task set as the smallest atom. The concept of tasks exists only within the task set. The running of tasks is allocated based on the fixed configuration resources and resource occupancy time in the task set.
[0055] The present invention also provides a task scheduling device, as described in the following embodiments. Since the principle of solving the problem by the device is similar to that of the task scheduling method, the implementation of the device can refer to the implementation of the task scheduling method, and the repeated parts will not be repeated.
[0056] like Figure 4FIG. 1 is a schematic diagram of a task scheduling device provided by an embodiment of the present invention, wherein the device comprises:
[0057] The acquisition module 401 is used to acquire tasks to be executed;
[0058] The task adding module 402 is used to add the task to be executed to the task set, where the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the execution time of the task to be executed is less than or equal to the resource occupation time of the task set, and the task set is pre-configured with the configured resources and the resource occupation time;
[0059] The task execution module 403 is used to execute the to-be-executed tasks in the task set within an execution time period corresponding to the resource occupation time of the task set.
[0060] In one embodiment of the present invention, the task execution module 403 is specifically used to:
[0061] During the execution time period corresponding to the resource occupation time of the task set, the tasks to be executed are executed in sequence according to the priority order of the tasks.
[0062] In one embodiment of the present invention, the task execution module 403 is further configured to:
[0063] During an execution time period corresponding to the resource occupation time of the task set, multiple tasks to be executed in the task set are executed in parallel.
[0064] In one embodiment of the present invention, Figure 5 As shown, Figure 4 The device shown also includes a time period division module 501:
[0065] It is used to divide the resource occupation time of the task set into multiple execution time periods, and the total duration of the multiple execution time periods is equal to the resource occupation time of the task set.
[0066] In one embodiment of the present invention, Figure 5 In the device shown, the task execution module 403 is further specifically used for:
[0067] Execute the to-be-executed tasks in the task set in the first execution time period of the multiple execution time periods;
[0068] If the tasks to be executed in the task set are not completed in the first execution time period, they are continued to be executed in the second execution time period of the multiple execution time periods, and so on, until the tasks to be executed in the task set are completed.
[0069] An embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned task scheduling method when executing the computer program.
[0070] An embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned task scheduling method is implemented.
[0071] An embodiment of the present invention further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the task scheduling method described above is implemented.
[0072] In an embodiment of the present invention, a task to be executed is obtained; when the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the execution time of the task to be executed is less than or equal to the resource occupancy time of the task set, the task to be executed is added to the task set, and the task set is pre-configured with the configured resources and the resource occupancy time; within the execution time period corresponding to the resource occupancy time of the task set, the task to be executed in the task set is executed. The present invention adds the task to be executed to the task set, configures fixed resources and resource occupancy time for the task set, and performs resource scheduling with the task set as the smallest atom, which can strengthen the unified management of tasks; and, linearly executes the tasks in the task set in a specified time period, thereby improving resource utilization and avoiding unnecessary waste of resources.
[0073] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0074] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0075] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0076] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0077] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.
Claims
1. A task scheduling method, characterized in that: include: Get the tasks to be executed; When the resources required to execute the task to be executed are less than or equal to the remaining configured resources of the task set, and the duration of executing the task to be executed is less than or equal to the resource occupancy duration of the task set, the task to be executed is added to the task set, and the task set is pre-configured with the configured resources and resource occupancy duration; The tasks to be executed in the task set are executed within an execution time period corresponding to the resource occupation time of the task set.
2. The method according to claim 1, characterized in that Executing the tasks to be executed in the task set within the execution time period corresponding to the resource occupation time of the task set, including: During the execution time period corresponding to the resource occupation time of the task set, the tasks to be executed are executed in sequence according to the priority order of the tasks.
3. The method according to claim 1, characterized in that Executing the tasks to be executed in the task set within the execution time period corresponding to the resource occupation time of the task set, including: During an execution time period corresponding to the resource occupation time of the task set, multiple tasks to be executed in the task set are executed in parallel.
4. The method according to claim 1, characterized in that Also includes: The resource occupation duration of the task set is divided into a plurality of execution time periods, and the total duration of the plurality of execution time periods is equal to the resource occupation duration of the task set.
5. The method according to claim 4, characterized in that Executing the tasks to be executed in the task set within the execution time period corresponding to the resource occupation time of the task set, including: Execute the to-be-executed tasks in the task set in the first execution time period of the multiple execution time periods; If the tasks to be executed in the task set are not completed in the first execution time period, they are continued to be executed in the second execution time period of the multiple execution time periods, and so on, until the tasks to be executed in the task set are completed.
6. A task scheduling device, characterized in that: include: The acquisition module is used to obtain tasks to be executed; A task adding module, used for adding the to-be-executed task to the task set when the resources required to execute the to-be-executed task are less than or equal to the remaining configured resources of the task set and the execution time of the to-be-executed task is less than or equal to the resource occupation time of the task set, wherein the task set is pre-configured with the configured resources and the resource occupation time; The task execution module is used to execute the tasks to be executed in the task set within the execution time period corresponding to the resource occupation time of the task set.
7. The device according to claim 6, wherein the task execution module is specifically used for: During the execution time period corresponding to the resource occupation time of the task set, the tasks to be executed are executed in sequence according to the priority order of the tasks.
8. The device according to claim 6, characterized in that Task execution module, specifically used for: During an execution time period corresponding to the resource occupation time of the task set, multiple tasks to be executed in the task set are executed in parallel.
9. The device according to claim 6, characterized in that Also includes: The time period division module is used to divide the resource occupation time of the task set into multiple execution time periods, and the total duration of the multiple execution time periods is equal to the resource occupation time of the task set.
10. The device according to claim 9, characterized in that Task execution module, specifically used for: Execute the to-be-executed tasks in the task set in the first execution time period of the multiple execution time periods; If the tasks to be executed in the task set are not completed in the first execution time period, they are continued to be executed in the second execution time period of the multiple execution time periods, and so on, until the tasks to be executed in the task set are completed.
11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
13. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
