Task scheduling method, device, system, electronic device and storage medium
By receiving task requests and assigning them to the target scheduler, the operation and maintenance cost problems caused by the increase in the number of schedulers are solved, and the unified operation and maintenance and development efficiency is improved.
Patent Information
- Application Number
- CN202011374451.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-11-30
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-11-30
AI Technical Summary
With the increase of business systems, the number of schedulers increases, resulting in a linear increase in operation and maintenance costs, affecting development and operation and maintenance efficiency.
A unified scheduling entrance is used to receive task execution requests, determine the target scheduler from the scheduler collection according to the task type, allocate the tasks to the idle work nodes for execution, and realize unified management of different schedulers.
It reduces development and operation and maintenance costs, realizes unified operation and maintenance, releases the manpower of business system development and scheduling systems, and improves development efficiency.
Smart Images

Figure CN112486648B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to computer technology, specifically to cloud computing technology, and in particular to a task scheduling method, device, system, electronic device, and storage medium. Background Art
[0002] In related technologies, a large number of tasks are generated in different business scenarios, and these tasks need to be completed using computing resources. In order to improve the efficiency of task processing, a scheduler can be used to schedule tasks and assign tasks to multiple working nodes for execution.
[0003] In the process of implementing this application, it was found that the schedulers used in different business systems are generally different, and each business system has its own exclusive scheduler. Therefore, with the increase of business systems, the number of scheduling systems will also increase accordingly. The schedulers are numerous and scattered, resulting in a linear increase in operation and maintenance costs, which is not conducive to development and operation and maintenance. Summary of the invention
[0004] Provided are a task scheduling method, device, system, electronic device and storage medium.
[0005] According to a first aspect, a task scheduling method is provided, comprising: receiving a task execution request based on a unified scheduling entry, wherein the task execution request carries the task type of the task to be executed; in response to the task execution request, determining a target scheduler for scheduling the task to be executed from a scheduler set according to the task type of the task to be executed, wherein the target scheduler is used to assign the task to be executed to an idle working node for execution, and the scheduler set includes multiple schedulers of different types; and assigning the task to be executed to the target scheduler.
[0006] According to the second aspect, a task scheduling device is provided, including: a receiving module, used to receive a task execution request based on a unified scheduling entry, wherein the task execution request carries the task type of the task to be executed; and a first response module, used to respond to the task execution request, and determine a target scheduler for scheduling the task to be executed from a scheduler set according to the task type of the task to be executed, wherein the target scheduler is used to assign the task to be executed to an idle working node for execution, and the scheduler set includes multiple schedulers of different types; and an allocation module, used to assign the task to be executed to the target scheduler.
[0007] According to a third aspect, a task scheduling system is provided, comprising: the above-mentioned task scheduling device; and a scheduler set, wherein the above-mentioned scheduler set includes a plurality of schedulers of different types.
[0008] According to a fourth aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the present application.
[0009] According to a fifth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the present application.
[0010] According to a sixth aspect, a computer program product is provided, comprising a computer program, wherein the computer program implements the above-mentioned method of the present application when executed by a processor.
[0011] Through the technical solution of the embodiment of the present application, a unified scheduling entrance is used to receive task execution requests, and the target scheduler for scheduling the task to be executed is determined from the scheduler set according to the task type of the task to be executed. The unified management of different schedulers is realized, and the technical problem that with the increase of business systems, the number of scheduling systems will also increase, and there are many and scattered schedulers, resulting in a linear increase in operation and maintenance costs, which is not conducive to development and operation and maintenance, is solved. Unified operation and maintenance is achieved, development and operation and maintenance costs are reduced, and the manpower of the business system development scheduling system is released, thereby developing the technical effect of the business system more quickly and more focused.
[0012] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.
[0014] Figure 1 An exemplary system architecture to which the task scheduling method and device according to an embodiment of the present disclosure can be applied is schematically shown;
[0015] Figure 2 A flowchart of a task scheduling method according to an embodiment of the present disclosure is schematically shown;
[0016] Figure 3 A schematic diagram schematically shows a task scheduling method according to an embodiment of the present disclosure;
[0017] Figure 4 A flowchart of a task scheduling method according to another embodiment of the present disclosure is schematically shown;
[0018] Figure 5A schematic diagram schematically shows a method of allocating tasks to be executed to a task queue that matches the priority of the tasks to be executed according to an embodiment of the present disclosure;
[0019] Figure 6 A flowchart of constructing a set of working nodes corresponding to a scheduler according to an embodiment of the present disclosure is schematically shown;
[0020] Figure 7 A block diagram schematically shows a task scheduling device according to an embodiment of the present disclosure; and
[0021] Figure 8 A block diagram of a computer system suitable for implementing a task scheduling method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION
[0022] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0023] In the process of implementing this application, it is found that if a single scheduler is set for business systems of different types and scenarios, each business system will have its own exclusive scheduler. However, as the number of business systems increases, the number of scheduling systems will also increase, and the number of schedulers will be large and scattered, resulting in a linear increase in operation and maintenance costs, which is not conducive to development and operation and maintenance.
[0024] An embodiment of the present application provides a task scheduling method, including: receiving a task execution request based on a unified scheduling entry, wherein the task execution request carries the task type of the task to be executed; in response to the task execution request, determining a target scheduler for scheduling the task to be executed from a scheduler set according to the task type of the task to be executed, wherein the target scheduler is used to assign the task to be executed to an idle working node for execution, and the scheduler set includes multiple schedulers of different types; and assigning the task to be executed to the target scheduler.
[0025] Figure 1 An exemplary system architecture to which the task scheduling method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.
[0026] It should be noted that Figure 1What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.
[0027] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 111, 112, a task scheduling device 120, schedulers 131, 132, and a work node set 140. The task scheduling device 120 is used as a unified scheduling entry between the terminal devices 111, 112 and the schedulers 131, 132. The work node set 140 includes multiple work nodes. The schedulers 131, 132 are two schedulers in a scheduler set.
[0028] The user can use the terminal devices 111 and 112 to send the scheduled tasks to the task scheduling device 120 through the network. Various communication client applications can be installed on the terminal devices 111 and 112, such as task generation applications, knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software (only for example).
[0029] The terminal devices 111 and 112 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.
[0030] The task scheduling device 120 may be an entry scheduler, which is later connected to schedulers required by various services to unify the service scheduling system.
[0031] The working nodes in the working node set 140 may be computing devices, for example, servers that provide various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 111 and 112. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.
[0032] It should be noted that the task scheduling method provided in the embodiments of the present application can generally be executed by the task scheduling device 120 .
[0033] According to an embodiment of the present application, a task scheduling system is provided, including: a task scheduling device and a scheduler set, wherein the scheduler set includes multiple schedulers of different types. The embodiment of the present application can connect all business systems to the task scheduling system through a unified task scheduling system, share a scheduling system, unify operation and maintenance, reduce development and operation and maintenance costs, and release the manpower of the business system development scheduling system, so that the business system can be developed more quickly and more focused.
[0034] According to an embodiment of the present application, the task scheduling system may further include a working node.
[0035] It should be understood that Figure 1 The number of terminal devices, task scheduling devices, schedulers, and working nodes in the embodiment is only for illustration. Any number of terminal devices, task scheduling devices, schedulers, and working nodes may be provided according to implementation requirements.
[0036] Figure 2 The flowchart of the task scheduling method according to the embodiment of the present disclosure is schematically shown.
[0037] like Figure 2 As shown, the task scheduling method includes operations S210 to S230.
[0038] In operation S210 , a task execution request is received based on a unified scheduling entry, wherein the task execution request carries a task type of a task to be executed.
[0039] In operation S220, in response to a task execution request, a target scheduler for scheduling the task to be executed is determined from a scheduler set according to the task type of the task to be executed, wherein the target scheduler is used to assign the task to be executed to an idle working node for execution, and the scheduler set includes multiple schedulers of different types.
[0040] In operation S230 , the task to be executed is allocated to a target scheduler.
[0041] According to an embodiment of the present application, a task execution request may be sent by a user through a client, or automatically by a client. The task execution request may include the task type of the task to be executed, and may also include user identification information, priority information of the task to be executed, and the like.
[0042] According to an embodiment of the present application, for all received tasks, a task type label can be marked when submitting to indicate which type of scheduler needs to be used. After the upper-level task scheduling device receives the task, it will assign the task to the corresponding scheduler according to the corresponding label, and the corresponding lower-level scheduler will then assign it to the corresponding working node for execution, thereby completing the task execution.
[0043] According to an embodiment of the present application, a unified scheduling portal can receive task execution requests from different clients, where different clients can use different programming languages. In related technologies, different clients sometimes use different programming languages, so the scheduler corresponding to each client must be able to parse the task corresponding to the programming language of the client. The present application accesses different types of scheduling systems through a unified scheduling portal to achieve the distribution of tasks for clients with different programming languages.
[0044] According to an embodiment of the present application, the target scheduler may determine a currently idle working node, and then assign the to-be-executed tasks to the idle working node for execution.
[0045] According to an embodiment of the present application, the schedulers in the scheduler set complete task delegation by exposing APIs to users, and the schedulers can load-balance and execute tasks concurrently on multiple working nodes.
[0046] According to an embodiment of the present application, task types include but are not limited to: online real-time tasks and offline scheduled tasks.
[0047] According to an embodiment of the present application, an online real-time task may be scheduled by, for example, an open source online scheduling system gearman, which is widely used in scheduling online tasks.
[0048] According to an embodiment of the present application, offline scheduled tasks can be scheduled by, for example, the open source directed acyclic task scheduling system airflow, which is suitable for offline complex task scheduling.
[0049] The above-mentioned scheduling system is merely exemplary. This application does not limit the type of scheduling system, and may also include other scheduling systems in related technologies, such as spark system, hadoop system, flink system, etc., which will not be described in detail here.
[0050] Reference below Figure 3 to Figure 6 , combined with specific embodiments Figure 2 The method shown is further explained.
[0051] Figure 3 A schematic diagram of a task scheduling method according to an embodiment of the present disclosure is schematically shown.
[0052] like Figure 3As shown, based on the unified scheduling entry 310 receiving multiple task execution requests, the multiple task execution requests may be requests for online tasks and offline tasks. The unified scheduling entry 310 may send online tasks to the online task scheduling submodule 321, and send offline tasks to the offline task scheduling submodule 322. The unified scheduling entry 310, the online task scheduling submodule 321, and the sending of offline tasks to the offline task scheduling submodule 322 may be part or all of the task scheduling device.
[0053] The unified scheduling entry 310 may be responsible for functions such as user identity authentication, request parsing, task distribution, and flow control.
[0054] According to the embodiment of the present application, the principle of task diversion and flow control may include: each task will carry the task type and the required cluster type, and all schedulers can match and forward according to the task configuration information and the currently connected cluster information. Before forwarding, the flow will be limited according to the load of the corresponding cluster. For example, after the utilization rate of the gearman cluster (scheduler) exceeds 80%, tasks will no longer be sent to the cluster until the resource utilization rate drops to 70%. Tasks that cannot be sent will be saved in the mysql task list, and will not cause task loss.
[0055] According to an embodiment of the present application, the online task scheduling submodule 321 and the offline task scheduling submodule 322 can be responsible for the dynamic construction of working nodes in the scheduling cluster, scheduling cluster tenant management and authority verification, online / offline task dependency uploading, task job submission, status query, etc.
[0056] Depending on the application scenario, the online task scheduling submodule 321 and the offline task scheduling submodule 322 are connected to different scheduling clusters at the bottom layer, such as the first scheduling cluster 330 and the second scheduling cluster 340. Among them, the first scheduling cluster 330 and the second scheduling cluster 340 may include gearman, airflow, etc., and the first scheduling cluster 330 and the second scheduling cluster 340 may be used for task queue management, task distribution, function loading and task execution. The first scheduling cluster 330 and the second scheduling cluster 340 may also include shared storage for storing intermediate results of the task execution process of the working node.
[0057] According to an embodiment of the present application, the scheduling cluster includes a collection of various types of schedulers, and each type of scheduler is responsible for the scheduling of tasks of that type, load balancing of tasks, and resource management of working nodes.
[0058] According to an embodiment of the present application, the scheduling cluster may also be connected to a computing platform. For example, the second scheduling cluster 340 is connected to the stream-batch integrated computing platform 350 to implement functions such as stream-batch task submission and task status query.
[0059] According to an embodiment of the present application, the task scheduling method can be executed by a unified scheduler (i.e., a task scheduling device), which includes a unified scheduling entry, an online task scheduling submodule, and an offline task scheduling submodule, and can access different types of scheduling systems (including schedulers and work nodes, etc.). The unified scheduler can distribute tasks to different scheduling systems according to the task type, thereby realizing unified management of different scheduling systems.
[0060] Through the embodiments of the present application, a unified scheduling portal is used to receive task execution requests, and a target scheduler for scheduling the task to be executed is determined from a set of schedulers according to the task type of the task to be executed, thereby realizing unified management of different schedulers. This solves the technical problem that as the number of business systems increases, the number of scheduling systems will also increase accordingly, and there are many and scattered schedulers, resulting in a linear increase in operation and maintenance costs, which is not conducive to development and operation and maintenance. This achieves unified operation and maintenance, reduces development and operation and maintenance costs, realizes the requirement of quickly meeting scheduling needs, and releases the manpower of the business system to develop the scheduling system, thereby developing the technical effect of the business system more quickly and more focused.
[0061] According to an embodiment of the present application, before operation S220, that is, before responding to the task execution request, the scheduling authority of the user who sent the task execution request is determined, wherein the user's scheduling authority includes the ability to request tasks of the scheduling target type; when the target type is the same as the task type of the task to be executed, the task execution request is responded to.
[0062] According to an embodiment of the present application, when the target type and the task type of the task to be executed are different, the task execution request is rejected.
[0063] According to an embodiment of the present application, for example, if the user's calling authority is to request to schedule online task types, if the task type of the task to be executed is an online task type, the task execution request can be responded to. If the task type of the task to be executed is an offline task type, the task execution request can be rejected.
[0064] According to an embodiment of the present application, after the scheduling system is applied, since all business lines are using the scheduling system, there will be clear regulations on what types of tasks each user has scheduling authority to perform, which can prevent someone from making mistakes and submitting many unauthorized scheduling tasks, causing the scheduler to crash.
[0065] Figure 4 A flowchart of a task scheduling method according to another embodiment of the present disclosure is schematically shown.
[0066] According to an embodiment of the present application, the task execution request also carries the priority of the task to be executed. In this embodiment, in addition to operations S210 to S230, Figure 4As shown, the method further includes operations S410 to S420.
[0067] In operation S410 , in response to a task execution request, the to-be-executed task is allocated to a task queue that matches the priority of the to-be-executed task according to the priority of the to-be-executed task, wherein tasks in the same task queue have the same priority.
[0068] In operation S420 , tasks in different task queues are scheduled in sequence according to priority order.
[0069] According to an embodiment of the present application, task queues of different levels may be maintained in the scheduler. When a target scheduler for scheduling the task to be executed is determined from a scheduler set according to the task type of the task to be executed, the task to be executed may be assigned to a task queue matching the priority of the task to be executed.
[0070] According to an embodiment of the present application, the higher the priority of a task queue is, the tasks in the task queue will be processed first.
[0071] It should be noted that the operations provided in the embodiments of the present application may be executed in parallel or sequentially, or in different orders, as long as the expected results of the technical solutions disclosed in the present application can be achieved, and this document does not impose any restrictions thereon.
[0072] Figure 5 The diagram schematically shows a schematic diagram of allocating tasks to be executed to task queues that match the priorities of the tasks to be executed according to an embodiment of the present disclosure.
[0073] like Figure 5 As shown, for all submitted tasks, for example, task n, task m, task p, there will be a corresponding priority label, which indicates the priority of the task. The priority label corresponding to task n is the highest level, the priority label corresponding to task m is the middle level, and the priority label corresponding to task p is the lowest level.
[0074] According to an embodiment of the present application, the scheduler 500 maintains task queues of different levels, for example, task queue 510, task queue 520 and task queue 530. Among them, the priority of task queue 510 is higher than that of task queue 520, and the priority of task queue 520 is higher than that of task queue 530.
[0075] According to an embodiment of the present application, after receiving task n, task m, and task p, the scheduler 500 assigns task n to the task queue 510 , assigns task m to the task queue 520 , and assigns task p to the task queue 530 .
[0076] According to an embodiment of the present application, the scheduler 500 may use a scheduling thread to perform task scheduling. The scheduling thread may give priority to scheduling tasks in the task queue 510 and assign tasks in the task queue 510 to idle working nodes.
[0077] According to an embodiment of the present application, after the tasks in task queue 510 are scheduled, the tasks in task queue 520 are then scheduled, and finally the tasks in task queue 530 are scheduled.
[0078] According to an embodiment of the present application, before allocating the task to be executed to the target scheduler, the load information of the target scheduler may be obtained; when the load information of the target scheduler meets a preset condition, the task to be executed is allocated to the target scheduler.
[0079] According to an embodiment of the present application, the load information of the target scheduler can reflect the resource utilization of the scheduler, and can also reflect the resource utilization of the working nodes associated with the target scheduler.
[0080] According to an embodiment of the present application, the preset condition may be, for example, a resource utilization threshold. For example, the preset condition may be that the resource utilization is less than or equal to 80%.
[0081] According to an embodiment of the present application, when the load information of the target scheduler does not meet the preset conditions, the tasks to be executed are saved in the task list; until the load information of the target scheduler meets the preset conditions, the tasks to be executed are assigned to the target scheduler.
[0082] According to an embodiment of the present application, when the load information of the target scheduler does not meet a preset condition, allocating tasks to the target scheduler may be stopped.
[0083] According to an embodiment of the present application, the tasks to be executed are saved in a task list, so that the tasks to be executed can be prevented from being lost.
[0084] Figure 6 A flowchart of constructing a set of working nodes corresponding to a scheduler according to an embodiment of the present disclosure is schematically shown.
[0085] like Figure 6 As shown, the method includes operations S610 to S620.
[0086] In operation S610, load information of each scheduler in a scheduler set is obtained.
[0087] In operation S620, a working node set corresponding to the scheduler is constructed according to the load information of each scheduler, wherein the working node set includes one or more working nodes for executing tasks.
[0088] According to an embodiment of the present application, a scheduler set may include multiple schedulers, and the load information of each scheduler may include the number of tasks received by the scheduler itself, and may also include resource usage information of the working nodes managed by each scheduler.
[0089] According to an embodiment of the present application, obtaining the load information of each scheduler in the scheduler set may include the following methods: each cluster will periodically collect the load information of all working nodes under the cluster, and aggregate the load information of a cluster. A cluster may include a scheduler and multiple working nodes managed by the scheduler, and the load information of each cluster may be reported to a unified scheduling portal. When a task is received, the scheduler will assign the task to the working node with the smallest load of the task type for execution based on the resource requirement type of the task, such as CPU-intensive or memory-intensive.
[0090] According to an embodiment of the present application, each type of scheduler may have a set of working nodes for executing specific tasks (such as a set of working nodes of schedulers such as gearman and airflow). The working nodes in the set of working nodes are responsible for executing specific online / offline jobs.
[0091] According to an embodiment of the present application, a set of working nodes corresponding to a scheduler can be dynamically constructed based on the load information of each scheduler.
[0092] According to the embodiments of the present application, when the number of working nodes is certain, the working nodes may not be exclusively owned by a certain scheduler, and the corresponding working nodes may be temporarily allocated according to the load information of each scheduler, so as to dynamically build a working node set corresponding to the scheduler. In this way, the machine resources can be fully utilized, and the cluster load of the class A task will not be too high and the cluster of the class B task will not be unloaded because the scheduler for scheduling the class A task has a large task volume and the scheduler for scheduling the class B task has a small task volume.
[0093] According to an embodiment of the present application, different types of working nodes may be arranged in a mixed manner, and each working node may be shared by schedulers in a scheduler set, thereby improving resource utilization.
[0094] According to an embodiment of the present application, constructing a working node set corresponding to a scheduler based on the load information of each scheduler includes: monitoring the load information of each scheduler in the scheduler set in different time periods; and generating working node sets corresponding to each scheduler in different time periods based on the load information of each scheduler in different time periods.
[0095] According to the embodiments of the present application, the load information of the scheduler in different historical time periods can be analyzed based on big data analysis to determine the working nodes required by the scheduler in different historical time periods, thereby achieving the technical effect of reasonably configuring the working nodes corresponding to each scheduler.
[0096] According to an embodiment of the present application, by dynamically constructing a set of working nodes corresponding to each scheduler in different time periods, machine resources can be fully utilized.
[0097] Figure 7 A block diagram of a task scheduling device according to an embodiment of the present disclosure is schematically shown.
[0098] like Figure 7 As shown, the task scheduling device 700 includes: a receiving module 710 , a first responding module 720 and an allocating module 730 .
[0099] The receiving module 710 is used to receive a task execution request based on the unified scheduling entry, wherein the task execution request carries the task type of the task to be executed.
[0100] The first response module 720 is used to respond to a task execution request and determine a target scheduler for scheduling the task to be executed from a scheduler set according to the task type of the task to be executed, wherein the target scheduler is used to assign the task to be executed to an idle working node for execution, and the scheduler set includes multiple schedulers of different types.
[0101] The allocation module 730 is used to allocate the tasks to be executed to the target scheduler.
[0102] Through the embodiments of the present application, a unified scheduling portal is used to receive task execution requests, and a target scheduler for scheduling the task to be executed is determined from a set of schedulers according to the task type of the task to be executed, thereby realizing unified management of different schedulers. This solves the technical problem that as the number of business systems increases, the number of scheduling systems will also increase accordingly, and there are many and scattered schedulers, resulting in a linear increase in operation and maintenance costs, which is not conducive to development and operation and maintenance. This achieves unified operation and maintenance, reduces development and operation and maintenance costs, realizes the requirement of quickly meeting scheduling needs, and releases the manpower of the business system to develop the scheduling system, thereby developing the technical effect of the business system more quickly and more focused.
[0103] According to an embodiment of the present application, the task scheduling device 700 also includes: a first determination module, used to determine the scheduling authority of the user who sends the task execution request before responding to the task execution request, wherein the user's scheduling authority includes the ability to request tasks of the scheduling target type; wherein the first response module 720 is used to respond to the task execution request when the target type is the same as the task type of the task to be executed.
[0104] According to an embodiment of the present application, the task execution request also carries the priority of the task to be executed, and the task scheduling device 700 further includes: a second response module and a scheduling module.
[0105] The second response module is used to respond to the task execution request and allocate the task to be executed to a task queue that matches the priority of the task to be executed according to the priority of the task to be executed, wherein the tasks in the same task queue have the same priority.
[0106] The scheduling module is used to schedule tasks in different task queues in order of priority.
[0107] According to an embodiment of the present application, the task scheduling device 700 further includes: a first acquisition module, configured to acquire load information of a target scheduler before allocating a task to be executed to the target scheduler;
[0108] The allocation module 730 is used to allocate the to-be-executed tasks to the target scheduler when the load information of the target scheduler meets a preset condition.
[0109] According to an embodiment of the present application, the task scheduling device 700 further includes: a saving module, configured to save the to-be-executed task in a task list when the load information of the target scheduler does not meet a preset condition;
[0110] The allocation module 730 is used to allocate the to-be-executed tasks to the target scheduler until the load information of the target scheduler meets a preset condition.
[0111] According to an embodiment of the present application, the task scheduling device 700 further includes: a second acquisition module and a construction module.
[0112] The second acquisition module is used to obtain the load information of each scheduler in the scheduler set.
[0113] A construction module is used to construct a work node set corresponding to the scheduler according to the load information of each scheduler, wherein the work node set includes one or more work nodes for executing tasks.
[0114] According to an embodiment of the present application, the construction module includes: a monitoring unit and a generating unit.
[0115] The monitoring unit is used to monitor the load information of each scheduler in the scheduler set in different time periods.
[0116] The generating unit is used to generate a set of working nodes corresponding to each scheduler in different time periods according to the load information of each scheduler in different time periods.
[0117] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
[0118] Figure 8 A block diagram of a computer system suitable for implementing a task scheduling method according to an embodiment of the present disclosure is schematically shown.
[0119] like Figure 8 , is a block diagram of an electronic device for executing the method of an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.
[0120] like Figure 8 As shown, the electronic device 800 includes: one or more processors 801, a memory 802, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 801 is taken as an example.
[0121] The memory 802 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the method provided in the present application.
[0122] The memory 802 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present application (for example, the attached Figure 7The processor 801 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 802, that is, implements the method in the above method embodiment.
[0123] The memory 802 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the use of the electronic device according to the above method, etc. In addition, the memory 802 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 802 may optionally include a memory remotely arranged relative to the processor 801, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0124] The electronic device may further include: an input device 803 and an output device 804. The processor 801, the memory 802, the input device 803 and the output device 804 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.
[0125] The input device 803 can receive input digital or character information, and generate key signal input related to user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 804 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.
[0126] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0127] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.
[0128] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0129] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0130] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.
[0131] According to the technical solution of the embodiment of the present application, a unified scheduling entrance is used to receive task execution requests, and a target scheduler for scheduling the task to be executed is determined from a set of schedulers according to the task type of the task to be executed, thereby realizing unified management of different schedulers and solving the technical problem that as the number of scheduling systems increases, the number of schedulers is also increased, and the operation and maintenance costs increase linearly, which is not conducive to development and operation and maintenance. Unified operation and maintenance is achieved, development and operation and maintenance costs are reduced, and the requirements for quickly meeting scheduling needs are realized, which releases the manpower of the business system to develop the scheduling system, thereby developing the technical effect of the business system more quickly and more focused.
[0132] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.
[0133] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.
Claims
1. A task scheduling method, comprising: Receiving task execution requests from different clients using different programming languages based on a unified scheduling portal, wherein the task execution requests carry a task type of the task to be executed; In response to a task execution request from a client, a target scheduler capable of parsing the task to be executed corresponding to the programming language adopted by the client and scheduling the task to be executed is determined from a set of schedulers according to the task type of the task to be executed, wherein the target scheduler is used to assign the task to be executed to an idle working node for execution, and the set of schedulers includes multiple schedulers of different types; Allocating the task to be executed to the target scheduler, The method further comprises: Analyzing the load information of each scheduler in the scheduler set in different time periods based on big data analysis; and According to the obtained load information of each of the schedulers in different time periods, a set of working nodes corresponding to each of the schedulers in different time periods is generated.
2. The method according to claim 1, further comprising: Before responding to the task execution request, determining the scheduling authority of the user who sent the task execution request, wherein the scheduling authority of the user includes being able to request a task of a scheduling target type; and In a case where the target type is the same as the task type of the task to be executed, respond to the task execution request.
3. The method according to claim 2, further comprising: In a case where the target type is different from the task type of the task to be executed, the task execution request is rejected.
4. The method according to claim 1, wherein: The task execution request also carries the priority of the task to be executed, and the method further includes: In response to the task execution request, assigning the task to be executed to a task queue that matches the priority of the task to be executed according to the priority of the task to be executed, wherein tasks in the same task queue have the same priority; and Schedule tasks in different task queues in order of priority.
5. The method according to claim 1, further comprising: Before allocating the to-be-executed task to the target scheduler, obtaining load information of the target scheduler; as well as When the load information of the target scheduler meets a preset condition, the to-be-executed task is allocated to the target scheduler.
6. The method according to claim 5, further comprising: When the load information of the target scheduler does not meet the preset condition, saving the task to be executed in a task list; Until the load information of the target scheduler meets the preset condition, the to-be-executed task is allocated to the target scheduler.
7. The method according to claim 6, further comprising: When the load information of the target scheduler does not meet the preset condition, stop allocating tasks to the target scheduler.
8. The method according to claim 1, wherein the set of working nodes includes one or more working nodes for executing tasks.
9. The method according to claim 1, wherein: The task types include: online real-time tasks and offline scheduled tasks.
10. A task scheduling device, comprising: A receiving module, configured to receive task execution requests from different clients using different programming languages based on a unified scheduling portal, wherein the task execution request carries a task type of a task to be executed; A first response module, configured to respond to a task execution request from a client, determine, according to the task type of the task to be executed, from a set of schedulers a target scheduler that can parse the task to be executed corresponding to the programming language used by the client and schedule the task to be executed, wherein the target scheduler is configured to allocate the task to be executed to an idle working node for execution, and the set of schedulers includes a plurality of schedulers of different types; an allocating module, configured to allocate the to-be-executed task to the target scheduler; and A construction module is used to analyze the load information of each scheduler in the scheduler set in different time periods based on big data analysis; and generate a set of working nodes corresponding to each scheduler in different time periods according to the obtained load information of each scheduler in different time periods.
11. The apparatus according to claim 10, further comprising: A first determination module is used to determine the scheduling authority of the user who sent the task execution request before responding to the task execution request, wherein the scheduling authority of the user includes the ability to request a task of a scheduling target type; The first response module is used to respond to the task execution request when the target type is the same as the task type of the task to be executed.
12. The device according to claim 10, wherein: The task execution request also carries the priority of the task to be executed, and the device further includes: A second response module is used to respond to the task execution request and allocate the task to be executed to a task queue that matches the priority of the task to be executed according to the priority of the task to be executed, wherein the tasks in the same task queue have the same priority; and The scheduling module is used to schedule tasks in different task queues in order of priority.
13. The apparatus according to claim 10, further comprising: A first acquisition module, used for acquiring load information of the target scheduler before allocating the to-be-executed task to the target scheduler; The allocation module is used to allocate the to-be-executed task to the target scheduler when the load information of the target scheduler meets a preset condition.
14. The apparatus according to claim 13, further comprising: A saving module, configured to save the to-be-executed task in a task list when the load information of the target scheduler does not meet the preset condition; The allocation module is used to allocate the to-be-executed task to the target scheduler until the load information of the target scheduler meets the preset condition.
15. The device according to claim 10, wherein: The working node set includes one or more working nodes for executing tasks.
16. A task scheduling system, comprising: The task scheduling device according to any one of claims 10 to 15; as well as A scheduler set, wherein the scheduler set includes a plurality of schedulers of different types.
17. An electronic device, characterized in that: include: At least one processor: and a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.
18. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 9.
