Task scheduling method and device, electronic equipment and storage medium

Abstract

The invention relates to a task scheduling method and device, electronic equipment and a storage medium, and relates to the field of task scheduling. The method comprises the steps of obtaining a target task and configuration information of the target task; obtaining resource state information of each actuator in the actuator cluster in real time; according to the configuration information and the resource state information, a load balancing strategy is utilized to allocate a target actuator for the target task; in the process that the target task waits for the execution of the target executor, in response to the detection that the target executor meets a preset condition, determining whether an idle executor exists in the executor cluster or not; and in response to determining that the idle executors exist in the executor cluster, allocating the target task to the idle executors. The system performance can be improved, and the resource utilization efficiency can be improved.

Description

Technical Field

[0001] This application relates to the technical field of task scheduling, and particularly to a task scheduling method, apparatus, electronic device, and storage medium. Background Art

[0002] With the rapid development of Internet technology, distributed systems have gradually become the main platform for processing large-scale data and complex tasks. However, in distributed systems, task scheduling faces many challenges, such as unbalanced node loads, task congestion, resource waste, and node failures. Existing task scheduling methods often fail to fully consider the specific situation of tasks and the real-time status of nodes, resulting in low system performance and inefficient resource utilization. Summary of the Invention

[0003] Embodiments of the present disclosure provide a task scheduling method, apparatus, electronic device, and storage medium, aiming to solve the deficiencies of existing task scheduling methods.

[0004] In a first aspect, embodiments of the present disclosure provide a task scheduling method, including: obtaining a target task and configuration information of the target task; obtaining in real time resource status information of each executor in an executor cluster; according to the configuration information and the resource status information, using a load balancing strategy to allocate a target executor for the target task; during the process of the target task waiting for the target executor to execute, in response to detecting that the target executor meets a preset condition, determining whether there is an idle executor in the executor cluster; in response to determining that there is an idle executor in the executor cluster, allocating the target task to the idle executor.

[0005] In a second aspect, embodiments of the present disclosure provide a task scheduling apparatus, including: a first obtaining unit configured to obtain a target task and configuration information of the target task; a second obtaining unit configured to obtain in real time resource status information of each executor in an executor cluster; a task allocation unit configured to, according to the configuration information and the resource status information, use a load balancing strategy to allocate a target executor for the target task; an idle detection unit configured to, during the process of the target task waiting for the target executor to execute, in response to detecting that the target executor meets a preset condition, determine whether there is an idle executor in the executor cluster; a task scheduling unit configured to, in response to determining that there is an idle executor in the executor cluster, allocate the target task to the idle executor.

[0006] In a third aspect, embodiments of the present disclosure provide an electronic device, including a memory, a processor, a bus, and a computer program stored on the memory and executable on the processor, where when the processor executes the computer program, the task scheduling method described in the first aspect is implemented.

[0007] Fourthly, an embodiment of the present disclosure provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the task scheduling method described in the first aspect is implemented.

[0008] By applying the technical solution of the present disclosure, the status information of each actuator can be monitored in real time, combined with the configuration information of the target task, and the target actuator can be determined by using the load balancing strategy. When it is determined that the target actuator cannot execute the target task, the target task is assigned to an idle actuator, so that dynamic task scheduling can be realized, the system performance is improved, and the resource utilization efficiency is increased.

[0009] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings are used to better understand the solution and do not constitute a limitation to the present disclosure. Among them: Figure 1 FIG. is an exemplary system architecture diagram to which an embodiment of the task scheduling method of the present disclosure can be applied; Figure 2 FIG. is a flowchart of an embodiment of the task scheduling method of the present disclosure; Figure 3 FIG. is a flowchart of another embodiment of the task scheduling method of the present disclosure; Figure 4 FIG. is a structural diagram of an embodiment of the task scheduling device of the present disclosure; Figure 5 FIG. is a structural diagram of an embodiment of the electronic device of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0011] It should be noted that the following detailed description is exemplary and is intended to provide further description of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those of ordinary skill in the technical field to which the present disclosure belongs.

[0012] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0013] In the case of no conflict, the embodiments in the present disclosure and the features in the embodiments may be combined with each other.

[0014] To make the technical solutions and advantages of the present disclosure more clear and understandable, the following further describes the present disclosure in detail with reference to the accompanying drawings and specific embodiments.

[0015] Figure 1 An exemplary system architecture 100 is shown, which can apply the embodiments of the task scheduling method or load balancing device of the present disclosure.

[0016] As Figure 1 shown, the system architecture 100 may include a task management device 101, a load balancing device 102, a resource monitoring and management device 103, a network 104, and an executor cluster 105. The network 104 is used to provide a medium for communication links between the task management device 101, the load balancing device 102, the resource monitoring and management device 103, and the node cluster 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.

[0017] The task management device 101 can monitor each task, determine the status and configuration information of each task. Here, the tasks can include various types of tasks, for example, they can include IO-intensive tasks, CPU-intensive and memory-intensive tasks. Among them, IO-intensive tasks mainly involve a large number of input / output operations, such as network requests, file reading and writing, database access, etc. CPU-intensive tasks refer to those tasks that require a large amount of CPU computing resources. When these tasks are executed, they almost completely occupy the CPU resources, so that other tasks cannot be processed by the CPU in a timely manner. Memory-intensive tasks mainly rely on a large amount of memory resources to execute tasks, and usually involve large-scale data processing. The task management device 101 can send the status and configuration information of each task to the load balancing device 102.

[0018] The load balancing device 102 can obtain the status and configuration information of each task from the task management device 101, and can also obtain the resource information of each executor in the executor cluster 105 from the resource monitoring and management device 103. Then, various load balancing algorithms can be used to schedule each task to each executor for task execution.

[0019] The resource monitoring and management device 103 can monitor the resource status information of each executor in the executor cluster 105 in real time to determine the resource occupancy information and resource idle information of each executor. And send the above resource status information to the load balancing device 102.

[0020] The actuator cluster 105 may include multiple actuators, and each actuator may be a physical resource or a virtual resource. The resources in each actuator can be used to execute each task. Each actuator can send resource status information to the resource monitoring and management device 103 in real time.

[0021] In some specific practices, a resource monitoring agent may be set in each actuator in the actuator cluster 105. The resource monitoring agent can monitor the resource status information of the actuator in real time and send it to the resource monitoring and management device 103.

[0022] It should be noted that the task scheduling method provided by the embodiments of the present disclosure is generally executed by the load balancing device 102. Correspondingly, the load balancing device is generally set in the load balancing device 102.

[0023] It should be understood that Figure 1 the number of task management devices, load balancing devices, resource monitoring and management devices, networks, and actuators in the actuator cluster in

[0024] Figure 2 shows a flow 200 of an embodiment of the task scheduling method of the present disclosure. As Figure 2 shown, the task scheduling method of this embodiment may include the following steps: Step 201, obtain a target task and configuration information of the target task.

[0025] In this embodiment, the execution subject of the task scheduling method (such as Figure 1 the load balancing device 102 shown) can obtain the target task and the configuration information of the target task. Here, the target task may include various types of tasks, such as IO-intensive tasks, CPU-intensive and memory-intensive tasks. At the same time, the configuration information of the target task can also be obtained. The configuration information may include the status, dependency conditions, priority, etc. of the target task. The status of the target task may include executed, unexecuted, executing, etc. The dependency condition may be a condition that needs to be satisfied when the target task is executed. The priority is used to indicate the priority level of the task.

[0026] In some specific practices, when the target task does not include preset partitioning parameters, the configuration information may also include partitioning parameters. The partitioning parameters may include time, location, identifier, etc. For example, if the target task is to obtain data from X hour on X day to Y hour on Y day, the target task can be divided into multiple subtasks by dividing the time period from X hour on X day to Y hour on Y day into multiple time segments. If the target task is to obtain data in data table identifiers O to M, the target task can be divided into multiple subtasks by dividing O to M into identifier segments. If the target task does not include these preset partitioning parameters, the task partitioning information can be set in the configuration information. The task partitioning information can exist in the form of annotations or code. The target task can be partitioned according to the above task partitioning information.

[0027] Step 202, obtain the resource status information of each executor in the executor cluster in real time.

[0028] In this embodiment, the execution entity may also obtain the resource status information of each executor in the executor cluster in real time. Specifically, the execution entity may obtain the resource status information of each executor in the executor cluster from the resource status management device in the form of a heartbeat. The executor cluster may include multiple executors. Each executor may be used to execute various types of tasks. Each executor may include hardware resources and software resources. The resource status information may represent the occupied amount, free amount, etc. of the resources.

[0029] Step 203, according to the configuration information and the resource status information, use the load balancing strategy to allocate a target executor for the target task.

[0030] The execution entity may further allocate a target executor for the target task according to the configuration information of the target task and the resource status information of each executor, using the load balancing strategy. Specifically, according to the resource demand of the target task in the configuration information and the resource free amount of each executor in the resource status information, multiple executors capable of executing the target task can be determined. Then, further combining the number of tasks to be executed by each executor, the executor with the smaller number of tasks can be selected as the target executor. Or, the executor with the shortest execution time can be selected as the target executor from each executor.

[0031] Step 204, during the process of the target task waiting for the target executor to execute, in response to detecting that the target executor meets the preset conditions, determine whether there is an idle executor in the executor cluster.

[0032] In this embodiment, during the process that the target task waits for the target executor to execute, if it is detected that the target executor cannot execute the target task, the target task can be assigned to other executors. Specifically, the target task can be added to the task buffer queue of the target executor. The target executor can sequentially retrieve tasks from the task buffer queue for execution. During the process that the target task waits in the task buffer queue, it can be detected whether the target executor meets a preset condition. If it does, it is considered that the target executor cannot execute the target task, and at this time, the target task can be assigned to other executors. Here, the preset condition can include but is not limited to: downtime, resource occupancy greater than a preset threshold, task processing duration greater than a preset duration, and so on.

[0033] After determining that the target executor meets the preset condition, it can be determined whether there is an idle executor in the executor cluster. An idle executor refers to an available executor that is not currently executing any tasks.

[0034] Step 205, in response to determining that there is an idle executor in the executor cluster, assign the target task to the idle executor.

[0035] If there is an idle executor in the executor cluster, a certain task can be assigned to the idle executor. In this way, the idle executor can quickly execute the target task, thereby improving the execution efficiency of the target task.

[0036] The task scheduling method provided in the above embodiments of the present disclosure can monitor the status information of each executor in real time, combine the above status information with the configuration information of the target task, use the load balancing strategy to determine the target executor, and when it is determined that the target executor cannot execute the target task, assign the target task to other idle executors, thereby enabling dynamic task scheduling, improving system performance, and enhancing resource utilization efficiency.

[0037] Continue to refer to Figure 3 , which shows the flow 300 of another embodiment of the task scheduling method according to the present disclosure. As Figure 3 shown, the method in this embodiment can include the following steps: Step 301, obtain the target task and the configuration information of the target task.

[0038] Step 302, obtain the resource status information of each executor in the executor cluster in real time.

[0039] Step 303, according to the configuration information, divide the target task into at least two subtasks; according to the resource status information, use the load balancing strategy to assign at least two executors to the at least two subtasks.

[0040] In this embodiment, to improve the execution efficiency of the target task, the target task may be divided into at least two subtasks according to the configuration information of the target task. Here, the configuration information may include division parameters, and the target task can be divided according to the above division parameters.

[0041] After obtaining at least two subtasks through division, the executors can be allocated to each subtask according to the resource status information of each executor and using the load balancing strategy.

[0042] Step 304: Determine the priority of the target task according to the configuration information; according to the priority of the target task, add the target task to the task buffer queue of the target available executor, so that the target available executor sequentially retrieves tasks from the task buffer queue for execution.

[0043] After determining the target executor for the target task, the target task can be added to the task buffer queue of the target executor. Specifically, the priority of the target task can be determined from the configuration information. Then, the target task is added to the above task buffer queue according to the priority. It can be understood that the higher the priority, the more forward the position in the task buffer queue.

[0044] Step 305: In response to determining that the available resources of the target executor are less than the preset threshold, or the waiting duration of the target task in the task buffer queue is greater than the preset duration, determine that the target executor meets the preset conditions.

[0045] In this embodiment, if the available resources of the target executor are less than the preset threshold, or the waiting duration of the target task in the task buffer queue is greater than the preset duration, it is considered that the target executor meets the preset conditions. Here, the preset threshold can be dynamic, and it can be determined by the required resources of the target task, or by the required resources of each task in the task buffer queue of the target executor.

[0046] In addition, if the waiting duration of the target task in the task buffer queue is greater than the preset duration, it is considered that the execution duration of the target task will time out, and then the executor for the target task needs to be replaced, that is, the target executor meets the preset conditions.

[0047] Step 306: During the process of the target task waiting for the target executor to execute, in response to detecting that the target executor meets the preset conditions, determine whether there is an idle executor in the executor cluster.

[0048] Step 307: In response to determining that there is an idle executor in the executor cluster, allocate the target task to the idle executor.

[0049] In some alternative implementation manners of this embodiment, when allocating a target task to an idle executor, other tasks to be allocated to this idle executor can be cancelled first, and the target task is added to the task buffer queue of the idle executor.

[0050] In some alternative implementation manners of this embodiment, before the idle executor executes the target task, the execution status of the dependent tasks of the target task can be determined first. The dependent tasks here refer to those tasks whose execution results are required for the execution of the target task, and these other tasks can be called dependent tasks. If the dependent tasks have not been executed, the target task can be added to the task buffer queue of the idle executor, and the idle executor is controlled to execute the above-mentioned dependent tasks.

[0051] Step 308, in response to determining that there is no idle executor in the executor cluster, control the target task to wait in the task buffer queue of the target executor.

[0052] In some alternative implementation manners of this embodiment, the execution entity can determine whether an executor is available by means of heartbeat. For example, a message is sent to the executor every 60 seconds. If a reply message from the executor is received within a preset duration, it is considered that the executor is available. If no reply message is received within the preset duration, it is considered that the executor is unavailable.

[0053] After determining that the executor is unavailable, the executor can be controlled to pause the execution of all tasks. The unexecuted tasks are allocated to other executors by using a load balancing strategy. The other executors here can be idle executors or non-idle executors.

[0054] Save the execution results of the tasks being executed, and send the execution results and the tasks being executed to the idle executor together, so that the idle executor can immediately execute these tasks.

[0055] The task scheduling method provided by the above embodiments of the present disclosure, by dynamically adjusting the task distribution, ensures that high-load tasks can be timely migrated to low-load nodes, effectively avoids node overload and performance bottlenecks, and thus significantly improves the overall processing capacity and response time of the system; fully considers the real-time load and health status of nodes, ensures that tasks are allocated to the most suitable nodes for execution, avoids resource idleness and waste, and realizes the efficient utilization of resources; introduces a fault tolerance mechanism and an elastic task allocation algorithm, so that the system can quickly respond when facing emergencies such as node failures and network anomalies, and automatically transfer tasks to other healthy nodes to continue execution, ensuring the stability and reliability of the system.

[0056] Further referring to Figure 4 As an implementation of the methods shown in the above figures, the present disclosure provides an embodiment of a task scheduling device. This device embodiment is related toFigure 2 The method embodiments shown correspond to this, and this device can be specifically applied to various electronic devices.

[0057] As Figure 4 As shown, the task scheduling device 400 in this embodiment includes: a first acquisition unit 401, a second acquisition unit 402, a task allocation unit 403, an idle detection unit 404, and a task scheduling unit 405.

[0058] The first acquisition unit 401 is configured to acquire a target task and configuration information of the target task.

[0059] The second acquisition unit 402 is configured to acquire resource status information of each executor in the executor cluster in real time.

[0060] The task allocation unit 403 is configured to allocate a target executor for the target task according to the configuration information and the resource status information by using a load balancing policy.

[0061] The idle detection unit 404 is configured to determine whether there is an idle executor in the executor cluster in response to detecting that the target executor meets a preset condition during the process of the target task waiting for the target executor to execute.

[0062] The task scheduling unit 405 is configured to allocate the target task to the idle executor in response to determining that there is an idle executor in the executor cluster.

[0063] In addition, in the technical solution of this application, an electronic device is also proposed.

[0064] Figure 5 The structure diagram of an electronic device provided by an embodiment of the present disclosure is shown.

[0065] As Figure 5 As shown, the electronic device may include a processor 501, a memory 502, a bus 503, and a computer program stored in the memory 502 and executable on the processor 501. Among them, the processor 501 and the memory 502 communicate with each other through the bus 503. When the processor 501 executes the computer program, the steps of the above method are implemented, for example, including: acquiring a target task and configuration information of the target task; acquiring resource status information of each executor in the executor cluster in real time; allocating a target executor for the target task according to the configuration information and the resource status information by using a load balancing policy; determining whether there is an idle executor in the executor cluster in response to detecting that the target executor meets a preset condition during the process of the target task waiting for the target executor to execute; allocating the target task to the idle executor in response to determining that there is an idle executor in the executor cluster.

[0066] In addition, an embodiment of the present disclosure also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented, for example, including: obtaining a target task and configuration information of the target task; obtaining resource status information of each executor in the executor cluster in real time; according to the configuration information and the resource status information, using a load balancing strategy to allocate a target executor for the target task; during the process of waiting for the target executor to execute the target task, in response to detecting that the target executor meets a preset condition, determining whether there is an idle executor in the executor cluster; in response to determining that there is an idle executor in the executor cluster, allocating the target task to the idle executor.

[0067] In summary, in the technical solution of the present disclosure, the status information of each available executor can be monitored in real time, combined with the configuration information of the target task, and a target available executor is determined using a load balancing strategy. When it is determined that the target available executor cannot execute the target task, the target task is allocated to other idle executors, so that dynamic task scheduling can be achieved, the system performance is improved, and the resource utilization efficiency is increased.

[0068] The above are only the preferred embodiments of the present disclosure and are not intended to limit the present disclosure. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present disclosure shall be included within the scope of protection of the present disclosure.

Claims

1. A task scheduling method, comprising: Obtaining a target task and configuration information of the target task; Obtain resource status information of each executor in the executor cluster in real time; According to the configuration information and the resource status information, using a load balancing strategy, allocating a target executor to the target task; In the process of the target task waiting for the target executor to be executed, in response to detecting that the target executor meets a preset condition, determining whether there is an idle executor in the executor cluster; In response to determining that there are idle executors in the executor cluster, the target task is assigned to the idle executor.

2. The method according to claim 1, wherein: The method of allocating a target executor to the target task by using a load balancing strategy according to the configuration information and the resource status information includes: According to the configuration information, the target task is divided into at least two subtasks; According to the resource status information, a load balancing strategy is used to allocate a target executor to the at least two subtasks.

3. The method according to claim 1, wherein: The method further comprises: Determining the priority of the target task according to the configuration information; According to the priority of the target task, the target task is added to the task buffer queue of the target executor, so that the target executor sequentially takes tasks from the task buffer queue for execution.

4. The method according to claim 3, wherein: The method further comprises: In response to determining that the available resources of the target executor are less than a preset threshold, or the waiting time of the target task in the task buffer queue is greater than a preset time, it is determined that the target executor meets the preset condition.

5. The method according to claim 3, wherein: The method further comprises: In response to determining that there is no idle executor in the executor cluster, the target task is controlled to wait in the task buffer queue of the target executor.

6. The method according to claim 1, wherein: The allocating the target task to the idle executor comprises: Other tasks assigned to the idle executor are canceled, and the idle executor is controlled to execute the target task.

7. The method according to claim 6, wherein: The controlling the idle executor to execute the target task includes: Determining the execution status of dependent tasks of the target task; In response to determining that the dependent task is not executed, the target task is placed in a task buffer queue of the idle executor, and the idle executor is controlled to execute the dependent task.

8. A task scheduling device, comprising: A first acquisition unit is configured to acquire a target task and configuration information of the target task; A second acquisition unit is configured to acquire resource status information of each executor in the executor cluster in real time; A task allocation unit is configured to allocate a target executor to the target task using a load balancing strategy according to the configuration information and the resource status information; An idle detection unit is configured to determine whether there is an idle executor in the executor cluster in response to detecting that the target executor meets a preset condition during the process in which the target task waits for the target executor to execute; The task scheduling unit is configured to allocate the target task to the idle executor in response to determining that there is an idle executor in the executor cluster.

9. An electronic device comprising a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the task scheduling method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the task scheduling method according to any one of claims 1 to 7 is implemented.

Images