Task scheduling method and device, electronic equipment and computer program product

Abstract

The invention discloses a task scheduling method and device, electronic equipment and a computer program product. The task scheduling method comprises the steps of obtaining task information of a to-be-scheduled task; obtaining historical resource use data of the to-be-scheduled task according to the task information of the to-be-scheduled task, wherein the historical resource use data is obtained based on task model monitoring; acquiring current resource use data of the plurality of task execution nodes; and scheduling the to-be-scheduled task to a target task execution node according to the historical resource use data of the to-be-scheduled task and the current resource use data of the plurality of task execution nodes, the target task execution node being one of the plurality of task execution nodes. According to the method, the task is abstracted into the task model, the execution process of the task is clarified, the historical resource use condition of the task can be accurately monitored and calculated based on the task model, historical experience data is used as a task scheduling basis, the accuracy and rationality of task scheduling are improved, and the resource allocation rationality and the resource utilization rate are improved.

Description

Technical Field

[0001] The present application relates to the technical field of task scheduling, and in particular to a task scheduling method, device, electronic equipment, and computer program product. Background Art

[0002] As technology continues to develop, more and more data is generated, and more and more data needs to be processed. The financial field represented by banks inevitably chooses to apply technologies such as big data, cloud computing and blockchain. In this transformation process, big data technology also plays an important role. However, there are more and more tasks with large amounts of data, which puts higher and higher requirements on batch task scheduling centers. A good scheduling method can give full play to machine resources, reduce resource waste and reduce failure rates.

[0003] Existing task scheduling technologies are either not lightweight enough or cannot maximize the resource utilization of each execution node. In addition, task scheduling is completely predictive and the scheduling effect needs to be improved. Summary of the invention

[0004] The embodiments of the present application provide a task scheduling method, device, electronic device, and computer program product to improve task scheduling effects, and improve rationality of resource allocation and resource utilization.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a task scheduling method, the task scheduling method comprising:

[0007] Get the task information of the task to be scheduled;

[0008] Acquire historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring;

[0009] Get the current resource usage data of multiple task execution nodes;

[0010] According to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes, the task to be scheduled is scheduled to a target task execution node, and the target task execution node is one of the multiple task execution nodes.

[0011] Optionally, the task to be scheduled is a multi-threaded task, and acquiring historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled includes:

[0012] Recording the task submission information and task running information of the multi-threaded task;

[0013] Constructing a task model of the multi-threaded task according to the task submission information and task running information of the multi-threaded task;

[0014] Monitoring resource usage data of the multi-threaded task using the task model of the multi-threaded task;

[0015] The historical resource usage data of the multi-threaded task is determined according to the monitored resource usage data of the multi-threaded task.

[0016] Optionally, monitoring the resource usage data of the multi-threaded task by using the task model of the multi-threaded task includes:

[0017] Monitoring resource usage data of each thread in the multi-threaded task according to the task model of the multi-threaded task;

[0018] The resource usage data of the multi-threaded task is determined according to the resource usage data of each thread.

[0019] Optionally, the task scheduling method further includes:

[0020] Receive heartbeat data sent by each task execution node, wherein the heartbeat data includes the current state of the task execution node and current resource usage data;

[0021] Store the current status of each task execution node and the current resource usage data in the database;

[0022] The obtaining of current resource usage data of multiple task execution nodes includes:

[0023] The current resource usage data of the plurality of task execution nodes is obtained from the database.

[0024] Optionally, scheduling the task to be scheduled to a target task execution node according to the historical resource usage data of the task to be scheduled and the current resource usage data of the plurality of task execution nodes comprises:

[0025] Calculating estimated resource usage data of the task to be scheduled based on historical resource usage data of the task to be scheduled and a preset safety factor;

[0026] The task to be scheduled is scheduled to a target task execution node according to the estimated resource usage data of the task to be scheduled and the current resource usage data of the plurality of task execution nodes.

[0027] Optionally, scheduling the task to be scheduled to a target task execution node according to the estimated resource usage data of the task to be scheduled and the current resource usage data of the plurality of task execution nodes comprises:

[0028] Determine current resource remaining data of each of the task execution nodes according to current resource usage data of each of the task execution nodes;

[0029] Calculating the difference between the current resource remaining data of each of the task execution nodes and the estimated resource usage data of the task to be scheduled;

[0030] Determining the target task execution node among the plurality of task execution nodes according to the difference;

[0031] Schedule the task to be scheduled to the target task execution node.

[0032] Optionally, the task scheduling method further includes:

[0033] In the case where the historical resource usage data of the task to be scheduled cannot be obtained, determining the task execution node with the most current resource remaining data according to the current resource usage data of the plurality of task execution nodes;

[0034] The task to be scheduled is scheduled to the task execution node with the most remaining resource data.

[0035] In a second aspect, an embodiment of the present application further provides a task scheduling device, the task scheduling device comprising:

[0036] A first acquisition unit, used to acquire task information of the task to be scheduled;

[0037] A second acquisition unit is used to acquire historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring;

[0038] A third acquisition unit is used to acquire current resource usage data of multiple task execution nodes;

[0039] The scheduling unit is used to schedule the task to be scheduled to a target task execution node according to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes. The target task execution node is one of the multiple task execution nodes.

[0040] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0041] A processor; and a memory arranged to store computer executable instructions, which, when executed, cause the processor to perform any of the aforementioned task scheduling methods.

[0042] In a fourth aspect, an embodiment of the present application further provides a computer program product, including a computer program or instructions, which implement any of the aforementioned task scheduling methods when executed by a processor.

[0043] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the task scheduling method of the embodiments of the present application first obtains the task information of the task to be scheduled; then obtains the historical resource usage data of the task to be scheduled based on the task information of the task to be scheduled, and the historical resource usage data is obtained based on the task model monitoring; then obtains the current resource usage data of multiple task execution nodes; finally, according to the historical resource usage data of the task to be scheduled and the current resource usage data of multiple task execution nodes, the task to be scheduled is scheduled to the target task execution node, and the target task execution node is one of the multiple task execution nodes. The task scheduling method of the embodiments of the present application abstracts the task into a task model, clarifies the execution process of the task, can accurately monitor and calculate the historical resource usage of the task based on the task model, uses the historical experience data as the basis for task scheduling, improves the accuracy and rationality of task scheduling, and improves the rationality of resource allocation and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] Figure 1 A flowchart of a task scheduling method in an embodiment of the present application;

[0046] Figure 2 This is a schematic diagram of an overall task scheduling process in an embodiment of the present application;

[0047] Figure 3 This is a structural diagram of a task model in an embodiment of the present application;

[0048] Figure 4 This is a structural diagram of a task scheduling device in an embodiment of the present application;

[0049] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0051] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0052] At present, the main distributed task scheduling centers are as follows:

[0053] 1) Apache DolphinScheduler: Aims to solve complex big data task dependencies and provide applications with data and relationships between various OPS orchestrations.

[0054] 2) Quartz Job Scheduler: It is written entirely in Java and designed for J2SE and J2EE applications. It provides great flexibility without sacrificing simplicity. It can be used to create simple or complex schedules for executing a job.

[0055] 3) Elastic-job: supports two ways to configure tasks: JAVAAPI and Spring configuration. Zookeeper is used to implement the registration center for registering, controlling and coordinating distributed task execution.

[0056] 4) XXL-JOB: It is a distributed task scheduling platform, whose core design goals are rapid development, simple learning, lightweight and easy to expand.

[0057] However, Apache DolphinScheduler is too heavy and not lightweight enough. The load between the task execution nodes of Quartz Job Scheduler is unbalanced. Elastic-job is based on cluster architecture, which requires multiple scheduling servers to continue to participate. The distributed system is complex and cannot achieve lightweight task scheduling. XXL-JOB is a lightweight scheduling platform, but its scheduling method is too simple and easily causes resource waste.

[0058] In summary, existing scheduling tools are either not lightweight enough or cannot maximize the utilization of execution node resources. In addition, all task scheduling is based on predictions and lacks a reasonable basis, and the scheduling effect needs to be improved.

[0059] Based on this, the embodiment of the present application provides a task scheduling method, such as Figure 1As shown, a flowchart of a task scheduling method in an embodiment of the present application is provided, and the task scheduling method at least includes the following steps S110 to S140:

[0060] Step S110, obtaining task information of the task to be scheduled.

[0061] Combination Figure 2 , provides a schematic diagram of the overall task scheduling process in an embodiment of the present application. When performing task scheduling, it is necessary to first obtain the task information of the task to be scheduled, and the task information may include basic information such as task ID, task name, and task type.

[0062] Step S120, acquiring historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring.

[0063] Based on the above task information, historical resource usage data corresponding to the task to be scheduled is further obtained from the maintained task database. The historical resource usage data refers to the resources used or occupied by the task to be scheduled when it was executed in the past, and may include, for example, CPU usage, memory usage, disk usage, etc.

[0064] The embodiment of the present application can abstract a task model for each task. The task model describes the execution process of each task and the dependency relationship between subtasks, etc. Based on the task model, the resource consumption generated by each task during the execution process can be monitored and calculated. Through continuous monitoring and analysis, the historical resource usage data of each task can be obtained. The historical resource usage data of the task can be dynamically updated as the task is continuously executed to reflect the recent resource usage characteristics of the task.

[0065] Step S130, obtaining current resource usage data of multiple task execution nodes.

[0066] The ultimate goal of task scheduling is to assign the scheduled tasks to a task execution node for execution. A task execution node may execute more than one task, but the resources of each task execution node are limited, and the number of tasks that can be executed simultaneously is also limited. Therefore, before assigning task execution nodes to the scheduled tasks, it is also necessary to obtain the current resource usage of all task execution nodes, such as CPU usage, memory usage, disk usage, traffic status, etc.

[0067] Step S140, scheduling the task to be scheduled to a target task execution node according to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes, wherein the target task execution node is one of the multiple task execution nodes.

[0068] The historical resource usage data of the tasks to be scheduled reflects the resource usage characteristics of the tasks to be scheduled in the past period of time, which can be used as the basis for predicting the resource usage of this scheduling. Further combined with the current resource usage of each task execution node, the tasks to be scheduled can be allocated to the target task execution nodes that meet the resource utilization and rationality requirements.

[0069] The task scheduling method of the embodiment of the present application abstracts the task into a task model, clarifies the execution process of the task, and can accurately monitor and calculate the historical resource usage of the task based on the task model. It uses historical experience data as the basis for task scheduling, improves the accuracy and rationality of task scheduling, and improves the rationality of resource allocation and resource utilization.

[0070] In some embodiments of the present application, the task to be scheduled is a multi-threaded task, and obtaining the historical resource usage data of the task to be scheduled based on the task information of the task to be scheduled includes: recording the task submission information and task running information of the multi-threaded task; building a task model of the multi-threaded task based on the task submission information and task running information of the multi-threaded task; monitoring the resource usage data of the multi-threaded task using the task model of the multi-threaded task; and determining the historical resource usage data of the multi-threaded task based on the monitored resource usage data of the multi-threaded task.

[0071] Before obtaining the historical resource usage data of the task to be scheduled, it is necessary to build and maintain a task model for each task. The task type here can be a multi-threaded task. For multi-threaded tasks, a task model of multi-threaded tasks is built based on DAG (Directed Acyclic Graph), such as Figure 3 As shown, a schematic diagram of the structure of a task model in an embodiment of the present application is provided. The main purpose of constructing a task model is to clearly identify the threads contained in each task and the relationship between the threads, so as to collect the resource usage of each task.

[0072] refer to Figure 3 , each node represents a thread, which can contain information such as thread ID, thread stack, task name, etc. The directed edge represents the relationship between threads, and the direction of the edge is from the parent thread to the child thread. Therefore, the task model can clearly represent all threads of a task and the relationship between threads, but the key is how to find all threads of a task and abstract them into this model.

[0073] The following is a detailed description of the specific steps for abstracting tasks into task models:

[0074] 1) The premise of multi-threaded task running is to submit the task to the thread pool. Therefore, the original thread pool of the task execution node needs to be transformed. First, it is necessary to record which thread each task is submitted by, and second, it is necessary to record which specific thread in the thread pool runs each submitted task, as well as the start time of the task running and other information. This information will be continuously persisted in the database.

[0075] 2) For a task, when it is called up for execution by the scheduling center, its main thread of execution can be obtained, that is, its main thread ID and its thread group can be obtained.

[0076] 3) When the main thread of the task is running, the task execution node will start the daemon thread to continue running. The daemon thread searches for all task running information submitted by it in the thread through the main thread ID.

[0077] (4) The relationship between the threads in these thread pools and the main thread ID can then be mapped into a task model.

[0078] Of course, it should be noted that the task model of the present application is also applicable to single-threaded tasks, that is, the task model can also be constructed for single-threaded tasks and resource usage data can be calculated.

[0079] Based on the above-abstracted task model, the task execution node will also periodically monitor the resource usage of the multi-threaded tasks corresponding to the task model, and finally calculate the historical resource usage data of the multi-threaded tasks according to the resource usage of the multi-threaded tasks and maintain it in the database.

[0080] The above embodiment can record and obtain all key information of the task running process by transforming the original thread pool and abstract the task model based on it, which provides feasibility for monitoring and calculating the historical resource usage data of the task.

[0081] In some embodiments of the present application, monitoring the resource usage data of the multi-threaded task using the task model of the multi-threaded task includes: monitoring the resource usage data of each thread in the multi-threaded task according to the task model of the multi-threaded task; and determining the resource usage data of the multi-threaded task according to the resource usage data of each thread.

[0082] The task execution node can periodically record the running status of each thread and resource usage data such as resource usage rate. For example, the thread stack and heap shared memory usage of each thread can be obtained through ThreadMXBean. At the same time, by calculating the ratio of the CPU time occupied by the thread to the total CPU time, the approximate value of the number of CPU cores used by the thread, that is, the CPU usage rate, can be obtained.

[0083] Based on the above task model and the recorded information such as the running status and resource occupancy rate of each thread, the resource usage of each task in the monitoring period can be obtained. For example, the resource usage data of all threads contained in each task can be summed up as the resource usage data of the multi-threaded task. Finally, the resource usage data of each task in the monitoring period can be averaged as the resource data required for each task's historical single execution, that is, the historical resource usage of the task.

[0084] In some embodiments of the present application, the task scheduling method also includes: receiving heartbeat data sent by each task execution node, the heartbeat data including the current status of the task execution node and current resource usage data; storing the current status of each task execution node and the current resource usage data in a database; obtaining the current resource usage data of multiple task execution nodes includes: obtaining the current resource usage data of multiple task execution nodes from the database.

[0085] In order to monitor the operating status and resource usage of each task execution node, the heartbeat mechanism of the task execution node can be modified. Whenever the task execution node sends heartbeat data to the task scheduling center, the current machine information of the task execution node, including the operating status information of the software and hardware and resource usage, is written into the memory database. The memory database will always maintain the latest operating status information and resource usage of each task execution node. The scheduling center can automatically obtain the current operating status and resource usage of each task execution node by accessing the memory database.

[0086] It should be noted that the operating status of the above-mentioned task execution node and the update frequency of resource usage data depend on the sending frequency of heartbeat data. Therefore, the sending frequency of heartbeat data can be flexibly set according to the actual needs of task scheduling and heartbeat detection, and no specific limitation is made here.

[0087] The embodiment of the present application can realize dynamic monitoring of the operating status and resource usage of each task execution node by appropriately modifying the original heartbeat mechanism of the task execution node. The modification cost is low and the execution burden of the task scheduling center is reduced.

[0088] In some embodiments of the present application, scheduling the task to be scheduled to a target task execution node based on the historical resource usage data of the task to be scheduled and the current resource usage data of multiple task execution nodes includes: calculating the estimated resource usage data of the task to be scheduled based on the historical resource usage data of the task to be scheduled and a preset safety factor; scheduling the task to be scheduled to the target task execution node based on the estimated resource usage data of the task to be scheduled and the current resource usage data of multiple task execution nodes.

[0089] For tasks with rich historical resource usage, the historical resource usage data reflects the resource consumption pattern of the task during its past execution. In order to cope with the uncertainty of resource usage (such as sudden resource demand, system load fluctuation, etc.), a preset safety factor can be introduced based on the historical resource usage data. For example, it can be set to a value greater than 1. The specific value can be dynamically set based on factors such as the fluctuation range of historical data and the redundancy of system resources.

[0090] The estimated resource usage corresponding to the task to be scheduled is calculated by multiplying the historical resource usage data by the above safety factor, and then the task to be scheduled is scheduled to the target task execution node based on the estimated resource usage of the task to be scheduled and the current resource usage data of each task execution node.

[0091] The embodiment of the present application estimates the current resource usage data by introducing a safety factor based on the historical resource usage data, which can cope with potential resource demand uncertainties, thereby improving the accuracy of task scheduling and the rationality of resource allocation, and reducing the occurrence of task execution waiting situations.

[0092] In some embodiments of the present application, scheduling the task to be scheduled to a target task execution node based on the estimated resource usage data of the task to be scheduled and the current resource usage data of multiple task execution nodes includes: determining the current resource remaining data of each task execution node based on the current resource usage data of each task execution node; calculating the difference between the current resource remaining data of each task execution node and the estimated resource usage data of the task to be scheduled; determining the target task execution node among the multiple task execution nodes based on the difference; and scheduling the task to be scheduled to the target task execution node.

[0093] Based on the data such as the current operating status and resource usage of each task execution node read from the database in the aforementioned embodiment, the resource remaining data of each task execution node can be calculated according to the resource usage data of each task execution node. For example, if the current CPU usage rate of the task execution node is 40%, the remaining usage data is 60%.

[0094] Then, the difference between the current remaining resource data of each task execution node and the estimated resource usage data of the task to be scheduled is calculated. If the difference is negative, it means that the current remaining resources of the task execution node cannot meet the usage requirements of the task to be scheduled, and the task cannot be assigned to the task execution node. On the contrary, if the difference is positive, it means that the current remaining resources of the task execution node can meet the usage requirements of the task to be scheduled. In order to avoid resource waste and the rationality and balance of resource allocation, the task execution node with a positive difference and the smallest difference can be used as the target task execution node of the task to be scheduled.

[0095] In addition, when determining the target task execution node, the current running status of each task execution node can also be considered. If the current running status of a task execution node is abnormal, such as a machine abnormality, the task execution node can be directly filtered out, thereby avoiding the task to be scheduled from being assigned to the execution node with abnormal operation, thereby affecting the normal execution of the task.

[0096] In some embodiments of the present application, the task scheduling method also includes: when the historical resource usage data of the task to be scheduled cannot be obtained, determining the task execution node with the most current resource remaining data based on the current resource usage data of multiple task execution nodes; and scheduling the task to be scheduled to the task execution node with the most current resource remaining data.

[0097] In actual scenarios, there may be situations where some tasks to be scheduled have no historical resource usage data. For example, if a task is scheduled for execution for the first time, in this case, the task execution node with the most remaining resources can be found among multiple task execution nodes, and then the task to be scheduled can be assigned to the execution node with the most remaining resources for execution, thereby ensuring that the task to be scheduled can be successfully executed.

[0098] In summary, the key points and technical effects of the task scheduling method of this application mainly include:

[0099] 1) By monitoring the resources of each task execution node in real time and scheduling each task based on the current resource status and the historical resource usage of the task, the goal of balanced resource utilization and non-waiting task execution is achieved.

[0100] 2) The task is abstracted into a task thread model, which can record the running status of multi-threaded tasks in detail and clearly. By transforming the thread pool, the relationship between the task main thread and its child threads is determined, solving the problem that the parent-child thread relationship between threads cannot be accurately determined in native JAVA.

[0101] 3) Scheduling based on historical experience of task execution makes task scheduling no longer groundless and resource allocation no longer based solely on prediction, making resource allocation of task execution nodes more reasonable and less risky.

[0102] The present application embodiment also provides a task scheduling device 400, such as Figure 4 As shown, a schematic diagram of the structure of a task scheduling device in an embodiment of the present application is provided, wherein the task scheduling device 400 includes: a first acquisition unit 410, a second acquisition unit 420, a third acquisition unit 430 and a scheduling unit 440, wherein:

[0103] A first acquisition unit 410 is used to acquire task information of a task to be scheduled;

[0104] A second acquisition unit 420 is used to acquire historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring;

[0105] The third acquisition unit 430 is used to acquire current resource usage data of multiple task execution nodes;

[0106] The scheduling unit 440 is used to schedule the task to be scheduled to a target task execution node according to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes. The target task execution node is one of the multiple task execution nodes.

[0107] In some embodiments of the present application, the task to be scheduled is a multi-threaded task, and the second acquisition unit 420 is specifically used to: record the task submission information and task running information of the multi-threaded task; construct a task model of the multi-threaded task based on the task submission information and task running information of the multi-threaded task; use the task model of the multi-threaded task to monitor the resource usage data of the multi-threaded task; and determine the historical resource usage data of the multi-threaded task based on the monitored resource usage data of the multi-threaded task.

[0108] In some embodiments of the present application, the second acquisition unit 420 is specifically used to: monitor the resource usage data of each thread in the multi-threaded task according to the task model of the multi-threaded task; and determine the resource usage data of the multi-threaded task according to the resource usage data of each thread.

[0109] In some embodiments of the present application, the task scheduling device 400 also includes: a receiving unit, used to receive heartbeat data sent by each task execution node, the heartbeat data including the current status of the task execution node and current resource usage data; a storage unit, used to store the current status of each task execution node and the current resource usage data in a database; the second acquisition unit 420 is specifically used to: obtain the current resource usage data of multiple task execution nodes from the database.

[0110] In some embodiments of the present application, the scheduling unit 440 is specifically used to: calculate the estimated resource usage data of the task to be scheduled based on the historical resource usage data of the task to be scheduled and a preset safety factor; and schedule the task to be scheduled to the target task execution node based on the estimated resource usage data of the task to be scheduled and the current resource usage data of multiple task execution nodes.

[0111] In some embodiments of the present application, the scheduling unit 440 is specifically used to: determine the current resource remaining data of each of the task execution nodes based on the current resource usage data of each of the task execution nodes; calculate the difference between the current resource remaining data of each of the task execution nodes and the estimated resource usage data of the task to be scheduled; determine the target task execution node among the multiple task execution nodes based on the difference; and schedule the task to be scheduled to the target task execution node.

[0112] In some embodiments of the present application, the task scheduling device 400 also includes: a determination unit, which is used to determine the task execution node with the most current resource remaining data based on the current resource usage data of multiple task execution nodes when the historical resource usage data of the task to be scheduled cannot be obtained; the scheduling unit is specifically used to schedule the task to be scheduled to the task execution node with the most current resource remaining data.

[0113] It can be understood that the above-mentioned task scheduling device can implement each step of the task scheduling method provided in the above-mentioned embodiment, and the relevant explanations about the task scheduling method are applicable to the task scheduling device and will not be repeated here.

[0114] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0115] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0116] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0117] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a task scheduling device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0118] Get the task information of the task to be scheduled;

[0119] Acquire historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring;

[0120] Get the current resource usage data of multiple task execution nodes;

[0121] According to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes, the task to be scheduled is scheduled to a target task execution node, and the target task execution node is one of the multiple task execution nodes.

[0122] The above application Figure 1The method performed by the task scheduling device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0123] The present application also provides a computer program product, including a computer program or an instruction, which is implemented when the computer program or the instruction is executed by a processor. Figure 1 The task scheduling method in the illustrated embodiment is specifically used to execute:

[0124] Get the task information of the task to be scheduled;

[0125] Acquire historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring;

[0126] Get the current resource usage data of multiple task execution nodes;

[0127] According to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes, the task to be scheduled is scheduled to a target task execution node, and the target task execution node is one of the multiple task execution nodes.

[0128] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0129] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0130] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0131] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0132] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0133] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0134] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0135] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0136] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0137] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A task scheduling method, characterized in that: The task scheduling method comprises: Get the task information of the task to be scheduled; Acquire historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring; Get the current resource usage data of multiple task execution nodes; According to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes, the task to be scheduled is scheduled to a target task execution node, and the target task execution node is one of the multiple task execution nodes.

2. The task scheduling method according to claim 1, characterized in that: The task to be scheduled is a multi-threaded task, and obtaining historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled includes: Recording the task submission information and task running information of the multi-threaded task; Constructing a task model of the multi-threaded task according to the task submission information and task running information of the multi-threaded task; Monitoring resource usage data of the multi-threaded task using the task model of the multi-threaded task; The historical resource usage data of the multi-threaded task is determined according to the monitored resource usage data of the multi-threaded task.

3. The task scheduling method according to claim 1, characterized in that: The monitoring of the resource usage data of the multi-threaded task by using the task model of the multi-threaded task includes: Monitoring resource usage data of each thread in the multi-threaded task according to the task model of the multi-threaded task; The resource usage data of the multi-threaded task is determined according to the resource usage data of each thread.

4. The task scheduling method according to claim 1, characterized in that: The task scheduling method further includes: Receive heartbeat data sent by each task execution node, wherein the heartbeat data includes the current state of the task execution node and current resource usage data; Store the current status of each task execution node and the current resource usage data in the database; The obtaining of current resource usage data of multiple task execution nodes includes: The current resource usage data of the plurality of task execution nodes is obtained from the database.

5. The task scheduling method according to claim 1, characterized in that: The step of scheduling the task to be scheduled to a target task execution node according to the historical resource usage data of the task to be scheduled and the current resource usage data of the plurality of task execution nodes comprises: Calculating estimated resource usage data of the task to be scheduled based on historical resource usage data of the task to be scheduled and a preset safety factor; The task to be scheduled is scheduled to a target task execution node according to the estimated resource usage data of the task to be scheduled and the current resource usage data of the plurality of task execution nodes.

6. The task scheduling method according to claim 5, characterized in that: The step of scheduling the task to be scheduled to a target task execution node according to the estimated resource usage data of the task to be scheduled and the current resource usage data of the plurality of task execution nodes comprises: Determine current resource remaining data of each of the task execution nodes according to current resource usage data of each of the task execution nodes; Calculating the difference between the current resource remaining data of each of the task execution nodes and the estimated resource usage data of the task to be scheduled; Determining the target task execution node among the plurality of task execution nodes according to the difference; Schedule the task to be scheduled to the target task execution node.

7. The task scheduling method according to claim 1, characterized in that: The task scheduling method further includes: In the case where the historical resource usage data of the task to be scheduled cannot be obtained, determining the task execution node with the most current resource remaining data according to the current resource usage data of the plurality of task execution nodes; The task to be scheduled is scheduled to the task execution node with the most remaining resource data.

8. A task scheduling device, characterized in that: The task scheduling device comprises: A first acquisition unit, used to acquire task information of the task to be scheduled; A second acquisition unit is used to acquire historical resource usage data of the task to be scheduled according to the task information of the task to be scheduled, wherein the historical resource usage data is obtained based on task model monitoring; A third acquisition unit is used to acquire current resource usage data of multiple task execution nodes; The scheduling unit is used to schedule the task to be scheduled to a target task execution node according to the historical resource usage data of the task to be scheduled and the current resource usage data of the multiple task execution nodes. The target task execution node is one of the multiple task execution nodes.

9. An electronic device, comprising: processor; and a memory arranged to store computer executable instructions, wherein when the executable instructions are executed, the processor executes the task scheduling method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implements the task scheduling method according to any one of claims 1 to 7.

Images