Task scheduling method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a task scheduling method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring task scales and node loads sent by a plurality of computing nodes; determining a node to be migrated out and a node to be migrated in from the plurality of computing nodes according to the node loads corresponding to the plurality of computing nodes respectively; determining a to-be-migrated task from the plurality of tasks according to the task scales corresponding to the plurality of tasks executed by the to-be-migrated node; and migrating the to-be-migrated task from the to-be-migrated node to the to-be-migrated node. According to the invention, the load balancing effect and the task processing timeliness can be improved.

Description

Technical Field

[0001] The present application relates to network technology, and in particular to a task scheduling method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] With the rapid development of computer technology, computing power has been improved and computing demand has increased. In order to cope with a large number of tasks to be calculated in specific scenarios, multiple computing nodes are usually used to jointly provide computing power.

[0003] In the solutions provided by the related art, the load balancing principle is usually used in the task allocation link, that is, the tasks to be processed are preferentially allocated to the computing nodes with the lowest node load. However, considering the differences between the computing capabilities of different computing nodes and / or the differences between different tasks, the computing nodes to which the tasks are assigned may have the problem of excessive node load and fail to process the tasks in time. Therefore, the load balancing effect of the solutions provided by the related art is poor, which may result in the inability to process the tasks in time. Summary of the invention

[0004] The embodiments of the present application provide a task scheduling method, device, electronic device, computer-readable storage medium and computer program product, which can improve the load balancing effect and the timeliness of task processing.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present application provides a task scheduling method, including:

[0007] Get the task scale and node load sent by multiple computing nodes;

[0008] Determining a node to be migrated out and a node to be migrated in from the plurality of computing nodes according to the node loads respectively corresponding to the plurality of computing nodes;

[0009] Determining the task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated;

[0010] Migrate the task to be migrated from the node to be migrated out to the node to be migrated in.

[0011] The present application provides a task scheduling method, including:

[0012] Perform task scale estimation on each task executed by the computing node to obtain the task scale of each task;

[0013] Obtaining the node load of the computing node through load detection;

[0014] Send the task scale of each task executed by the computing node and the node load of the computing node to the scheduling node, so that the scheduling node performs the following processing:

[0015] Get the task scale and node load sent by multiple computing nodes;

[0016] Determining a node to be migrated out and a node to be migrated in from the plurality of computing nodes according to the node loads respectively corresponding to the plurality of computing nodes;

[0017] Determining the task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated;

[0018] Migrate the task to be migrated from the node to be migrated out to the node to be migrated in.

[0019] The present application provides a task scheduling device, including:

[0020] An acquisition module is used to obtain the task scale and node load sent by multiple computing nodes;

[0021] A node determination module, configured to determine a node to be migrated out and a node to be migrated in from the plurality of computing nodes according to the node loads respectively corresponding to the plurality of computing nodes;

[0022] A task determination module, configured to determine a task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated;

[0023] A migration module is used to migrate the task to be migrated from the node to be migrated out to the node to be migrated in.

[0024] The present application provides a task scheduling device, including:

[0025] A scale estimation module is used to estimate the task scale of each task executed by the computing node to obtain the task scale of each task;

[0026] A load detection module, used to obtain the node load of the computing node through load detection;

[0027] A sending module is used to send the task scale of each task executed by the computing node and the node load of the computing node to the scheduling node, so that the scheduling node performs the following processing:

[0028] Get the task scale and node load sent by multiple computing nodes;

[0029] Determining a node to be migrated out and a node to be migrated in from the plurality of computing nodes according to the node loads respectively corresponding to the plurality of computing nodes;

[0030] Determining the task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated;

[0031] Migrate the task to be migrated from the node to be migrated out to the node to be migrated in.

[0032] An embodiment of the present application provides an electronic device, including:

[0033] A memory for storing executable instructions;

[0034] The processor is used to implement the task scheduling method provided in the embodiment of the present application when executing the executable instructions stored in the memory.

[0035] An embodiment of the present application provides a computer-readable storage medium storing executable instructions for causing a processor to execute and implement the task scheduling method provided in the embodiment of the present application.

[0036] An embodiment of the present application provides a computer program product, which includes executable instructions for causing a processor to execute and implement the task scheduling method provided in the embodiment of the present application.

[0037] The embodiments of the present application have the following beneficial effects:

[0038] According to the node loads corresponding to multiple computing nodes, the nodes to be migrated out that need to migrate out the tasks and the nodes to be migrated in that can continue to carry the tasks are determined from the multiple computing nodes. Since migrating tasks from the nodes to be migrated out means that the node load of the nodes to be migrated out is relieved, and migrating tasks to the nodes to be migrated in means that the node load of the nodes to be migrated in is increased, the embodiment of the present application determines the tasks to be migrated of appropriate scale according to the task scale of each task in the nodes to be migrated out, so that the node load of the nodes to be migrated out can be effectively relieved after migrating out the tasks to be migrated, and the node load of the nodes to be migrated in will not be too large after migrating in the tasks to be migrated. In this way, it is possible to ensure that the tasks to be migrated can be successfully executed in the nodes to be migrated in, and avoid too many task scheduling times caused by re-triggering task scheduling. In summary, the embodiment of the present application can realize reasonable task scheduling for the tasks executed by computing nodes, improve the timeliness of task processing, and better realize load balancing between multiple computing nodes. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0040] Figure 1 It is a schematic diagram of the architecture of the task scheduling system provided in the embodiment of the present application;

[0041] Figure 2A It is a structural diagram of a server provided in an embodiment of the present application;

[0042] Figure 2B is another structural diagram of a server provided in an embodiment of the present application;

[0043] Figure 3A It is a flowchart of a task scheduling method provided in an embodiment of the present application;

[0044] Figure 3B is another flowchart of the task scheduling method provided in an embodiment of the present application;

[0045] Figure 3C is another flowchart of the task scheduling method provided in an embodiment of the present application;

[0046] Figure 3D is another flowchart of the task scheduling method provided in an embodiment of the present application;

[0047] Figure 3E is another flowchart of the task scheduling method provided in an embodiment of the present application;

[0048] Figure 3F is another flowchart of the task scheduling method provided in an embodiment of the present application;

[0049] Figure 4 This is a schematic diagram of the architecture of the log service platform provided by the embodiment of the present application;

[0050] Figure 5 It is a schematic diagram of the interaction between the Scheduler and the Worker provided in the embodiment of the present application;

[0051] Figure 6 It is a schematic diagram of task migration provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0053] In the following description, reference is made to "some embodiments", which describes a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. In the following description, reference is made to the term "multiple" to mean at least two.

[0054] In the following description, the terms "first\second\third\fourth" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third\fourth" can be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0056] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0057] 1) Task: refers to the computing task that needs to be performed. The present application embodiment does not limit the task type, and can be determined according to the actual application scenario. For example, for a log service platform, tasks may include index calculation, object storage backup, log delivery, and other types.

[0058] 2) Cloud Computing: It is a computing model that distributes computing tasks on a resource pool consisting of a large number of computing nodes, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use.

[0059] As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an Infrastructure as a Service (IaaS) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (such as virtualized machines, including operating systems), storage devices, and network devices.

[0060] According to the logical function division, the Platform as a Service (PaaS) layer can be deployed on the IaaS layer, and the Software as a Service (SaaS) layer can be deployed on the PaaS layer, or SaaS can be directly deployed on IaaS. PaaS is a platform for software operation, such as databases, web containers, etc. SaaS is a variety of business software, such as web portals, SMS mass senders, etc. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0061] In the embodiment of the present application, multiple tasks can be performed based on the computing mode of cloud computing.

[0062] 3) Compute node: refers to the part used to provide computing power in a computing cluster (such as the resource pool mentioned above). A compute node can be a physical node (i.e., a physical machine, such as a terminal device or server); it can also be a virtual node, such as a virtual machine or container; it can also be a component in a physical machine, virtual machine, or container that provides computing power. In the case where the compute node is a virtual node, one physical machine can run one or more compute nodes.

[0063] Among them, a physical machine refers to a physical hardware device that runs an operating system and other applications; a virtual machine is a complete computer system simulated by software that has complete hardware system functions and runs in an isolated environment. It can run independent operating systems and applications. Virtual machines have a certain degree of independence and can be managed and maintained like physical machines; containers use a technology that virtualizes the operating system and separates it into independent software units. That is, a container is a virtualized environment created on an operating system. Containers do not require an additional operating system or virtualization layer, but share the kernel of the host operating system.

[0064] In an embodiment of the present application, the multiple computing nodes involved may all belong to the same node type, for example, all are physical machines; or they may respectively belong to different node types, for example, some computing nodes are physical machines and other computing nodes are virtual machines.

[0065] 4) Scheduling node: used to determine which computing node a task is executed on. A scheduling node can manage multiple computing nodes. A scheduling node can be a physical machine, virtual machine, or container, or a component in a physical machine, virtual machine, or container that provides management capabilities.

[0066] 5) Task scale: used to indicate the size and execution difficulty of the task. The larger the task scale, the higher the execution difficulty of the task, and the computing node needs to consume more computing resources to execute the task.

[0067] 6) Node load: used to indicate the load of computing nodes. Node load can include one or more indicators, such as central processing unit (CPU) usage, memory usage, disk usage, etc.; node load can also be obtained by fusion processing of multiple indicators, where indicator fusion processing includes but is not limited to weighted summation.

[0068] 7) Load Balancing: It is a technology that distributes computing tasks to multiple computing nodes to achieve balance among multiple computing nodes, so as to optimize performance, improve reliability and increase scalability. In the embodiment of the present application, load balancing is mainly achieved by migrating tasks between different computing nodes.

[0069] 8) Data consistency: When there are multiple computing nodes working together in a computing cluster, data inconsistency may occur in each computing node due to network, hardware and other reasons. There are three levels of data consistency: Exactly once, At least once and At most once. Exactly once means that the task is executed only once, Atleast once means that the task is executed at least once, and At most once means that the task is executed at most once.

[0070] The embodiments of the present application provide a task scheduling method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the load balancing effect and the timeliness of task processing. The following describes an exemplary application of the electronic device provided by the embodiments of the present application. The electronic device provided by the embodiments of the present application can be implemented as various types of terminal devices or as a server.

[0071] See also Figure 1 , Figure 1 1 is a schematic diagram of the architecture of the task scheduling system 100 provided in the embodiment of the present application. For ease of understanding, the case where both the scheduling node and the computing node are servers is used as an example for explanation, but this does not constitute a limitation on the embodiment of the present application. Figure 1 The nodes involved can also be terminal devices, virtual machines, containers, etc. Figure 1 In the figure, server 200 is connected to server 400-1, server 400-2 and server 400-3 through network 300, wherein server 200 serves as a scheduling node, server 400-1, server 400-2 and server 400-3 serve as computing nodes, and network 300 may be a wide area network or a local area network, or a combination of the two.

[0072] First, server 200 may generate multiple tasks to be executed according to business conditions and send them to server 400-1, server 400-2, and server 400-3 respectively, wherein the tasks sent to server 400-1, the tasks sent to server 400-2, and the tasks sent to server 400-3 are different. Figure 1 In the example, tasks 1 to 200 are sent to server 400-1, tasks 201 to 400 are sent to server 400-2, and tasks 401 to 600 are sent to server 400-3, where task 1 represents the task with sequence number 1, and the rest are similar.

[0073] For the server as a computing node, each received task is executed, and the task scale of each executed task is estimated to obtain the task scale of each task; at the same time, the node load of the server itself is obtained through load detection (the node load here is the server load). Then, the server as a computing node sends the task scale and node load to the server 200. For example, for the server 400-1, the task scale of each task from task 1 to task 200 and the node load of the server 400-1 itself are sent to the server 200.

[0074] Server 200 obtains the task scale and node load sent by server 400-1, server 400-2 and server 400-3; determines the node to be migrated out and the node to be migrated in from server 400-1, server 400-2 and server 400-3 according to the node loads corresponding to server 400-1, server 400-2 and server 400-3 respectively; determines the task to be migrated from the multiple tasks executed by the node to be migrated out according to the task scales corresponding to the multiple tasks executed by the node to be migrated out; migrates the task to be migrated from the node to be migrated out to the node to be migrated in. Figure 1 In the example, the node to be migrated is server 400-1, the node to be migrated is server 400-2, and the tasks to be migrated are tasks 3 to task 100. Then, server 200 migrating tasks 3 to task 100 from server 400-1 to server 400-2 may mean: server 200 notifies server 400-1 to stop executing tasks 3 to task 100, and server 200 resends tasks 3 to task 100 to server 400-2; or server 200 notifies server 400-1 to stop executing tasks 3 to task 100, and notifies server 400-1 to send tasks 3 to task 100 to server 400-2.

[0075] In some embodiments, a terminal device or a server can implement the task scheduling method provided in the embodiments of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run; it can also be a small program, that is, a program that can be run only by downloading it to a browser environment; it can also be a small program that can be embedded in any APP, wherein the small program can be controlled to run or closed by the user. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0076] In some embodiments, the server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal device may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart TV, a smart watch, etc., but is not limited thereto. The terminal device and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0077] See also Figure 2A , Figure 2A is a schematic diagram of the structure of the server 200 provided in an embodiment of the present application, Figure 2A The server 200 shown includes: at least one processor 210, a memory 250 and at least one network interface 220. The various components in the server 200 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 2A Various buses are labeled as bus system 240 .

[0078] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0079] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0080] The memory 250 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0081] In some embodiments, memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0082] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0083] A network communication module 252, used to reach other computing devices via one or more (wired or wireless) network interfaces 220, exemplary network interfaces 220 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB);

[0084] In some embodiments, the task scheduling device provided in the embodiments of the present application can be implemented in software. Figure 2A The task scheduling device 255 stored in the memory 250 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: an acquisition module 2551, a node determination module 2552, a task determination module 2553 and a migration module 2554. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0085] See also Figure 2B , Figure 2B 4 is a schematic diagram of the structure of the server 400-1 (the same applies to the server 400-2 and the server 400-3) provided in the embodiment of the present application. Figure 2BThe server 400-1 shown includes: at least one processor 410, a memory 450 and at least one network interface 420. The components in the server 400-1 are coupled together via a bus system 440. The memory 450 includes an operating system 451 and a network communication module 452. Figure 2B The functions of each structure can be found in Figure 2A Description of the corresponding embodiment.

[0086] In some embodiments, the task scheduling device provided in the embodiments of the present application can be implemented in software. Figure 2B The task scheduling device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a scale estimation module 4551, a load detection module 4552, and a sending module 4553. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0087] The task scheduling method provided in the embodiment of the present application will be described in conjunction with the exemplary application and implementation of the electronic device provided in the embodiment of the present application.

[0088] See also Figure 3A , Figure 3A is a flowchart of a task scheduling method provided in an embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0089] In step 101, the computing node performs task scale estimation processing on each executed task to obtain the task scale of each task.

[0090] In an embodiment of the present application, a scheduling node is used to manage multiple computing nodes. The scheduling node can generate tasks to be executed according to business conditions, and send the tasks to the computing nodes so that the computing nodes execute the received tasks, wherein the tasks sent to different computing nodes are different. For example, the scheduling node can divide the number of multiple tasks generated by the number of computing nodes to obtain the average number of tasks, and send the multiple tasks generated to each computing node on average according to the average number of tasks; for another example, the scheduling node can preferentially send the generated tasks to the computing node with the lowest node load.

[0091] The computing node can execute some or all of the received tasks in parallel, and for each task executed, the task scale is estimated during the execution process to obtain the task scale. It is worth noting that the task migration involved in the embodiment of the present application is for the tasks in execution, because: for the tasks that have not started to be executed, the corresponding task scale cannot be determined; for the tasks that have been executed, there is no need to repeat the task through task migration.

[0092] The embodiment of the present application does not limit the method of task scale estimation processing. For example, the current input data volume when executing each task can be used as the task scale of the corresponding task, where the current input data volume is the instantaneous input data volume currently counted. For example, when the computing node executes Task 1, the current input data volume of Task 1 is 10 megabytes (MB), then 10MB is used as the task scale of Task 1.

[0093] In step 102, the computing node obtains the node load through load detection.

[0094] When executing tasks, computing nodes also perform load detection on themselves to obtain the node load of computing nodes. The greater the node load, the more serious the computing power consumption of the computing node, that is, the greater the pressure on the computing node. Node load can include one or more indicators, such as CPU usage, memory usage, and hard disk usage.

[0095] The embodiment of the present application does not limit the method of load detection, and it can be implemented through a specific detection command (such as obtaining CPU utilization through the mpstat command) or a detection program.

[0096] In step 103, the computing node sends the task scale and node load of each task to be executed to the scheduling node.

[0097] In order to facilitate the scheduling node to know the status of the computing node, the computing node sends the task scale of each task to be executed and its own node load to the scheduling node.

[0098] In some embodiments, steps 101 to 103 may be performed periodically, that is, the computing node may send heartbeat information to the scheduling node through a heartbeat mechanism, and each heartbeat information sent includes the task scale of each task executed and the node load.

[0099] In step 104, the scheduling node obtains the task scales and node loads sent by multiple computing nodes.

[0100] Here, the scheduling node obtains the task scale and node load sent by the multiple computing nodes it manages.

[0101] In some embodiments, when the scheduling node fails to obtain the task scale and node load sent by any computing node for more than a preset period of time, the task in any computing node is migrated to other computing nodes.

[0102] During the communication process between the computing node and the scheduling node, the scheduling node may fail to receive the information sent by the computing node in time due to some abnormal reasons (such as computing node failure, etc.). In this case, task migration can also be performed. Taking any computing node such as computing node A as an example, when the scheduling node fails to obtain the task scale and node load sent by computing node A for more than a preset time (such as 10 minutes), in order to ensure that the task can be processed in time, the scheduling node migrates the task in computing node A (referring to the unfinished task known to the scheduling node) to other computing nodes, and notifies computing node A to stop executing the task. Among them, the task in computing node A can be regarded as a task to be migrated, and the computing node to be migrated can be obtained according to the method of determining the node to be migrated in the following text. Through the above method, task migration can be carried out in time when an abnormal situation occurs, ensuring that the task can be processed in time, and improving the efficiency and success rate of task execution.

[0103] In step 105, the scheduling node determines the nodes to be migrated out and the nodes to be migrated in from the plurality of computing nodes according to the node loads corresponding to the plurality of computing nodes respectively.

[0104] Since the node load intuitively reflects the pressure situation of the computing node, the scheduling node determines the nodes to be migrated out and the nodes to be migrated in from the multiple computing nodes according to the node loads corresponding to the multiple computing nodes respectively, wherein the nodes to be migrated out refer to the computing nodes that need to migrate the tasks out, for example, the nodes to be migrated out may be several computing nodes with the largest node load; the nodes to be migrated in refer to the computing nodes that can receive or undertake other tasks, for example, the nodes to be migrated out may be several computing nodes with the smallest node load. The embodiment of the present application does not limit the method of determining the nodes to be migrated out and the nodes to be migrated in. For example, the multiple computing nodes may be sorted in the order of the node load from small to large to obtain the node order, and the first several computing nodes in the node order are taken as the nodes to be migrated in, and the last several computing nodes in the node order are taken as the nodes to be migrated out; the nodes to be migrated out and the nodes to be migrated in may also be determined by setting a load threshold.

[0105] In some embodiments, the above-mentioned scheduling node can be implemented in the following way: the scheduling node determines the nodes to be migrated out and the nodes to be migrated in from multiple computing nodes according to the node loads corresponding to the multiple computing nodes respectively and the maximum bearing loads corresponding to the multiple computing nodes respectively.

[0106] The maximum load bearing loads (referring to the maximum value of the node load that the computing node can bear, which can be preset) of multiple computing nodes managed by the same scheduling node may be different. In this case, the nodes to be migrated out and the nodes to be migrated in can be determined according to the node loads corresponding to the multiple computing nodes and the maximum load bearing loads of each computing node. For example, the remaining borne load parameter of each computing node can be calculated. The remaining borne load parameter can be the remaining borne load amount or the remaining borne load rate. The remaining borne load amount is equal to the difference between the maximum bearing load of the computing node and the current node load of the computing node. The remaining borne load rate is equal to the ratio between the remaining borne load amount of the computing node and the maximum bearing load of the computing node. It can be understood that if the remaining borne load parameter of the computing node is larger, it means that there are more remaining available resources on the computing node, and the computing node can be used to undertake more tasks. Therefore, the computing node whose remaining borne load parameter among the multiple computing nodes is greater than the first parameter threshold can be determined as the node to be migrated in, and the computing node whose remaining borne load parameter among the multiple computing nodes is less than the second parameter threshold can be determined as the node to be migrated out, wherein the first parameter threshold is greater than the second parameter threshold. The first parameter threshold and the second parameter threshold can be set according to actual needs and are not specifically limited here. In other embodiments, multiple computing nodes can also be sorted in order from small to large in the order of the remaining load parameters to obtain a first sorting, and the computing nodes in the first first quantity or the first proportion in the first sorting are used as nodes to be migrated out, and the computing nodes in the second second quantity or the second proportion in the first sorting are used as nodes to be migrated in, wherein the first quantity and the second quantity may be the same or different, and the first proportion and the second proportion may be the same or different. The above method combines the maximum load bearing capacity to determine the nodes to be migrated out and the nodes to be migrated in, taking into account the differences between different computing nodes, and can better achieve load balancing.

[0107] It is worth noting that if the number of nodes to be migrated determined in step 105 is zero, then there is no need to perform subsequent steps.

[0108] In step 106, the scheduling node determines the task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated.

[0109] Through step 105, the migration direction can be determined, that is, from which computing node to which computing node to migrate, and in the embodiment of the present application, the migration content needs to be determined, that is, which tasks in the node to be migrated are most suitable for migration. Therefore, the scheduling node determines the tasks to be migrated from the multiple tasks executed by the node to be migrated according to the task scales corresponding to the multiple tasks executed by the node to be migrated. For example, the multiple tasks executed by the node to be migrated can be sorted according to the task scale to obtain the task sequence, and the tasks in a specific position in the task sequence are taken as the tasks to be migrated; the tasks to be migrated can also be determined by setting a scale threshold, for example, the tasks whose task scale falls within the preset scale range are taken as the tasks to be migrated. It is worth noting that when the number of nodes to be migrated includes multiple ones, step 106 is performed for each node to be migrated.

[0110] In some embodiments, the above-mentioned scheduling node can be implemented in such a way that it determines the task to be migrated from multiple tasks according to the task scales corresponding to the multiple tasks executed by the node to be migrated: the scheduling node determines the task whose task scale is within a preset scale range as the task to be migrated among the multiple tasks executed by the node to be migrated.

[0111] For the node to be migrated out, if the task with a smaller task scale on the node to be migrated out is used as the task to be migrated, the effect on alleviating the node load of the node to be migrated out is small. In this way, a large number of tasks may need to be migrated from the node to be migrated out to effectively alleviate the node load on the node to be migrated out; if the task with a larger task scale on the node to be migrated out is used as the task to be migrated, there may be a situation where the node to be migrated out cannot carry the task to be migrated, or there may be a situation where the node load of the node to be migrated in is too high after the task to be migrated is migrated to the node to be migrated in. Therefore, based on this principle, for multiple tasks executed on the node to be migrated out, according to the task scale corresponding to each task, the task whose task scale is within the preset scale range can be determined as the task to be migrated. Among them, the minimum value in the preset scale range is greater than the minimum task scale among the multiple tasks, and the maximum value in the preset scale range is less than the maximum task scale among the multiple tasks. The preset scale range can be set according to actual needs and is not specifically limited here. In this way, the task with a medium task scale on the node to be migrated out is used as the task to be migrated.

[0112] In some embodiments, the above-mentioned scheduling node can be implemented in the following way: determining the task to be migrated from multiple tasks according to the task scales respectively corresponding to the multiple tasks performed by the node to be migrated is implemented: the scheduling node determines the load to be migrated according to the standard carrying load of the node to be migrated and the node load of the node to be migrated, and determines the task to be migrated from multiple tasks according to the load to be migrated of the node to be migrated and the task scales respectively corresponding to the multiple tasks performed by the node to be migrated.

[0113] Here, the standard bearing load refers to the standard value (or ideal value) of the node load that the computing node can bear, which can be pre-set, and the standard bearing loads of different computing nodes can be the same or different. In order to make the node load of the node to be migrated close to the standard bearing load, the scheduling node can calculate the difference between the node load of the node to be migrated (referring to the current node load) and the standard bearing load of the node to be migrated as the load to be migrated. In an embodiment of the present application, the mapping relationship between the node load and the task scale can be pre-set, that is, how many units of task scale correspond to 1 unit of node load. In this way, after determining the load to be migrated, the task scale corresponding to the load to be migrated is determined according to the mapping relationship as the task scale to be migrated. The task scale to be migrated is the task scale that needs to be migrated from the node to be migrated.

[0114] Then, multiple tasks to be executed by the node to be migrated are selected according to the scale of the tasks to be migrated, until the total scale of all selected tasks reaches the scale of the tasks to be migrated. The selection here can be random selection; it can be selected in order from large to small task numbers, where the larger the task number, the later the task generation time. The purpose here is to use tasks with larger task numbers as tasks to be migrated, and tasks with smaller task numbers can be executed and completed as soon as possible in the node to be migrated, so as to avoid excessive delays in task execution; it can be selected in order from low to high task progress, so that the node to be migrated continues to execute tasks with higher task progress, while tasks with lower task progress are migrated out as tasks to be migrated, thereby reducing the migration cost; it can also be selected in order from low to high task priority, so that the node to be migrated continues to execute tasks with higher priority, while tasks with lower priority are migrated out as tasks to be migrated, thereby ensuring that tasks with higher priority are executed and completed as soon as possible. Through the above method, the tasks to be migrated are determined in combination with the standard load-bearing capacity of the nodes to be migrated, thereby alleviating the node load pressure of the nodes to be migrated while ensuring that the nodes to be migrated still have enough tasks to execute after migrating the tasks to be migrated, without wasting the computing resources of the nodes to be migrated themselves, thereby improving the overall task execution efficiency of the computing cluster.

[0115] In some embodiments, the above-mentioned scheduling node can determine the task to be migrated from multiple tasks according to the task scales corresponding to the multiple tasks executed by the node to be migrated in the following way: the scheduling node screens the multiple tasks according to the task parameters corresponding to the multiple tasks executed by the node to be migrated, and determines the task to be migrated from the screened tasks.

[0116] Here, the task parameters may include at least one of a task number, a task progress, and a task priority, wherein the task number is generated by the scheduling node when generating the task, and the larger the task number, the later the task is generated; the task progress is counted by the computing node when executing the task, for example, if the task progress is 5% executed, the computing node may send the task progress together with the task scale and node load when sending it to the scheduling node; the task priority may be preset, for example, corresponding priorities are set in advance for different types of tasks.

[0117] The scheduling node screens multiple tasks according to the task parameters corresponding to the multiple tasks executed by the node to be migrated. According to the specific type of the task parameters, the screening process can be divided into the following three cases:

[0118] 1) The scheduling node selects several tasks with the largest task numbers according to the task numbers corresponding to the multiple tasks executed by the node to be migrated. For example, the multiple tasks executed by the node to be migrated are sorted in order from small to large task numbers, and a preset number or a preset proportion of tasks at the end of the sorting are selected. In this way, tasks with smaller task numbers are ensured to continue to be executed in the node to be migrated, and tasks with larger task numbers are selected to be included in the consideration range of the tasks to be migrated, which can effectively avoid excessive delays in task execution.

[0119] 2) The scheduling node selects several tasks with the lowest task progress according to the task progress corresponding to the multiple tasks executed by the node to be migrated. For example, the multiple tasks executed by the node to be migrated are sorted in order from low to high task progress, and the preset number or preset proportion of tasks at the top of the sorting are selected. In this way, for tasks with higher task progress, since the tasks are about to be completed and the node to be migrated has consumed a lot of time and computing resources to execute the tasks, the node to be migrated continues to execute the tasks with higher task progress, and the tasks with lower task progress are selected to be included in the consideration scope of the tasks to be migrated, which can effectively reduce the cost and price of task migration.

[0120] 3) The scheduling node selects several tasks with the lowest priority according to the priorities of the multiple tasks executed by the node to be migrated. For example, the multiple tasks executed by the node to be migrated are sorted in order from low to high priority, and a preset number or a preset proportion of tasks at the top of the sort are selected. In this way, the node to be migrated continues to execute the tasks with higher priority, and the tasks with lower priority are selected to be included in the scope of consideration for the tasks to be migrated, so that the tasks with higher priority can be completed as soon as possible, thereby meeting the priority requirements in the business.

[0121] After the scheduling node screens multiple tasks to be executed by the node to be migrated according to the task parameters (equivalent to the first screening), it determines the tasks to be migrated from the screened tasks (equivalent to the second screening).

[0122] In step 107, the scheduling node migrates the tasks to be migrated from the node to be migrated out to the node to be migrated in.

[0123] Here, the scheduling node migrates the tasks to be migrated from the node to be migrated out to the node to be migrated in, that is, the node to be migrated in will then execute the tasks to be migrated. In this way, for the node to be migrated out with too large a node load, migrating the tasks to be migrated out can reduce the node load of the node to be migrated out and reduce the probability of node abnormalities (such as crashes); for the node to be migrated in with too small a node load, migrating the tasks to be migrated in can utilize idle computing resources and improve the effective utilization rate of computing resources.

[0124] In some embodiments, the above-mentioned scheduling node can migrate the tasks to be migrated from the node to be migrated out to the node to be migrated in by performing any of the following processing: the scheduling node notifies the node to be migrated out to stop executing the tasks to be migrated, and the scheduling node sends the tasks to be migrated to the node to be migrated in; the scheduling node notifies the node to be migrated out so that the node to be migrated out stops executing the tasks to be migrated, and sends the tasks to be migrated to the node to be migrated in.

[0125] Two methods are provided here to implement task migration. The first method is that the scheduling node notifies the node to be migrated out to stop executing the task to be migrated, and at the same time, the scheduling node sends the task to be migrated to the node to be migrated in. That is, the scheduling node centrally manages the task migration process, which is suitable for scenarios where computing nodes cannot communicate or other scenarios that require centralized management. The second method is that the scheduling node notifies the node to be migrated out to stop executing the task to be migrated, and sends the task to be migrated to the node to be migrated in. This method is used by the node to be migrated out to send the task to be migrated, which is suitable for scenarios where computing nodes can communicate with each other.

[0126] In some embodiments, between any steps, the task scheduling method also includes: when the scheduling node generates a task to be executed, the scheduling node determines the task execution node from multiple computing nodes based on the node loads corresponding to the multiple computing nodes respectively; the scheduling node sends the generated task to the task execution node, so that the task execution node executes the received task.

[0127] Here, whenever the scheduling node generates a task to be executed, it can determine which computing node will execute the task based on the principle of load balancing, that is, determine the task execution node corresponding to the task to be executed. For example, the scheduling node determines the task execution node from multiple computing nodes according to the node loads corresponding to the multiple computing nodes (for example, the node load sent by the computing node most recently, that is, the latest node load), and then the scheduling node sends the generated task to the task execution node so that the task execution node executes the received task. Among them, the task execution node can be determined by the method of determining the node to be migrated, for example, several computing nodes with the smallest node load can be used as task execution nodes, or computing nodes with a node load less than a specific load threshold (such as a second load threshold) can be used as task execution nodes; the method of sending the task to be migrated to the node to be migrated can be used to send the task to the task execution node. Through the above method, the reasonable allocation of the generated tasks is achieved, which can effectively reduce the number of subsequent task migrations.

[0128] like Figure 3A As shown, the scheduling node determines the nodes to be migrated out that need to migrate out the tasks and the nodes to be migrated in that can continue to carry the tasks from the multiple computing nodes according to the node loads corresponding to the multiple computing nodes respectively. Since migrating tasks from the nodes to be migrated out means that the node load of the nodes to be migrated out is relieved, and migrating tasks to the nodes to be migrated in means that the node load of the nodes to be migrated in is increased, the embodiment of the present application determines the tasks to be migrated of appropriate scale according to the task scale of each task on the nodes to be migrated out, so that the node load of the nodes to be migrated out can be effectively relieved after migrating out the tasks to be migrated, and the node load of the nodes to be migrated in will not be too large after migrating in the tasks to be migrated. In this way, the tasks to be migrated can be effectively executed on the nodes to be migrated in as much as possible, and the task scheduling times caused by re-triggering the task scheduling are avoided. In summary, the embodiment of the present application can realize reasonable task scheduling for the tasks executed by the computing nodes, improve the timeliness of task processing, and better realize the load balancing between multiple computing nodes; at the same time, the implementation cost of the embodiment of the present application is low, easy to deploy, and has a wide range of applications, such as task scheduling in log scenarios.

[0129] In some embodiments, see Figure 3B , Figure 3B is a flowchart of a task scheduling method provided in an embodiment of the present application. Figure 3A The illustrated step 105 can be implemented through steps 201 to 202 , which will be described in conjunction with each step.

[0130] In step 201, the scheduling node selects computing nodes whose node loads are greater than a first load threshold from among a plurality of computing nodes as nodes to be migrated.

[0131] Here, since the node load can intuitively reflect the pressure of the computing node, compared with the method of sorting nodes according to the node load, determining the nodes to be migrated out and the nodes to be migrated in by threshold comparison can more accurately balance the node load of different computing nodes.

[0132] For example, a first load threshold and a second load threshold may be preset in the scheduling node, and the scheduling node may select computing nodes among multiple computing nodes whose node load is greater than the first load threshold as nodes to be migrated, wherein the first load threshold is greater than the second load threshold.

[0133] It is worth noting that if the node load includes multiple indicators, the first load threshold includes thresholds corresponding to the multiple indicators, and the second load threshold is the same. For example, the node load includes CPU usage, memory usage, and disk usage. The threshold corresponding to the CPU usage in the first load threshold is 90%, the threshold corresponding to the memory usage is 85%, and the threshold corresponding to the disk usage is 85%.

[0134] It is worth noting that multiple computing nodes may correspond to the same first load threshold and the same second load threshold. Alternatively, considering that the computing capabilities and hardware configurations of different computing nodes may be different, different first load thresholds may be set for different computing nodes, and different second load thresholds may be set for different computing nodes.

[0135] In step 202, the scheduling node selects computing nodes whose node loads are less than a second load threshold from among a plurality of computing nodes as nodes to be migrated.

[0136] Here, the scheduling node selects computing nodes whose node loads are less than a second load threshold among the multiple computing nodes as nodes to be migrated.

[0137] In some embodiments, the above-mentioned scheduling node obtaining the task scale and node load sent by multiple computing nodes can be implemented in the following manner: the scheduling node obtains the task scale and node load sent by multiple computing nodes at multiple time points;

[0138] The above-mentioned scheduling node screening out computing nodes whose node loads are greater than the first load threshold from among the multiple computing nodes as nodes to be migrated out can be implemented in the following manner: the scheduling node screens out computing nodes whose node loads are greater than the first load threshold from among the multiple computing nodes in the first historical time period as nodes to be migrated out;

[0139] The above-mentioned scheduling node screening out computing nodes whose node loads are less than the second load threshold from among multiple computing nodes can be implemented in the following manner: the scheduling node screens out computing nodes whose node loads are less than the second load threshold within the second historical time period from among multiple computing nodes as nodes to be migrated.

[0140] In the case where the computing node periodically sends the task scale and node load to the scheduling node, for the same computing node, the scheduling node can obtain the node load sent by the computing node at different time points. Since the node load may jump up and down over time, the embodiment of the present application also takes into account the stability factor. For example, the scheduling node selects the computing nodes whose multiple node loads in the first historical time period are greater than the first load threshold from multiple computing nodes as the nodes to be migrated out; the scheduling node selects the computing nodes whose multiple node loads in the second historical time period are less than the second load threshold from multiple computing nodes as the nodes to be migrated in. Among them, the first historical time period and the second historical time period can be the same or different, for example, both are within the last 5 minutes, and for example, considering that the node load is too high usually occurs in a short time, in order to quickly identify the node to be migrated out, the length of the first historical time period can be set to be less than the length of the second historical time period. It is worth noting that the historical time periods involved in the embodiment of the present application (such as the first historical time period, the second historical time period, and the third historical time period) are calculated from the current time forward, and the length of time from the current time does not exceed a specific length of time period, such as within the last 5 minutes. Through the above method, the accuracy of determining the nodes to be migrated out and the nodes to be migrated in can be improved, the task migration effect can be improved, and the waste of computing resources caused by unnecessary task migration can be avoided.

[0141] like Figure 3B As shown, since the node load can clearly reflect the pressure situation of the computing node, the nodes to be migrated out and the nodes to be migrated in are determined by threshold comparison, which can ensure that the determined nodes to be migrated out need to migrate out tasks, and the determined nodes to be migrated in need to migrate in tasks, thereby improving the effect of task migration and avoiding the waste of computing resources caused by unnecessary task migration.

[0142] In some embodiments, see Figure 3C , Figure 3C is a flowchart of a task scheduling method provided in an embodiment of the present application. Figure 3A The illustrated step 107 can be implemented through steps 301 to 303 , which will be described in conjunction with each step.

[0143] In step 301, the scheduling node performs load-bearing scale estimation processing according to the node loads corresponding to the multiple nodes to be migrated, and obtains the loadable task scales corresponding to the multiple nodes to be migrated.

[0144] Here, when there are multiple tasks to be migrated and multiple nodes to be migrated, an allocation problem is involved, that is, which task to be migrated is migrated to which node to be migrated. In view of this, the scheduling node performs a load-bearing scale estimation process based on the node loads corresponding to the multiple nodes to be migrated, and obtains the loadable task scales corresponding to the multiple nodes to be migrated. It can be understood that the larger the node load, the smaller the corresponding loadable task scale.

[0145] In some embodiments, the above-mentioned scheduling node can be implemented in the following way: the scheduling node performs load-bearing scale estimation processing according to the node loads corresponding to multiple nodes to be migrated, and obtains the load-bearing task scales corresponding to multiple nodes to be migrated: the scheduling node determines the load to be migrated of each node to be migrated according to the standard load-bearing load of each node to be migrated and the node load of each node to be migrated, and determines the corresponding load-bearing task scale according to the load to be migrated of each node to be migrated.

[0146] For example, for each node to be migrated, the scheduling node calculates the difference between the standard load of the node to be migrated and the node load of the node to be migrated (referring to the current node load) as the load to be migrated. In an embodiment of the present application, the mapping relationship between the node load and the task scale can be preset, that is, 1 unit of node load corresponds to how many units of task scale. In this way, after determining the load to be migrated, the task scale corresponding to the load to be migrated is determined according to the mapping relationship as the loadable task scale. The loadable task scale is the task scale of the task that needs to be migrated to the node to be migrated. In the above manner, a reasonable loadable task scale can be determined, and then after the task is migrated, the node load of the node to be migrated can be close to the standard load, and the node load will not be too large.

[0147] In step 302, the scheduling node allocates tasks to be migrated according to the loadable task scales corresponding to the multiple nodes to be migrated and the task scales corresponding to the multiple tasks to be migrated; wherein the task scale of the tasks to be migrated allocated to each node to be migrated does not exceed the loadable task scale of the corresponding node to be migrated.

[0148] Here, based on the principle that the task size of each to-be-migrated node assigned to the to-be-migrated task does not exceed the task size that the corresponding to-be-migrated node can carry, the scheduling node performs the to-be-migrated task allocation process. On this basis, other constraints can also be added according to the actual application scenario, for example, preferentially allocating to-be-migrated nodes with smaller node loads, or allocating at least one to-be-migrated task to each to-be-migrated node, etc.

[0149] In some embodiments, the above-mentioned scheduling node can be implemented to allocate tasks to be migrated according to the loadable task scales corresponding to the multiple nodes to be migrated and the task scales corresponding to the multiple tasks to be migrated in the following manner: the scheduling node performs node sorting processing on the multiple nodes to be migrated according to the node loads corresponding to the multiple nodes to be migrated to obtain a node order; the multiple nodes to be migrated are traversed according to the node order, and the following processing is performed on the traversed nodes to be migrated: from the unallocated tasks to be migrated, the tasks to be migrated whose total task scale does not exceed the loadable task scale of the traversed nodes to be migrated are selected; and the selected tasks to be migrated are allocated to the traversed nodes to be migrated.

[0150] Here, the allocation of tasks to be migrated can be implemented based on the principle of giving priority to allocating tasks to be migrated to nodes with smaller node loads. For example, the scheduling node performs node sorting on multiple nodes to be migrated according to the node loads corresponding to the multiple nodes to be migrated, obtains a node order, and then traverses the multiple nodes to be migrated according to the node order, wherein, if the node sorting is performed in order from small to large according to the node load, the traversal refers to traversing from the first computing node in the node order; if the node sorting is performed in order from large to small according to the node load, the traversal refers to traversing from the last computing node in the node order.

[0151] For the traversed node to be migrated, the scheduling node selects the tasks to be migrated whose total task scale does not exceed the loadable task scale of the traversed node to be migrated from the unassigned tasks to be migrated, and allocates the selected tasks to be migrated to the traversed node to be migrated. On this basis, a stop condition can also be set for the selection process to fully select and speed up the allocation efficiency. The stop condition is such as: the total task scale of the selected tasks to be migrated does not exceed the loadable task scale of the traversed node to be migrated, but if the next task to be migrated is selected, the total task scale will exceed the loadable task scale of the traversed node to be migrated.

[0152] Among them, the selection can be random or in a specific order. For example, when the task itself has a serial number, the unassigned multiple tasks to be migrated can be selected in the order of task serial numbers from small to large, so as to give priority to the tasks to be migrated with smaller task serial numbers; or, the unassigned multiple tasks to be migrated can be selected in the order of their priorities from high to low, so as to give priority to the tasks to be migrated with higher priorities.

[0153] In addition, during the traversal process, if all the tasks to be migrated have been assigned, the traversal is stopped.

[0154] Through the above method, tasks can be allocated preferentially to the nodes to be migrated with smaller node loads, which is more conducive to the realization of load balancing and can also speed up the allocation efficiency.

[0155] In step 303, the scheduling node migrates the to-be-migrated tasks assigned to each to-be-migrated node to the corresponding to-be-migrated node.

[0156] For each node to be migrated into, the scheduling node migrates the tasks to be migrated that are assigned to the node to be migrated into from the node to be migrated out to which the tasks to be migrated originally belong to, to the node to be migrated into.

[0157] like Figure 3C As shown, the scheduling node allocates tasks according to the loadable task scale of the node to be migrated, and ensures that the task scale of the task to be migrated assigned to each node to be migrated does not exceed the loadable task scale of the corresponding node to be migrated. Thus, it can be ensured that after the task to be migrated assigned to each node to be migrated is migrated to the corresponding node to be migrated, the node to be migrated has sufficient resources to execute the task to be migrated, which can improve the effect of task migration and effectively avoid the situation where the task to be migrated to the node to be migrated exceeds the load-bearing capacity of the node to be migrated.

[0158] In some embodiments, see Figure 3D , Figure 3D is a flowchart of a task scheduling method provided in an embodiment of the present application. Figure 3A The illustrated step 106 can be implemented through steps 401 to 402 , which will be described in conjunction with each step.

[0159] In step 401, the scheduling node selects tasks whose migration frequency is less than a frequency threshold from among multiple tasks executed by the node to be migrated, and uses them as candidate tasks to be migrated.

[0160] If a task is frequently migrated, it will have an adverse effect on the overall data consistency. Moreover, for a task, the computing nodes that migrate out of the task and the computing nodes that migrate into the task may repeatedly calculate part of the data in the task. If a task is frequently migrated, some data may be repeatedly calculated more times. This will reduce the resource utilization of the computing nodes as a whole and also reduce the overall data consistency. Therefore, in an embodiment of the present application, the migration frequency of tasks can be limited. For example, based on the determined node to be migrated, the scheduling node selects tasks whose migration frequency is less than the frequency threshold from multiple tasks executed by the node to be migrated as candidate tasks to be migrated. Tasks other than candidate tasks to be migrated are not considered for task migration.

[0161] For each task that has been migrated in the past, the scheduling node can create a migration record, which includes the task identifier (such as the task name or task serial number) and the migration time (i.e., the time point at which the migration was performed). For example, for Task 1, the migration record can be in the form of "Task 1; September 1, 10 am". In this way, for each task executed by the node to be migrated, the migration frequency can be calculated based on the migration time in the migration record corresponding to the task.

[0162] In some embodiments, the above-mentioned scheduling node can be implemented in the following way: the scheduling node filters out tasks whose migration frequency is less than the frequency threshold from multiple tasks executed by the node to be migrated, as candidate tasks to be migrated: the scheduling node filters out tasks whose migration frequency is less than the frequency threshold in the migration time period from multiple tasks executed by the node to be migrated, as candidate tasks to be migrated.

[0163] Here, the migration frequency can be calculated within a preset migration time period, and the migration time period used to calculate the migration frequency can be the same for different tasks. In the process of calculating the migration frequency, for each task executed by the node to be migrated out, the number of migrations of the task within the migration time period can be counted, and the number of migrations can be divided by the length of the migration time period to obtain the migration frequency of the task.

[0164] For example, the current time is 11:00 a.m. on September 1, and the migration time period is within the last 10 minutes, that is, [10:50 a.m. on September 1, 11:00 a.m. on September 1], and the migration records of Task 1 obtained include "Task 1; 10:40 a.m. on September 1", "Task 1; 10:51 a.m. on September 1", and "Task 1; 10:55 a.m. on September 1". The two time points "10:51 a.m. on September 1" and "10:55 a.m. on September 1" fall into the migration time period, that is, Task 1 has migrated twice within the migration time period, and the migration frequency can be obtained as 2 times / 10 minutes. It is worth noting that if it is found that the migration time in some migration records does not fall into the migration time period during the calculation of the migration frequency, it proves that the migration records are no longer useful for calculating the migration frequency, so the migration records can be discarded to save storage resources, such as discarding the migration record "Task 1; 10:40 a.m. on September 1" above. Through the above method, it is possible to ensure that the statistical caliber of the migration frequency of each task is the same, so that it is convenient to accurately screen out candidate tasks to be migrated according to the migration frequency.

[0165] In some embodiments, the same frequency threshold applies to multiple tasks executed by the node to be migrated.

[0166] Here, the frequency threshold may be preset and applicable to multiple tasks executed by the node to be migrated. For example, if the frequency threshold is 1 time / 5 minutes, then for task 1 executed by the node to be migrated, it is determined whether the migration frequency of task 1 is less than 1 time / 5 minutes; for task 2 executed by the node to be migrated, it is also determined whether the migration frequency of task 2 is less than 1 time / 5 minutes.

[0167] In some embodiments, before the scheduling node filters out tasks whose migration frequency is less than a frequency threshold from multiple tasks executed by the node to be migrated, the task scheduling method also includes: the scheduling node calculates the frequency thresholds corresponding to the multiple tasks executed by the node to be migrated based on the task parameters corresponding to the multiple tasks executed by the node to be migrated.

[0168] Here, the frequency threshold can also be calculated based on the task parameters, that is, the frequency threshold of the task is calculated based on the task parameters of each task executed by the node to be migrated. Among them, the task parameters can include at least one of the task number, task progress and task priority, wherein the task number is generated by the scheduling node when generating the task, the smaller the task number, the earlier the task is generated, and the less it is desired to be a candidate task to be migrated, so the task number and the frequency threshold are set to be positively correlated, so that the earlier the task is generated, the faster it is executed; the task progress is counted by the computing node when executing the task, for example, the task progress is 5% executed, the higher the task progress of a task, the greater the cost of migrating the task, and the less it is desired to use the task as a candidate task to be migrated, so the task progress and the frequency threshold are set to be negatively correlated, so that the higher the task progress, the faster the task is executed; the task priority can be preset, for example, the corresponding priorities are set in advance for different types of tasks, the higher the priority of a task, the faster it is desired to execute and complete the task, that is, the less it is desired to use the task as a candidate task to be migrated, so the priority and the frequency threshold are set to be negatively correlated, so that the higher the priority, the faster the task is executed and completed.

[0169] Exemplarily, when the task parameters include the task number, task progress, and task priority at the same time, the calculation formula of the frequency threshold is as follows: frequency threshold = reference frequency threshold × (a × task number / (b × task progress + c × priority)). Wherein, "×" represents a multiplication operation, and " / " represents a division operation; the reference frequency threshold can be preset; a is the weight of the task number, b is the weight of the task progress, and c is the weight of the priority. A, b, and c are all numbers greater than 0, and the specific values ​​can be set according to the actual application scenario. Of course, in actual application scenarios, the calculation formula for the frequency threshold is not limited to the above example. As long as the calculation formula that satisfies the positive correlation between the task number and the frequency threshold, the negative correlation between the task progress and the frequency threshold, and the negative correlation between the priority and the frequency threshold can be applied to the embodiments of the present application.

[0170] In some embodiments, before the scheduling node selects tasks whose migration frequency is less than a frequency threshold from a plurality of tasks executed by the nodes to be migrated, the task scheduling method further includes: the scheduling node calculating the frequency threshold according to the number of nodes to be migrated.

[0171] Here, the more nodes to be migrated out, the more it is necessary to control the number of candidate tasks to be migrated determined in each node to be migrated out, so as to prevent too many tasks from migrating to the node to be migrated in, resulting in a surge in the node load of the node to be migrated in. Therefore, the number of nodes to be migrated out and the frequency threshold can be set to be negatively correlated, that is, the more nodes to be migrated out, the smaller the calculated frequency threshold, and the stricter the frequency limit based on the frequency threshold, which can effectively reduce the number of candidate tasks to be migrated determined in each node to be migrated out. For example, the calculation formula of the frequency threshold is as follows: frequency threshold = reference frequency threshold / (d × the number of nodes to be migrated out), where d is a number greater than 0, indicating the weight of the number of nodes to be migrated out. It is worth noting that the frequency threshold calculated here is applicable to all tasks in all nodes to be migrated out.

[0172] In some embodiments, before the scheduling node filters out tasks whose migration frequency is less than a frequency threshold from multiple tasks executed by the node to be migrated out, the task scheduling method also includes: the scheduling node matches the computing node where the task history is located with the node to be migrated in for each task executed by the node to be migrated out, and calculates the frequency threshold of the task based on the number of successfully matched nodes.

[0173] Here, targeted frequency limiting can be implemented based on the reason why the task was migrated. The reason why the task was migrated indicates the computing nodes involved in the task migration process (i.e., the computing nodes where the history was located). For each task executed by the node to be migrated, the computing node where the task history was located can be determined based on the migration record of the task. For example, for each task that has been migrated in history, the scheduling node can create a migration record based on each migration process of the task. The migration record includes the task identifier (such as the task name or task serial number) and the computing nodes involved in the migration process (which can also include the migration time). The format of the migration record is such as "Task 1; migrate from computing node A to computing node B", that is, it can be determined based on the migration record that the computing nodes where the history of Task 1 was located include computing node A and computing node B.

[0174] Since the nodes to be migrated have been determined, the computing nodes where the task history is located can be matched with the nodes to be migrated to obtain the number of nodes that have successfully matched. For example, the computing nodes where the task 1 history is located include computing nodes A and computing nodes B, and the nodes to be migrated include computing nodes A, computing nodes C, and computing nodes D. Then, computing node A in the computing node where the task 1 history is located is successfully matched with computing node A in the node to be migrated, and the number of nodes that have successfully matched is 1.

[0175] Then, the frequency threshold of the task is calculated based on the number of successfully matched nodes. In order to avoid the task being repeatedly migrated to the same computing node, the number of successfully matched nodes and the frequency threshold are set to be negatively correlated. In this way, the more nodes a task successfully matches, the stricter the frequency limit, and the less likely the task is to be determined as a candidate task to be migrated. For example, the calculation formula for the frequency threshold is as follows: frequency threshold = reference frequency threshold / (1 + e × number of successfully matched nodes), where e is a number greater than 0, indicating the weight of the number of successfully matched nodes. It is worth noting that the frequency threshold here is calculated for a single task and is only applicable to that task.

[0176] In some embodiments, before the scheduling node filters out tasks whose migration frequency is less than a frequency threshold from multiple tasks executed by the node to be migrated, the task scheduling method also includes: the scheduling node calculates the frequency threshold of the task for each task executed by the node to be migrated based on the number of nodes to be migrated, the task parameters of the task, and at least part of the number of successfully matched nodes corresponding to the task.

[0177] Here, all or part of the above-mentioned number of nodes to be migrated, task parameters, and number of successfully matched nodes can also be considered to calculate the frequency threshold. For example, the frequency threshold can be calculated based on the number of nodes to be migrated, task parameters, and number of successfully matched nodes. For each task performed by the node to be migrated, the frequency threshold calculation formula is as follows: frequency threshold = reference frequency threshold × (a × task number / ((b × task progress + c × priority) × (d × number of nodes to be migrated) × (1 + e × number of successfully matched nodes))); the frequency threshold can also be calculated only based on task parameters and the number of successfully matched nodes. For each task performed by the node to be migrated, the frequency threshold calculation formula is as follows: frequency threshold = reference frequency threshold × (a × task number / ((b × task progress + c × priority) × (1 + e × number of successfully matched nodes))), and so on. In this way, the flexibility of calculating the frequency threshold can be improved, and it is suitable for various frequency-limited scenarios.

[0178] In step 402, the scheduling node determines a task to be migrated from a plurality of candidate tasks to be migrated according to the task scales respectively corresponding to the plurality of candidate tasks to be migrated.

[0179] Here, the scheduling node screens the multiple candidate tasks to be migrated according to the task scales respectively corresponding to the multiple candidate tasks to be migrated, and obtains the tasks to be migrated.

[0180] like Figure 3DAs shown, the embodiment of the present application performs a first screening on multiple tasks to be executed on the node to be migrated according to the migration time of the tasks, that is, the tasks that have not been migrated in the third historical time period are screened out from the multiple tasks as candidate tasks to be migrated. After the first screening is completed, a second screening is performed based on the task scale, which can limit the migration frequency of the tasks, effectively avoid frequent migration of the same task, reduce the data duplication rate caused by task migration, and improve the overall data consistency.

[0181] In some embodiments, see Figure 3E , Figure 3E is a flowchart of a task scheduling method provided in an embodiment of the present application. Figure 3A The illustrated step 106 can be implemented through steps 501 to 502 , which will be described in conjunction with each step.

[0182] In step 501, the scheduling node performs task sorting processing on multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated, and obtains a task sequence.

[0183] Here, for multiple tasks executed by the node to be migrated out, if only tasks with a smaller task scale are migrated out, the effect on relieving the pressure of the node to be migrated out is limited, and the node load of the node to be migrated out may still be high after the migration; if only tasks with a larger task scale are migrated out, it will cause greater pressure on the node to be migrated in. Therefore, in the embodiment of the present application, tasks with a medium scale in the node to be migrated out are migrated, so that the pressure of the node to be migrated out can be effectively relieved, and excessive pressure will not be caused to the node to be migrated in, wherein the medium scale is determined based on the relative value of the task scale.

[0184] For example, the scheduling node sorts the multiple tasks according to the task scales corresponding to the multiple tasks executed by the node to be migrated, and obtains the task sequence, wherein the task sequence may be sorted in order of task scale from small to large or from large to small.

[0185] In some embodiments, the above-mentioned scheduling node can be implemented in the following way: according to the task scales corresponding to the multiple tasks executed by the node to be migrated, the scheduling node performs task sorting processing on the multiple tasks to obtain the task order: the scheduling node selects tasks whose migration frequency is less than the frequency threshold from the multiple tasks executed by the node to be migrated as candidate tasks to be migrated; according to the task scales corresponding to the multiple candidate tasks to be migrated, the scheduling node performs task sorting processing on the multiple candidate tasks to be migrated to obtain the task order.

[0186] Here, the scheduling node may also perform a first screening according to the migration times corresponding to the multiple tasks executed by the node to be migrated. After completing the first screening, the multiple candidate tasks to be migrated are sorted according to the task scales corresponding to the multiple candidate tasks to be migrated to obtain the task sequence.

[0187] In step 502, the scheduling node determines tasks in the middle sequence range in the task sequence as tasks to be migrated.

[0188] After completing the task sorting process and obtaining the task sequence, the scheduling node determines the tasks in the middle sequence range in the task sequence as tasks to be migrated. The middle sequence range can be pre-agreed with the number of tasks, for example, the third to the 100th tasks in the task sequence are all determined as tasks to be migrated; or the task ratio can be pre-agreed with, for example, the tasks in the middle of the task sequence and accounting for 50% of the total number of tasks are determined as tasks to be migrated.

[0189] It is worth noting that step 501 to step 502 may be performed once for each node to be migrated out, so as to filter out the tasks to be migrated from the multiple tasks executed by each node to be migrated out.

[0190] like Figure 3E As shown, the embodiment of the present application selects medium-sized tasks from multiple tasks as tasks to be migrated according to the size of the task scales of each task in the node to be migrated out. In this way, on the one hand, migrating the tasks to be migrated from the node to be migrated out can effectively relieve the pressure of the node to be migrated out; on the other hand, migrating the tasks to be migrated to the node to be migrated in will not cause excessive pressure on the node to be migrated in, so that the tasks to be migrated can be executed smoothly after migration, thereby improving the rationality of task migration and effectively reducing the number of task migrations.

[0191] In some embodiments, see Figure 3F , Figure 3F is a flowchart of a task scheduling method provided in an embodiment of the present application. Figure 3A The illustrated step 101 can be implemented through steps 601 to 603 , which will be described in conjunction with each step.

[0192] In step 601 , a computing node determines the current amount of input data for each task executed by the computing node.

[0193] In the embodiment of the present application, for a task, there are two kinds of data, input data and output data, during its execution. Taking the index calculation task in the log scenario as an example, the input data is the log data reported by the user, and the output data is the log index calculated based on the log data. However, the input data and the output data are often not of the same data order, so the task scale of the task can be determined by combining the input data and the output data.

[0194] First, the computing node can determine the current input data volume of each task executed by itself, where the current input data volume refers to the data volume of the current instantaneous input data.

[0195] In step 602, the computing node determines the data volume magnification ratio of each task according to the output data volume and the input data volume of each task in the third historical time period.

[0196] Secondly, the computing node can count the output data volume and input data volume of each task in the third historical time period, and divide the statistical output data volume by the input data volume to obtain the data volume magnification ratio. The output data volume and input data volume here are not instantaneous data volumes, but the total data volume counted in a time period. Among them, the third historical time period can be set according to the actual application scenario, such as within the last 5 minutes.

[0197] For example, the output data volume of task 1 in the third historical time period is 1000MB, and the input data volume of task 1 in the third historical time period is 10MB. Then, the data volume magnification ratio of task 1 can be obtained as 1000MB / 10MB=100.

[0198] In step 603, the computing node performs parameter fusion processing on the current input data volume, data volume magnification ratio and task complexity for calculating output data of each task to obtain the task scale of each task.

[0199] Thirdly, the computing node also determines the task complexity for calculating the output data in each task, which can be a pre-configured task parameter. For example, if the index calculation task is used to calculate the key value index, the number of key-value pairs configured by the user in the index calculation task can be used as the task complexity. The larger the number of key-value pairs, the more complex the task calculation.

[0200] Finally, the computing node performs parameter fusion processing on the current input data volume, data volume magnification rate, and task complexity for calculating output data of each task to obtain the task scale of each task. The embodiment of the present application does not limit the method of parameter fusion processing, including but not limited to multiplication, direct summation, weighted summation, etc., that is, the relationship between the current input data volume and the task scale, the relationship between the data volume magnification rate and the task scale, and the relationship between the task complexity and the task scale are all positively correlated.

[0201] like Figure 3F As shown, the embodiment of the present application combines three dimensions (current input data volume, data volume magnification rate and task complexity) to calculate the task scale, which can accurately measure the execution difficulty of the task and facilitate the scheduling node to screen out the tasks that really need to be migrated according to the task scale.

[0202] Below, we will explain the exemplary application of the embodiment of the present application in an actual application scenario. For ease of understanding, we will take the log service platform as an example. The embodiment of the present application ensures efficient and stable task scheduling through refined task scale assessment, computing node load scheduling, and task migration frequency limiting, supports cloud native service elasticity, and allows users to use the cloud native log service provided by the log service platform without paying attention to system capacity, and can open hundreds of log topics at the same time; for the log service platform, when facing huge traffic, it can ensure that data is not delayed, repeated, or lost, and at the same time, it has excellent cost performance and low implementation cost.

[0203] The present application embodiment provides Figure 4 The architecture diagram of the log service platform shown in the figure can abstract the components in the log service platform into two roles, namely the scheduling node (Scheduler) and the computing node (Worker). Among them, the Scheduler is responsible for the generation and scheduling of various tasks, and the Worker is used to receive the tasks sent by the Scheduler and perform specific computing work.

[0204] The present application embodiment provides Figure 5 The interaction diagram between Scheduler and Worker is shown in Figure 5 As shown, Scheduler is used to detect business conditions and generate corresponding tasks according to business conditions. For example, several log topics can be divided in the log service platform, and each log topic includes tasks such as index calculation, object storage backup, and log delivery. For the generated tasks, Scheduler selects several workers with the lowest node load (corresponding to the task execution nodes mentioned above) according to the node loads corresponding to multiple workers, and issues the tasks.

[0205] When a Worker receives a task from the Scheduler, it starts to perform specific calculations and periodically reports heartbeat information to the Scheduler. The heartbeat information includes the task size corresponding to each task executed by the Worker and the node load of the Worker itself.

[0206] The Scheduler will continue to receive heartbeat information reported by the Worker. If it finds that the node load of a Worker exceeds the first load threshold, it will migrate part of the Worker's tasks to other low-load Workers.

[0207] The above-mentioned refined task scale assessment is performed by Worker, while computing node load scheduling and task migration frequency limiting are performed by Scheduler, which will be explained separately below.

[0208] 1) Refine the task scale assessment.

[0209] The task size assessment faces the following challenges, which makes the task size determined using only the current input data volume inaccurate:

[0210] ① Data volume magnification rate difference: For different tasks, the data volume magnification rate between the data received by the Worker when executing the task (i.e., input data) and the output data may be very different. Even for different data received for the same task, the data volume magnification rate may be very different. Taking the index calculation task as an example, based on the functional design of the log service platform, the data received by the Worker when executing the index calculation task is in the LogGroup package format. A LogGroup package mainly consists of tags (topics) and log lines. Only one tag is stored in a LogGroup package, and when the LogGroup package is parsed into an index, a tag is stored for each line of log. Since a LogGroup package can contain up to 10,000 lines of logs, the data volume magnification rate may differ by dozens of times when different LogGroup packages are parsed.

[0211] ② Difference in task complexity: Taking the index calculation task as an example, in the log service platform, if the user only enables the full-text indexing function, the Worker hardly needs to perform any calculations when executing the index calculation task; however, if the user configures multiple layers of complex key-value indexes, the Worker needs to perform very heavy calculations.

[0212] Therefore, the embodiment of the present application uses the current input data volume of the task as a benchmark, combined with the data volume magnification rate and task complexity, to finely evaluate the scale of a task. The calculation formula for the task scale is as follows:

[0213]

[0214] Among them, inputflow represents the current input data volume; represents the output data volume in the time period [t1, t2] (corresponding to the third historical time period above); Indicates the amount of input data in the time period [t1, t2]; Indicates the data volume magnification rate; keylength indicates the number of key-value pairs corresponding to the index, that is, the task complexity.

[0215] 2) Computing node load scheduling.

[0216] The heartbeat information of the Worker received by the Scheduler contains the task scale of each task executed by the Worker and the node load of the Worker itself. The Scheduler determines whether to migrate some tasks from WorkerA to WorkerB based on the heartbeat information of each Worker. The Scheduler mainly solves three problems: how to determine whether a Worker needs to migrate out tasks; how to determine whether a Worker can migrate in tasks; and for Workers that need to migrate out tasks, which tasks are more suitable to migrate out.

[0217] The first two questions are how to evaluate the node load size of the Worker, that is, when the node load of the Worker is large enough, the task needs to be migrated out, and when the node load of the Worker is small enough, the task can be migrated in. Therefore, a first load threshold and a second load threshold can be set. When the node load of a Worker is greater than the first load threshold, the Worker is used as a Worker to be migrated out (corresponding to the node to be migrated out above); when the node load of a Worker is less than the second load threshold, the Worker is used as a Worker to be migrated in (corresponding to the node to be migrated in above). It is worth noting that when the node load includes multiple indicators, a threshold can be set for each indicator. For example, the node load includes CPU usage, memory usage, and disk usage. In the first load threshold, the threshold corresponding to the CPU usage is 90%, the threshold corresponding to the memory usage is 85%, and the threshold corresponding to the disk usage is 85%; in the second load threshold, the threshold corresponding to the CPU usage is 20%, the threshold corresponding to the memory usage is 15%, and the threshold corresponding to the disk usage is 15%.

[0218] On this basis, since the node load of a Worker may fluctuate over time, the stability issue can be resolved by evaluating the node load in a time period to determine whether to migrate in or out. For example, when the multiple node loads of a Worker in the first historical time period (such as the last 5 minutes) are greater than the first load threshold, the Worker is considered a Worker to be migrated out; when the multiple node loads of a Worker in the second historical time period (such as the last 1 hour) are less than the second load threshold, the Worker is considered a Worker to be migrated in.

[0219] Regarding the question of what tasks are suitable for migration, such as Figure 6 As shown ( Figure 6 The size of the area filled with the pattern indicates the size of the task scale. The larger the area of ​​the filled pattern, the larger the task scale of the task, and the smaller the area of ​​the filled pattern, the smaller the task scale of the task). If only large-scale tasks are migrated, it is easy for the node load of the Worker to which they are migrated to to surge, that is, the node load of the Worker may be too high after the migration, which is not conducive to the goal of load balancing. If only small-scale tasks are migrated, it will have no effect on reducing the node load of the Worker to be migrated. Therefore, the embodiment of the present application migrates medium-sized tasks. For example, for each Worker to be migrated, according to the task scales corresponding to the multiple tasks executed by the Worker to be migrated, the multiple tasks are sorted to obtain a task order, and the tasks from the 3rd to the 100th (corresponding to the middle order range above) in the task order are determined as tasks to be migrated.

[0220] If the number of tasks to be migrated and the number of workers to be migrated are both multiple, then during the migration process, the multiple workers to be migrated can be sorted in order from small to large according to the node load to obtain the node order. Then, the multiple workers to be migrated are traversed in the node order, and unassigned tasks to be migrated are selected and assigned to the traversed workers to be migrated, wherein the total task scale of the selected tasks to be migrated does not exceed the loadable task scale (i.e., the load limit) of the traversed workers to be migrated.

[0221] 3) Task migration frequency limit.

[0222] In the embodiment of the present application, a migration frequency limiter is abstracted in the Scheduler to implement task migration frequency limiting. The migration frequency limiter is used to skip tasks that have been migrated in the most recent time period (such as 5 minutes) when selecting tasks to be migrated, thereby avoiding frequent scheduling of the same task.

[0223] The embodiments of the present application can at least achieve the following technical effects: 1) It can significantly reduce the number of task migrations and effectively improve the success rate of task execution. Even in a failure scenario, log data is no longer discarded, and the overall data consistency reaches a quasi-Exactly once level; 2) The implementation cost of the embodiments of the present application is low, there is no need to configure complex features, it is easy to deploy, and it is suitable for task scheduling in various scenarios.

[0224] The following is a description of an exemplary structure of the task scheduling device 255 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the task scheduling device 255 of the memory 250 may include: an acquisition module 2551, used to acquire the task scales and node loads sent by multiple computing nodes; a node determination module 2552, used to determine the nodes to be migrated out and the nodes to be migrated in from multiple computing nodes according to the node loads corresponding to the multiple computing nodes respectively; a task determination module 2553, used to determine the tasks to be migrated from multiple tasks according to the task scales corresponding to the multiple tasks executed by the nodes to be migrated out; and a migration module 2554, used to migrate the tasks to be migrated from the nodes to be migrated out to the nodes to be migrated in.

[0225] In some embodiments, the node determination module 2552 is also used to: screen out computing nodes whose node load is greater than a first load threshold from multiple computing nodes as nodes to be migrated out; screen out computing nodes whose node load is less than a second load threshold from multiple computing nodes as nodes to be migrated in; wherein the first load threshold is greater than the second load threshold.

[0226] In some embodiments, the acquisition module 2551 is also used to obtain the task scale and node load sent by multiple computing nodes at multiple time points; the node determination module 2552 is also used to: screen out, from multiple computing nodes, computing nodes whose multiple node loads in a first historical time period are greater than a first load threshold as nodes to be migrated out; screen out, from multiple computing nodes, computing nodes whose multiple node loads in a second historical time period are less than a second load threshold as nodes to be migrated in.

[0227] In some embodiments, the number of tasks to be migrated includes multiple, and the number of nodes to be migrated includes multiple; the migration module 2554 is also used to: perform load-bearing scale estimation processing according to the node loads corresponding to the multiple nodes to be migrated, and obtain the loadable task scales corresponding to the multiple nodes to be migrated; perform task allocation processing according to the loadable task scales corresponding to the multiple nodes to be migrated, and the task scales corresponding to the multiple tasks to be migrated; wherein the task scale of the task to be migrated assigned to each node to be migrated does not exceed the loadable task scale of the corresponding node to be migrated; migrate the task to be migrated assigned to each node to be migrated to the corresponding node to be migrated.

[0228] In some embodiments, the migration module 2554 is also used to: sort the multiple nodes to be migrated according to the node loads corresponding to the multiple nodes to be migrated, and obtain a node order; traverse the multiple nodes to be migrated according to the node order, and perform the following processing on the traversed nodes to be migrated: select tasks to be migrated whose total task scale does not exceed the loadable task scale of the traversed nodes to be migrated from the unassigned tasks to be migrated; and assign the selected tasks to be migrated to the traversed nodes to be migrated.

[0229] In some embodiments, the task determination module 2553 is also used to: filter out tasks whose migration frequency is less than a frequency threshold from multiple tasks executed by the node to be migrated, as candidate tasks to be migrated; and determine the task to be migrated from multiple candidate tasks to be migrated according to the task scales corresponding to the multiple candidate tasks to be migrated.

[0230] In some embodiments, the task determination module 2553 is further used to: sort the multiple tasks according to the task scales corresponding to the multiple tasks executed by the node to be migrated, and obtain a task sequence; and determine the tasks in the middle sequence range in the task sequence as the tasks to be migrated.

[0231] In some embodiments, the task scheduling device 255 also includes a task sending module, which is used to: when generating a task to be executed, determine a task execution node from multiple computing nodes according to the node loads corresponding to the multiple computing nodes; send the generated task to the task execution node so that the task execution node executes the received task.

[0232] The following is a description of an exemplary structure of the task scheduling device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2BAs shown, the software modules stored in the task scheduling device 455 of the memory 450 may include: a scale estimation module 4551, which is used to perform task scale estimation processing on each task executed by the computing node to obtain the task scale of each task; a load detection module 4552, which is used to obtain the node load of the computing node through load detection; a sending module 4553, which is used to send the task scale of each task executed by the computing node and the node load of the computing node to the scheduling node, so that the scheduling node performs the following processing: obtain the task scales and node loads sent by multiple computing nodes; determine the nodes to be migrated out and the nodes to be migrated in from the multiple computing nodes according to the node loads respectively corresponding to the multiple computing nodes; determine the tasks to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the nodes to be migrated out; migrate the tasks to be migrated from the nodes to be migrated out to the nodes to be migrated in.

[0233] In some embodiments, the scale estimation module 4551 is also used to: determine the current input data volume of each task executed by the computing node; determine the data volume magnification ratio of each task based on the output data volume and input data volume of each task in the third historical time period; perform parameter fusion processing on the current input data volume, data volume magnification ratio and task complexity for calculating the output data of each task to obtain the task scale of each task.

[0234] The embodiment of the present application provides a computer program product or a computer program, which includes executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the executable instructions from the computer-readable storage medium, and the processor executes the executable instructions, so that the electronic device executes the task scheduling method described in the embodiment of the present application.

[0235] An embodiment of the present application provides a computer-readable storage medium storing executable instructions, wherein executable instructions are stored. When the executable instructions are executed by a processor, the processor will execute the task scheduling method provided by the embodiment of the present application.

[0236] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0237] In some embodiments, executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0238] As an example, executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0239] By way of example, executable instructions may be deployed to be executed on one computing device, or on multiple computing devices located at one site, or on multiple computing devices distributed across multiple sites and interconnected by a communication network.

[0240] The above are only embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A task scheduling method, characterized in that: include: Get the task scale and node load sent by multiple computing nodes; Determining a node to be migrated out and a node to be migrated in from the plurality of computing nodes according to the node loads respectively corresponding to the plurality of computing nodes; Determining the task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated; Migrate the task to be migrated from the node to be migrated out to the node to be migrated in.

2. The method according to claim 1, characterized in that The determining, from the plurality of computing nodes, nodes to be migrated out and nodes to be migrated in according to the node loads respectively corresponding to the plurality of computing nodes, comprises: Filtering out computing nodes whose node loads are greater than a first load threshold from among the multiple computing nodes as nodes to be migrated out; Filtering out computing nodes whose node loads are less than a second load threshold from the plurality of computing nodes as nodes to be migrated; The first load threshold is greater than the second load threshold.

3. The method according to claim 2, characterized in that The obtaining of task scales and node loads sent by multiple computing nodes includes: Obtain the task scale and node load sent by multiple computing nodes at multiple time points; The step of selecting computing nodes whose node load is greater than a first load threshold from among the plurality of computing nodes as nodes to be migrated out includes: Among the multiple computing nodes, select computing nodes whose multiple node loads in a first historical time period are all greater than the first load threshold as nodes to be migrated; The step of selecting computing nodes whose node load is less than a second load threshold from among the plurality of computing nodes as nodes to be migrated includes: Among the multiple computing nodes, multiple computing nodes whose node loads in the second historical time period are all less than the second load threshold are screened out as nodes to be migrated.

4. The method according to claim 1, characterized in that The number of tasks to be migrated includes multiple, and the number of nodes to be migrated includes multiple; the migrating the tasks to be migrated from the nodes to be migrated out to the nodes to be migrated in includes: Performing load estimation processing according to the node loads corresponding to the multiple nodes to be migrated, thereby obtaining the loadable task scales corresponding to the multiple nodes to be migrated; Performing task allocation processing according to the loadable task scales corresponding to the plurality of nodes to be migrated and the task scales corresponding to the plurality of tasks to be migrated; wherein the task scale of the task to be migrated allocated to each node to be migrated does not exceed the loadable task scale of the corresponding node to be migrated; Migrate the tasks to be migrated assigned to each node to be migrated to the corresponding node to be migrated.

5. The method according to claim 4, characterized in that The processing of allocating tasks to be migrated according to the loadable task scales respectively corresponding to the multiple nodes to be migrated and the task scales respectively corresponding to the multiple tasks to be migrated includes: According to the node loads respectively corresponding to the multiple nodes to be migrated, the multiple nodes to be migrated are sorted to obtain a node sequence; The plurality of nodes to be migrated are traversed in the node order, and the following processing is performed on the traversed nodes to be migrated: Selecting, from the unallocated tasks to be migrated, the tasks to be migrated whose total task size does not exceed the task size that can be carried by the traversed node to be migrated; The selected tasks to be migrated are assigned to the traversed nodes to be migrated into.

6. The method according to claim 1, characterized in that The step of determining the task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated includes: Selecting tasks whose migration frequency is less than a frequency threshold from among the multiple tasks executed by the node to be migrated, as candidate tasks to be migrated; A task to be migrated is determined from the multiple candidate tasks to be migrated according to task scales respectively corresponding to the multiple candidate tasks to be migrated.

7. The method according to claim 1, characterized in that The step of determining the task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated includes: According to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated, the multiple tasks are sorted to obtain a task sequence; Tasks in the middle sequence range in the task sequence are determined as tasks to be migrated.

8. The method according to any one of claims 1 to 7, characterized in that: The method further comprises: When generating a task to be executed, determining a task execution node from the multiple computing nodes according to the node loads respectively corresponding to the multiple computing nodes; The generated task is sent to the task execution node, so that the task execution node executes the received task.

9. A task scheduling method, characterized in that: include: Perform task scale estimation on each task executed by the computing node to obtain the task scale of each task; Obtaining the node load of the computing node through load detection; The task scale of each task executed by the computing node and the node load of the computing node are sent to the scheduling node, so that the scheduling node executes the task scheduling method described in any one of claims 1 to 8.

10. The method according to claim 9, characterized in that The task scale estimation process is performed on each task executed by the computing node to obtain the task scale of each task, including: Determining a current amount of input data for each task executed by the computing node; Determine a data volume magnification rate of each task according to the output data volume and the input data volume of each task in the third historical time period; The current input data volume, data volume magnification ratio and task complexity for calculating output data of each task are subjected to parameter fusion processing to obtain the task scale of each task.

11. A task scheduling device, characterized in that: include: An acquisition module is used to obtain the task scale and node load sent by multiple computing nodes; A node determination module, configured to determine a node to be migrated out and a node to be migrated in from the plurality of computing nodes according to the node loads respectively corresponding to the plurality of computing nodes; A task determination module, configured to determine a task to be migrated from the multiple tasks according to the task scales respectively corresponding to the multiple tasks executed by the node to be migrated; A migration module is used to migrate the task to be migrated from the node to be migrated out to the node to be migrated in.

12. A task scheduling device, characterized in that: include: A scale estimation module is used to estimate the task scale of each task executed by the computing node to obtain the task scale of each task; A load detection module, used to obtain the node load of the computing node through load detection; A sending module is used to send the task scale of each task executed by the computing node and the node load of the computing node to the scheduling node, so that the scheduling node executes the task scheduling method described in any one of claims 1 to 8.

13. An electronic device, characterized in that: include: A memory for storing executable instructions; The processor is used to implement the task scheduling method described in any one of claims 1 to 8, or the task scheduling method described in any one of claims 9 to 10 when executing the executable instructions stored in the memory.

14. A computer-readable storage medium, characterized in that: Executable instructions are stored, and when executed by a processor, the task scheduling method described in any one of claims 1 to 8 or the task scheduling method described in any one of claims 9 to 10 is implemented.

15. A computer program product, characterized in that It includes executable instructions, which are used to implement the task scheduling method described in any one of claims 1 to 8, or the task scheduling method described in any one of claims 9 to 10 when executed by a processor.

Images