Resource scheduling system and method

Abstract

The embodiments of the present specification provide a resource scheduling system and method, wherein the resource scheduling system comprises: a task processor, a scheduler and a resource processing node; wherein the task processor is configured to determine a target subtask according to task information of a to-be-processed task, determine a subtask unit in the target subtask, and submit a resource acquisition request of the target subtask to the scheduler; the scheduler is configured to acquire schedulable resource information of an initial processing node in response to the resource acquisition request, calculate a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information, and send the resource scheduling result to the task processor and the resource processing node; the task processor is configured to send the subtask unit to the resource processing node according to the resource scheduling result; and the resource processing node is configured to receive the resource scheduling result and the subtask unit, and execute the subtask unit based on a target resource corresponding to the resource scheduling result.

Description

TECHNICAL FIELD

[0001] Embodiments of the present specification relate to the technical field of computer technology, and in particular to a resource scheduling system and method. BACKGROUND

[0002] With the development of the Internet, a huge resource system is usually needed to support rich online project forms such as search, e-commerce, big data, database, etc. The scale of online machines is so large that if the utilization rate is improved by one percent, it will bring huge cost savings. Therefore, improving cluster utilization has always been one of the core goals of the scheduling team of each group. However, the challenge of mixed deployment is also huge, and the most core challenge is how to provide a resource sharing mechanism (global, single machine) to achieve the goal of improving cluster utilization rate under the premise of guaranteeing the SLA (Service-Level Agreement) of each application.

[0003] The current online cluster is characterized by low-concurrency scheduling (containers only need to be started once and do not need to be frequently scheduled), mainly solving the problems of container orchestration and management, and is more suitable for scenarios where tasks run for a long time. However, in the high-concurrency job scenario of big data computing, there is no effective solution. And the K8S scheduling system is for low-concurrency, fine-grained resource orchestration and management scheduling scenarios. Neither in the scheduling protocol nor in the scheduler design, the K8S scheduling system can support high-concurrency scheduling capabilities. Therefore, there is an urgent need to provide a solution to the above problems. SUMMARY

[0004] Therefore, the embodiments of the present specification provide a resource scheduling system. One or more embodiments of the present specification simultaneously relate to three kinds of resource scheduling methods, a computing device, a computer readable storage medium and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of the embodiments of the present specification, a resource scheduling system is provided, comprising:

[0006] a task processor, a scheduler and a resource processing node;

[0007] The task processor is configured to determine a target subtask according to task information of a to-be-processed task, determine a subtask unit in the target subtask, and submit a resource acquisition request of the target subtask to the scheduler.

[0008] The scheduler is configured to acquire schedulable resource information of an initial processing node in response to the resource acquisition request, calculate a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information, and send the resource scheduling result to the task processor and the resource processing node.

[0009] The task processor is configured to send the subtask unit to the resource processing node according to the resource scheduling result.

[0010] The resource processing node is configured to receive the resource scheduling result and the subtask unit, and execute the subtask unit based on a target resource corresponding to the resource scheduling result.

[0011] Optionally, the task processor is further configured to:

[0012] determine subtask information and a subtask execution order according to task information of the to-be-processed task, extract a target subtask in the to-be-processed task according to the subtask information and the subtask execution order, create a subtask unit in the target subtask according to target subtask information of the target subtask, and determine resource acquisition sub-information of the subtask unit, create a resource acquisition request of the target subtask based on the resource acquisition sub-information, and submit the resource acquisition request to the scheduler.

[0013] Optionally, the task processor is further configured to:

[0014] screen a target subtask unit in the subtask unit according to the resource scheduling result, and send the target subtask unit to the resource processing node based on the resource scheduling result.

[0015] Correspondingly, the resource processing node is further configured to:

[0016] receive the target subtask unit, determine a target resource corresponding to the resource scheduling result, and execute the target subtask unit through the target resource, and send an execution result of the target subtask unit to the task processor in a case where the target subtask unit is executed completely.

[0017] Optionally, the task processor is further configured to:

[0018] determine a remaining subtask unit in the subtask unit based on the execution result, and send the remaining subtask unit to the resource processing node.

[0019] Correspondingly, the resource processing node is further configured to:

[0020] receive the remaining sub-task units sent by the task processor; and execute the remaining sub-task units through the target resources corresponding to the target sub-task units.

[0021] Optionally, the task processor is further configured to:

[0022] in a case where there is an unexecuted sub-task unit in the target sub-task, create sub-resource revocation information based on the execution result, and send the sub-resource revocation information to the scheduler;

[0023] Correspondingly, the scheduler is further configured to:

[0024] update the unscheduled resource information corresponding to the resource acquisition request based on the sub-resource revocation information.

[0025] Optionally, the task processor is further configured to:

[0026] in a case where there is no unexecuted sub-task unit in the target sub-task, create sub-resource return information based on the execution result, and send the sub-resource return information to the scheduler;

[0027] Correspondingly, the scheduler is further configured to:

[0028] update the resource scheduling information corresponding to the resource acquisition request based on the sub-resource return information.

[0029] Optionally, the task processor is further configured to:

[0030] in a case where the execution of the sub-task unit is completed, determine a second target sub-task to be executed after the target sub-task according to the task information of the to-be-processed task, take the second target sub-task as the target sub-task, and return to the step of determining the sub-task unit in the target sub-task.

[0031] Optionally, the scheduler is further configured to:

[0032] determine a preset scheduling algorithm corresponding to the resource acquisition request; determine a resource processing node corresponding to the resource acquisition request and resource scheduling information corresponding to the resource processing node according to the schedulable resource information and the preset scheduling algorithm; and take the resource scheduling information as the resource scheduling result.

[0033] Optionally, the scheduler is further configured to:

[0034] screening candidate processing nodes in the initial processing node according to the schedulable resource information and resource acquisition information carried by the resource acquisition request; scoring the candidate processing nodes according to node information corresponding to the candidate processing nodes and a preset scoring rule to obtain a node score of each candidate processing node; screening a resource processing node in the candidate processing nodes based on the node score; and determining resource scheduling information corresponding to the resource processing node according to the resource acquisition information.

[0035] Optionally, the scheduler is further configured to:

[0036] in a case where it is detected that the schedulable resource information of the initial processing node has changed, determining a candidate resource processing node to be scheduled and a target request queue to be traversed in the initial processing node;

[0037] traversing a queue resource acquisition request in the target request queue based on target schedulable resource information of the candidate resource processing node and a preset traversal order;

[0038] in a case where the target schedulable resource information meets a resource acquisition condition corresponding to the resource acquisition request, taking the candidate resource processing node as a resource processing node of the resource acquisition request;

[0039] determining resource scheduling information corresponding to the resource processing node according to resource acquisition information of the resource acquisition request.

[0040] Optionally, the scheduler is further configured to:

[0041] acquiring a queue resource acquisition request in the target request queue; constructing a request tree according to a request priority and request resource information of the queue resource acquisition request; and traversing the queue resource acquisition request in the request tree.

[0042] Optionally, the scheduler is further configured to:

[0043] in a case where a task to be processed is detected, determining a target processing node in an initial processing node, and sending task information of the task to be processed to the target processing node;

[0044] Correspondingly, the target processing node is configured to:

[0045] receive the task information, and start a task processor corresponding to the task to be processed based on the task information.

[0046] According to a second aspect of an embodiment of the present specification, a resource scheduling method applied to a task processor is provided, comprising:

[0047] determine a target subtask according to task information of a task to be processed, determine a subtask unit in the target subtask, and submit a resource acquisition request of the target subtask to a scheduler;

[0048] receive a resource scheduling result of the resource acquisition request returned by the scheduler;

[0049] send the subtask unit to a resource processing node according to the resource scheduling result.

[0050] According to a third aspect of the embodiments of the present specification, a resource scheduling method is provided, applied to a scheduler, including:

[0051] receiving a resource acquisition request of a target subtask sent by a task processor;

[0052] obtaining schedulable resource information of an initial processing node in response to the resource acquisition request;

[0053] calculating a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information;

[0054] and sending the resource scheduling result to the task processor and the resource processing node.

[0055] According to a fourth aspect of the embodiments of the present specification, a resource scheduling method is provided, applied to a resource processing node, including:

[0056] receiving a resource scheduling result of a target subtask sent by a scheduler, and receiving a subtask unit of the target subtask sent by a task processor;

[0057] executing the subtask unit based on a target resource corresponding to the resource scheduling result.

[0058] According to a fifth aspect of the embodiments of the present specification, a computing device is provided, including:

[0059] a memory and a processor;

[0060] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions, which realize the steps of the above resource scheduling method when executed by the processor.

[0061] According to a sixth aspect of the embodiments of the present specification, a computer readable storage medium is provided, which stores computer executable instructions, which realize the steps of the above resource scheduling method when executed by the processor.

[0062] According to a seventh aspect of the embodiments of the present specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the resource scheduling method.

[0063] The resource scheduling system provided by the present specification realizes resource scheduling on the to-be-processed task through interaction between the task processor, the scheduler and the resource processing node; the task processor is configured to determine a target subtask according to task information of the to-be-processed task, determine a subtask unit in the target subtask, and submit a resource acquisition request of the target subtask to the scheduler; the scheduler is configured to acquire schedulable resource information of the resource processing node in response to the resource acquisition request, calculate a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information, and send the resource scheduling result to the task processor and the resource processing node; the task processor is configured to send the subtask unit to the resource processing node according to the resource scheduling result; and the resource processing node is configured to receive the resource scheduling result and the subtask unit, and execute the subtask unit based on a target resource corresponding to the resource scheduling result. The resource scheduling system avoids resource interaction caused by separate resource application for each subtask unit, improves resource scheduling efficiency, and improves processing efficiency of the to-be-processed task. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 FIG. 1 is a structural schematic diagram of a resource scheduling system according to an embodiment of the present specification;

[0065] Figure 2 FIG. 2 is an architectural diagram of a resource scheduling system according to an embodiment of the present specification;

[0066] Figure 3 FIG. 3 is an implementation schematic diagram of a resource scheduling system according to an embodiment of the present specification;

[0067] Figure 4 FIG. 4 is a schematic diagram of a request queue in a resource scheduling system according to an embodiment of the present specification;

[0068] Figure 5 FIG. 5 is a schematic diagram of a scheduling process in a resource scheduling system according to an embodiment of the present specification;

[0069] Figure 6 FIG. 6 is an interaction schematic diagram of sub-resource information in a resource scheduling system according to an embodiment of the present specification;

[0070] Figure 7 FIG. 7 is a schematic diagram of traversing a queue resource acquisition request in a resource scheduling system according to an embodiment of the present specification;

[0071] Figure 8is a flowchart of a first resource scheduling method provided by an embodiment of the present specification.

[0072] Figure 9 is a flowchart of a second resource scheduling method provided by an embodiment of the present specification.

[0073] Figure 10 is a flowchart of a third resource scheduling method provided by an embodiment of the present specification.

[0074] Figure 11 is a structural block diagram of a computing device provided by an embodiment of the present specification. DETAILED DESCRIPTION

[0075] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present specification. However, the present specification can be practiced without the specific details, other than in the examples, and it can be apparent to those skilled in the art that the present specification can be practiced without the specific details. In other instances, well-known methods, procedures, components, and networks have not been described in detail so as not to unnecessarily obscure aspects of the present specification.

[0076] The terminology used in one or more embodiments of the present specification is for the purpose of describing particular embodiments only and is not intended to be limiting of one or more embodiments of the present specification. As used in one or more embodiments of the present specification and the accompanying claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in one or more embodiments of the present specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0077] It will be understood that, although the terms first, second, etc. can be used herein to describe various information, these terms are not intended to denote a temporal or chronological order. Rather, these terms are used solely to distinguish one from another only. For example, without departing from the scope of one or more embodiments of the present specification, first can be termed second, and similarly, second can be termed first. The term "if' as used herein, can be interpreted as meaning "when" or "in response to determining" depending on the context.

[0078] First, the noun terms related to one or more embodiments of the present specification are explained.

[0079] Kubernetes (K8S): is a container cluster management system, which is an open source platform that can realize functions such as automatic deployment, automatic scaling, maintenance, etc. of a container cluster.

[0080] Pod: can be understood as the role of "virtual machine" in the traditional deployment environment. Through the Pod, users can make the migration from the traditional environment (virtual machine environment) to Kubernetes (container environment) more smoothly. For example, all scheduling, network, storage, and security-related attributes are basically at the Pod level. The common feature of these attributes is that they describe the "machine" as a whole, rather than the "program" running inside.

[0081] In the present specification, a resource scheduling system is provided, and the present specification also relates to three resource scheduling methods, a computing device, and a computer readable storage medium, which are described in detail one by one in the following embodiments.

[0082] Referring to Figure 1 , Figure 1 A structural schematic diagram of a resource scheduling system provided according to an embodiment of the present specification is shown. The resource scheduling system comprises:

[0083] a task processor 102, a scheduler 104, and a resource processing node 106;

[0084] The task processor 102 is configured to create a target subtask according to task information of a to-be-processed task, determine a subtask unit in the target subtask, and submit a resource acquisition request of the target subtask to the scheduler 104.

[0085] The scheduler 104 is configured to acquire schedulable resource information of an initial processing node in response to the resource acquisition request, calculate a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information, and send the resource scheduling result to the task processor 102 and the resource processing node 106.

[0086] The task processor 102 is configured to send the subtask unit to the resource processing node 106 according to the resource scheduling result.

[0087] The resource processing node 106 is configured to receive the resource scheduling result and the subtask unit, and execute the subtask unit according to the resource scheduling result.

[0088] Specifically, the task processor 102 refers to a program for controlling a to-be-processed task in a distributed system, and is responsible for applying for task resources on behalf of the to-be-processed task to the scheduler 104 and sending a task program to the resource processing node 106. The scheduler 104 (Resource Manager) refers to a server or a scheduling program for coordinating and allocating available resources in a distributed system. The to-be-processed task refers to an instruction, a task or a program to be executed. In actual applications, the to-be-processed task can be a SQL program, a query instruction, an order placement task, etc., which is not limited herein. The resource processing node 106 refers to a server or a virtual service environment in a distributed system responsible for supervising the execution of a task program of a to-be-processed task. The initial processing node can be understood as a machine node available for resource scheduling by the scheduler 104.

[0089] In actual applications, in an offline big data scenario, the to-be-processed task can apply for thousands of resources for the execution of each task unit. In addition, the execution time of each task unit in the to-be-processed task can be very short (80% < 10 seconds). Furthermore, the resource limitation of the quota group causes a large number of resource acquisition requests to be queued. In addition, all task units in the same computing stage have the same task resource description (i.e., the same resource requirement). In order to avoid frequent process start and stop caused by resource application and resource scheduling on the basis of task units, the embodiments of the present specification divide the to-be-processed task into sub-tasks, and divide the sub-tasks into sub-task units on the basis of the sub-tasks. In addition, resource acquisition is performed on the basis of the sub-tasks, i.e., a resource acquisition request corresponding to a target sub-task is submitted. Then, the scheduler 104 determines a corresponding resource scheduling result for the target sub-task in the initial processing node based on the resource acquisition request, wherein the resource scheduling result can be a resource scheduling result of all resources requested in the resource acquisition request, or a resource scheduling result of part of the resources requested in the resource acquisition request, which is not limited herein. Based on the determination of the resource scheduling node, the resource processing node 106 executes the sub-task units in the target sub-task based on the resource scheduling result.

[0090] In a distributed system, there are usually three roles: a scheduler 104, a task processor 102 and subordinate sub-tasks, and a resource processing node 106. Among them, the resource processing node 106 reports its resource quantity to the scheduler 104 for unified resource management and scheduling by the scheduler 104; when a user needs to submit a job (to-be-processed task), a job manager (task processor 102) applies for a certain amount of machine resources (such as CPU, memory) to the scheduler 104 for use by a job node (sub-task unit); after receiving the resource application of the to-be-processed task, the scheduler 104 calculates the available resource list that can be allocated to the job manager according to the remaining available resources in the distributed system (for example: 1 core CPU and 1 GB memory are available on machine A, and 2 core CPU and 2 GB memory are available on machine B); then, the scheduler 104 sends the resource list to the job manager and the corresponding machine node. After receiving the available resource list, the job manager allocates the job node (job program) to the corresponding machine node (resource processing node 106) to execute the job program (sub-task unit). When the job node is executed, the job manager informs the resource manager to return the resources of the job node. After receiving the return application, the resource manager informs the machine node to release the resources of the job node, and the released resources can be allocated to other jobs again.

[0091] The above schematic process can be implemented through five steps as shown in Figure 2 : 1. The user submits a job and starts a job manager; 2. The job manager applies for resources to the resource manager (scheduler 104); 3. The resource manager allocates resources to the job manager and the machine node; 4. The job manager distributes the job program to the corresponding machine node; 5. The machine node executes the job program after obtaining the resources.

[0092] When implemented specifically, as shown in Figure 3 , the resource scheduling system may need to interact with the task processor (AppMaster), the scheduler (Scheduler), the node (machine node can also be referred to as node), and the user through the interface server (ApiServer, used for interaction between nodes and services, and used for interaction with external users) to realize the resource scheduling and execution of the to-be-processed task. The specific steps are as follows:

[0093] Wherein, 1. A user (User) creates a to-be-processed task to an interface server; 2. A scheduler listens to the to-be-processed task from the interface server, and generates a scheduling result for the to-be-processed task; 3. The scheduler sends the scheduling result to the interface server; 4. A node listens to a scheduling event of the to-be-processed task through the interface server, and pulls up a task processor on a single machine; 5. The task processor sends a plurality of worker objects (Worker, i.e., sub-task units) to the interface server, expecting the scheduler to schedule; 6. The scheduler listens to the worker objects from the interface server, and generates a scheduling result for the worker objects; 7. The scheduler sends the scheduling result to the interface server; 8. The task processor listens to the scheduling result through the interface server, and manages the life cycle of the worker objects; 9. The node listens to the scheduling event of the to-be-processed task through the interface server, and executes the worker objects on the single machine.

[0094] Since each to-be-processed task needs to be managed by a task processor 102, and the task processor 102 also needs to be started in a processing node, after a user submits a to-be-processed task, the scheduler 104 needs to allocate a target processing node for the to-be-processed task in an initial processing node, so as to start the task processor 102 corresponding to the to-be-processed task. In actual implementation, the scheduler 104 is further configured to:

[0095] In the case of listening to the to-be-processed task, determining a target processing node in an initial processing node, and sending task information of the to-be-processed task to the target processing node;

[0096] Correspondingly, the target processing node is further configured to:

[0097] Receive the task information, and start the task processor 102 corresponding to the to-be-processed task based on the task information.

[0098] In actual application, in order to start the task processor 102 dedicated to the to-be-processed task, the scheduler 104 needs to allocate a target processing node for the to-be-processed task, so as to start the task processor 102 in the target processing node. In actual implementation, since the task processor 102 needs to manage the to-be-processed task, the task processor 102 needs to be started based on the task information of the to-be-processed task.

[0099] The task information of the to-be-processed task can include task name, task type, task details, task required resources and the like of the to-be-processed task, which is not limited herein.

[0100] In summary, when the scheduler 104 detects a to-be-processed task, the task processor 102 corresponding to the to-be-processed task is started in the target processing node scheduled by the scheduler 104, so as to start the task processor 102 and manage the to-be-processed task based on the task processor 102.

[0101] In specific implementation, the task processor 102 is further configured to:

[0102] determine subtask information and a subtask execution order according to the task information of the to-be-processed task, extract a target subtask in the to-be-processed task according to the subtask information and the subtask execution order, create a subtask unit in the target subtask according to target subtask information of the target subtask, and determine resource acquisition sub-information of the subtask unit, create a resource acquisition request of the target subtask based on the resource acquisition sub-information, and submit the resource acquisition request to the scheduler 104.

[0103] In actual application, because the processing amount of the to-be-processed task can be huge, it is not conducive to resource scheduling of the to-be-processed task. Therefore, the to-be-processed task can be divided into multiple stage subtasks, and each subtask can be scheduled and executed.

[0104] Specifically, the subtask information includes task-related information of the separated subtask in the to-be-processed task, and the subtask information can include subtask details, subtask data amount, subtask resource information, etc., which are not limited herein. The subtask execution order can be understood as the execution order of the subtask. In addition, because one subtask can need to be processed based on the execution result of another subtask, there can be an execution order between these subtasks. These subtasks need to be executed according to the execution order to ensure the correctness of processing the to-be-processed task. Therefore, the target subtask to be executed in the subtask needs to be determined according to the subtask information and the subtask execution order.

[0105] Because the processing amount of a single subtask can also be huge, in this case, the single subtask can be divided into multiple subtask units, which can be understood as homogeneous task units. These subtask units can be executed in parallel or sequentially, which are not limited herein.

[0106] In specific implementation, at least one subtask unit in the target subtask can be created according to the subtask unit data amount or the preset subtask unit quantity of the subtask, etc. For example, if the data amount of the subtask is 10T and the preset subtask unit data amount is 1T, 10 subtask units can be created based on the subtask.

[0107] Further, according to the subtask resource information of the target subtask and the number of subtask units or the data volume and the like, the information of the subtask resource required by the subtask unit, i.e., the resource acquisition sub-information, is determined. Further, the resource acquisition sub-information is combined, and the resource acquisition request of the target subtask is created. The resource acquisition request is used to request the scheduler 104 to request the resource required by the subtask unit in the target subtask. The subtask resource information refers to the information of the resource required by the subtask. In actual application, the subtask resource information can also be the task resource information of the to-be-processed task. For example, the task resource information of the to-be-processed task is 1000-core CPU and 500 GB memory, and the subtask resource information is also 1000-core CPU and 500 GB memory.

[0108] Taking the to-be-processed task as the SQL statement 1, the task information of the SQL statement 1 includes the task type "offline task" of the SQL statement 1, the task resource information "1000-core CPU, 3000 GB memory", and the SQL statement details "select*from table1 join select*from table2". According to the task information, the subtask information of the three subtasks is determined. The subtask information of the subtask 1 includes select*from table1 and the data volume 1000 GB; the subtask information of the subtask 2 includes select*from table2 and the data volume 500 GB; and the subtask information of the subtask 3 includes table1 join table2 and the data volume 1500 GB. Since the execution order of the SQL keywords in the three subtasks is select in the subtask 1, select in the subtask 2, and join in the subtask 3, the subtask execution order is determined as the subtask 1, the subtask 2, and the subtask 3. Based on the subtask execution order, the target subtask is determined as the subtask 1. According to the data volume 1000 GB of the subtask 1 and the maximum processing data volume of the preset subtask unit being 1 GB, 1000 subtask units are created for the subtask 1. Since the task resource information of the SQL statement 1 is 1000-core CPU and 3000 GB memory, the resource acquisition sub-information of each subtask unit is determined as 1-core CPU and 3 GB memory. Based on the resource acquisition sub-information of the 1000 subtask units, the resource acquisition request corresponding to the subtask 1 is created, and the resource acquisition sub-information of the 1000 subtask units is carried in the resource acquisition request.

[0109] In summary, the to-be-processed task is divided into subtasks and subtask units, and resources are applied for the subtasks based on the subtask units. The resources required by the subtask units are jointly applied, avoiding separate resource application for each subtask unit, accelerating resource application efficiency, and improving resource scheduling efficiency.

[0110] On the basis that the task processor 102 creates a resource acquisition request of the target subtask, the scheduler 104 needs to schedule corresponding resources for the resource acquisition request to ensure the smooth execution of the target subtask. In the embodiments of the present specification, the scheduler 104 is further configured to:

[0111] determine a preset scheduling algorithm corresponding to the resource acquisition request; determine a resource processing node 106 corresponding to the resource acquisition request and resource scheduling information corresponding to the resource processing node 106 according to the schedulable resource information and the preset scheduling algorithm; and take the resource scheduling information as the resource scheduling result.

[0112] The preset scheduling algorithm refers to an algorithm that is set in advance to schedule resources for a to-be-processed task. In actual applications, the preset scheduling algorithm can be determined according to the request type (such as an offline request or an online request) of the resource acquisition request. The preset scheduling algorithm can be a bestfit algorithm based on a processing node that processes the resource acquisition request, or a first-fit algorithm based on an available processing node that can satisfy the resource acquisition request. In addition, it can also be other types of algorithms, which are not limited herein.

[0113] In specific implementations, since the types of preset scheduling algorithms are various, in a first optional implementation provided by the present specification, the scheduler 104 is further configured to:

[0114] screen candidate processing nodes in the initial processing node according to the schedulable resource information and resource acquisition information carried by the resource acquisition request; score the candidate processing nodes according to node information corresponding to the candidate processing nodes and a preset scoring rule to obtain a node score of each candidate processing node; screen a resource processing node 106 from the candidate processing nodes based on the node score; and determine resource scheduling information corresponding to the resource processing node 106 according to the resource acquisition information.

[0115] The candidate processing node can be understood as a processing node whose available resources in the initial processing node can satisfy part or all of the resources applied for in the resource acquisition information. The node information can be available resource information, task amount information, and the like of the candidate node. Accordingly, the preset scoring rule refers to a rule preset for scoring the candidate processing node. The preset scoring rule can be scoring according to information such as how many resources and how many tasks, which is not limited herein.

[0116] According to the node information and the preset scoring rule, the candidate processing node is scored, that is, the node score of the candidate processing node is obtained, such as 90 points, 80 points, 60 points, and the like. Based on the node score, the resource processing node 106 is screened according to the preset screening condition (such as selecting a score greater than a score threshold or a maximum node score) in the candidate processing node. In the invocable resources of the resource processing node 106, the information that can satisfy part or all of the resources applied for in the resource acquisition information is determined, that is, the resource scheduling information. In actual application, the number of the resource processing node 106 can be one or multiple, which is not limited herein.

[0117] In summary, by selecting the available candidate processing node in the initial processing node as the resource processing node 106 corresponding to the resource acquisition request, the quality of resource scheduling for the resource acquisition request is ensured.

[0118] In addition, in the second optional implementation provided in the application, the scheduler 104 is further configured to:

[0119] In a case where it is detected that the schedulable resource information of the initial processing node is changed, determining a candidate resource processing node to be scheduled and a target request queue to be traversed in the initial processing node;

[0120] Traversing a queue resource acquisition request in the target request queue based on the target schedulable resource information of the candidate resource processing node and a preset traversal order;

[0121] In a case where the target schedulable resource information satisfies a resource acquisition condition corresponding to the resource acquisition request, taking the candidate resource processing node as the resource processing node 106 of the resource acquisition request;

[0122] According to the resource acquisition information of the resource acquisition request, determining resource scheduling information corresponding to the resource processing node 106.

[0123] The change in the schedulable resource information of the initial processing node can be understood as a case where any one of the initial processing nodes has resource release, that is, the available resources of the initial processing node are increased, and then the initial processing node with the change is taken as the candidate resource processing node to be scheduled to process the task of the requested resource.

[0124] In actual applications, multiple request queues can be created according to the request types (such as offline or online) and / or request demand levels (high or low) of the resource acquisition requests. When the scheduler 104 receives a resource acquisition request, the resource acquisition request is inserted into a corresponding request queue according to its request type and / or request demand level. The request demand level can be understood as the demand level of the resource acquisition request for the scheduling quality. For example, a resource acquisition request with low demand for the scheduling quality can be processed by a multi-threaded resource, and a resource acquisition request with high demand for the scheduling quality needs to be processed by a single-threaded resource to avoid competition between multiple threads. Based on this, the request queues in the scheduler 104 can be divided into online queues, offline queues, multi-threaded queues, single-threaded queues, and the like, without limitation.

[0125] Specifically, the deployment of the request queue can be as shown in Figure 4 There are two queue pools in the scheduler 104. The queue pool 1 is used to process offline resource acquisition requests, and the queue pool 2 is used to process online resource acquisition requests. In addition, each queue pool includes two request queues. The request queue A is used to process resource acquisition requests with high demand, and single-thread execution is adopted for the resource acquisition requests in the request queue A. The request queue B is used to process resource acquisition requests with low demand, and multi-thread execution is adopted for the resource acquisition requests in the request queue A.

[0126] Further, the execution of the above scheduling process can be as shown in Figure 5 When the scheduler 104 determines that there is a change in the schedulable resource information of the node (initial processing node), candidate resource processing nodes such as node 1 and node 2 with changed schedulable resource information are determined in the initial processing node. Node 1 is a node in quota group 1 (Quota1), and node 2 is a node in quota group 2 (Quota2). Quota group 1 is a resource quota divided for project 1. Quota group 2 is a resource quota divided for project 2. Node 1 determines that the request queue to be traversed in the queue pool 2 corresponding to quota group 1 is request queue 1, and then filters the queue resource acquisition requests that can be met in request queue 1 (this process can be understood as executing filtering logic when filtering queue resource acquisition requests).

[0127] The preset traversal order can be the order of arrangement of the queue resource acquisition requests in the target request queue. Alternatively, a request tree can be constructed from the queue resource acquisition requests in the request queue, and the traversal order of the request tree can be used as the preset traversal order.

[0128] Taking the resource scheduling system as an example of K8S, each sub-task unit or task processor 102 in the resource scheduling system is a Pod, and the task processor 102 applies for resources, the scheduler 104 returns resources, and the resource processing node 106 executes the sub-task unit are all taken as a carrier. Since the online service is an independent process, each process is represented by an independent Pod. In order to make the K8S system support the online deployment of the resource scheduling framework, the K8S system can support the offline service, and the K8S system can support the online fine resource arrangement management and the high-concurrency scheduling capability in the offline big data scene. An offline protocol needs to be proposed for the offline service. The protocol supports the above-mentioned multi-level sub-task division of the to-be-processed task, and supports the division of the request queue according to the type of the resource acquisition request.

[0129] In specific implementation, the scheduler 104 is further configured to:

[0130] acquire the queue resource acquisition request in the target request queue; construct a request tree according to the request priority and the resource acquisition information of the queue resource acquisition request; and traverse the queue resource acquisition request in the request tree.

[0131] In actual application, the higher the request priority is, the more important the queue resource acquisition request is, and the more the queue resource acquisition request needs to be satisfied in priority. The resource acquisition information can be understood as the information of the resource requested by the queue resource acquisition request, such as the request of the scheduler 104 to allocate 3-core CPU and 2 GB memory in the queue resource acquisition request 1. The resource acquisition information can be simply written as [3, 2]. The first dimension 2 indicates that 3-core CPU is requested to be allocated; and the second dimension 2 indicates that 2 GB memory is requested to be allocated.

[0132] In specific implementation, the request tree can be a red-black tree or other binary tree, which is not limited here. After the request tree is constructed, the queue resource acquisition request in the request tree is traversed according to the traversal order defined by the request tree. In addition, the request tree can also be directly used as a request queue. When any queue resource acquisition request is received, the queue resource acquisition request is directly inserted into the corresponding position in the request tree.

[0133] Taking the red-black tree as an example, since the resource requested by the resource acquisition request is usually multi-dimensional, the resource of each node in each red-black tree is also multi-dimensional, and then each node in the red-black tree saves the minimum value in each dimension in the left and right child trees of the node. As shown in the following table: Figure 6

[0134] ​It can be seen that the target available resource of the candidate resource node is 2-core CPU and 2GB memory, which is referred to as [2, 2]. The minimum value of the left subtree of the root node 5 is [2, 2], but actually there is no node satisfying the condition [2, 2]. Therefore, in the traversal process, the case of "forward failure and backtracking" must be considered. When the node 2 is reached, it is found that the minimum value [2, 3] of the left subtree of the node 2 cannot be satisfied by the available resource [2, 2], that is, the left subtree does not meet the resource constraint, therefore, the left subtree cannot be further advanced. It is necessary to continue to confirm whether the node 2 itself meets the resource constraint; the value of the node 2 is [3, 3] and does not meet the condition, then it is necessary to continue to confirm whether the right subtree of the node 2 has a node satisfying the condition; since the minimum value of the right subtree is [3, 2] and does not meet the condition, it is necessary to backtrack to the node 5, and so on. Finally, it is determined that the node 7 meets the condition.

[0135] It can be seen that the worst case of useless traversal of the whole tree under the condition of considering "backtracking" is O(N / 2) (because the leaf node is not traversed uselessly when it is known whether the leaf node meets the condition when reaching the upper layer of the leaf node).

[0136] Although the preset traversal order corresponding to the construction of the red-black tree inevitably causes useless traversal, the cost of "useless traversal in the tree" and "useless traversal caused by the calculation of the remaining resources of the nodes or the quota in the periphery after the queue resource acquisition request is selected" is completely different, which can be verified from the experimental results (the effect before and after optimization differs by 10-1000 times). In fact, the total overhead according to the preset traversal order corresponding to the construction of the red-black tree = the number of useless traversals * the overhead caused by a single useless traversal. The "overhead caused by a single useless traversal in the tree" is the minimum cost, and from the outside, it also ensures that "the queue resource acquisition request selected from the tree must be satisfiable".

[0137] In specific implementation, the purpose of traversing the queue resource acquisition requests in the target request queue according to the preset traversal order is to filter the queue resource acquisition requests that can be first satisfied by the candidate resource processing node among the queue resource acquisition requests. In actual application, the request is searched in the target request queue, and filtering logic is performed on the request in the searching process. Only one Queue is traversed by the same machine (one candidate resource processing node) at the same time, and only one machine (one candidate resource processing node) can traverse the same Queue at the same time. In addition, the sorting rule of the Queue supports the FIFO (First In First Out) and / or fairness principle.

[0138] And in the case that the queue resource acquisition request is the above resource acquisition request, the resource scheduling information corresponding to the resource processing node 106 is determined according to the resource acquisition information of the resource acquisition request.

[0139] In summary, the resource acquisition request is selected through the available candidate resource processing nodes, which improves the utilization efficiency of the resource processing nodes. In addition, the preset traversal order corresponding to the red-black tree is traversed, which reduces the number of useless traversals and improves the scheduling efficiency.

[0140] In practical applications, since the resource scheduling result may only meet the execution needs of a part of the sub-task units, the task processor 102 is further configured to:

[0141] screen a target sub-task unit from the sub-task units according to the resource scheduling result, and send the target sub-task unit to the resource processing node 106 based on the resource scheduling result.

[0142] Correspondingly, the resource processing node 106 is further configured to:

[0143] receive the target sub-task unit, determine a target resource corresponding to the resource scheduling result, and execute the target sub-task unit through the target resource, and send an execution result of the target sub-task unit to the task processor 102 in the case that the target sub-task unit is executed.

[0144] Specifically, screening a target sub-task unit from the sub-task units according to the resource scheduling result means screening a sub-task unit that can be executed by a target resource corresponding to the resource scheduling result from the sub-task units. The target sub-task unit can be part or all of the sub-task units, which is not limited here.

[0145] In addition, since the resource scheduling result may not complete the execution of all sub-task units at one time, only the execution of the target sub-task unit can be completed. Therefore, the task processor 102 sends the target sub-task unit to the resource processing node 106 first. And receives the execution result of the target sub-task unit by the resource processing node 106 through the target resource corresponding to the resource scheduling result.

[0146] In addition, for the remaining sub-task units in the sub-task units except the target sub-task unit, the task processor 102 is further configured to:

[0147] determine the remaining sub-task units from the sub-task units based on the execution result, and send the remaining sub-task units to the resource processing node 106;

[0148] Correspondingly, the resource processing node 106 is further configured to:

[0149] receive the remaining sub-task units sent by the task processor 102; and execute the remaining sub-task units by the target resources corresponding to the target sub-task units.

[0150] After completing the execution of the target sub-task units, in order to complete the execution of the remaining sub-task units in the sub-task units, the task processor 102 needs to send the remaining sub-task units to the resource processing node 106, and receive the execution results of the resource processing node 106 executing the remaining sub-task units by multiplexing the target resources.

[0151] In specific implementation, after completing one target sub-task unit, a remaining sub-task unit can be determined in the sub-task units according to the execution result, and the remaining sub-task unit can be executed by the unit resources corresponding to the target sub-task unit.

[0152] In summary, when a resource acquisition request obtains part of resources that are insufficient to meet the number of all waiting sub-task units for execution, the resource can continue to be used by the next sub-task unit after completing the current sub-task unit without being returned, thereby avoiding resource interaction and single machine process start-stop caused by frequent resource return and resource scheduling, and improving the processing efficiency of the target sub-task.

[0153] In addition, since the remaining sub-task units executed by multiplexing resources do not need the scheduler 104 to continuously perform resource scheduling for the remaining sub-task units, the task processor 102 in the embodiment of the present specification is further configured to:

[0154] in a case where there is an unexecuted sub-task unit in the target sub-task, create sub-resource revocation information based on the execution result, and send the sub-resource revocation information to the scheduler 104;

[0155] Correspondingly, the scheduler 104 is further configured to:

[0156] update the unscheduled resource information corresponding to the resource acquisition request based on the sub-resource revocation information.

[0157] The sub-resource revocation information refers to information for requesting the scheduler 104 to revoke the sub-resource scheduling. Since the unexecuted sub-task units can be executed by multiplexing resources, in a case where one target sub-task unit is executed, the resources corresponding to the target sub-task unit can be multiplexed to continue executing other sub-task units. In this case, the task processor 102 sends the sub-resource revocation information to the scheduler 104, which is to inform the scheduler 104 to return a sub-resource request.

[0158] Specifically, the sub-resource revocation information can be in the form of a sub-task label. For example, label=req. After the scheduler 104 receives the sub-resource revocation information, the unscheduled resource information corresponding to the resource acquisition request is updated according to the sub-resource revocation information, so as to reduce the unscheduled resources of the resource acquisition request.

[0159] In addition, the task processor 102 is further configured to:

[0160] In the case where there is no unexecuted sub-task unit in the target sub-task, create sub-resource return information based on the execution result, and send the sub-resource return information to the scheduler 104;

[0161] Correspondingly, the scheduler 104 is further configured to:

[0162] Update the resource scheduling information corresponding to the resource acquisition request based on the sub-resource return information.

[0163] The sub-resource return information refers to information for requesting the scheduler 104 to release resources. Since there is no unexecuted sub-task unit in the target sub-task, the idle resources after executing the target sub-task unit do not need to be used. In this case, the task processor 102 sends the sub-resource return information to the scheduler 104, informs the scheduler 104 that the resource acquisition request no longer needs to use the sub-resource, and the scheduler 104 can release the resources. Specifically, the sub-resource return information can also be in the form of a sub-task label. For example, label=res. After the scheduler 104 receives the sub-resource return information, the resource scheduling information corresponding to the resource acquisition request is updated according to the sub-resource return information, so as to release the resources returned by the resource acquisition request.

[0164] In specific implementation, if a large number of sub-task units are updated (start and stop) in a short time, the update of the target sub-task is also quite uneconomical, because the resource change only accounts for a very small part of the entire target sub-task content, but the entire target sub-task needs to be updated to the storage. Therefore, by creating sub-resource revocation information and sub-resource return information, the flow of resources is represented, the change of resources will not cause the update of the entire target sub-task, and the sub-resource revocation information and the sub-resource return information are extremely small in size compared with the target sub-task, so the pressure generated by event update can be significantly reduced.

[0165] Specifically, as Figure 7As shown, T0: the task processor creates a task group (which can be understood as the above target subtask), which includes two unexecuted jobs (which can be understood as the above subtask unit), namely worker0 and worker1. The task processor applies for two resources through the interface server; T1: the scheduler allocates one resource to the task group; T2: worker0 starts execution through the allocated resource, and the worker0 is a running job; T3: woker0 execution ends, and the task processor reuses the resource to continue running worker1. Wherein, the worker0 is a completed job. In order to make the scheduler schedule one less resource, the task processor creates a sub-resource revocation information, wherein label=req, indicating that the scheduler is notified to return one request (i.e. to revoke one resource); T4: worker1 execution ends, and the task processor finds that there is no worker to be scheduled, so it creates a sub-resource return information, label=req, indicating that the scheduler is notified to return one resource, wherein worker0 and worker1 are both completed jobs. In summary, the number of workers to be scheduled by the scheduler 104 is: the number of subtask units in the target subtask - Sum (revoked subtask units of sub-resources) - Sum (returned subtask units of sub-resources).

[0166] In addition, since there are other subtasks in addition to the target subtask to be processed in the to-be-processed task, the task processor 102 is further configured to:

[0167] In the case where the subtask unit is executed, the second target subtask to be executed after determining the target subtask of the to-be-processed task is determined, the second target subtask is taken as the target subtask, and the step of determining the subtask unit in the target subtask is returned.

[0168] In specific implementation, after executing the target subtask, it is necessary to determine other subtasks to be executed in the to-be-processed task. The other subtasks are processed in the same way as the target subtask, so as to complete the execution of the to-be-processed task.

[0169] Wherein, the second target subtask can be understood as a subtask to be executed after the target subtask in the to-be-processed task. Specifically, the second target subtask can be determined according to the above determined subtask execution sequence.

[0170] The resource scheduling system provided in the specification realizes resource scheduling on a to-be-processed task through interaction among a task processor 102, a scheduler 104, and a resource processing node 106; wherein the task processor 102 is configured to determine a target subtask according to task information of the to-be-processed task, determine a subtask unit in the target subtask, and submit a resource acquisition request of the target subtask to the scheduler 104; the scheduler 104 is configured to acquire schedulable resource information of the resource processing node 106 in response to the resource acquisition request, calculate a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information, and send the resource scheduling result to the task processor 102 and the resource processing node 106; the task processor 102 is configured to send the subtask unit to the resource processing node 106 according to the resource scheduling result; and the resource processing node 106 is configured to receive the resource scheduling result and the subtask unit, and execute the subtask unit based on a target resource corresponding to the resource scheduling result. The resource scheduling system avoids resource interaction caused by separate resource application for each subtask unit, improves resource scheduling efficiency, and improves processing efficiency of the to-be-processed task.

[0171] Corresponding to the above system embodiments, the specification also provides system scheduling method embodiments, which can correspond to the above system embodiments when read. Figure 8 A flowchart of a first system scheduling method provided by an embodiment of the specification is shown, and the method is applied to a task processor and specifically includes the following steps:

[0172] Step 802: determining a target subtask according to task information of a to-be-processed task, determining a subtask unit in the target subtask, and submitting a resource acquisition request of the target subtask to a scheduler;

[0173] Step 804: receiving a resource scheduling result of the resource acquisition request returned by the scheduler;

[0174] Step 806: sending the subtask unit to a resource processing node according to the resource scheduling result.

[0175] Optionally, the step of determining a target subtask according to task information of a to-be-processed task, determining a subtask unit in the target subtask, and submitting a resource acquisition request of the target subtask to a scheduler includes:

[0176] determining subtask information and a subtask execution order according to the task information of the to-be-processed task;

[0177] extracting a target subtask in the to-be-processed task according to the subtask information and the subtask execution order;

[0178] creating a subtask unit in the target subtask according to target subtask information of the target subtask;

[0179] determining resource acquisition sub-information of the subtask unit;

[0180] creating a resource acquisition request of the target subtask based on the resource acquisition sub-information, and submitting the resource acquisition request to the scheduler.

[0181] Optionally, after receiving the resource scheduling result of the resource acquisition request returned by the scheduler, the method further comprises:

[0182] screening a target subtask unit from the subtask units according to the resource scheduling result;

[0183] sending the target subtask unit to the resource processing node based on the resource scheduling result;

[0184] receiving an execution result of the target subtask unit sent by the resource processing node.

[0185] Optionally, after receiving the execution result of the target subtask unit sent by the resource processing node, the method further comprises:

[0186] determining a remaining subtask unit from the subtask units based on the execution result;

[0187] sending the remaining subtask unit to the resource processing node.

[0188] Optionally, after receiving the execution result of the target subtask unit sent by the resource processing node, the method further comprises:

[0189] in a case where there is an unexecuted subtask unit in the target subtask, creating sub-resource revocation information based on the execution result, and sending the sub-resource revocation information to the scheduler.

[0190] Optionally, after receiving the execution result of the target subtask unit sent by the resource processing node, the method further comprises:

[0191] in a case where there is no unexecuted subtask unit in the target subtask, creating sub-resource return information based on the execution result, and sending the sub-resource return information to the scheduler.

[0192] Optionally, after sending the subtask unit to the resource processing node based on the resource scheduling result, the method further comprises:

[0193] In a case where the subtask unit is executed completely, a second target subtask to be executed is determined according to the task information of the to-be-processed task after the target subtask is determined, the second target subtask is taken as the target subtask, and the step of determining the subtask unit in the target subtask is returned to be executed.

[0194] Optionally, before the target subtask is determined according to the task information of the to-be-processed task, the method further includes:

[0195] In a case where the to-be-processed task is listened to, a target processing node is determined in an initial processing node, and the task information of the to-be-processed task is sent to the target processing node.

[0196] The target processing node can be understood as a processing node that receives the task information and starts a task processor corresponding to the to-be-processed task based on the task information.

[0197] The resource scheduling method provided in the specification is applied to a task processor, the task processor determines a target subtask according to task information of a to-be-processed task, determines a subtask unit in the target subtask, and submits a resource acquisition request of the target subtask to a scheduler. The target subtask in the to-be-processed task is determined based on the division of the to-be-processed task into subtasks, and the resource is applied to the scheduler based on the target subtask. The resource scheduling result of the resource acquisition request returned by the scheduler is received, and the subtask unit is sent to a resource processing node according to the resource scheduling result. The resource interaction caused by the resource application of each subtask unit is avoided, the resource scheduling efficiency is improved, and the processing efficiency of the to-be-processed task is improved.

[0198] The resource scheduling method provided in the specification determines a target subtask according to task information of a to-be-processed task, determines a subtask unit in the target subtask, and submits a resource acquisition request of the target subtask to a scheduler. The schedulable resource information of the resource processing node is acquired in response to the resource acquisition request, the resource scheduling result of the target subtask is calculated according to the resource acquisition request and the schedulable resource information, and the resource scheduling result is sent to the task processor and the resource processing node. The subtask unit is sent to the resource processing node according to the resource scheduling result. The resource scheduling result and the subtask unit are received, and the subtask unit is executed based on the target resource corresponding to the resource scheduling result. The resource interaction caused by the resource application of each subtask unit is avoided, the resource scheduling efficiency is improved, and the processing efficiency of the to-be-processed task is improved.

[0199] Corresponding to the above system embodiments, the specification also provides system scheduling method embodiments, which can be read in correspondence with the above system embodiments. Figure 9A flow chart of a second system scheduling method provided by one embodiment of the present specification is shown, which is applied to a scheduler and specifically includes the following steps:

[0200] Step 902: receiving a resource acquisition request of a target subtask sent by a task processor;

[0201] Step 904: acquiring schedulable resource information of an initial processing node in response to the resource acquisition request;

[0202] Step 906: calculating a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information;

[0203] Step 908: sending the resource scheduling result to the task processor and a resource processing node.

[0204] The resource acquisition request of the target subtask carries sub-resource acquisition information of a subtask unit of the target subtask. The scheduler can batch schedule resources for the subtask unit according to the resource acquisition request. The resource scheduling result can include scheduling results of scheduling resources for part or all of the subtask units.

[0205] Optionally, after the resource scheduling result is sent to the task processor and the resource processing node, the method further includes:

[0206] receiving sub-resource revocation information sent by the task processor based on an execution result of a target subtask unit in the target subtask;

[0207] updating unscheduled resource information corresponding to the resource acquisition request based on the sub-resource revocation information.

[0208] Optionally, after the resource scheduling result is sent to the task processor and the resource processing node, the method further includes:

[0209] receiving sub-resource return information sent by the task processor based on an execution result of a target subtask unit in the target subtask;

[0210] updating resource scheduling information corresponding to the resource acquisition request based on the sub-resource return information.

[0211] Optionally, the calculating a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information includes:

[0212] determining a preset scheduling algorithm corresponding to the resource acquisition request;

[0213] determine the resource processing node corresponding to the resource acquisition request and the resource scheduling information corresponding to the resource processing node according to the schedulable resource information and the preset scheduling algorithm;

[0214] use the resource scheduling information as the resource scheduling result.

[0215] Optionally, the determining the resource processing node corresponding to the resource acquisition request and the resource scheduling information corresponding to the resource processing node according to the schedulable resource information and the preset scheduling algorithm comprises:

[0216] screening candidate processing nodes in an initial processing node according to the schedulable resource information and resource acquisition information carried by the resource acquisition request;

[0217] scoring the candidate processing nodes according to node information corresponding to the candidate processing nodes and a preset scoring rule to obtain a node score of each candidate processing node;

[0218] screening a resource processing node from the candidate processing nodes based on the node score;

[0219] determining the resource scheduling information corresponding to the resource processing node according to resource acquisition information.

[0220] Optionally, the determining the resource processing node corresponding to the resource acquisition request and the resource scheduling information corresponding to the resource processing node according to the schedulable resource information and the preset scheduling algorithm comprises:

[0221] in a case where it is detected that the schedulable resource information of an initial processing node has changed, determining a candidate resource processing node to be scheduled and a target request queue to be traversed in the initial processing node;

[0222] traversing a queue resource acquisition request in the target request queue based on target schedulable resource information of the candidate resource processing node and a preset traversal order;

[0223] in a case where the target schedulable resource information meets a resource acquisition condition corresponding to the resource acquisition request, using the candidate resource processing node as the resource processing node of the resource acquisition request;

[0224] determining the resource scheduling information corresponding to the resource processing node according to resource acquisition information of the resource acquisition request.

[0225] Optionally, the traversing the queue resource acquisition request in the target request queue is implemented in the following manner:

[0226] obtaining a queue resource acquisition request in the target request queue;

[0227] construct a request tree according to the request priority and the resource acquisition information of the queue resource acquisition request;

[0228] traverse the queue resource acquisition request in the request tree.

[0229] The resource scheduling method provided in the specification is applied to a scheduler. The scheduler receives a resource acquisition request of a target subtask sent by a task processor. In response to the resource acquisition request, the scheduler acquires schedulable resource information of an initial processing node. The scheduler calculates a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information. The scheduler sends the resource scheduling result to the task processor and a resource processing node. The resource scheduling is implemented on the basis of the target subtask. The resource interaction caused by separately scheduling each subtask unit is avoided. The resource scheduling efficiency is improved, and the processing efficiency of the task to be processed is improved.

[0230] Corresponding to the system embodiments described above, the specification also provides system scheduling method embodiments. When reading this embodiment, the system scheduling method embodiments can correspond to the system embodiments described above. Figure 10 A flowchart of a third system scheduling method provided by an embodiment of the specification is shown. The method is applied to a resource processing node and specifically includes the following steps:

[0231] Step 1002: receiving a resource scheduling result of a target subtask sent by a scheduler;

[0232] Step 1004: receiving a subtask unit of the target subtask sent by a task processor;

[0233] Step 1006: executing the subtask unit based on a target resource corresponding to the resource scheduling result.

[0234] Optionally, after the receiving of the resource scheduling result of the target subtask sent by the scheduler, the method further includes:

[0235] receiving a target subtask unit of the target subtask sent by the task processor;

[0236] determining a target resource corresponding to the resource scheduling result, and executing the target subtask unit through the target resource;

[0237] In the case where the target subtask unit is executed, sending an execution result of the target subtask unit to the task processor.

[0238] Optionally, after the sending of the execution result of the target subtask unit to the task processor, the method further includes:

[0239] receive the remaining subtask units sent by the task processor based on the execution result;

[0240] execute the remaining subtask units through the target resource corresponding to the target subtask unit.

[0241] Specifically, the specific implementation process of executing the target subtask unit and the remaining subtask units is the specific implementation process of executing the subtask units in the target subtask.

[0242] The resource scheduling method provided in the specification is applied to a resource processing node. The resource processing node receives a resource scheduling result of a target subtask sent by a scheduler. The resource processing node receives subtask units of the target subtask sent by a task processor. The resource processing node executes the subtask units based on a target resource corresponding to the resource scheduling result. This implementation realizes receiving a resource scheduling result based on a target subtask and executing subtask units based on a resource scheduling result. This implementation avoids resource interaction caused by individually receiving resource scheduling for each subtask unit, improves processing efficiency, and improves the processing efficiency of the task to be processed.

[0243] Figure 11 A structural block diagram of a computing device 1100 is shown according to an embodiment of the specification. The components of the computing device 1100 include but are not limited to a memory 1110 and a processor 1120. The processor 1120 is connected to the memory 1110 through a bus 1130. A database 1150 is used to save data.

[0244] The computing device 1100 also includes an access device 1140, which enables the computing device 1100 to communicate via one or more networks 1160. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 1140 can include one or more of any type of network interface (for example, a network interface card (NIC)) such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a worldwide microwave interconnection access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0245] In an embodiment of the specification, the above-mentioned components of the computing device 1100 and Figure 11 other components not shown in the specification can be connected to each other, for example, through a bus. It should be understood that Figure 11 The structural block diagram of the computing device shown is only for the purpose of example, and is not a limitation on the scope of the specification. Those skilled in the art can add or replace other components as needed.

[0246] The computing device 1100 can be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other type of mobile device, or a stationary computing device such as a desktop computer or PC. The computing device 1100 can also be a mobile or stationary server.

[0247] The processor 1120 is configured to execute computer-executable instructions to perform the steps of the resource scheduling method described above.

[0248] The above is a schematic solution of the computing device according to the embodiment. It should be noted that the technical solution of the computing device and the technical solution of the resource scheduling method described above belong to the same concept, and the details of the technical solution of the computing device that are not described in detail can be referred to the description of the technical solution of the resource scheduling method.

[0249] An embodiment of the present specification further provides a computer-readable storage medium, which stores computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the steps of the resource scheduling method described above.

[0250] The above is a schematic solution of the computer-readable storage medium according to the embodiment. It should be noted that the technical solution of the storage medium and the technical solution of the resource scheduling method described above belong to the same concept, and the details of the technical solution of the storage medium that are not described in detail can be referred to the description of the technical solution of the resource scheduling method.

[0251] An embodiment of the present specification further provides a computer program, which causes a computer to perform the steps of the resource scheduling method described above when the computer program is executed in the computer.

[0252] The above is a schematic solution of the computer program according to the embodiment. It should be noted that the technical solution of the computer program and the technical solution of the resource scheduling method described above belong to the same concept, and the details of the technical solution of the computer program that are not described in detail can be referred to the description of the technical solution of the resource scheduling method.

[0253] The above-described embodiments of the application have several aspects, no single one of which is solely responsible for the application's desirable attributes. Without limiting the scope of the application as expressed by the claims which follow, some further embodiments make these aspects even more useful. Other embodiments can result in less desirable attributes.

[0254] The computer readable medium can include any entity or apparatus capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, software distribution medium, etc.

[0255] It should be noted that, for the foregoing method embodiments, the acts described can be performed in serial, parallel, or some other order. In other words, the order of the acts of the embodiments can be modified without changing the underlying attachment structure. Moreover, the embodiments can additionally be modified in other ways. For example, one or more acts shown and / or described in connection with one embodiment can be incorporated into another embodiment.

[0256] In the above-described embodiments, the description of each embodiment focuses on different aspects. The parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0257] The above-disclosed specification and drawings provide a complete description of the application. The application can be modified to incorporate any additional features without departing from the basic application. The application is not to be restricted, except by the claims and their equivalents.

Claims

1. A resource scheduling system, comprising: a task processor, a scheduler and a resource processing node; wherein the task processor is configured to determine a target subtask according to task information of a to-be-processed task, determine a plurality of subtask units in the target subtask, and submit a resource acquisition request of the target subtask to the scheduler, wherein the target subtask is obtained by dividing the to-be-processed task, the subtask units are obtained by dividing the target subtask, and the plurality of subtask units are homogeneous subtask units; the scheduler is configured to acquire schedulable resource information of an initial processing node in response to the resource acquisition request, calculate a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information, and send the resource scheduling result to the task processor and the resource processing node; the task processor is configured to send the subtask units to the resource processing node according to the resource scheduling result; the resource processing node is configured to receive the resource scheduling result and the subtask units, execute the subtask units based on target resources corresponding to the resource scheduling result, and when the target resources are insufficient to execute all the subtask units, the resource processing node does not return the target resources after executing part of the subtask units, and executes subtask units that have not been executed in the subtask units. 2.The resource scheduling system of claim 1, wherein the task processor is further configured to: determine subtask information and a subtask execution sequence according to task information of the to-be-processed task; extract a target subtask in the to-be-processed task according to the subtask information and the subtask execution sequence; create subtask units in the target subtask according to target subtask information of the target subtask, and determine resource acquisition sub-information of the subtask units; create a resource acquisition request of the target subtask based on the resource acquisition sub-information, and submit the resource acquisition request to the scheduler. 3.The resource scheduling system of claim 1, wherein the task processor is further configured to: filter target subtask units in the subtask units according to the resource scheduling result, and send the target subtask units to the resource processing node based on the resource scheduling result; correspondingly, the resource processing node is further configured to: receive the target subtask units, determine target resources corresponding to the resource scheduling result, and execute the target subtask units through the target resources; and in a case where the target subtask units are executed, send an execution result of the target subtask units to the task processor. 4.The resource scheduling system of claim 3, wherein the task processor is further configured to: determine remaining subtask units in the subtask units based on the execution result, and send the remaining subtask units to the resource processing node; correspondingly, the resource processing node is further configured to: receive the remaining subtask units sent by the task processor, and execute the remaining subtask units through target resources corresponding to the target subtask units. 5.The resource scheduling system of claim 3, wherein the task processor is further configured to: In a case where there is an unexecuted subtask unit in the target subtask, sub-resource revocation information is created based on the execution result, and the sub-resource revocation information is sent to the scheduler; Correspondingly, the scheduler is further configured to: Based on the sub-resource revocation information, the unscheduled resource information corresponding to the resource acquisition request is updated.

6. The resource scheduling system of claim 3, the task processor is further configured to: In a case where there is no unexecuted subtask unit in the target subtask, sub-resource return information is created based on the execution result, and the sub-resource return information is sent to the scheduler; Correspondingly, the scheduler is further configured to: Based on the sub-resource return information, the resource scheduling information corresponding to the resource acquisition request is updated.

7. The resource scheduling system of claim 1, the task processor is further configured to: In a case where the subtask unit execution is completed, a second target subtask to be executed after the target subtask is determined according to the task information of the to-be-processed task, the second target subtask is taken as the target subtask, and the step of determining the subtask unit in the target subtask is returned to be executed.

8. The resource scheduling system of claim 1, the scheduler is further configured to: A preset scheduling algorithm corresponding to the resource acquisition request is determined; a resource processing node corresponding to the resource acquisition request and resource scheduling information corresponding to the resource processing node are determined according to the schedulable resource information and the preset scheduling algorithm; and the resource scheduling information is taken as the resource scheduling result.

9. The resource scheduling system of claim 8, the scheduler is further configured to: According to the schedulable resource information and resource acquisition information carried by the resource acquisition request, candidate processing nodes are screened in an initial processing node; and according to node information corresponding to the candidate processing nodes and a preset scoring rule, the candidate processing nodes are scored to obtain node scores of each candidate processing node. Based on the node score, a resource processing node is screened from the candidate processing node; According to the resource acquisition information, the resource scheduling information corresponding to the resource processing node is determined.

10. The resource scheduling system of claim 8, the scheduler is further configured to: In a case where it is monitored that the schedulable resource information of the initial processing node exists change, a candidate resource processing node to be scheduled and a target request queue to be traversed are determined in the initial processing node. Based on the target schedulable resource information of the candidate resource processing node and a preset traversal order, a queue resource acquisition request in the target request queue is traversed; in a case where the target schedulable resource information meets a resource acquisition condition corresponding to the resource acquisition request, the candidate resource processing node is taken as the resource processing node of the resource acquisition request; According to the resource acquisition information of the resource acquisition request, the resource scheduling information corresponding to the resource processing node is determined.

11. The resource scheduling system of claim 1, the scheduler is further configured to: A queue resource acquisition request in a target request queue is acquired; a request tree is constructed according to a request priority and resource acquisition information of the queue resource acquisition request; and the queue resource acquisition request in the request tree is traversed.

12. The resource scheduling system of claim 1, the scheduler is further configured to: In a case where a to-be-processed task is listened to, a target processing node is determined in an initial processing node, and task information of the to-be-processed task is sent to the target processing node. Correspondingly, the target processing node is configured to: receive the task information, and start a task processor corresponding to the to-be-processed task based on the task information.

13. A resource scheduling method applied to a task processor, comprising: determining a target subtask according to task information of a to-be-processed task, and determining a plurality of subtask units in the target subtask, and submitting a resource acquisition request of the target subtask to a scheduler, wherein the target subtask is obtained based on division of the to-be-processed task, the subtask units are obtained based on division of the target subtask, and the plurality of subtask units are homogeneous subtask units; receiving a resource scheduling result of the resource acquisition request returned by the scheduler; sending the subtask units to a resource processing node according to the resource scheduling result, the subtask units are executed by the resource processing node, and when target resources are insufficient to execute all the subtask units, the resource processing node does not return the target resources after executing part of the subtask units, and executes subtask units not executed in the subtask units.

14. A resource scheduling method applied to a scheduler, comprising: receiving a resource acquisition request of a target subtask sent by a task processor, wherein the resource acquisition request is submitted by the task processor after determining a target subtask according to task information of a to-be-processed task, and determining a plurality of subtask units in the target subtask, the target subtask is obtained based on division of the to-be-processed task, the plurality of subtask units are obtained based on division of the target subtask, and the plurality of subtask units are homogeneous subtask units; obtaining schedulable resource information of an initial processing node in response to the resource acquisition request; calculating a resource scheduling result of the target subtask according to the resource acquisition request and the schedulable resource information; and sending the resource scheduling result to the task processor and a resource processing node, when the resource processing node receives the subtask units of the target subtask sent by the task processor, executes the subtask units based on target resources corresponding to the resource scheduling result, and the target resources are insufficient to execute all the subtask units, the resource processing node does not return the target resources after executing part of the subtask units, and executes subtask units not executed in the subtask units.

15. A resource scheduling method applied to a resource processing node, comprising: receiving a resource scheduling result of a target subtask sent by a scheduler, wherein the target subtask is obtained based on division of a to-be-processed task, subtask units are obtained based on division of the target subtask, and a plurality of the subtask units are homogeneous subtask units; receiving the subtask units of the target subtask sent by a task processor; executing the subtask units based on target resources corresponding to the resource scheduling result, and when the target resources are insufficient to execute all the subtask units, the resource processing node does not return the target resources after executing part of the subtask units, and executes subtask units not executed in the subtask units.

16. A computing device, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the resource scheduling method according to any one of claims 13-15.

17. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the resource scheduling method according to any one of claims 13-15.

Images