YARN cluster resource scheduling method, device, medium and computer equipment

Abstract

The present disclosure relates to a YARN cluster resource scheduling method, apparatus, medium and computer equipment, wherein the method includes: receiving a job resource request of a job task sent by an application manager; when determining that the job resource request is a rigid resource request, adding the rigid resource request to a request queue, the request queue includes multiple request units, and multiple rigid resource requests belonging to different job tasks are added to corresponding different request units; starting multiple first threads, each first thread obtains multiple rigid resource requests in each corresponding request unit from the request queue to perform resource pre-allocation processing; after resource pre-allocation, each first thread determines whether each pre-allocation result meets the resource demand of the corresponding job task; if so, each first thread submits the pre-allocation result to the application manager respectively. The implementation scheme of the present disclosure can improve the resource scheduling performance of the cluster.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the technical field of computer cluster resource management, and in particular to a YARN cluster resource scheduling method, a YARN cluster resource scheduling device, and a computer-readable storage medium and a computer device for implementing the YARN cluster resource scheduling method. Background Art

[0002] YARN is a new Hadoop resource management system that separates resource management from the computing framework and turns it into a general resource management system that can provide unified resource management and scheduling for upper-level applications.

[0003] The YARN system is usually composed of a resource manager RM (Resource Manager), a node manager NM (NodeManager), an application manager AM (Application Master), and a resource container. When scheduling resources, there are some rigid resource requests, such as resource requests for scientific computing or machine learning training tasks. If these jobs apply for 1000 cores of CPU resources, for example, then all 1000 cores of CPU resources must be allocated to users within the specified time, otherwise the task will fail. For example, if 500 cores of CPU resources are allocated, the total resources required by the job task are not met, and the job task cannot be started.

[0004] In the related art, a rigid scheduler (Gang Scheduler) is designed to process rigid resource requests. However, the current implementation of the rigid scheduler is a synchronous method. The AM and RM of all applications communicate through heartbeat timing. The RM uses a lock to ensure that the multiple heartbeats of each application are serial. The Gang Scheduler will synchronously perform a resource scheduling in each heartbeat and synchronously return the scheduling result such as success or failure. This method of resource scheduling is coupled with the heartbeat, resulting in the problem that it is impossible to flexibly customize the queuing strategy for the resource requests of multiple applications, affecting the scheduling performance. In addition, since the synchronous method is used, concurrency is not supported, so when the amount of resource requests increases, some performance bottlenecks will appear. Summary of the invention

[0005] In order to solve the above technical problems or at least partially solve the above technical problems, the embodiments of the present disclosure provide a YARN cluster resource scheduling method, a YARN cluster resource scheduling device, and a computer-readable storage medium and a computer device for implementing the YARN cluster resource scheduling method.

[0006] In a first aspect, an embodiment of the present disclosure provides a YARN cluster resource scheduling method, which is applied to a resource manager. The method includes:

[0007] Receive a job resource request for a job task sent by an application manager;

[0008] When it is determined that the job resource request is a rigid resource request, the rigid resource request is added to a request queue; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units;

[0009] Starting a plurality of first threads, each of the first threads obtaining a plurality of rigid resource requests in a corresponding each of the request units from the request queue to perform resource pre-allocation processing;

[0010] After pre-allocating resources, each of the first threads determines whether each pre-allocation result meets the resource demand of the corresponding job task;

[0011] If so, each of the first threads submits the pre-allocation result to the application manager respectively.

[0012] In some embodiments of the present disclosure, each of the first threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource pre-allocation processing, including:

[0013] Get the node list of all nodes in the cluster;

[0014] Filter out nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated;

[0015] Calculate a node score for each node in the to-be-allocated node list based on at least one constraint condition;

[0016] Determine a target candidate node from the list of nodes to be assigned based on the node score;

[0017] Resources are pre-allocated on the target candidate node for the multiple rigid resource requests in the corresponding request units.

[0018] In some embodiments of the present disclosure, the preset filtering condition at least includes that the node load is greater than a preset load; and / or the at least one constraint condition includes a weak constraint condition of a node attribute.

[0019] In some embodiments of the present disclosure, the method further includes:

[0020] After pre-allocating resources, each of the first threads cancels the pre-allocation result to release the resources in the pre-allocation result when it is determined that the pre-allocation result does not meet the resource demand of the corresponding job task.

[0021] In some embodiments of the present disclosure, the method further includes:

[0022] When the pre-allocation result does not meet the resource demand of the corresponding job task, determining whether the rigid resource request to which the job task belongs carries designated retry information;

[0023] If so, the rigid resource request to which the job task belongs is added back to the request queue.

[0024] In a second aspect, an embodiment of the present disclosure provides a YARN cluster resource scheduling method, which is applied to a resource manager. The method includes:

[0025] Receive a job resource request for a job task sent by an application manager;

[0026] When it is determined that the job resource request is a rigid resource request, the rigid resource request is added to a request queue; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units;

[0027] A plurality of second threads are started, and each of the second threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing.

[0028] In some embodiments of the present disclosure, each of the second threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing, including:

[0029] Get the node list of all nodes in the cluster;

[0030] Filter out nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated;

[0031] Calculate a node score for each node in the to-be-allocated node list based on at least one constraint condition;

[0032] Resource allocation is performed for a plurality of rigid resource requests in a corresponding request unit based on the node score of each node.

[0033] In some embodiments of the present disclosure, allocating resources for multiple rigid resource requests in the corresponding request unit based on the node score of each node includes:

[0034] Sort the nodes based on the node scores of each node, and obtain the allocatable resources of each sorted node;

[0035] The resources are allocated in sequence based on the sorted allocatable resources of each node.

[0036] In some embodiments of the present disclosure, the method further includes:

[0037] Before calculating the node score for each node in the list of nodes to be allocated, obtaining snapshot information of the allocatable resources of each node;

[0038] A node score is calculated for each node based on the snapshot information.

[0039] In some embodiments of the present disclosure, the method further includes:

[0040] After allocating resources, each of the second threads, when determining that the allocation result does not meet the resource demand of the corresponding job task, re-adds the rigid resource request to which the job task belongs to the request queue.

[0041] In a third aspect, an embodiment of the present disclosure provides a YARN cluster resource scheduling device, which is applied to a resource manager, and the device includes:

[0042] A request receiving module, used for receiving a job resource request of a job task sent by an application manager;

[0043] A queue adding module, for adding the rigid resource request to a request queue when determining that the job resource request is a rigid resource request; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units;

[0044] A pre-allocation module, used for starting a plurality of first threads, each of which obtains a plurality of rigid resource requests in a corresponding each of the request units from the request queue to perform resource pre-allocation processing;

[0045] An allocation result judgment module, used to enable each of the first threads to judge whether each pre-allocation result meets the resource demand of the corresponding job task after pre-allocation of resources;

[0046] The allocation result submission module is used to enable each of the first threads to submit the pre-allocation result to the application manager respectively when the judgment result of the allocation result judgment module is yes.

[0047] In a fourth aspect, an embodiment of the present disclosure provides a YARN cluster resource scheduling device, which is applied to a resource manager, and the device includes:

[0048] A request receiving module, used for receiving a job resource request of a job task sent by an application manager;

[0049] A queue adding module, configured to add the rigid resource request to a request queue when determining that the job resource request is a rigid resource request; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units;

[0050] The allocation module is used to start a plurality of second threads, each of which obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing.

[0051] In a fifth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the YARN cluster resource scheduling method described in any of the above embodiments.

[0052] In a sixth aspect, an embodiment of the present disclosure provides an electronic device, including:

[0053] Processor; and

[0054] Memory for storing computer programs;

[0055] The processor is configured to execute the steps of the YARN cluster resource scheduling method described in any of the above embodiments by executing the computer program.

[0056] Compared with the prior art, the technical solution provided by the embodiments of the present disclosure has the following advantages:

[0057] In the YARN cluster resource scheduling method, apparatus, medium and computer equipment provided by the embodiments of the present disclosure, the resource manager RM receives a job resource request of a job task sent by the application manager AM, and when it is determined that the job resource request is a rigid resource request, the resource manager RM adds the rigid resource request to a request queue, wherein the request queue includes multiple request units, and multiple rigid resource requests belonging to different job tasks are added to corresponding different request units; multiple first threads are started, and each of the first threads obtains the corresponding multiple rigid resource requests in each of the request units from the request queue to perform resource pre-allocation processing; after resource pre-allocation, each of the first threads determines whether each pre-allocation result meets the resource demand of the corresponding job task; if so, each of the first threads submits the pre-allocation result to the AM respectively. In this way, the solution of this embodiment can add the job resource request to the request queue, and then take the job resource request from the request queue for resource scheduling, so as to decouple the heartbeat of the AM from the resource scheduling. After the AM heartbeat sends the job resource request, the RM does not need to immediately return the resource scheduling allocation result, and can asynchronously take the job resource request from the request queue for resource allocation, which improves the scheduling performance to a certain extent. At the same time, multiple threads can concurrently take the job resource request of the corresponding job task from the corresponding request unit in the request queue for resource pre-allocation and submit the allocation result, which alleviates the bottleneck problem of scheduling performance when the amount of resource requests increases. In addition, in the case of concurrent job resource requests for multiple job tasks, the allocation process of the node resources involved in the rigid resource request to which each job task belongs can be uniformly allocated as a whole or uniformly rolled back, which improves the scheduling performance to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0060] Figure 1 A schematic diagram of a YARN cluster system according to an embodiment of the present disclosure;

[0061] Figure 2 This is a flow chart of the YARN cluster resource scheduling method according to an embodiment of the present disclosure;

[0062] Figure 3 A schematic diagram of the YARN cluster resource scheduling method process according to an embodiment of the present disclosure;

[0063] Figure 4 This is a flow chart of a YARN cluster resource scheduling method according to another embodiment of the present disclosure;

[0064] Figure 5 This is a flow chart of a YARN cluster resource scheduling method according to another embodiment of the present disclosure;

[0065] Figure 6 This is a flow chart of a YARN cluster resource scheduling method according to another embodiment of the present disclosure;

[0066] Figure 7 A schematic diagram of a YARN cluster resource scheduling device according to an embodiment of the present disclosure;

[0067] Figure 8 This is a schematic diagram of a YARN cluster resource scheduling device according to another embodiment of the present disclosure;

[0068] Fig. 9 A schematic diagram of a computer device for implementing a YARN cluster resource scheduling method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0069] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0070] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0071] It should be understood that, in the following, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0072] Figure 1 This is a schematic diagram of the YARN cluster system. Figure 2A YARN cluster resource scheduling method is provided in an embodiment of the present disclosure. The method can be based on Figure 1 The cluster system implementation shown can be specifically applied to the resource manager RM. The method may include the following steps:

[0073] Step S201: receiving a job resource request of a job task sent by an application manager.

[0074] For example, a job resource request may carry information such as the type and quantity of resources required for the corresponding job task. The resource type may be different types of resources such as CPU, memory, etc. The same job task may have multiple job resource requests in succession. Specifically, the AM may send the job resource request of the job task of the application submitted by the client to the RM based on the heartbeat.

[0075] Step S202: When it is determined that the job resource request is a rigid resource request, the rigid resource request is added to a request queue; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units.

[0076] Exemplarily, a job resource request may be a flexible resource request or a rigid resource request. For example, if a rigid resource request requests 1000 CPU cores, the RM must successfully allocate sufficient 1000 CPU cores before returning the allocation result to the AM, otherwise it will fail directly.

[0077] In this embodiment, the RM can call a pre-configured rigid scheduler (Gang Scheduler) to process rigid resource requests. All existing schedulers can be configured in the RM in a pluggable manner, which can be understood by referring to the existing technology and will not be described here.

[0078] When RM determines that the received job resource request is a rigid resource request, it adds the rigid resource request to the request queue (RequestQueue), and the heartbeat of AM returns immediately without performing a lock (Block) operation. GangScheduler then asynchronously retrieves the rigid resource request from the Request Queue to allocate resources. That is, in this embodiment, GangScheduler is in an asynchronous scenario. At this time, in order to adaptively record the situation of multiple rigid resource requests belonging to the same job task, in the asynchronous scenario, refer to Figure 3As shown in , in this embodiment, multiple rigid resource requests belonging to the same job task are encapsulated into a request cell (Request Cell), and multiple rigid resource requests belonging to the same job task are added to a request cell (Request Cell) in the request queue (Request Queue), and different job tasks correspond to different request cells (Request Cell). For example, job task 1 corresponds to Request Cell1, job task 2 corresponds to Request Cell2, and so on, job task N corresponds to RequestCellN, and so on. The unique identification ID of each Request Cell can be the unique identification ID of the corresponding job task, and all job resource requests belonging to a job task can carry the unique identification ID of the job task, so that multiple rigid resource requests belonging to different job tasks can be distinguished and added to their respective corresponding different request cells.

[0079] Step S203: Start a plurality of first threads, each of which obtains a plurality of rigid resource requests in a corresponding request unit from the request queue to perform resource pre-allocation processing.

[0080] Exemplarily, each first thread can be considered as a transaction, and each first thread can centrally process multiple rigid resource requests in a corresponding request cell in the request queue, such as thread 1 corresponding to Request Cell 1, thread 2 corresponding to Request Cell 2, and thread N corresponding to Request Cell N. Specifically, Gang Scheduler can start multiple first threads, that is, start transactions, and each first thread obtains multiple rigid resource requests in the corresponding request cell from the request queue for resource pre-allocation processing. No real resource allocation is performed here, only resource pre-allocation, which can be understood as the transaction is not submitted.

[0081] Step S204: After pre-allocating resources, each of the first threads determines whether each pre-allocation result meets the resource demand of the corresponding job task.

[0082] Exemplarily, after pre-allocating resources, each first thread determines whether its own pre-allocation result meets the resource requirement of the corresponding job task. For example, thread 1 processes the job resource request in request cell Request Cell1, obtains pre-allocation result 1, and then determines whether pre-allocation result 1 meets the resource requirement of the corresponding job task 1. The resource requirement can be calculated based on the sum of the resource requirements carried by multiple rigid resource requests belonging to job task 1.

[0083] Step S205: If yes, each of the first threads submits the pre-allocation result to the AM respectively.

[0084] Specifically, when each first thread determines that its own pre-allocation result meets the resource demand of the corresponding job task, it submits its own pre-allocation result to the AM respectively, which can be understood here as the final actual allocation and transaction submission.

[0085] In this embodiment, the resource allocation of the Gang Scheduler is scheduled based on the overall scheduling of the request unit Request Cell, and the pre-allocation result may at least include a group of resource container lists.

[0086] The YARN cluster resource scheduling method of this embodiment can add the job resource request to the request queue, and then take the job resource request from the request queue for resource scheduling, so as to decouple the heartbeat of the AM from the resource scheduling. After the AM heartbeat sends the job resource request, the RM does not need to immediately return the resource scheduling allocation result, and can asynchronously take the job resource request from the request queue for resource allocation, which improves the scheduling performance to a certain extent. At the same time, multiple threads can concurrently take the job resource request of the corresponding job task from the corresponding request unit in the request queue for resource pre-allocation and submit the allocation result. When the amount of resource requests increases, the bottleneck problem of scheduling performance can be alleviated. In addition, in the case of concurrent job resource requests for multiple job tasks, the allocation process of multiple node resources involved in the rigid resource request to which each job task belongs can be uniformly allocated as a whole or uniformly rolled back, which improves the scheduling performance to a certain extent.

[0087] Optionally, in some embodiments of the present disclosure, Figure 4 As shown in , in step S203, each of the first threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource pre-allocation processing, which may specifically include the following steps:

[0088] Step S401: Obtain a node list consisting of all nodes in the cluster.

[0089] Specifically, each node in the cluster, such as a server, may include a node manager NM (NodeManager). NM may send heartbeat information to RM to report its own load information and / or resource container information. RM may obtain a node list of all nodes in the cluster, and the node list may include information such as node identification, node load, node attributes, etc., but is not limited thereto.

[0090] Step S402: Filter out nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated.

[0091] Exemplarily, in some embodiments of the present disclosure, the preset filtering condition may at least include but is not limited to the node load being greater than the preset load. The preset load can be set by a user, and there is no restriction on this. In this embodiment, the nodes with large node loads in the cluster can be filtered out, and a list of nodes to be allocated is obtained based on the remaining nodes. The remaining nodes can be understood as nodes with small loads and more idle resources.

[0092] For example, the node list includes four nodes, such as N1, N2, N3 and N4. The node list to be allocated obtained after filtering includes three nodes, such as N1, N2 and N3.

[0093] Step S403: Calculate a node score for each node in the list of nodes to be allocated based on at least one constraint condition.

[0094] Exemplarily, calculating the node score is to score the node, that is, to calculate the expected amount of allocated resources. The at least one constraint condition may include, but is not limited to, weak constraints on node attributes, that is, conditions that do not necessarily have to be met, and a best effort is made. Node attributes, for example, may include, but are not limited to, memory attributes of the node, etc. For example, there is a first constraint condition and a second constraint condition, and the first constraint condition may include, but is not limited to, at least one of the following: the load corresponding to the node is lower than the predetermined load, and the memory corresponding to the node is greater than the predetermined memory. The second constraint condition may include, but is not limited to, at least one of the following: nodes are evenly distributed, and high-load nodes are skipped. Among them, node distribution evenly means that after the resource allocation is completed, the used resources of each node are as consistent as possible. Skipping high-load nodes may mean that resource requests skip nodes that do not meet the load threshold as much as possible.

[0095] Specifically, each first thread can score each node in the list of nodes to be allocated, such as N1, N2, and N3, based on the first constraint to obtain a score score1, and can score each node in the list of nodes to be allocated, such as N1, N2, and N3, based on the second constraint to obtain a score score2, and then perform weighted calculation on the scores score1 and score2 of each node, such as N1, N2, and N3 to obtain a final score. The specific scoring calculation method can be understood by referring to the prior art, and will not be repeated here.

[0096] Step S404: determining a target candidate node from the list of nodes to be assigned based on the node scores.

[0097] Specifically, the larger the node score, the higher the priority of the corresponding node as the target candidate node. For example, the target candidate nodes determined based on the final score are N1 and N3. Figure 3As shown, in order to adaptively match the above request cell, the target candidate node may exist in the form of an allocation candidate cell, that is, the allocation candidate cell includes, for example, the determined target candidate nodes N1 and N3.

[0098] Step S405: pre-allocate resources for the multiple rigid resource requests in the corresponding request units on the target candidate node.

[0099] Specifically, for example, resources are pre-allocated for multiple rigid resource requests in corresponding request cells at target candidate nodes N1 and N3.

[0100] After step S405, the process jumps to step S204. When it is determined that the pre-allocation result meets the resource demand of the corresponding job task, the allocation candidate cell (Allocation Candidate Cell) can be converted into an allocation cell (AllocationCell), and the pre-allocation result can be submitted to the AM in units of the allocation cell.

[0101] In this embodiment, the final suitable nodes are determined by the above-mentioned screening and filtering, node score calculation and other methods, and the resource pre-allocation based on the determined nodes greatly improves the accuracy of the resource allocation result.

[0102] Optionally, in some embodiments of the present disclosure, the method may further include the following steps: after pre-allocating resources, each of the first threads determines that the pre-allocation result does not meet the resource requirements of the corresponding job task, canceling the pre-allocation result to release the resources in the pre-allocation result. In this way, when the resource amount of the rigid resource request of a job task cannot be met, it can directly fail and release the pre-allocated resources, so as to facilitate timely resource allocation for other job tasks.

[0103] Optionally, in some embodiments of the present disclosure, the method may further include the following steps:

[0104] Step i): when the pre-allocation result does not meet the resource demand of the corresponding job task, it is determined whether the rigid resource request to which the job task belongs carries designated retry information.

[0105] In this embodiment, any rigid resource request belonging to a job task may carry designated retry information, so that if the rigid resource request cannot be satisfied after allocating resources once, the resource can be retried for allocation again.

[0106] Step ii): If yes, then the rigid resource request to which the job task belongs is added back into the request queue.

[0107] Specifically, when it is determined that the allocation needs to be retried again, the rigid resource request can be re-added to the request queue. Referring to the details in step S202, the re-added rigid resource request, for example, belongs to job task 1, then it is re-added to the request cell Request Cell1 in the request queue; if it belongs to job task 2, then it is re-added to the request cell Request Cell2 in the request queue.

[0108] The present disclosure also provides a YARN cluster resource scheduling method, which is applied to a resource manager, such as Figure 5 As shown in , the method comprises the following steps:

[0109] Step S501: receiving a job resource request of a job task sent by an application manager;

[0110] Step S502: When it is determined that the job resource request is a rigid resource request, the rigid resource request is added to a request queue; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units.

[0111] It can be understood that the above steps S501 to S502 may refer to the corresponding detailed description of steps S201 to S202 in the above embodiments, which will not be repeated here.

[0112] Step S503: Start a plurality of second threads, each of which obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing.

[0113] For example, Figure 3 As shown, each second thread can be considered as a transaction, and each second thread can centrally process multiple rigid resource requests in a corresponding request cell in the request queue, such as thread 1 corresponding to Request Cell 1, thread 2 corresponding to Request Cell 2, and thread N corresponding to Request Cell N. Specifically, the Gang Scheduler can start multiple second threads, that is, start transactions, and each second thread obtains multiple rigid resource requests in the corresponding request cell from the request queue for resource allocation processing.

[0114] The solution of this embodiment can add the job resource request to the request queue, and then take the job resource request from the request queue for resource scheduling, so as to decouple the heartbeat of the AM from the resource scheduling. After the AM heartbeat sends the job resource request, the RM does not need to immediately return the resource scheduling allocation result, and can asynchronously take the job resource request from the request queue for resource allocation, thereby improving the scheduling performance to a certain extent. In addition, in the case of concurrent job resource requests for multiple job tasks, the allocation process of the node resources involved in the rigid resource request to which each job task belongs can be uniformly allocated as a whole, thereby improving the scheduling performance to a certain extent.

[0115] Optionally, in some embodiments of the present disclosure, Figure 6 As shown in the above step S503, each of the second threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing, which may specifically include the following steps:

[0116] Step S601: Obtain a node list consisting of all nodes in the cluster.

[0117] Step S602: Filter out nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated.

[0118] Step S603: Calculate a node score for each node in the list of nodes to be allocated based on at least one constraint condition.

[0119] It can be understood that the specific implementation of steps S601 to S603 can refer to the detailed description of the above steps S401 to S403, which will not be repeated here.

[0120] Step S604: Allocate resources for multiple rigid resource requests in the corresponding request unit based on the node score of each node.

[0121] The difference between this embodiment and the above embodiments is that after calculating the node score for a single node, resources are directly allocated without pre-allocating resources, which can improve the efficiency of resource scheduling to a certain extent.

[0122] Further, optionally, in some embodiments of the present disclosure, in step S604, resource allocation is performed for multiple rigid resource requests in the corresponding request unit based on the node score of each node, which may specifically include: sorting the nodes based on the node score of each node, and obtaining the allocatable resources of each sorted node; and allocating resources in sequence based on the allocatable resources of each sorted node.

[0123] Exemplarily, nodes may be sorted based on the size of the node score of each node, and resource allocation may be prioritized based on the allocatable resources of nodes with large scores. Optionally, in some embodiments, multiple rounds of allocation may be performed in this way: for example, when node N1 is allocated after the score is calculated, node N1 may not be allocated the expected resources, such as other transactions have allocated resources from the node during this period. In this case, it is a pity to directly cancel the release of resources. Considering that the definition of weak constraints itself is to do its best, the optimization strategy may be to first allocate the remaining resources of the node. If the node still has surplus resources, it will participate in another round of allocation.

[0124] Optionally, in some embodiments of the present disclosure, the method may further include the following steps: before calculating the node score for each node in the list of nodes to be allocated, obtaining snapshot information of the allocatable resources of each node; and calculating the node score for each node based on the snapshot information.

[0125] Specifically, because multiple threads are performing resource allocation concurrently, there is a concurrency conflict, which causes a problem such as dirty reads in the allocatable resource data of a node obtained by a certain thread. In this embodiment, snapshot information of allocatable resources of each node, such as snapshot available, is obtained, and scores are given to each node based on the snapshot information, thereby realizing subsequent resource allocation.

[0126] For example, when calculating the score for node N1, each second thread does not directly take the actual latest value of the current resource, but initially obtains the latest snapshot available resource from node N1 and performs the score accordingly, which can solve problems such as dirty reads caused by concurrency conflicts. Alternatively, the snapshot available of the node can be used for calculation during the entire second thread processing to solve the repeatable read problem.

[0127] In some cases, a snapshot of the node is obtained for resource allocation. If the node data changes during the allocation, the transaction can be aborted early, that is, the thread can be restarted.

[0128] Optionally, in some embodiments of the present disclosure, the method may further include the following steps: after resource allocation, when each of the second threads determines that the allocation result does not meet the resource demand of the corresponding job task, it adds the rigid resource request to which the job task belongs back to the request queue.

[0129] Specifically, for example, after each second thread scores node N1 and calculates the allocatable resources, it directly allocates the resource container and performs the same operation on the subsequent nodes. After a round of operation on all the nodes after screening, it is checked whether the resource requirements of the job task are met. If they are met, the submission is successful, and if they are not met, the submission fails. If it fails, the corresponding rigid resource request is put back into the request queue to continue resource allocation. In this way, there is no need to directly release the allocated resources but to continue to allocate them, which improves the scheduling performance to a certain extent.

[0130] It should be noted that, although the steps of the method in the present disclosure are described in a specific order in the accompanying drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step for execution, and / or one step may be decomposed into multiple steps for execution, etc. In addition, it is also easy to understand that these steps may be, for example, executed synchronously or asynchronously in multiple modules / processes / threads.

[0131] Figure 7 The YARN cluster resource scheduling device of the embodiment of the present disclosure is shown, which is applied to a resource manager. The device may include:

[0132] The request receiving module 701 is used to receive a job resource request of a job task sent by the application manager;

[0133] The queue adding module 702 is used to add the rigid resource request to the request queue when determining that the job resource request is a rigid resource request; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units;

[0134] A pre-allocation module 703 is used to start a plurality of first threads, each of which obtains a plurality of rigid resource requests in each of the request units from the request queue to perform resource pre-allocation processing;

[0135] The allocation result judgment module 704 is used to enable each of the first threads to judge whether each pre-allocation result meets the resource demand of the corresponding job task after pre-allocation of resources;

[0136] The allocation result submitting module 705 is used to enable each of the first threads to submit the pre-allocation result to the application manager respectively when the judgment result of the allocation result judging module is yes.

[0137] The YARN cluster resource scheduling method of this embodiment can add the job resource request to the request queue, and then take the job resource request from the request queue for resource scheduling, so as to decouple the heartbeat of the application manager AM from the resource scheduling. After the AM heartbeat sends the job resource request, the RM does not need to immediately return the resource scheduling allocation result, and can asynchronously take the job resource request from the request queue for resource allocation, which improves the scheduling performance to a certain extent. At the same time, multiple threads can concurrently take the job resource request of the corresponding job task from the corresponding request unit in the request queue for resource pre-allocation and submit the allocation result, which alleviates the bottleneck problem of scheduling performance when the resource request amount increases. In addition, in the case of concurrent job resource requests for multiple job tasks, the allocation process of multiple node resources involved in the rigid resource request to which each job task belongs can be uniformly allocated as a whole or uniformly rolled back, which improves the scheduling performance to a certain extent.

[0138] In some embodiments of the present disclosure, the pre-allocation module is used to: obtain a node list consisting of all nodes in the cluster; filter out nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated; calculate a node score for each node in the list of nodes to be allocated based on at least one constraint condition; determine a target candidate node from the list of nodes to be allocated based on the node score; and pre-allocate resources on the target candidate node for multiple rigid resource requests in the corresponding request unit.

[0139] In some embodiments of the present disclosure, the preset filtering condition at least includes that the node load is greater than a preset load; and / or the at least one constraint condition includes a weak constraint condition of a node attribute.

[0140] In some embodiments of the present disclosure, the device also includes a resource release module, which is used to enable each of the first threads to cancel the pre-allocation result to release the resources in the pre-allocation result when it is determined that the pre-allocation result does not meet the resource demand of the corresponding job task after resource pre-allocation.

[0141] In some embodiments of the present disclosure, the device also includes a request retry module, which is used to determine whether the rigid resource request to which the job task belongs carries specified retry information when the pre-allocation result does not meet the resource demand of the corresponding job task; if so, the rigid resource request to which the job task belongs is re-added to the request queue.

[0142] The present disclosure also provides a YARN cluster resource scheduling device, which is applied to a resource manager, such as Figure 8 As shown, the device comprises:

[0143] The request receiving module 801 is used to receive a job resource request of a job task sent by the application manager;

[0144] The queue adding module 802 is used to add the rigid resource request to the request queue when determining that the job resource request is a rigid resource request; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units;

[0145] The allocation module 803 is used to start multiple second threads, each of which obtains multiple rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing.

[0146] In some embodiments of the present disclosure, each of the second threads started by the allocation module 803 obtains multiple rigid resource requests in the corresponding each request unit from the request queue to perform resource allocation processing, which may specifically include: obtaining a node list consisting of all nodes in the cluster; filtering out nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated; calculating a node score for each node in the list of nodes to be allocated based on at least one constraint condition; and allocating resources to the multiple rigid resource requests in the corresponding request unit based on the node score of each node.

[0147] In some embodiments of the present disclosure, the allocation module 803 performs resource allocation for multiple rigid resource requests in the corresponding request unit based on the scoring results of each node, including: sorting the nodes based on the node scores of each node, and obtaining the allocatable resources of each sorted node; and allocating resources in sequence based on the allocatable resources of each sorted node.

[0148] In some embodiments of the present disclosure, the device further includes a snapshot acquisition module, which is used to obtain snapshot information of the allocatable resources of each node before calculating the node score for each node in the list of nodes to be allocated. The allocation module 803 is also used to calculate the node score for each node based on the snapshot information.

[0149] In some embodiments of the present disclosure, the device may also include a request retry module, which is used to re-add the rigid resource request belonging to the job task to the request queue when, after each second thread allocates resources, it is determined that the allocation result does not meet the resource requirement of the corresponding job task.

[0150] Regarding the devices in the above-mentioned embodiments, the specific manner in which each module performs operations and the corresponding technical effects brought about have been described in detail in the corresponding embodiments of the method above, and will not be elaborated here.

[0151] It should be noted that although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the present disclosure, the features and functions of two or more modules or units described above can be concretized in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units for concretization. The components displayed as modules or units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the disclosed scheme. Those of ordinary skill in the art can understand and implement it without paying creative work.

[0152] The embodiments of the present disclosure further provide a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the YARN cluster resource scheduling method described in any one of the above embodiments are implemented.

[0153] Exemplarily, the readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0154] The computer readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by an instruction execution system, an apparatus, or a device or used in combination with it. The program code contained on the readable storage medium may be transmitted with any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0155] The present disclosure also provides a computer device, such as Fig. 9 The computer device shown in the figure may include a processor 901 and a memory 902, and the memory 902 is used to store a computer program. The processor 901 is configured to execute the steps of the YARN cluster resource scheduling method in any of the above embodiments by executing the computer program.

[0156] The various aspects, implementations, implementations or features of the described embodiments can be used alone or in any combination. The various aspects of the described embodiments can be implemented by software, hardware or a combination of software and hardware. The described embodiments can also be embodied by a computer-readable medium storing computer-readable code, which includes instructions that can be executed by at least one computing device. The computer-readable medium can be associated with any data storage device capable of storing data that can be read by a computer system. Computer-readable media for example can include read-only memory, random access memory, CD-ROM, HDD, DVD, magnetic tape, and optical data storage devices. The computer-readable medium can also be distributed in computer systems connected via a network so that the computer-readable code can be stored and executed in a distributed manner.

[0157] The above technical description may refer to the accompanying drawings, which form a part of the present application, and the implementation methods according to the described embodiments are shown in the accompanying drawings by description. Although these embodiments are described in sufficient detail to enable those skilled in the art to implement these embodiments, these embodiments are non-restrictive; other embodiments can be used in this way, and changes can be made without departing from the scope of the described embodiments. For example, the order of operations described in the flowchart is non-restrictive, so the order of two or more operations illustrated in the flowchart and described according to the flowchart can be changed according to several embodiments. As another example, in several embodiments, one or more operations illustrated in the flowchart and described according to the flowchart are optional or deletable. In addition, certain steps or functions can be added to the disclosed embodiments, or the order of two or more steps can be replaced. All these changes are considered to be included in the disclosed embodiments and claims.

[0158] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0159] The above description is only a specific embodiment of the present disclosure, so that those skilled in the art can understand or implement the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A YARN cluster resource scheduling method, characterized in that: Applied to a resource manager, the method comprises: Receive a job resource request for a job task sent by the application manager based on a heartbeat; When it is determined that the job resource request is a rigid resource request, the rigid resource request is added to a request queue; the heartbeat returns immediately without performing a lock operation; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units; Starting a plurality of first threads, each of the first threads obtaining a plurality of rigid resource requests in a corresponding each of the request units from the request queue to perform resource pre-allocation processing; After pre-allocating resources, each of the first threads determines whether each pre-allocation result meets the resource demand of the corresponding job task; If so, each of the first threads submits the pre-allocation result to the application manager respectively.

2. The resource scheduling method according to claim 1, characterized in that: Each of the first threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource pre-allocation processing, including: Get the node list of all nodes in the cluster; Filtering nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated; Calculate a node score for each node in the to-be-allocated node list based on at least one constraint condition; Determine a target candidate node from the list of nodes to be assigned based on the node score; Resources are pre-allocated on the target candidate node for the multiple rigid resource requests in the corresponding request units.

3. The resource scheduling method according to claim 2, characterized in that: The preset filtering condition at least includes that the node load is greater than a preset load; and / or the at least one constraint condition includes a weak constraint condition of a node attribute.

4. The resource scheduling method according to claim 1 or 2, characterized in that: The method further comprises: After pre-allocating resources, each of the first threads cancels the pre-allocation result to release the resources in the pre-allocation result when it is determined that the pre-allocation result does not meet the resource demand of the corresponding job task.

5. The resource scheduling method according to claim 4, characterized in that: The method further comprises: When the pre-allocation result does not meet the resource demand of the corresponding job task, determining whether the rigid resource request to which the job task belongs carries designated retry information; If so, the rigid resource request to which the job task belongs is added back to the request queue.

6. A YARN cluster resource scheduling method, characterized in that: Applied to a resource manager, the method comprises: Receive a job resource request for a job task sent by the application manager based on a heartbeat; When it is determined that the job resource request is a rigid resource request, the rigid resource request is added to a request queue; the heartbeat returns immediately without performing a lock operation; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units; A plurality of second threads are started, and each of the second threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing.

7. The resource scheduling method according to claim 6, characterized in that: Each of the second threads obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing, including: Get the node list of all nodes in the cluster; Filter out nodes that meet preset filtering conditions from the node list to obtain a list of nodes to be allocated; Calculate a node score for each node in the to-be-allocated node list based on at least one constraint condition; Resource allocation is performed for a plurality of rigid resource requests in a corresponding request unit based on the node score of each node.

8. The resource scheduling method according to claim 7, characterized in that: The performing resource allocation for a plurality of rigid resource requests in a corresponding request unit based on the node score of each node includes: Sort the nodes based on the node scores of each node, and obtain the allocatable resources of each sorted node; The resources are allocated in sequence based on the sorted allocatable resources of each node.

9. The resource scheduling method according to claim 7, characterized in that: The method further comprises: Before calculating the node score for each node in the list of nodes to be allocated, obtaining snapshot information of the allocatable resources of each node; A node score is calculated for each node based on the snapshot information.

10. The resource scheduling method according to claim 8 or 9, characterized in that: The method further comprises: After allocating resources, each of the second threads, when determining that the allocation result does not meet the resource demand of the corresponding job task, re-adds the rigid resource request to which the job task belongs to the request queue.

11. A YARN cluster resource scheduling device, characterized in that: Applied to a resource manager, the device comprises: A request receiving module, used for receiving a job resource request of a job task sent by the application manager based on the heartbeat; A queue adding module is used to add the rigid resource request to the request queue when determining that the job resource request is a rigid resource request; the heartbeat returns immediately without performing a lock operation; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units; A pre-allocation module, used for starting a plurality of first threads, each of which obtains a plurality of rigid resource requests in a corresponding each of the request units from the request queue to perform resource pre-allocation processing; An allocation result judgment module, used to enable each of the first threads to judge whether each pre-allocation result meets the resource demand of the corresponding job task after pre-allocation of resources; The allocation result submitting module is used to enable each of the first threads to submit the pre-allocation result to the application manager respectively when the judgment result of the allocation result judging module is yes.

12. A YARN cluster resource scheduling device, characterized in that: Applied to a resource manager, the device comprises: A request receiving module, used for receiving a job resource request of a job task sent by the application manager based on the heartbeat; A queue adding module is used to add the rigid resource request to the request queue when determining that the job resource request is a rigid resource request; the heartbeat returns immediately without performing a lock operation; wherein the request queue includes a plurality of request units, and a plurality of rigid resource requests belonging to different job tasks are added to corresponding different request units; The allocation module is used to start a plurality of second threads, each of which obtains a plurality of rigid resource requests in each corresponding request unit from the request queue to perform resource allocation processing.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the YARN cluster resource scheduling method according to any one of claims 1 to 10 are implemented.

14. A computer device, characterized in that: include: processor; as well as Memory for storing computer programs; The processor is configured to execute the steps of the YARN cluster resource scheduling method according to any one of claims 1 to 10 by executing the computer program.

Images