Resource scheduling method, device, electronic device and storage medium

Abstract

The present application provides a resource scheduling method, device, electronic device and storage medium, which relates to the field of cluster scheduling technology, including: obtaining resource trees corresponding to different resource devices; in response to training tasks issued by users, obtaining the required number of GPU cards for training tasks, and, based on multiple resource trees of different resource devices, obtaining the total number of idle GPU cards; determining the predicted rate based on the total number of idle GPU cards and the required number of GPU cards; determining the target first-level node of the target resource device based on the communication rate and predicted rate of the first-level nodes in different resource devices; executing the training task based on multiple GPU cards under the target first-level node. The present application increases the perception of NVLINK and NVSWITCH resources with inter-card communication functions in the scheduler, and automatically selects GPU cards with efficient communication resources for training tasks based on the predicted rate, thereby effectively improving the efficiency of training and reasoning tasks.

Description

Technical Field

[0001] Embodiments of the present application relate to the technical field of cluster scheduling, and in particular, to a resource scheduling method, device, electronic device, and storage medium. Background Art

[0002] NVLINK and NVSWITCH are high-speed card-to-card communication technologies launched by NVIDIA based on custom hardware for multiple GPU cards in the same server. Adding NVLINK or NVSWITCH in scenarios such as AI distributed training, multi-card training, and multi-card reasoning can greatly improve task training efficiency. The communication efficiency of NVLINK and NVSWITCH also varies greatly between generations.

[0003] On the one hand, the existing cluster resource scheduling technology for training tasks only manages GPU resources, lacks the perception of NVLINK and NVSWITCH resources. In a cluster where GPU devices without NVLINK or NVSWITCH and GPU devices with NVLINK or NVSWITCH are mixed, it is impossible to reasonably and automatically allocate matching resources, resulting in the inability to improve training efficiency. On the other hand, the existing scheduling technology also lacks the perception of the NVLINK and NVSWITCH communication rates within the cluster. In a cluster where NVLINK and NVSWITCH of different generations are mixed, it is impossible to reasonably and automatically select resources with uniform rates, resulting in resource waste. Therefore, how to improve the efficiency of training and reasoning tasks based on cluster scheduling has become a problem that needs to be solved urgently in this field. Summary of the invention

[0004] The embodiments of the present application provide a resource scheduling method, device, electronic device and storage medium, aiming to solve the problem of how to improve the efficiency of training and reasoning tasks based on cluster scheduling.

[0005] A first aspect of an embodiment of the present application provides a resource scheduling method, the method comprising:

[0006] Obtain resource trees corresponding to different resource devices, respectively, wherein the resource tree includes at least one first-level node, and different first-level nodes are used to represent multiple GPU clusters with inter-card communication functions and different communication rates;

[0007] In response to a training task issued by a user, the number of GPU cards required for the training task is obtained, and the total number of idle GPU cards is obtained based on multiple resource trees of different resource devices;

[0008] Determine a predicted rate based on the total number of idle GPU cards and the required number of GPU cards;

[0009] Determining a target first-level node of a target resource device based on the communication rates of the first-level nodes in the different resource devices and the predicted rate;

[0010] The training task is executed based on multiple GPU cards in the lower layer of the target first-level node.

[0011] In an optional implementation manner, the obtaining resource trees corresponding to different resource devices respectively includes:

[0012] Get the resource topology matrix on different resource devices;

[0013] Build the root node and create the initial node under the root node;

[0014] Based on the resource topology matrix, generating a plurality of first-level nodes corresponding to the resource topology information with a target label in the resource topology matrix, wherein the first-level nodes are arranged at the same layer as the initial nodes, and the target label is used to indicate that the GPU cluster has the inter-card communication function;

[0015] The resource topology information in the resource topology matrix is ​​traversed in a preset traversal order to determine a plurality of second-level nodes under each first-level node, wherein the second-level nodes are used to represent the GPU cards included in the first-level nodes.

[0016] In an optional implementation manner, the obtaining the total number of idle GPU cards based on multiple resource trees of different resource devices includes:

[0017] Based on the initial rate parameter, determining a first matching node in the plurality of resource trees, the first matching node being a first-level node in each resource tree with a communication rate greater than or equal to the initial rate parameter;

[0018] Obtaining the sum of the number of idle second-level nodes in the lower layer of each of the first matching nodes in all resource trees as the total number of idle GPU cards;

[0019] The determining the predicted rate based on the total number of idle GPU cards and the required number of GPU cards includes:

[0020] When the total number of idle GPU cards is greater than or equal to the required number of GPU cards, the initial rate parameter is used as the predicted rate;

[0021] In the case that the total number of idle GPU cards is less than the required number of GPU cards, the preset rate variable is reduced based on the initial rate parameter to obtain the predicted rate.

[0022] In an optional implementation, after determining the predicted rate, the method further includes:

[0023] In the case where the user presets an expected rate, comparing the expected rate with the predicted rate;

[0024] In the case where the predicted rate is greater than or equal to the expected rate, determining a second matching node in the plurality of resource trees based on the predicted rate, the second matching node being a first-level node in each resource tree whose communication rate is greater than or equal to the predicted rate;

[0025] Obtaining the sum of the number of second-level nodes in an idle state in the lower layers of each of the second matching nodes in all resource trees as the updated total number of idle GPU cards;

[0026] When the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards, maintaining the predicted rate;

[0027] When the updated total number of idle GPU cards is less than the required number of GPU cards, the predicted rate is reduced by a preset rate variable.

[0028] In an optional implementation, after determining the predicted rate, the method further includes:

[0029] If the user does not set an expected rate, the predicted rate is compared with 0;

[0030] If the predicted rate is greater than or equal to 0, determining a second matching node in the plurality of resource trees based on the predicted rate, the second matching node being a first-level node in each resource tree with a communication rate greater than or equal to the predicted rate;

[0031] Obtaining the sum of the number of second-level nodes in an idle state in the lower layers of each of the second matching nodes in all resource trees as the updated total number of idle GPU cards;

[0032] When the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards, maintaining the predicted rate;

[0033] When the updated total number of idle GPU cards is less than the required number of GPU cards, the predicted rate is reduced by a preset rate variable.

[0034] In an optional implementation manner, determining the target first-level node of the target resource device based on the communication rate of the first-level node in the different resource devices and the predicted rate includes:

[0035] Based on the first-level nodes of the plurality of resource trees, obtaining first-level nodes whose communication rate is greater than or equal to the preset rate as nodes to be confirmed;

[0036] Obtain the sum of the number of idle second-level nodes in the lower layer of each node to be confirmed as the total number of idle GPU cards of the node to be confirmed;

[0037] When the total number of idle GPU cards of the node to be confirmed is greater than or equal to the required number of GPU cards, the node to be confirmed is used as a candidate node;

[0038] The candidate nodes are screened to obtain the target first-level node.

[0039] In an optional implementation manner, the obtaining the target first-level node based on screening the candidate nodes includes:

[0040] Obtaining a difference between the communication rate of each candidate node and the predicted rate;

[0041] Generate a resource score for each candidate node based on the difference between the communication rate of each candidate node and the predicted rate, wherein the greater the difference between the communication rate of the candidate node and the predicted rate, the higher the resource score of the candidate node;

[0042] The candidate node with the highest resource score is used as the target first-level node.

[0043] A second aspect of an embodiment of the present application provides a resource scheduling device, the device comprising:

[0044] A resource tree acquisition module, used to acquire resource trees corresponding to different resource devices, wherein the resource tree includes at least one first-level node, and different first-level nodes are used to represent multiple GPU clusters with inter-card communication functions and different communication rates;

[0045] The rate determination data module is used to obtain the number of GPU cards required for the training task in response to the training task issued by the user, and to obtain the total number of idle GPU cards based on multiple resource trees of different resource devices;

[0046] A prediction rate module, used to determine a prediction rate based on the total number of idle GPU cards and the required number of GPU cards;

[0047] A target node determination module, configured to determine a target first-level node of a target resource device based on the communication rates of the first-level nodes in the different resource devices and the predicted rate;

[0048] The task execution module is used to execute the training task based on the multiple GPU cards under the target first-level node.

[0049] A third aspect of an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the resource scheduling method described in any one of the first aspect of the embodiments of the present application.

[0050] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium having a computer program / instruction stored thereon. When the computer program / instruction is executed by a processor, the steps of the resource scheduling method described in any one of the first aspect of the embodiment of the present application are implemented.

[0051] Beneficial effects:

[0052] The present application provides a resource scheduling method, device, electronic device and storage medium, the method comprising: obtaining resource trees corresponding to different resource devices respectively, the resource tree comprising at least one first-level node, different first-level nodes being used to characterize multiple GPU clusters with inter-card communication functions and different communication rates; in response to a training task posted by a user, obtaining the required number of GPU cards for the training task, and, based on multiple resource trees of different resource devices, obtaining the total number of idle GPU cards; determining a predicted rate based on the total number of idle GPU cards and the required number of GPU cards; determining a target first-level node of a target resource device based on the communication rate of the first-level nodes in the different resource devices and the predicted rate; and executing the training task based on multiple GPU cards under the target first-level node. The present application increases the perception of NVLINK and NVSWITCH resources with inter-card communication functions in the scheduler, and automatically selects GPU cards with efficient communication resources for training tasks based on the predicted rate, thereby effectively improving the efficiency of training and reasoning tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0054] Figure 1 It is a schematic diagram of the steps of a resource scheduling method proposed in an embodiment of the present application;

[0055] Figure 2 is an example diagram of a resource tree in a resource scheduling method proposed in an embodiment of the present application;

[0056] Figure 3 It is a schematic diagram of a process example of a resource scheduling method proposed in an embodiment of the present application;

[0057] Figure 4 It is a schematic diagram of a process of constructing a resource tree in a resource scheduling method proposed in an embodiment of the present application;

[0058] Figure 5 It is a flowchart of determining a predicted rate in a resource scheduling method proposed in an embodiment of the present application;

[0059] Figure 6 It is a schematic diagram of a process of determining a candidate node in a resource scheduling method proposed in an embodiment of the present application;

[0060] Figure 7 It is a structural diagram of a resource scheduling device proposed in an embodiment of the present application;

[0061] Figure 8 It is a schematic diagram of an electronic device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0063] In the drawings, the size of the constituent elements, the thickness of the layer or the area may be exaggerated for the sake of clarity. Therefore, any implementation of the present disclosure is not necessarily limited to the size shown in the drawings, and the shapes and sizes of the components in the drawings do not reflect the true proportions. In addition, the drawings schematically show ideal examples, and any implementation of the present disclosure is not limited to the shapes or values ​​shown in the drawings.

[0064] First, the proper nouns mentioned in the embodiments of the present application are explained:

[0065] AI: Artificial Intelligence (AI) is a technology that uses computer technology to realize human intelligence.

[0066] GPU: (Graphic Processing Unit, referred to as GPU) GPU is a concept relative to CPU. As graphics processing becomes more and more important in modern computers, a dedicated graphics core processor is needed. At the same time, in the fields of scientific computing and deep learning, GPU can also be used as a computing accelerator, which can greatly improve computing speed and efficiency.

[0067] NVLINK: is a direct interconnect between GPUs that scales multi-GPU input / output (I / O) within a server.

[0068] NVSWITCH: Multiple NVLINKs can be connected to achieve many-to-many GPU communication within a single node at the highest speed NVLINK can achieve.

[0069] Multi-card training: AI training tasks that use multiple GPU cards at the same time.

[0070] Multi-card inference: AI inference tasks that use multiple GPU cards at the same time.

[0071] Distributed training tasks: Start AI training tasks with multiple instances working together on multiple devices. Each instance uses the same specifications and number of GPU cards to keep the time spent on each training phase of each instance close.

[0072] NVLINK and NVSWITCH are high-speed card-to-card communication technologies launched by NVIDIA based on custom hardware for multiple GPU cards in the same server. Adding NVLINK or NVSWITCH in scenarios such as AI distributed training, multi-card training, and multi-card reasoning can greatly improve task training efficiency. The communication efficiency of NVLINK and NVSWITCH also varies greatly between generations.

[0073] In the related technology, the existing cluster resource scheduling technology for training tasks only manages GPU resources on the one hand, and lacks the perception of NVLINK and NVSWITCH resources. In a cluster where GPU devices without NVLINK or NVSWITCH and GPU devices with NVLINK or NVSWITCH are mixed, it is impossible to reasonably and automatically allocate matching resources, resulting in the inability to improve training efficiency; on the other hand, the existing scheduling technology also lacks the perception of the NVLINK and NVSWITCH communication rates within the cluster. In a cluster where NVLINK and NVSWITCH of different generations are mixed, it is impossible to reasonably and automatically select resources with uniform rates, resulting in resource waste; in addition, the existing cluster scheduling technology cannot support users to specify the expected communication rate between GPU cards according to their own task needs, and automatically allocate scheduling resources for users. Therefore, how to improve the efficiency of training and reasoning tasks based on cluster scheduling has become a problem that needs to be solved urgently in this field.

[0074] In view of this, an embodiment of the present application proposes a resource scheduling method. Figure 1 A schematic diagram of the steps of a resource scheduling method proposed in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method comprises the following steps:

[0075] S101, obtaining resource trees corresponding to different resource devices, wherein the resource tree includes at least one first-level node, and different first-level nodes are used to represent multiple GPU clusters with inter-card communication functions and different communication rates.

[0076] When the step S101 is specifically implemented, firstly, a resource topology matrix on different resource devices is obtained, and the resource topology matrix is ​​used to characterize the topology information of the connection between GPU cards. Optionally, the topology information of the connection between GPU cards can be obtained by calling the tool provided by NVIDIA, and the resource topology matrix can be constructed based on the topology information. For example, Table 1 shows an example data table of resource topology information in a resource topology matrix. As shown in Table 1, the information structure of the resource topology matrix is ​​as follows:

[0077] Table 1. Example data table of resource topology information in a resource topology matrix

[0078] GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 GPU0 X NV2 NV1 NV2 NV1 NODE NODE NODE GPU1 NV2 X NV1 NV1 NODE NV2 NODE NODE GPU2 NV1 NV1 X NV2 NODE NODE NV2 NODE GPU3 NV2 NV1 NV2 X NODE NODE NODE NV1 GPU4 NV1 NODE NODE NODE X NV2 NV1 NV2 GPU5 NODE NV2 NODE NODE NV2 X NV1 NV1 GPU6 NODE NODE NV2 NODE NV1 NV1 X NV2 GPU7 NODE NODE NODE NV1 NV2 NV1 NV2 X

[0079] Taking the data in Table 1 as an example, the GPU0-GPU7 in the first row and the GPU0-GPU7 in the first column of the resource topology matrix represent the GPU numbers, and the symbol at the intersection of the rows and columns represents the connection method between the two GPUs. For example, the NV2 in the second column of the first row represents that GPU0 and GPU1 are connected via NVLINK, 2 represents the number of channels, and each channel is 25GB / s, so the inter-card communication rate between GPU0 and GPU1 is 50GB / s; the NV1 in the third column of the second row represents that GPU1 and GPU2 are connected via NVLINK, 1 represents the number of channels, so the inter-card communication rate is 25GB. The sixth column of the first row represents that there is no NVLINK connection between GPU0 and GPU5, and the inter-card communication rate is between 5GB-15GB.

[0080] It should be noted that in the embodiment of the present application, NVSWITCH is also implemented by connecting multiple NVLINKs. Except for the topology and rate, other forms of expression are the same as NVLINK. Therefore, they are collectively referred to as NVLINK in the description of the embodiment of the present application. The rate of NVLINK increases linearly with the number of channels, with one channel at 25GB. The highest rate in the prior art is 18 channels, reaching 450GB. Other card-to-card communication methods are all lower than 25GB.

[0081] In the embodiment of the present application, after constructing the resource topology matrix, a resource tree corresponding to each resource device is constructed based on the resource topology matrices corresponding to different resource devices. Figure 4 A schematic diagram of a process of constructing a resource tree in a resource scheduling method proposed in an embodiment of the present application is shown. Figure 2 An example diagram of a resource tree in a resource scheduling method proposed in an embodiment of the present application is shown. Figure 2 and Figure 4 As shown, first build the root node (corresponding to Figure 4 The root node shown in ,) and create the initial node (corresponding to Figure 4 The second-layer node of "other 0" shown in the figure, wherein 0 is only convenient for subsequent matching and does not represent actual meaning; the second-layer node refers to the case where the root node is the first-layer node, and the "other 0" node of the initial node is the second-layer node); a third-layer node is established under the initial node, and the GPU card in the resource device is used as the third-layer node; based on the resource topology matrix, a plurality of first-level nodes corresponding to the resource topology information with a target label in the resource topology matrix are generated, the first-level nodes are set at the same layer as the initial node, and the target label is used to characterize that the GPU cluster has the inter-card communication function, for example, the target label can be "NV#"; the resource topology information in the resource topology matrix is ​​traversed according to a preset traversal order, and a plurality of second-level nodes under each first-level node are determined, and the second-level nodes are used to characterize the GPU cards contained in the first-level node; finally, the nodes are constructed into a tree structure according to the subordinate relationship between the nodes to obtain the resource tree.

[0082] Exemplarily, taking the resource devices corresponding to the resource topology matrix shown in Table 1 as an example, GPU0 to GPU7 are used as GPU nodes under the initial node "other 0", and the communication efficiency between GPU0 and GPU7 cards can meet the regular rate of 5GB / s-15GB / s; traverse the resource topology matrix to remove the NV# tags to deduplicate, and create a second-layer node of NVLINK (#*25) GB / s for each NV# tag. As shown in the example, NVLINK only appears NV1 and NV2. Under the root node, two second-layer nodes of NVLINK 25GB / s and NVLINK 50GB / s are established (corresponding to the first-level nodes).

[0083] In some optional embodiments, the preset traversal order refers to traversing the resource topology information in the resource topology matrix in the order of row N and column N+1, and each time the target tag is detected, a second-level node corresponding to the GPU card is established under the first-level node corresponding to the target tag, and the second-level node corresponds to the GPU card corresponding to the row head and column head of the resource topology information corresponding to the target tag in the resource topology matrix.

[0084] In some optional implementations, after determining the second-level node, based on the communication rates of different first-level nodes, determine the first-level node with the highest communication rate (corresponding to the Figure 4 The second-layer node in the first-layer node with the highest rate (NVLINK second-layer node) is the second-layer node (corresponding to Figure 4 For example, all second-tier nodes under the first-tier nodes of NVLINK 50GB / s are copied to the first-tier nodes of NVLINK 25GB / s. This means that the resources that can meet the requirements of 50GB / s inter-card communication can also meet the requirements of 25GB / s inter-card communication.

[0085] In some optional implementations, deduplication is performed on multiple second-level nodes under each first-level node. Specifically, the first-level nodes are taken out one by one, and the second-level nodes contained therein are merged under the first-level node. The principle of merging is that if there are duplicate GPU card resources in two second-level nodes, the two second-level nodes are merged, and the GPU cards corresponding to the two second-level nodes before the merger are placed in the merged second-level node, and the merged GPU card is deduplicated; the above operation is repeated until there are no identical GPU card resources between all second-level nodes under each first-level node.

[0086] S102: In response to a training task issued by a user, obtain the number of GPU cards required for the training task, and obtain the total number of idle GPU cards based on multiple resource trees of different resource devices.

[0087] S103: Determine a predicted rate based on the total number of idle GPU cards and the required number of GPU cards.

[0088] When implementing steps S102-S103, Figure 5 FIG. 4 shows a flow chart of determining a predicted rate in a resource scheduling method proposed in an embodiment of the present application. Figure 5As shown, first, in response to the training task posted by the user, the required number of GPU cards for the training task is obtained; then, based on the initial rate parameter, a first matching node is determined in the multiple resource trees, and the first matching node is a first-level node in each resource tree whose communication rate is greater than or equal to the initial rate parameter, wherein the initial rate parameter is a preset parameter, and exemplarily, the initial rate parameter can be 450GB / s; the sum of the number of second-level nodes in an idle state at the lower layer of each first matching node in all resource trees is obtained as the total number of idle GPU cards.

[0089] For example, Figure 5 For example, take 450GB / s as the initial rate parameter; match each resource tree under the resource manager with the rate parameter, and determine whether a first-level node (corresponding to Figure 5 ); if it is not found in the current resource tree, then loop through the above steps in the next resource tree; if a first-level node of a resource tree that meets the above conditions is found in the current resource tree, take out the second-level node with the most free GPU cards under the first-level node (corresponding to Figure 5 The number of idle GPU cards is added to the total number of idle GPU cards; the above steps are repeated to traverse all resource trees under the resource manager, and the total number of idle GPU cards is obtained by accumulation.

[0090] In the embodiment of the present application, when the total number of idle GPU cards is greater than or equal to the required number of GPU cards, the initial rate parameter is used as the predicted rate; when the total number of idle GPU cards is less than the required number of GPU cards, the preset rate variable is reduced based on the initial rate parameter to obtain the predicted rate. Exemplarily, the preset rate variable can be 25 GB / s.

[0091] In addition, when the total number of idle GPU cards is less than the required number of GPU cards, it indicates that the cluster resources of the resource device are insufficient, and sufficient GPU resources cannot be scheduled through the cluster resources to support the execution of the training task. The scheduling fails and the process ends.

[0092] In some optional embodiments, after determining the predicted rate, if the user has pre-set an expected rate, the expected rate is compared with the predicted rate; if the predicted rate is greater than or equal to the expected rate, a second matching node is determined in the multiple resource trees based on the predicted rate, and the second matching node is a first-level node in each resource tree whose communication rate is greater than or equal to the predicted rate; the sum of the number of second-level nodes in the lower layer of each second matching node in all resource trees that are idle is obtained as the updated total number of idle GPU cards; if the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards, the predicted rate is maintained; if the updated total number of idle GPU cards is less than the required number of GPU cards, the predicted rate is reduced by a preset rate variable.

[0093] In some optional embodiments, after determining the predicted rate, if the user has not set an expected rate, the predicted rate is compared with 0; if the predicted rate is greater than or equal to 0, a second matching node is determined in the multiple resource trees based on the predicted rate, and the second matching node is a first-level node in each resource tree whose communication rate is greater than or equal to the predicted rate; the sum of the number of second-level nodes in the lower layer of each second matching node in all resource trees that are idle is obtained as the updated total number of idle GPU cards; when the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards, the predicted rate is maintained; when the updated total number of idle GPU cards is less than the required number of GPU cards, the predicted rate is reduced by a preset rate variable.

[0094] S104: Determine a target first-level node of a target resource device based on the communication rates of the first-level nodes in the different resource devices and the predicted rate.

[0095] When step S104 is specifically implemented, the target first-level node of the target resource device is determined based on the communication rate of the first-level nodes in the different resource devices and the predicted rate. Specifically, Figure 6 A schematic diagram of a process for determining candidate nodes in a resource scheduling method proposed in an embodiment of the present application is shown. Figure 6As shown, first, based on the first-level nodes of the multiple resource trees, first-level nodes with a communication rate greater than or equal to the preset rate are obtained as nodes to be confirmed; the sum of the number of second-level nodes in an idle state in the lower layer of each of the nodes to be confirmed is obtained as the total number of idle GPU cards of the nodes to be confirmed; when the total number of idle GPU cards of the nodes to be confirmed is greater than or equal to the required number of GPU cards, the nodes to be confirmed are used as candidate nodes; the candidate nodes are screened to obtain the target first-level nodes.

[0096] In some optional embodiments, the candidate nodes are screened as follows: the difference between the communication rate of each candidate node and the predicted rate is obtained; based on the difference between the communication rate of each candidate node and the predicted rate, a resource score of each candidate node is generated, wherein the greater the difference between the communication rate of the candidate node and the predicted rate, the higher the resource score of the candidate node; the candidate node with the highest resource score is used as the target first-level node.

[0097] S105. Execute the training task based on the multiple GPU cards in the lower layer of the target first-level node.

[0098] When step S105 is specifically implemented, the resource allocator selects the resource tree of the target first-level node device with the highest score for the instance to be created in the user's training task, takes out GPU cards of the same number as the required number of GPU cards of the training task from the second-level node under the target first-level node, writes the GPU card information into the task, and marks the GPU cards under the target first-level node of the target resource device as occupied; repeats the process of step S104 until all instances to be created in the user's training task are traversed and taken out; starts the instance on the corresponding device according to all instances to be created, and allocates the corresponding numbered GPU card resources to each instance to be created for training; after the training task is completed, the scheduler destroys the instance, and marks the GPU card information in the resource tree corresponding to the device GPU card occupied by the instance as idle in the resource manager.

[0099] In order to enable those skilled in the art to better understand the resource scheduling method provided in this application, this application describes the resource scheduling method provided in this application in detail through the following specific embodiments:

[0100] Figure 3 A schematic diagram showing a process example of a resource scheduling method proposed in an embodiment of the present application is shown as follows: Figure 3As shown in the figure, the GPU cluster includes three mixed resource devices: two 8-GPU card devices, each device has NVLINK fully connected between cards, 8 channels, and a communication rate of 200GB / s between cards; one 8-GPU card device, the device has NVLINK fully connected between cards, 12 channels, and a communication rate of 300GB / s between cards. Users need 2 instances to create distributed training tasks, and each instance requires 8 cards.

[0101] First, the collector runs on each resource device and calls the tool provided by NVIDIA to obtain the resource topology matrix of the connection between GPU cards. Among them, one is an 8-card device NVLINK fully connected channel 12, so the resource topology matrix is ​​all NV12; two are 8-card devices NVLINK fully connected channel 8, so the resource topology matrix is ​​all NV8.

[0102] Subsequently, the topology parser obtains three resource trees through the parsing process according to the information reported by the collector; corresponding to 8 cards, the resource tree of the NV12 device has two second-layer nodes, the initial node "other 0" and the first-level node "NVLINK 300GB / s", and each of the two nodes has a third-layer node (second-level node) with 8 cards, indicating that the 8 cards can meet the inter-card communication rate of 0-300GB; corresponding to the 2 resource trees of the 8-card NV8 device, each has two second-layer nodes, the initial node "other 0" and the first-level node "NVLINK 200GB / s", and each of the two nodes has a third-layer node (second-level node) with 8 cards, indicating that the 8 cards can meet the inter-card communication rate of 0-200GB.

[0103] In response to 2 instances of training tasks issued by the user, each instance requires 8 cards. The total number of GPU cards required for all instances to be built is added up, and a total of 16 GPU cards are required, which is the required number of GPU cards; the total number of idle GPU cards that meet the communication rate of 300GB / s through the rate prediction process is only 8 cards, which is less than the required number of GPU cards of 16 cards; the total number of idle GPU cards that meet the communication rate of 200GB / s is 24 cards, which is greater than the required number of GPU cards of 16 cards, so the predicted rate is determined to be 200GB / s.

[0104] The resource preselector first takes out the first instance to be created in the training task, traverses all resource devices, and matches suitable resource devices. Since the three devices all meet the inter-card communication rate of 200GB / s, the three devices in the cluster enter the candidate as candidate devices with candidate nodes. The resource scorer scores the candidate nodes of the candidate device of the instance to be built, wherein the closer the inter-card communication rate of the GPU matched by the candidate node of the candidate device is to the predicted rate, the higher the score, that is, it is determined that the score of the two devices with a communication rate of 200GB / s will be higher than the communication rate of 300GB / s. The resource allocator selects the resource tree of the device with the highest score for the instance to be created in the user task, that is, one of the two devices with an inter-card communication rate of 200GB / s, and finds the first-level node under the resource tree of the corresponding device with a speed not less than and closest to 200GB / s. Find 8 cards of idle GPU under the node. Take out 8 GPU cards from the node, write the GPU information into the task to be built, and set the corresponding GPU under the resource tree node to be occupied.

[0105] Repeat the above process to select resource devices for the second instance to be created of the training task. Similarly, the remaining device with a communication rate of 200GB / s will be scored higher than the device with a communication rate of 300GB / s and will be selected first. The task launcher starts the corresponding instance on the corresponding two GPU cards with a communication rate of 200GB / s based on all the instances to be created for the user task, and allocates the GPU card resources with the corresponding number to the instance to execute the training task. After the training task is completed, the scheduler destroys the task instance and marks the GPU information under the resource tree corresponding to the GPU card label of the device occupied by the instance as idle in the resource manager.

[0106] The present application provides a resource scheduling method, device, electronic device and storage medium, the method comprising: obtaining resource trees corresponding to different resource devices respectively, the resource tree comprising at least one first-level node, different first-level nodes being used to characterize multiple GPU clusters with inter-card communication functions and different communication rates; in response to a training task posted by a user, obtaining the required number of GPU cards for the training task, and, based on multiple resource trees of different resource devices, obtaining the total number of idle GPU cards; determining a predicted rate based on the total number of idle GPU cards and the required number of GPU cards; determining a target first-level node of a target resource device based on the communication rate of the first-level nodes in the different resource devices and the predicted rate; and executing the training task based on multiple GPU cards under the target first-level node. The present application increases the perception of NVLINK and NVSWITCH resources with inter-card communication functions in the scheduler, and automatically selects GPU cards with efficient communication resources for training tasks based on the predicted rate, thereby effectively improving the efficiency of training and reasoning tasks.

[0107] Based on the same inventive concept, an embodiment of the present application proposes an Yizong resource scheduling system, and the resource scheduling method is applied to the resource scheduling system. The system includes: a collector and a scheduler. The collector is used to run on a resource device and report NVLINK information on the collection device; the scheduler is responsible for completing resource management, resource selection, and resource scheduling.

[0108] The scheduler includes five units: topology parser, resource manager, rate predictor, resource preselector, resource scorer, and resource allocator. The functions of each module are as follows: the resource manager is used to store and manage the resource tree, which records the NVLINK rate, the corresponding GPU resource occupancy and idleness in a tree structure; the rate predictor is used to quickly compare the total number of resources and the total number of cluster resources of the task that the user expects to start, and predict the reasonable communication rate and record it in the user task; the resource preselector is used to select all devices and GPU resources matching the task according to the task information, resource application information and predicted rate information delegated by the user; the resource scorer is used to score the resource devices from all matching resource devices, and the closer the GPU communication rate is to the predicted rate, the higher the score; the resource allocator is used to select the resource device with the highest score for the user task, and mark the corresponding resource as occupied under the corresponding resource tree of the resource manager; the task launcher is used to start an instance on the corresponding resource device for the user task and allocate the corresponding GPU card resources.

[0109] Based on the same inventive concept, the embodiment of the present application discloses a resource scheduling device. Figure 7 A schematic diagram of the structure of a resource scheduling device proposed in an embodiment of the present application is shown. Figure 7 As shown, the device comprises:

[0110] A resource tree acquisition module, used to acquire resource trees corresponding to different resource devices, wherein the resource tree includes at least one first-level node, and different first-level nodes are used to represent multiple GPU clusters with inter-card communication functions and different communication rates;

[0111] The rate determination data module is used to obtain the number of GPU cards required for the training task in response to the training task issued by the user, and to obtain the total number of idle GPU cards based on multiple resource trees of different resource devices;

[0112] A prediction rate module, used to determine a prediction rate based on the total number of idle GPU cards and the required number of GPU cards;

[0113] A target node determination module, configured to determine a target first-level node of a target resource device based on the communication rates of the first-level nodes in the different resource devices and the predicted rate;

[0114] The task execution module is used to execute the training task based on the multiple GPU cards under the target first-level node.

[0115] In an optional implementation manner, the resource tree acquisition module includes:

[0116] The resource topology matrix submodule is used to obtain the resource topology matrix on different resource devices;

[0117] The root node construction submodule is used to construct the root node and create the initial node under the root node;

[0118] A first-level node construction submodule is used to generate, based on the resource topology matrix, a plurality of first-level nodes corresponding to the resource topology information with a target label in the resource topology matrix, wherein the first-level nodes are arranged at the same layer as the initial nodes, and the target label is used to indicate that the GPU cluster has the inter-card communication function;

[0119] The second-level node construction submodule is used to traverse the resource topology information in the resource topology matrix according to a preset traversal order, and determine multiple second-level nodes under each first-level node, where the second-level nodes are used to represent the GPU cards included in the first-level nodes.

[0120] In an optional implementation, in an optional implementation, the rate determination data module includes:

[0121] A first matching node submodule, used to determine a first matching node in the plurality of resource trees based on an initial rate parameter, the first matching node being a first-level node in each resource tree with a communication rate greater than or equal to the initial rate parameter;

[0122] An idle GPU card determination submodule is used to obtain the sum of the number of idle second-level nodes in the lower layer of each of the first matching nodes in all resource trees as the total number of idle GPU cards;

[0123] In an optional implementation, the prediction rate module includes:

[0124] A first prediction rate determination submodule, configured to use the initial rate parameter as the prediction rate when the total number of idle GPU cards is greater than or equal to the required number of GPU cards;

[0125] The second predicted rate determination submodule is used to reduce a preset rate variable based on the initial rate parameter to obtain the predicted rate when the total number of idle GPU cards is less than the required number of GPU cards.

[0126] In an optional embodiment, the device further comprises:

[0127] A first rate comparison module, configured to compare the expected rate with the predicted rate when the user presets an expected rate;

[0128] A second matching node module, configured to determine a second matching node in the plurality of resource trees based on the predicted rate when the predicted rate is greater than or equal to the expected rate, the second matching node being a first-level node in each resource tree whose communication rate is greater than or equal to the predicted rate;

[0129] An idle GPU card updating module is used to obtain the sum of the number of idle second-level nodes under each of the second matching nodes in all resource trees as the updated total number of idle GPU cards;

[0130] A first rate updating module, configured to maintain the predicted rate when the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards;

[0131] The second rate updating module is used to reduce the predicted rate by a preset rate variable when the updated total number of idle GPU cards is less than the required number of GPU cards.

[0132] In an optional embodiment, the device further comprises:

[0133] A second rate comparison module, configured to compare the predicted rate with 0 when the user does not set an expected rate;

[0134] A second matching node module, configured to determine a second matching node in the plurality of resource trees based on the predicted rate if the predicted rate is greater than or equal to 0, wherein the second matching node is a first-level node in each resource tree whose communication rate is greater than or equal to the predicted rate;

[0135] An idle GPU card updating module is used to obtain the sum of the number of idle second-level nodes under each of the second matching nodes in all resource trees as the updated total number of idle GPU cards;

[0136] A third rate updating module, configured to maintain the predicted rate when the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards;

[0137] The fourth rate updating module is used to reduce the predicted rate by a preset rate variable when the updated total number of idle GPU cards is less than the required number of GPU cards.

[0138] In an optional implementation, the target node determination module includes:

[0139] The node to be confirmed submodule is used to obtain, based on the first-level nodes of the plurality of resource trees, the first-level nodes whose communication rate is greater than or equal to the preset rate as the nodes to be confirmed;

[0140] The idle GPU card submodule of the node to be confirmed is used to obtain the sum of the number of second-level nodes in the lower layer of each node to be confirmed that are in an idle state as the total number of idle GPU cards of the node to be confirmed;

[0141] A candidate node submodule, used to use the node to be confirmed as a candidate node when the total number of idle GPU cards of the node to be confirmed is greater than or equal to the required number of GPU cards;

[0142] The target node determination submodule is used to screen the candidate nodes to obtain the target first-level node.

[0143] In an optional implementation, the target node determination submodule includes:

[0144] A rate difference unit, used to obtain the difference between the communication rate of each candidate node and the predicted rate;

[0145] A scoring unit, configured to generate a resource score for each candidate node based on a difference between the communication rate of each candidate node and the predicted rate, wherein the greater the difference between the communication rate of the candidate node and the predicted rate, the higher the resource score of the candidate node;

[0146] The target node determination unit is used to select the candidate node with the highest resource score as the target first-level node.

[0147] Based on the same inventive concept, the present application embodiment discloses an electronic device, Figure 8 A schematic diagram of an electronic device proposed in an embodiment of the present application is shown, such as Figure 8 As shown, the electronic device 100 includes: a memory 110 and a processor 120, the memory 110 and the processor 120 are connected to each other through a bus communication, the memory 110 stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, the at least one program, the code set or instruction set can be loaded and run on the processor 120 to implement the steps in the resource scheduling method disclosed in the embodiment of the present application.

[0148] Based on the same inventive concept, an embodiment of the present application discloses a computer-readable storage medium, on which is stored at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement the steps in the resource scheduling method disclosed in the embodiment of the present application.

[0149] Based on the same inventive concept, an embodiment of the present application discloses a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps in the resource scheduling method disclosed in the embodiment of the present application.

[0150] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0151] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, apparatuses, electronic devices, and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0152] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0153] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0154] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0155] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "including one..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0156] The above is a detailed introduction to a resource scheduling method, device, electronic device and storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A resource scheduling method, characterized in that: The method comprises: Obtain resource trees corresponding to different resource devices, respectively, wherein the resource tree includes at least one first-level node, and different first-level nodes are used to represent multiple GPU clusters with inter-card communication functions and different communication rates; In response to a training task posted by a user, obtain the number of GPU cards required for the training task, and obtain the total number of idle GPU cards based on multiple resource trees of different resource devices; Determine a predicted rate based on the total number of idle GPU cards and the required number of GPU cards; Determining a target first-level node of a target resource device based on the communication rates of the first-level nodes in the different resource devices and the predicted rate; The training task is executed based on multiple GPU cards in the lower layer of the target first-level node.

2. The resource scheduling method according to claim 1, characterized in that: The obtaining of resource trees corresponding to different resource devices respectively includes: Get the resource topology matrix on different resource devices; Build the root node and create the initial node under the root node; Based on the resource topology matrix, generating a plurality of first-level nodes corresponding to the resource topology information with a target label in the resource topology matrix, wherein the first-level nodes are arranged at the same layer as the initial nodes, and the target label is used to indicate that the GPU cluster has the inter-card communication function; The resource topology information in the resource topology matrix is ​​traversed in a preset traversal order to determine a plurality of second-level nodes under each first-level node, wherein the second-level nodes are used to represent the GPU cards included in the first-level nodes.

3. The resource scheduling method according to claim 1, characterized in that: The method of obtaining the total number of idle GPU cards based on multiple resource trees of different resource devices includes: Based on the initial rate parameter, determining a first matching node in the plurality of resource trees, the first matching node being a first-level node in each resource tree with a communication rate greater than or equal to the initial rate parameter; Obtaining the sum of the number of idle second-level nodes in the lower layer of each of the first matching nodes in all resource trees as the total number of idle GPU cards; The determining the predicted rate based on the total number of idle GPU cards and the required number of GPU cards includes: When the total number of idle GPU cards is greater than or equal to the required number of GPU cards, the initial rate parameter is used as the predicted rate; In the case that the total number of idle GPU cards is less than the required number of GPU cards, the preset rate variable is reduced based on the initial rate parameter to obtain the predicted rate.

4. The resource scheduling method according to claim 1, characterized in that: After determining the predicted rate, the method further comprises: In the case where the user presets an expected rate, comparing the expected rate with the predicted rate; In the case where the predicted rate is greater than or equal to the expected rate, determining a second matching node in the plurality of resource trees based on the predicted rate, the second matching node being a first-level node in each resource tree whose communication rate is greater than or equal to the predicted rate; Obtaining the sum of the number of second-level nodes in an idle state in the lower layers of each of the second matching nodes in all resource trees as the updated total number of idle GPU cards; When the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards, maintaining the predicted rate; When the updated total number of idle GPU cards is less than the required number of GPU cards, the predicted rate is reduced by a preset rate variable.

5. The resource scheduling method according to claim 1, characterized in that: After determining the predicted rate, the method further comprises: If the user does not set an expected rate, the predicted rate is compared with 0; If the predicted rate is greater than or equal to 0, determining a second matching node in the plurality of resource trees based on the predicted rate, the second matching node being a first-level node in each resource tree with a communication rate greater than or equal to the predicted rate; Obtaining the sum of the number of second-level nodes in an idle state in the lower layers of each of the second matching nodes in all resource trees as the updated total number of idle GPU cards; When the updated total number of idle GPU cards is greater than or equal to the required number of GPU cards, maintaining the predicted rate; When the updated total number of idle GPU cards is less than the required number of GPU cards, the predicted rate is reduced by a preset rate variable.

6. The resource scheduling method according to claim 1, characterized in that: The determining of a target first-level node of a target resource device based on the communication rate of the first-level node in the different resource devices and the predicted rate includes: Based on the first-level nodes of the plurality of resource trees, obtaining first-level nodes whose communication rate is greater than or equal to the preset rate as nodes to be confirmed; Obtain the sum of the number of idle second-level nodes in the lower layer of each node to be confirmed as the total number of idle GPU cards of the node to be confirmed; When the total number of idle GPU cards of the node to be confirmed is greater than or equal to the required number of GPU cards, the node to be confirmed is used as a candidate node; The candidate nodes are screened to obtain the target first-level node.

7. The resource scheduling method according to claim 6, characterized in that: The step of obtaining the target first-level node based on screening the candidate nodes includes: Obtaining a difference between the communication rate of each candidate node and the predicted rate; Generate a resource score for each candidate node based on the difference between the communication rate of each candidate node and the predicted rate, wherein the greater the difference between the communication rate of the candidate node and the predicted rate, the higher the resource score of the candidate node; The candidate node with the highest resource score is used as the target first-level node.

8. A resource scheduling device, characterized in that: The device comprises: A resource tree acquisition module, used to acquire resource trees corresponding to different resource devices, wherein the resource tree includes at least one first-level node, and different first-level nodes are used to represent multiple GPU clusters with inter-card communication functions and different communication rates; The rate determination data module is used to obtain the number of GPU cards required for the training task in response to the training task issued by the user, and to obtain the total number of idle GPU cards based on multiple resource trees of different resource devices; A prediction rate module, used to determine a prediction rate based on the total number of idle GPU cards and the required number of GPU cards; A target node determination module, configured to determine a target first-level node of a target resource device based on the communication rates of the first-level nodes in the different resource devices and the predicted rate; The task execution module is used to execute the training task based on the multiple GPU cards under the target first-level node.

9. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps in the resource scheduling method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the steps in the resource scheduling method described in any one of claims 1 to 7 are implemented.

Images