A method, apparatus, device and medium for scheduling a GPU task

Abstract

A GPU task deployment method, device, equipment and medium are disclosed. When a global scheduler receives a GPU task, the GPU task corresponding GPU requirement quantity is obtained, and the available resource information corresponding to each cluster node is obtained from an API server. Further, the available resource information is matched based on the GPU requirement quantity to screen at least one candidate node from multiple cluster nodes and generate a GPU available list corresponding thereto. If the number of candidate nodes is not less than two, the GPU available list is sent to a GPU monitoring server to obtain multiple target connection matrices. According to the comparison result of the GPU combination in the ring communication link in each target connection matrix, a decision node is selected from the candidate nodes, and the GPU task is allocated to the decision node to execute the GPU task. Thus, the optimal allocation of nodes is realized according to the requirements of the GPU task, and the execution efficiency of the GPU task is effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of resource allocation, and particularly relates to a GPU task deployment method and device, equipment and a medium. BACKGROUND

[0002] With the development of society, in order to further solve the labor cost and improve the efficiency of logistics and transportation, more and more automatic driving vehicles are used in more and more fields.

[0003] For the automatic driving vehicle, its driving safety as one of the primary considerations, the functions of the automatic driving vehicle need to be continuously tested to improve its driving safety as much as possible, and the automatic driving vehicle is usually realized by multiple road tests. Since the cost of road test is high, and some functions cannot be detected by single road test, the simulation test of the automatic driving vehicle can be realized by the virtualization device in the Kubernetes cluster.

[0004] With the increasing size of the deep learning model in the automatic driving vehicle, multiple GPUs are often needed for collaborative training of the model. However, the existing Kubernetes cluster usually uniformly marks the GPU resources as GPU resources when registering the GPU resources, and it is difficult to optimize the allocation of GPU resources according to various tasks, and the utilization rate of GPU resources is low. SUMMARY

[0005] The present application provides a GPU task deployment method, device, equipment and medium, which solves the technical problem that the existing Kubernetes cluster usually uniformly marks the GPU resources as GPU resources when registering the GPU resources, and it is difficult to optimize the allocation of GPU resources according to various tasks, and the utilization rate of GPU resources is low.

[0006] The first aspect of the present application provides a GPU task deployment method applied to a global scheduler in a resource cluster, the global scheduler, an API server and a GPU monitoring server are communicatively connected with each other, the resource cluster includes a plurality of cluster nodes, and the method comprises:

[0007] When receiving a GPU task, the number of GPU requirements in the GPU task is obtained, and the available resource information corresponding to each cluster node is obtained from the API server;

[0008] According to the number of GPU requirements and the available resource information, at least one candidate node is selected from a plurality of cluster nodes, and a corresponding GPU available list is generated;

[0009] when the number of the candidate nodes is not less than two, sending the GPU available list to the GPU monitoring server;

[0010] when receiving a plurality of target connection matrices returned by the GPU monitoring server in response to the GPU available list, selecting a decision node from the plurality of candidate nodes according to each of the target connection matrices;

[0011] allocating the GPU task to the decision node, and executing the GPU task through the decision node.

[0012] Optionally, the cluster node comprises a resource plug-in and a plurality of GPUs to be determined; the method further comprises:

[0013] acquiring, by the resource plug-in, a unique device identifier corresponding to each available GPU from the plurality of GPUs to be determined;

[0014] constructing available resource information by the resource plug-in using the unique device identifier and a node identifier corresponding to the cluster node, and uploading the available resource information to the API server;

[0015] iterating through the available GPUs by the resource plug-in, constructing a connection matrix, and uploading the connection matrix to the GPU monitoring server.

[0016] Optionally, the step of iterating through the available GPUs by the resource plug-in, constructing a connection matrix, and uploading the connection matrix to the GPU monitoring server comprises:

[0017] iterating through the available GPUs by the resource plug-in, determining a plurality of initial link information between each two of the available GPUs;

[0018] selecting initial link information with the highest link level as target link information of the available GPUs;

[0019] constructing an initial connection matrix using all the target link information, and uploading the initial connection matrix to the GPU monitoring server.

[0020] Optionally, the method further comprises:

[0021] sending heartbeat update information to the GPU monitoring server at a regular time by the resource plug-in;

[0022] determining updated resource information corresponding to each of the available GPUs by the resource plug-in by regularly scanning each of the available GPUs;

[0023] judging whether the updated resource information exists in the target connection matrix;

[0024] If not, the available resource information in the API server is updated by the updated resource information, and state update information is sent to the GPU monitoring server;

[0025] If yes, the step of scanning each available GPU by the resource plugin to determine the updated resource information corresponding to each available GPU is executed.

[0026] Optionally, the GPU monitoring server comprises a plurality of listening ports and a server body; the method further comprises:

[0027] The initial connection matrix, the state update information and the heartbeat update information uploaded by the resource plugin are received through the listening port;

[0028] The GPU matrix table is constructed by the server body using the initial connection matrix, the state update information, the heartbeat update information and the node identifier of the cluster node to which the resource plugin belongs;

[0029] Each GPU matrix table is polled by the server body, and the GPU matrix table that has not been updated by the heartbeat update information within a preset time period is removed;

[0030] When the GPU available list is received, the corresponding target connection matrix is queried according to the GPU available list by the server body, and returned to the global scheduler.

[0031] Optionally, the step of selecting at least one candidate node from a plurality of cluster nodes according to the GPU demand quantity and each available resource information and generating a corresponding GPU available list comprises:

[0032] The GPU available quantity of the cluster node to which each available resource information belongs is counted;

[0033] The GPU available quantity and the GPU demand quantity are compared, and the cluster node whose GPU available quantity is greater than or equal to the GPU demand quantity is selected as a candidate node;

[0034] The GPU available list is generated by using the available resource information corresponding to each candidate node.

[0035] Optionally, the step of selecting a decision node from a plurality of candidate nodes according to each target connection matrix when a plurality of target connection matrices returned by the GPU monitoring server in response to the GPU available list are received comprises:

[0036] When receiving the plurality of target connection matrices returned by the GPU monitoring server in response to the GPU available list, each of the target connection matrices is traversed to construct a plurality of initial GPU combinations respectively;

[0037] A plurality of ring communication links corresponding to each of the initial GPU combinations are constructed respectively;

[0038] Each of the ring communication links of the initial GPU combinations is compared, and a ring communication link corresponding to a minimum communication bottleneck speed is selected as a target communication link;

[0039] Each of the target communication links is compared, and an intermediate GPU combination to which a maximum value of the minimum communication bottleneck speed belongs is selected according to a comparison result;

[0040] If the number of the intermediate GPU combinations is single, a candidate node to which the intermediate GPU combination belongs is determined as a decision node.

[0041] Optionally, the method further comprises:

[0042] If the number of the intermediate GPU combinations exceeds one, a number of GPUs in each of the intermediate GPU combinations whose connection speed is equal to or greater than a preset high-speed level is obtained respectively;

[0043] An intermediate GPU combination corresponding to a minimum value of the number of the GPUs is selected, and a candidate node to which the intermediate GPU combination belongs is determined as a decision node.

[0044] Optionally, the method further comprises:

[0045] When the number of the candidate nodes is single, the candidate node is determined as the decision node.

[0046] The second aspect of the application provides a GPU task deployment device, which is applied to a global scheduler in a resource cluster, the global scheduler, an API server and a GPU monitoring server are in communication connection with each other, the resource cluster comprises a plurality of cluster nodes, and the device comprises:

[0047] An information obtaining module is configured to obtain a GPU demand quantity in a GPU task and obtain available resource information corresponding to each of the cluster nodes from the API server when the GPU task is received;

[0048] A node screening module is configured to screen at least one candidate node from the plurality of cluster nodes and generate a GPU available list according to the GPU demand quantity and the available resource information corresponding to each of the cluster nodes;

[0049] A list sending module is configured to send the GPU available list to the GPU monitoring server when the number of the candidate nodes is not less than two.

[0050] a decision node selection module, configured to select a decision node from the plurality of candidate nodes according to each of the plurality of target connection matrices when the GPU monitoring server returns the plurality of target connection matrices in response to the GPU available list;

[0051] a GPU task execution module, configured to allocate the GPU task to the decision node and execute the GPU task through the decision node.

[0052] The third aspect of the present application provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the GPU task deployment method according to any one of the first aspect of the present application.

[0053] The fourth aspect of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the GPU task deployment method according to any one of the first aspect of the present application.

[0054] From the above technical solutions, the present application has the following advantages:

[0055] When the global scheduler receives a GPU task, the GPU task corresponding GPU requirement quantity is obtained, and the available resource information corresponding to each cluster node is obtained from the API server. Further, the GPU requirement quantity is matched with the available resource information to screen at least one candidate node from the plurality of cluster nodes, and a GPU available list corresponding to the candidate node is generated. If the number of candidate nodes is not less than two, the GPU available list is sent to the GPU monitoring server to obtain a plurality of target connection matrices corresponding thereto. According to the comparison result of the GPU combination under the annular communication link constructed according to each target connection matrix, a single decision node is selected from the candidate nodes. Finally, the GPU task is allocated to the decision node, and the GPU task is executed through the decision node. Thus, the optimal allocation of nodes is realized according to the requirements of the GPU task, and the execution efficiency of the GPU task is effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0056] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0057] Figure 1 A step flow chart of a GPU task deployment method provided by the first embodiment of the present application;

[0058] Figure 2 A step flow chart of a GPU task deployment method provided for the second embodiment of the present application is provided.

[0059] Figure 3 A structural block diagram of a GPU task deployment device provided for the third embodiment of the present application is provided. DETAILED DESCRIPTION

[0060] The embodiments of the present application provide a GPU task deployment method, device, equipment and medium, and are used for solving the technical problem that the existing Kubernetes cluster is difficult to perform optimal allocation of GPU resources according to various tasks when registering GPU resources, and the utilization rate of GPU resources is low.

[0061] In order to make the technical scheme of the present application clearer and easier to understand, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the embodiments described below are only some of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of the present application.

[0062] Please refer to Figure 1 , Figure 1 A step flow chart of a GPU task deployment method provided for the first embodiment of the present application is provided.

[0063] The GPU task deployment method provided by the present application is applied to a global scheduler in a resource cluster, and the global scheduler, an API server and a GPU monitoring server are in communication connection with each other. The resource cluster includes a plurality of cluster nodes, and the method includes the following steps.

[0064] In step 101, when a GPU task is received, the number of GPU requirements in the GPU task is obtained, and the available resource information corresponding to each cluster node is obtained from the API server.

[0065] The global scheduler refers to a scheduler triggered in response to each GPU task, and is used to select a corresponding decision node to execute the GPU task according to the task requirements of the GPU task.

[0066] The API server refers to an interface server providing functions such as adding, deleting, modifying and querying for various resource objects, and is a center for data interaction and communication between various cluster nodes in the resource cluster, for example, a Kubernetes API server.

[0067] The GPU monitoring server refers to an independently running application program for collecting available CPU resources of each cluster node and link query function of GPU resources.

[0068] The GPU task refers to a task of calling a graphic processing unit (GPU) to perform corresponding graphic or image calculation in response to a rendering command issued by a CPU.

[0069] In the embodiment of the present application, when the global scheduler receives a GPU task input by a user, the GPU demand quantity carried in the GPU task is acquired, and the available resource information corresponding to each cluster node is acquired from the API server.

[0070] It should be noted that the GPU demand quantity refers to the resource demand quantity required for calling the GPU to complete the GPU task. The available resource information refers to the identifier corresponding to each GPU in the cluster node, for example, GPU / UUID-node ID.

[0071] Step 102, filtering at least one candidate node from the plurality of cluster nodes according to the GPU demand quantity and the available resource information, and generating a corresponding GPU available list;

[0072] After acquiring the GPU demand quantity and the available resource information, the available GPU quantity in each available resource information is counted, and by comparing the GPU demand quantity and the available GPU quantity, the candidate node whose available GPU quantity is greater than or equal to the GPU demand quantity is determined, and the GPU available list is generated by using the available resource information corresponding to the candidate node.

[0073] It should be noted that when only a single cluster node meets the GPU demand quantity, it is used as a candidate node, at this time, there is only a single candidate node in the GPU available list, which can be directly used as a decision node and the GPU task is allocated, and the GPU task is executed through the decision node.

[0074] Step 103, when the number of candidate nodes is not less than two, the GPU available list is sent to the GPU monitoring server;

[0075] In the embodiment of the present application, when the number of candidate nodes is not less than two, it indicates that there are multiple candidate nodes at this time, and in order to further select the node with the optimal connection, the GPU available list can be sent to the GPU monitoring server to acquire the connection matrix of the available GPU on each candidate node.

[0076] Step 104, when receiving a plurality of target connection matrices returned by the GPU monitoring server in response to the GPU available list, a decision node is selected from the plurality of candidate nodes according to the target connection matrices;

[0077] When receiving the multiple connection matrices returned by the GPU monitoring server in response to the GPU available list, the optimal connection combination of the available GPUs is selected from each connection matrix, and the optimal connection combinations of each connection matrix are further compared to select a decision node from the multiple candidate nodes.

[0078] In step 105, the GPU task is allocated to the decision node, and the GPU task is executed through the decision node.

[0079] After the decision node is selected, the GPU task can be allocated to the decision node, and the GPU task is executed through the decision node to realize the optimal allocation of the GPU task, effectively improve the execution efficiency of the task, and reduce the IO waiting time between GPUs.

[0080] In the embodiment of the application, when the global scheduler receives a GPU task, the number of GPU requirements corresponding to the GPU task is obtained, and the available resource information corresponding to each cluster node is obtained from the API server. Further, the available resource information is matched based on the number of GPU requirements to filter at least one candidate node from the multiple cluster nodes, and a GPU available list corresponding to the candidate node is generated. If the number of candidate nodes is not less than two, the GPU available list is sent to the GPU monitoring server to obtain a plurality of target connection matrices corresponding thereto. According to the comparison result of the GPU combination constructed under the ring communication link of each target connection matrix, a single decision node is selected from the candidate nodes. Finally, the GPU task is allocated to the decision node, and the GPU task is executed through the decision node. Thus, the optimal allocation of the node is realized according to the requirements of the GPU task, and the execution efficiency of the GPU task is effectively improved.

[0081] Please refer to Figure 2 , Figure 2 A step flowchart of a GPU task allocation method provided by the second embodiment of the application.

[0082] The GPU task allocation method provided by the application is applied to a global scheduler in a resource cluster. The global scheduler, the API server and the GPU monitoring server are in communication connection with each other. The resource cluster includes multiple cluster nodes. The method includes the following steps:

[0083] In step 201, when receiving a GPU task, the number of GPU requirements in the GPU task is obtained, and the available resource information corresponding to each cluster node is obtained from the API server.

[0084] In the embodiment of the application, the specific implementation process of step 201 is similar to that of step 101, and will not be repeated here.

[0085] In an example of the present application, the cluster node includes a resource plug-in and a plurality of GPUs to be determined; the method further includes steps S11-S13:

[0086] S11, acquiring, by the resource plug-in, a unique device identifier corresponding to each available GPU from the plurality of GPUs to be determined;

[0087] S12, constructing, by the resource plug-in, available resource information using the unique device identifier and a node identifier corresponding to the cluster node, and uploading the available resource information to an API server;

[0088] In an embodiment of the present application, each cluster node includes a resource plug-in and a plurality of GPUs to be determined. Through the GPU driver in the resource plug-in, all available GPUs on each cluster node are acquired, and a unique device identifier corresponding to each available GPU is acquired. Through the resource plug-in, available resource information is constructed using the unique device identifier and a node identifier corresponding to the cluster node to which the unique device identifier belongs, and the available resource information is uploaded to an API server.

[0089] For example, through the driver of an Nvidia GPU, all available GPU resources on the node can be acquired. According to the internal device UUID of the GPU, since the internal device UUID is globally unique and representative, each GPU device name is set as GPU / UUID, and the id of the node is associated to obtain available resource information, which is reported to the API server of Kubernetes for use by a global scheduler to obtain available resource information when allocating resources.

[0090] S13, traversing, by the resource plug-in, the available GPUs, constructing a connection matrix, and uploading the connection matrix to a GPU monitoring server.

[0091] Step S13 can include the following sub-steps:

[0092] Traversing, by the resource plug-in, the available GPUs, determining a plurality of initial link information between each available GPU and any available CPU;

[0093] Selecting the initial link information with the highest link level as target link information of the available GPU;

[0094] Using all the target link information, constructing an initial connection matrix, and uploading the initial connection matrix to a GPU monitoring server.

[0095] In an embodiment of the present application, by traversing all available GPUs through the resource plug-in, a plurality of initial link information between each available GPU and any available CPU is determined, and the initial link information with the highest link level between each available GPU and other available GPUs is selected as target link information corresponding to the available GPU. After all the target link information is determined, a connection matrix is constructed using all the target link information and uploaded to a GPU monitoring server.

[0096] The specific connection matrix can be shown in Table 1 as follows:

[0097] GPU / 0 GPU / 1 GPU / 2 GPU / 3 GPU / 4 GPU / 5 GPU / 6 GPU / 7 GPU / 0 X NVlink NVlink NVlink QPI QPI QPI NVlink GPU / 1 NVlink X NVlink NVlink QPI QPI NVlink QPI … … … … … … … … …

[0098] Table 1

[0099] In the presence of high-speed links, the low-speed links are not used, at this time the connection matrix only needs to record the highest speed link, after obtaining the connection matrix of each cluster node, the registration information of the connection matrix is sent to the GPU monitoring server to upload it to the GPU monitoring server.

[0100] Further, the method further comprises the following steps S21-S25:

[0101] S21, sending heartbeat update information to the GPU monitoring server by the resource plug-in at a regular time;

[0102] S22, scanning each available GPU by the resource plug-in at a regular time to determine the update resource information corresponding to each available GPU;

[0103] The heartbeat update information refers to the heartbeat information indicating that there is available GPU resource in the cluster node.

[0104] The update resource information refers to the information indicating the communication speed or link level of the available GPU in the cluster node at the current time, and the information indicating the state of the available GPU resource.

[0105] In the embodiment, the heartbeat update information is sent to the GPU monitoring server by the resource plug-in at a regular time, and each available GPU is scanned at a regular time to determine the update resource information of each available GPU at the current time.

[0106] S23, judging whether the target connection matrix has update resource information;

[0107] S24, if not, updating the available resource information in the API server with the update resource information, and sending state update information to the GPU monitoring server;

[0108] S25, if yes, jumping to the step of scanning each available GPU by the resource plug-in at a regular time to determine the update resource information corresponding to each available GPU.

[0109] In the embodiment of the present application, by judging whether the connection matrix exists the same update resource information, if not, it indicates that the state of the available GPU has changed at this time, at this time, the update resource information can be used to update the available resource information in the API server, and the state update information is sent to the GPU server. If it exists, it indicates that the state of the available GPU has not changed at this time, and the step S22 can be executed to continue to detect the use of the available GPU.

[0110] It should be noted that the state update information is an array of a set of values representing changes, for example, the gpu3 of the node2 of the cluster node is unavailable, and the expression [node:node2, gpu:gpu3_UUID, action:delete] is used.

[0111] In another example of the present application, the GPU monitoring server includes a plurality of listening ports and a server body; the method can further include steps S31-S34:

[0112] S31, receiving the initial connection matrix, the state update information and the heartbeat update information uploaded by the resource plug-in through the listening port;

[0113] S32, constructing the GPU matrix table by the server body using the initial connection matrix, the state update information, the heartbeat update information and the node identifier of the cluster node to which the resource plug-in belongs;

[0114] S33, polling each GPU matrix table by the server body to remove the GPU matrix table which is not updated within a preset period of time by the heartbeat update information;

[0115] In the embodiment of the present application, the GPU monitoring server can include a plurality of listening ports and a server body, the connection matrix, the state update information and the heartbeat update information uploaded by the resource plug-in are received through the listening port and delivered to the server body, the connection matrix, the state update information and the heartbeat update are combined with the node identifier corresponding to the cluster node to which the resource plug-in belongs, and the GPU matrix table is constructed, which can be specifically referred to Table 2 as follows:

[0116] Node Name Initial Connection Matrix Last Update Time Last Heartbeat Time Node 1

[0117] Table 2

[0118] Among them, the last update time refers to the receiving time of the connection matrix, and the last heartbeat time refers to the time of the last received heartbeat update information.

[0119] At the same time, the GPU matrix table is scanned at a fixed time, and the cluster node which is overdue (the last heartbeat time is not updated for a certain time) is removed, which indicates that the cluster node cannot be used.

[0120] S34, when receiving the GPU available list, querying the corresponding target connection matrix according to the GPU available list by the server body, and returning to the global scheduler.

[0121] When the GPU monitoring server receives the GPU available list, the corresponding initial connection matrix can be searched in the GPU matrix table according to the sending node name of the GPU available list by the server body. In actual operation, the initial connection matrix GPU can have changed from the available state to the unavailable state. At this time, the initial connection matrix can be further intercepted according to the name of the available GPU in the GPU available list to generate the target connection matrix corresponding to the GPU available list and return to the global scheduler.

[0122] Step 202, according to the GPU demand quantity and the available resource information of each cluster node, at least one candidate node is selected from a plurality of cluster nodes and a corresponding GPU available list is generated;

[0123] Optionally, step 202 can include the following sub-steps:

[0124] The number of available GPUs of each cluster node to which the available resource information belongs is counted;

[0125] The number of available GPUs of each cluster node to which the available resource information belongs is counted;

[0126] The number of available GPUs of each cluster node to which the available resource information belongs is counted;

[0127] In the embodiment of the present application, the number of available GPUs on each cluster node can be determined by counting the number of available GPUs of each cluster node to which the available resource information belongs. The cluster node whose number of available GPUs is greater than or equal to the number of GPU demands is selected as a candidate node by further comparing the number of GPU demands with the number of available GPUs. Finally, the available resource information corresponding to the candidate node is used to generate the GPU available list.

[0128] It should be noted that in addition to the number of GPU demands, the demand information such as the type of GPU demand, whether communication between multiple GPUs is needed and / or the communication mode between GPUs can be further combined to select the candidate node. For example, the type of each GPU is obtained through the available resource information, and it is determined whether it is the same as the type of GPU demand. The candidate node with the GPU that meets the communication mode between GPUs is further selected, and the specific implementation manner is not limited in the embodiment of the present application.

[0129] Step 203, when the number of candidate nodes is single, the candidate node is determined as a decision node.

[0130] In one example of the present application, when the number of candidate nodes is a single, it indicates that the candidate node is the only cluster node meeting the GPU requirement number, and the candidate node can be determined as the decision node.

[0131] Step 204, when the number of candidate nodes is not less than two, sending the GPU available list to the GPU monitoring server;

[0132] In another example of the present application, when the number of candidate nodes is not less than two, it indicates that the candidate nodes have advantages and disadvantages, and cannot be directly used as decision nodes. The GPU available list corresponding to each candidate node can be sent to the GPU monitoring server.

[0133] Step 205, when receiving the multiple target connection matrices returned by the GPU monitoring server in response to the GPU available list, selecting the decision node from the multiple candidate nodes according to the target connection matrices;

[0134] Optionally, step 205 can include the following sub-steps S11-S15:

[0135] S11, when receiving the multiple target connection matrices returned by the GPU monitoring server in response to the GPU available list, traversing the target connection matrices to respectively construct multiple initial GPU combinations;

[0136] The initial GPU combination refers to traversing all GPUs in the target connection matrix to construct a combination according to the GPU requirement number. For example, the number of GPUs in a single node is n, and the number of GPUs required by the task is m, so there are C(n, m) combination ways (S1, S2, S3, …). Since the number of GPUs in a single node is generally not more than 8, the number of combinations here is at most C(8, 4) = 70.

[0137] In the embodiment of the present application, when receiving the multiple target connection matrices returned by the GPU monitoring server in response to the GPU available list, multiple GPUs are selected from each target connection matrix by traversing each target connection matrix, and the selected multiple GPUs are used to construct multiple initial GPU combinations.

[0138] S12, respectively constructing multiple ring communication links corresponding to each initial GPU combination;

[0139] The ring communication link refers to traversing the initial GPU combination, selecting each GPU in the initial GPU combination as a starting point GPU and an ending point GPU one by one, and connecting the remaining GPUs in turn to form a communication link. For example, a multi-dimensional array d[i][j][s] can represent the maximum low-speed link value of the target communication link constructed under the condition that the selected starting point GPU is i, the ending point GPU is j, and the used GPU is s.

[0140] In the embodiment of the present application, the GPUs are selected one by one from the initial GPU combination as the starting point and the ending point and the remaining GPUs are connected respectively, so as to construct a plurality of ring communication links corresponding to the same initial GPU combination.

[0141] S13, comparing each ring communication link of the initial GPU combination, selecting the ring communication link with the minimum communication bottleneck speed as the target communication link;

[0142] In the embodiment of the present application, the ring communication links corresponding to the same initial GPU combination are traversed, and by comparing each ring communication link, the ring communication link with the minimum communication bottleneck speed is selected as the target communication link.

[0143] It should be noted that due to the different connection modes between the GPUs in the ring communication link, there are a plurality of different communication bottleneck speeds, and the minimum value of the communication bottleneck speed is selected as the minimum communication bottleneck speed.

[0144] S14, comparing each target communication link, and selecting the intermediate GPU combination to which the maximum value of each minimum communication bottleneck speed belongs according to the comparison result;

[0145] S15, if the number of intermediate GPU combinations is single, the candidate node to which the intermediate GPU combination belongs is determined as the decision node.

[0146] In the embodiment of the present application, after selecting the target communication link corresponding to each initial GPU combination, the maximum value of the minimum communication bottleneck speed corresponding to each target communication link is selected, and the GPU combination to which the maximum value belongs is selected as the intermediate GPU combination and the number of intermediate GPU combinations is counted.

[0147] If the number of intermediate GPU combinations is single, the candidate node to which the intermediate GPU combination belongs is determined as the decision node.

[0148] Further, step 205 can further include sub-steps S16-S17:

[0149] S16, if the number of intermediate GPU combinations exceeds one, the number of GPUs with connection speed equal to or greater than the preset high-speed level in each intermediate GPU combination is obtained respectively;

[0150] S17, selecting the candidate node to which the intermediate GPU combination corresponding to the minimum value of the number of GPUs belongs as the decision node.

[0151] In another embodiment of the present application, if the number of intermediate GPU combinations exceeds one, the number of GPUs in each intermediate GPU combination whose connection speed is greater than the preset high-speed level is obtained respectively, and the candidate node to which the intermediate GPU combination corresponding to the minimum number of GPUs is selected as the decision node.

[0152] In step 206, the GPU task is allocated to the decision node, and the GPU task is executed through the decision node.

[0153] After the decision node is selected, the GPU task is allocated to the decision node, and the GPU task is executed through the decision node.

[0154] In the embodiment of the present application, when the global scheduler receives a GPU task, the number of GPU requirements corresponding to the GPU task is obtained, and the available resource information corresponding to each cluster node is obtained from the API server. Further, the available resource information is matched based on the number of GPU requirements to filter at least one candidate node from the plurality of cluster nodes, and a GPU available list corresponding thereto is generated. If the number of candidate nodes is not less than two, the GPU available list is sent to the GPU monitoring server to obtain a plurality of target connection matrices corresponding thereto. According to the comparison results of the GPU combinations constructed based on each target connection matrix under the ring communication link, a single decision node is selected from the candidate nodes. Finally, the GPU task is allocated to the decision node, and the GPU task is executed through the decision node. Thus, the optimal allocation of nodes is realized according to the requirements of the GPU task, and the execution efficiency of the GPU task is effectively improved.

[0155] Please refer to Figure 3 , Figure 3 A structural block diagram of a GPU task deployment device provided in Embodiment Three of the present application.

[0156] The embodiment of the present application provides a GPU task deployment device applied to a global scheduler in a resource cluster. The global scheduler, the API server and the GPU monitoring server are in communication connection with each other. The resource cluster includes a plurality of cluster nodes. The device includes:

[0157] The information acquisition module 301 is configured to acquire the number of GPU requirements in the GPU task and obtain the available resource information corresponding to each cluster node from the API server when the GPU task is received.

[0158] The node filtering module 302 is configured to filter at least one candidate node from the plurality of cluster nodes and generate a corresponding GPU available list according to the number of GPU requirements and the available resource information.

[0159] The list sending module 303 is configured to send the GPU available list to the GPU monitoring server when the number of candidate nodes is not less than two.

[0160] The decision node selection module 304 is configured to select a decision node from a plurality of candidate nodes according to each target connection matrix when receiving a plurality of target connection matrices returned by the GPU monitoring server in response to the GPU available list;

[0161] The GPU task execution module 305 is configured to allocate a GPU task to the decision node and execute the GPU task through the decision node.

[0162] Optionally, the cluster node comprises a resource plug-in and a plurality of GPUs to be determined; the apparatus further comprises:

[0163] The identification acquisition module is configured to acquire, through the resource plug-in, a unique device identification corresponding to each available GPU from the plurality of GPUs to be determined;

[0164] The available resource information construction module is configured to construct available resource information by using the unique device identification and a node identification corresponding to the cluster node through the resource plug-in, and upload the available resource information to the API server;

[0165] The connection matrix construction module is configured to traverse the available GPUs through the resource plug-in, construct a connection matrix, and upload the connection matrix to the GPU monitoring server.

[0166] Optionally, the connection matrix construction module is specifically configured to:

[0167] traverse the available GPUs through the resource plug-in, and determine a plurality of initial link information between each two of the available GPUs;

[0168] select the initial link information with the highest level as target link information of the available GPUs;

[0169] construct an initial connection matrix by using all the target link information, and upload the initial connection matrix to the GPU monitoring server.

[0170] Optionally, the apparatus further comprises:

[0171] The heartbeat update module is configured to send heartbeat update information to the GPU monitoring server at a regular time through the resource plug-in;

[0172] The regular scanning module is configured to scan each available GPU at a regular time through the resource plug-in, and determine updated resource information corresponding to each available GPU;

[0173] The resource information existence judgment module is configured to judge whether the target connection matrix has the updated resource information;

[0174] The available resource information update module is configured to, if not, update the available resource information in the API server by using the updated resource information, and send state update information to the GPU monitoring server;

[0175] a jump module, configured to jump to perform the step of determining the update resource information corresponding to each available GPU by timing scanning each available GPU through the resource plug-in if the resource plug-in exists.

[0176] Optionally, the GPU monitoring server comprises a plurality of listening ports and a server body; the device further comprises:

[0177] an information receiving module, configured to receive the initial connection matrix, the state update information and the heartbeat update information uploaded by the resource plug-in through the listening port;

[0178] a matrix table constructing module, configured to construct the GPU matrix table by using the initial connection matrix, the state update information, the heartbeat update information and the node identifier of the cluster node to which the resource plug-in belongs through the server body;

[0179] a matrix table polling module, configured to poll each GPU matrix table through the server body, and remove the GPU matrix table which is not updated within a preset time period by the heartbeat update information;

[0180] a matrix returning module, configured to, when the GPU available list is received, query the corresponding target connection matrix according to the GPU available list through the server body, and return to the global scheduler.

[0181] Optionally, the node screening module 302 is specifically configured to:

[0182] count the available number of GPUs of the cluster node to which each available resource information belongs;

[0183] compare the GPU required number and the GPU available number, and select the cluster node whose GPU available number is greater than or equal to the GPU required number as a candidate node;

[0184] generate the GPU available list by using the available resource information corresponding to each candidate node.

[0185] Optionally, the decision node selecting module 304 is specifically configured to:

[0186] when a plurality of target connection matrices returned by the GPU monitoring server in response to the GPU available list are received, traverse each target connection matrix, and respectively construct a plurality of initial GPU combinations corresponding to each target connection matrix;

[0187] respectively construct a plurality of ring communication links corresponding to each initial GPU combination;

[0188] compare each ring communication link of the initial GPU combination, and select the ring communication link corresponding to the minimum communication bottleneck speed as a target communication link;

[0189] According to the comparison result, a maximum value of each minimum communication bottleneck speed is selected from each target communication link;

[0190] If the number of intermediate GPU combinations is one, a candidate node to which the intermediate GPU combination belongs is determined as the decision node.

[0191] Optionally, the decision node selection module 304 is specifically configured to:

[0192] If the number of intermediate GPU combinations is more than one, the number of GPUs whose connection speed is equal to or greater than the preset high speed level in each intermediate GPU combination is obtained respectively;

[0193] The candidate node to which the intermediate GPU combination corresponding to the minimum number of GPUs is belongs is selected as the decision node.

[0194] Optionally, the apparatus further comprises:

[0195] The single point selection module is configured to determine the candidate node as the decision node when the number of candidate nodes is one.

[0196] Embodiments of the present application provide an electronic device, comprising a memory and a processor, the memory stores a computer program, and the computer program is executed by the processor to make the processor execute the steps of the GPU task deployment method according to any embodiment of the present application.

[0197] Embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executed to implement the GPU task deployment method according to any embodiment of the present application.

[0198] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described apparatus and modules can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0199] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0200] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed on multiple network units. Part or all of the units may be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0201] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0202] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the application, essentially or the part that contributes to the prior art, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0203] The above-described embodiments are only used to illustrate the technical solutions of the application, rather than limit them; although the application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to part of the technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the application.

Claims

1. A method for allocating GPU tasks, characterized in that, A global scheduler is applied within a resource cluster, wherein the global scheduler, API server, and GPU monitoring server are interconnected in pairs, and the resource cluster includes multiple cluster nodes. The method includes: When a GPU task is received, the number of GPUs required within the GPU task is obtained, and the available resource information corresponding to each cluster node is obtained from the API server. Based on the GPU demand and available resource information, at least one candidate node is selected from the multiple cluster nodes and a corresponding list of available GPUs is generated; When the number of candidate nodes is not less than two, the list of available GPUs is sent to the GPU monitoring server. When the GPU monitoring server receives a response from the GPU available list and returns multiple target connection matrices, a decision node is selected from the multiple candidate nodes based on each target connection matrix. The GPU task is assigned to the decision node, and the GPU task is executed through the decision node; The step of selecting a decision node from multiple candidate nodes based on each of the target connection matrices when receiving multiple target connection matrices returned by the GPU monitoring server in response to the GPU available list includes: When the GPU monitoring server receives a response from the GPU available list and returns multiple target connection matrices, it iterates through each target connection matrix and constructs the corresponding multiple initial GPU combinations. Construct multiple ring communication links corresponding to each of the initial GPU combinations; Compare each of the ring communication links in the initial GPU combination, and select the ring communication link with the minimum communication bottleneck speed as the target communication link. Compare each of the target communication links, and select the intermediate GPU combination to which the maximum value of the minimum communication bottleneck speed belongs based on the comparison results; If the number of intermediate GPU combinations is a single one, then the candidate node to which the intermediate GPU combination belongs is determined as the decision node.

2. The method according to claim 1, characterized in that, The cluster node includes resource plugins and multiple GPUs to be evaluated; the method further includes: The resource plugin is used to obtain the unique device identifier corresponding to each available GPU from multiple GPUs to be determined. The resource plugin uses the unique device identifier and the node identifier corresponding to the cluster node to construct available resource information and upload it to the API server. The available GPUs are traversed through the resource plugin, a connection matrix is ​​constructed, and uploaded to the GPU monitoring server.

3. The method according to claim 2, characterized in that, The step of traversing the available GPUs through the resource plugin, constructing a connection matrix, and uploading it to the GPU monitoring server includes: The resource plugin is used to traverse the available GPUs and determine multiple initial link information between each pair of available GPUs. The initial link information with the highest link level is selected as the target link information for the available GPUs; Using all the target link information, an initial connection matrix is ​​constructed and uploaded to the GPU monitoring server.

4. The method according to claim 3, characterized in that, The method further includes: The resource plugin periodically sends heartbeat update information to the GPU monitoring server. The resource plugin periodically scans each available GPU to determine the updated resource information corresponding to each available GPU. Determine whether the target connection matrix contains the updated resource information; If not, the available resource information in the API server is updated using the updated resource information, and status update information is sent to the GPU monitoring server. If it exists, then proceed to the step of periodically scanning each available GPU through the resource plugin to determine the updated resource information corresponding to each available GPU.

5. The method according to claim 4, characterized in that, The GPU monitoring server includes multiple listening ports and a server body; the method further includes: The initial connection matrix, status update information, and heartbeat update information uploaded by the resource plugin are received through the listening port. The server body constructs a GPU matrix table using the initial connection matrix, the status update information, the heartbeat update information, and the node identifier of the cluster node to which the resource plugin belongs. The server body polls each of the GPU matrix tables and removes GPU matrix tables whose heartbeat update information has not been updated within a preset time period. When the list of available GPUs is received, the server body queries the corresponding target connection matrix according to the list of available GPUs and returns it to the global scheduler.

6. The method according to claim 1, characterized in that, The step of selecting at least one candidate node from multiple cluster nodes and generating a corresponding list of available GPUs based on the GPU demand and the available resource information includes: Calculate the number of available GPUs on the cluster nodes to which each of the available resource information belongs; Compare the number of GPUs required with the number of GPUs available, and select cluster nodes whose number of available GPUs is greater than or equal to the number of GPUs required as candidate nodes; A list of available GPUs is generated using the available resource information corresponding to each candidate node.

7. The method according to claim 1, characterized in that, Also includes: If the number of intermediate GPU combinations exceeds one, then obtain the number of GPUs in each intermediate GPU combination whose connection speed is equal to or greater than a preset high-speed level. The candidate node to which the intermediate GPU combination corresponding to the minimum number of GPUs belongs is selected as the decision node.

8. The method according to any one of claims 1-7, characterized in that, The method further includes: When the number of candidate nodes is a single node, the candidate node is determined as the decision node.

9. A GPU task allocation device, characterized in that, A global scheduler applied within a resource cluster, wherein the global scheduler, API server, and GPU monitoring server are interconnected in pairs, the resource cluster includes multiple cluster nodes, and the device includes: The information acquisition module is used to obtain the number of GPUs required in the GPU task and obtain the available resource information corresponding to each cluster node from the API server when a GPU task is received. The node filtering module is used to filter at least one candidate node from multiple cluster nodes and generate a corresponding list of available GPUs based on the number of GPUs required and the available resource information of each node. The list sending module is used to send the list of available GPUs to the GPU monitoring server when the number of candidate nodes is not less than two. The decision node selection module is used to select a decision node from multiple candidate nodes based on each of the target connection matrices when it receives multiple target connection matrices returned by the GPU monitoring server in response to the GPU available list. The GPU task execution module is used to allocate the GPU task to the decision node and execute the GPU task through the decision node; The decision node selection module is specifically used for: When the GPU monitoring server receives a response from the GPU available list and returns multiple target connection matrices, it iterates through each target connection matrix and constructs the corresponding multiple initial GPU combinations. Construct multiple ring communication links corresponding to each of the initial GPU combinations; Compare each of the ring communication links in the initial GPU combination, and select the ring communication link with the minimum communication bottleneck speed as the target communication link. Compare each of the target communication links, and select the intermediate GPU combination to which the maximum value of the minimum communication bottleneck speed belongs based on the comparison results; If the number of intermediate GPU combinations is a single one, then the candidate node to which the intermediate GPU combination belongs is determined as the decision node.

10. An electronic device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the GPU task allocation method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed, it implements the GPU task allocation method as described in any one of claims 1-8.

Images