GPU (Graphics Processing Unit) distribution method and device based on GPU load information
By collecting and parsing GPU information, calculating load or idle rate, and selecting GPUs with the smallest load or the highest idle rate, the problem that AI computing algorithm service fails to automatically select GPUs with less load when selecting GPUs, and improves GPU utilization rate and AI computing power.
Patent Information
- Application Number
- CN202411023269.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-29
- Publication Date
- 2024-12-24
AI Technical Summary
When selecting a GPU, AI computing algorithm services often randomly select or statically specify it. They fail to automatically select GPUs with less loads according to the GPU usage, resulting in more loads being selected, resulting in low GPU utilization and insufficient computing power.
By receiving GPU allocation requests from AI computing algorithm services, collecting and parsing information values of multiple GPUs, including total memory, used memory and available memory, calculating the load or idle rate of each GPU, and selecting the GPU with the lowest load or the highest idle rate of the GPU, and selecting the GPU with the lowest load or the highest idle rate Return to the AI computing algorithm service.
It effectively improves the utilization rate of GPUs, avoids unreasonable GPU selection caused by random selection, improves AI computing capabilities, and enables GPUs with lighter loads to dynamically automatically select, improving computing performance.
Smart Images
Figure CN119179565A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computers, and in particular to a GPU allocation method and device based on GPU load information. Background Art
[0002] The AI computing algorithm service selects a GPU randomly, and may not select a GPU with less load, or it may be statically specified and cannot be automatically selected based on GPU usage. The AI computing algorithm service needs to select a GPU based on GPU usage and dynamically and automatically select a GPU with less load.
[0003] The AI computing algorithm service selects a GPU randomly or statically. Random selection does not consider GPU usage, which may cause a GPU with a heavier load to be selected and a GPU with a lighter load not to be selected. The statically specified algorithm service starts the specified GPU. The algorithm service does not obtain GPU usage and selects a GPU based on usage. Summary of the invention
[0004] In view of the above technical problems, the present invention provides a GPU allocation method and device based on GPU load information, which can improve the utilization rate of GPU.
[0005] A first aspect of the present invention provides a GPU allocation method based on GPU load information, comprising: receiving a GPU allocation request, the GPU allocation request coming from an AI computing algorithm service; collecting GPU information, and after successfully collecting the GPU information, parsing the GPU information to obtain information values and GPU index identifiers of multiple GPUs; calculating the GPU with the smallest load based on the information values of the multiple GPUs, selecting the GPU index identifier of the GPU with the smallest load and returning it to the AI computing algorithm service.
[0006] In an optional embodiment, after successfully collecting the GPU information, parsing the GPU information to obtain information values and GPU index identifiers of multiple GPUs includes:
[0007] Get the total memory value, used memory value, available memory value, and GPU index of multiple GPUs.
[0008] In an optional embodiment, the collecting of GPU information, and parsing the GPU information to obtain information values and GPU index identifiers of multiple GPUs after the GPU information is successfully collected, includes:
[0009] Initiate an http request to the server where the GPU and GPU exporter are installed to obtain GPU information. The http response returns a message body, which is parsed to obtain the total memory value, used memory value, available memory value, and GPU index identifier.
[0010] In an optional embodiment, the calculating the GPU with the smallest load according to the information values of the multiple GPUs includes:
[0011] Calculate the loads of the multiple GPUs according to the ratios of the used memory values and the total memory values of the information values of the multiple GPUs respectively;
[0012] Compare the loads of multiple GPUs and determine the GPU with the least load.
[0013] In an optional embodiment, the calculating the GPU with the smallest load according to the information values of the multiple GPUs includes:
[0014] Calculate the idle rates of multiple GPUs according to the ratios of the available memory values to the total memory values of the information values of the multiple GPUs;
[0015] The idle rates of multiple GPUs are compared, and the GPU with the largest idle rate is determined as the GPU with the smallest load.
[0016] A second aspect of the present invention provides a GPU allocation method and device based on GPU load information, comprising:
[0017] A receiving module, configured to receive a GPU allocation request, where the GPU allocation request comes from an AI computing algorithm service;
[0018] A collection and analysis module, used to collect GPU information, and after successfully collecting the GPU information, analyze the GPU information to obtain information values and GPU index identifiers of multiple GPUs;
[0019] An allocation module is used to calculate the GPU with the smallest load according to the information values of the multiple GPUs, select the GPU index identifier of the GPU with the smallest load and return it to the AI computing algorithm service.
[0020] In an optional embodiment, the allocation module also includes a first submodule, which is used to calculate the load of multiple GPUs based on the ratio of the used memory value to the total memory value of the information values of multiple GPUs; compare the loads of multiple GPUs to determine the GPU with the smallest load.
[0021] In an optional embodiment, the allocation module also includes a second sub-module, which is used to calculate the idle rates of multiple GPUs based on the ratio of the available memory value to the total memory value of the information values of the multiple GPUs; compare the idle rates of the multiple GPUs, and determine that the GPU with the largest idle rate is the GPU with the smallest load.
[0022] According to a third aspect of the present invention, there is provided an electronic device, comprising:
[0023] At least one processor; and at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the method described in the first aspect of the embodiment of the present invention.
[0024] According to a fourth aspect of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a computer, the method according to the first aspect of the embodiment of the present invention is executed.
[0025] The present invention calculates the GPU with the smallest load according to the information values of the multiple GPUs, selects the GPU index identifier of the GPU with the smallest load and returns it to the AI computing algorithm service, thereby avoiding unreasonable GPU selection caused by random selection and improving AI computing capability. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 The figure is a flow chart of a GPU allocation method based on GPU load information in an embodiment of the present invention.
[0027] Figure 2 The figure is a flow chart of another GPU allocation method based on GPU load information in an embodiment of the present invention.
[0028] Figure 3 The module diagram of a GPU allocation method device based on GPU load information in an embodiment of the present invention. DETAILED DESCRIPTION
[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0030] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0031] See also Figure 1 The present invention provides a GPU allocation method based on GPU load information, which includes the following steps.
[0032] S10: Receive a GPU allocation request, where the GPU allocation request comes from an AI computing algorithm service.
[0033] Usually, the AI computing algorithm service requests a GPU with less load from the GPU allocation service. The GPU allocation service initiates an http request from the server where the GPU and GPU exporter are installed to obtain GPU information. The http response returns a 200 status code for success, and the rest indicates failure.
[0034] S20: Collect GPU information, and after successfully collecting the GPU information, parse the GPU information to obtain information values and GPU index identifiers of multiple GPUs.
[0035] After the GPU allocation service initiates an http request to obtain GPU information, the http response returns a message body. The message body is parsed to select the target row containing the names corresponding to GPU index, total memory, used memory, and available memory, and extract the value corresponding to the target row name.
[0036] In some embodiments, the collected GPU information is stored in a table, and each row in the table is filled with a row name and a value corresponding to the row name. For example, the total memory value, used memory value, available memory value, and GPU index identifier of the information values of multiple GPUs are obtained through the message body.
[0037] The GPU index identifier may be the name of the GPU or the identification number of the GPU, such as GPU1, GPU2, GPU3, etc. The total capacity of the GPU is, for example, 16 GB, the used memory is 8 GB, and the available memory is 7 GB.
[0038] S30: Calculate the GPU with the smallest load according to the information values of the multiple GPUs, select the GPU index identifier of the GPU with the smallest load and return it to the AI computing algorithm service.
[0039] After the information value of the GPU is known, the load of each GPU can be calculated separately, and the GPU with the smallest load can be selected to allocate the corresponding GPU for the AI computing algorithm service.
[0040] In one embodiment, GPUs may be allocated based on load values. For example, the loads of multiple GPUs are calculated based on the ratio of the used memory values to the total memory values of the information values of multiple GPUs, and the load of each GPU is calculated. The loads of multiple GPUs are compared to determine the GPU with the smallest load.
[0041] For example, the total amount of GPU1 is 16G, the used memory is 8G, and the available memory is 7G. The load of this GPU is 8 / 16*100%=50%. The total amount of GPU2 is 16G, the used memory is 4G, and the available memory is 11G. The load of this GPU is 4 / 16*100%=25%. Obviously, the load of GPU2 is less than that of GPU1, and GPU2 is allocated to the AI computing algorithm service.
[0042] In another embodiment, the GPUs may be allocated according to the memory idle rate value. For example, the idle rates of the multiple GPUs are calculated according to the ratio of the available memory value to the total memory value of the information values of the multiple GPUs; that is, the memory idle rate of each GPU is calculated respectively. The idle rates of the multiple GPUs are compared, and the GPU with the maximum idle rate is determined as the GPU with the smallest load.
[0043] For example, the total amount of GPU1 is 16G, the used memory is 7G, and the available memory is 8G. The idle rate value of this GPU is 8 / 16*100%=50%. The total amount of GPU2 is 16G, the used memory is 11G, and the available memory is 4G. The idle rate value of this GPU is 4 / 16*100%=25%. Obviously, the load of GPU1 is greater than the load of GPU2, and GPU1 is allocated to the AI computing algorithm service.
[0044] The AI computing algorithm service allocates the GPU with the least service request load to the GPU, which can avoid unreasonable GPU selection caused by random selection. The AI computing algorithm service dynamically and automatically selects the GPU with the least load, allowing the GPU to run at a low load, thereby improving AI computing capabilities.
[0045] See also Figure 2 , Figure 2 Flowchart for GPU allocation. The GPU allocation method based on GPU load information includes the following steps.
[0046] S41: Receive a request to allocate a GPU.
[0047] S42: Collect GPU information.
[0048] S43: Determine whether the collection is successful.
[0049] S44: If successful, parse the GPU information.
[0050] S45: Calculate the load.
[0051] S46: Select the GPU with the smallest load.
[0052] S47: Return the GPU ID and then end.
[0053] S48: Return an invalid GPU identifier and then end.
[0054] See also Figure 3 The present invention also provides a GPU allocation method and device based on GPU load information, including: a receiving module 31, a collection and analysis module 32, and an allocation module 33.
[0055] The receiving module 31 is used to receive a GPU allocation request, where the GPU allocation request comes from an AI computing algorithm service.
[0056] The collection and analysis module 32 is used to collect GPU information, and after successfully collecting the GPU information, analyze the GPU information to obtain information values and GPU index identifiers of multiple GPUs.
[0057] For example, an http request is initiated to a server with a GPU and a GPU exporter installed to obtain GPU information. The http response returns a message body, which is parsed to obtain the total memory value, used memory value, available memory value, and GPU index identifier.
[0058] For example, the total memory value, used memory value, available memory value, and GPU index identifier of the information values of multiple GPUs are obtained.
[0059] The allocation module 33 is used to calculate the GPU with the smallest load according to the information values of the multiple GPUs, select the GPU index identifier of the GPU with the smallest load and return it to the AI computing algorithm service.
[0060] Furthermore, the allocation module 33 also includes a first submodule, which is used to calculate the loads of multiple GPUs according to the ratios of the used memory values and the total memory values of the information values of multiple GPUs; compare the loads of multiple GPUs to determine the GPU with the smallest load.
[0061] For example, the total amount of GPU1 is 16G, the used memory is 8G, and the available memory is 7G. The load of this GPU is 8 / 16*100%=50%. The total amount of GPU2 is 16G, the used memory is 4G, and the available memory is 11G. The load of this GPU is 4 / 16*100%=25%. Obviously, the load of GPU2 is less than that of GPU1, and GPU2 is allocated to the AI computing algorithm service.
[0062] Furthermore, the allocation module 33 also includes a second sub-module, which is used to calculate the idle rates of multiple GPUs according to the ratio of the available memory value to the total memory value of the information values of multiple GPUs; compare the idle rates of multiple GPUs, and determine that the GPU with the largest idle rate is the GPU with the smallest load.
[0063] For example, the total amount of GPU1 is 16G, the used memory is 7G, and the available memory is 8G. The idle rate value of this GPU is 8 / 16*100%=50%. The total amount of GPU2 is 16G, the used memory is 11G, and the available memory is 4G. The idle rate value of this GPU is 4 / 16*100%=25%. Obviously, the load of GPU1 is greater than the load of GPU2, and GPU1 is allocated to the AI computing algorithm service.
[0064] The functions or methods performed by the above modules can refer to the description of the GPU allocation method based on GPU load information, which will not be repeated here.
[0065] The present invention also provides an electronic device, comprising:
[0066] At least one processor; and at least one memory in communication with the processor, wherein: the memory stores program instructions that can be executed by the processor, and the processor calls the program instructions to execute the above-mentioned GPU allocation method based on GPU load information.
[0067] The present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned GPU allocation method based on GPU load information is implemented.
[0068] It can be understood that computer-readable storage media may include: any entity or device capable of carrying a computer program, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc. A computer program includes computer program code. The computer program code may be in source code form, object code form, executable file, or some intermediate form, etc. A computer-readable storage medium may include: any entity or device capable of carrying a computer program code, recording media, USB flash drives, mobile hard disks, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution media, etc.
[0069] In certain embodiments of the present invention, the device may include a controller, which is a single-chip microcomputer chip that integrates a processor, a memory, a communication module, etc. The processor may refer to the processor included in the controller. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc.
[0070] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, segment or portion of code that includes one or more executable instructions for implementing the steps of a specific logical function or process, and the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order depending on the functions involved, which should be understood by technicians in the technical field to which the embodiments of the present invention belong.
[0071] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.
[0072] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A GPU allocation method based on GPU load information, characterized in that: include: Receive a GPU allocation request, where the GPU allocation request comes from an AI computing algorithm service; Collecting GPU information, and parsing the GPU information to obtain information values and GPUindex identifiers of multiple GPUs after successfully collecting the GPU information; The GPU with the smallest load is calculated according to the information values of the multiple GPUs, and the GPU index identifier of the GPU with the smallest load is selected to return to the AI computing algorithm service.
2. The GPU allocation method based on GPU load information according to claim 1, characterized in that: After successfully collecting the GPU information, the GPU information is parsed to obtain information values and GPU index identifiers of multiple GPUs, including: Get the total memory value, used memory value, available memory value, and GPU index of multiple GPUs.
3. The GPU allocation method based on GPU load information according to claim 1, characterized in that: The collecting of GPU information, and parsing the GPU information to obtain information values and GPU index identifiers of multiple GPUs after the GPU information is successfully collected, includes: Initiate an http request to the server where the GPU and GPU exporter are installed to obtain GPU information. The http response returns a message body, which is parsed to obtain the total memory value, used memory value, available memory value, and GPU index identifier.
4. The GPU allocation method based on GPU load information according to claim 2 or 3, characterized in that: The step of calculating the GPU with the smallest load according to the information values of the multiple GPUs includes: Calculate the loads of multiple GPUs according to the ratios of the used memory values and the total memory values of the information values of the multiple GPUs respectively; Compare the loads of multiple GPUs and determine the GPU with the least load.
5. The GPU allocation method based on GPU load information according to claim 2 or 3, characterized in that: The step of calculating the GPU with the smallest load according to the information values of the multiple GPUs includes: Calculate the idle rates of multiple GPUs according to the ratios of the available memory values to the total memory values of the information values of the multiple GPUs; The idle rates of multiple GPUs are compared, and the GPU with the largest idle rate is determined as the GPU with the smallest load.
6. A GPU allocation method and device based on GPU load information, characterized in that: include: A receiving module, configured to receive a GPU allocation request, where the GPU allocation request comes from an AI computing algorithm service; A collection and analysis module, used to collect GPU information, and after successfully collecting the GPU information, analyze the GPU information to obtain information values and GPU index identifiers of multiple GPUs; An allocation module is used to calculate the GPU with the smallest load according to the information values of the multiple GPUs, select the GPU index identifier of the GPU with the smallest load and return it to the AI computing algorithm service.
7. The GPU allocation device based on GPU load information according to claim 6, characterized in that: The allocation module also includes a first submodule, which is used to calculate the loads of multiple GPUs according to the ratios of the used memory values and the total memory values of the information values of multiple GPUs; compare the loads of multiple GPUs to determine the GPU with the smallest load.
8. The GPU allocation device based on GPU load information according to claim 6, characterized in that: The allocation module also includes a second submodule, which is used to calculate the idle rates of multiple GPUs according to the ratios of the available memory values to the total memory values of the information values of the multiple GPUs; compare the idle rates of the multiple GPUs, and determine the GPU with the maximum idle rate as the GPU with the minimum load.
9. An electronic device, characterized in that: include: at least one processor; And at least one memory communicatively connected to the processor, wherein: the memory stores program instructions executable by the processor, and the processor calls the program instructions to execute the GPU allocation method based on GPU load information as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: A computer program is stored thereon, and when the computer program is executed by a computer, the GPU allocation method based on GPU load information as claimed in any one of claims 1 to 7 is executed.