GPU resource allocation methods, devices, electronic equipment, and storage media

Abstract

This invention provides a GPU resource allocation method, apparatus, electronic device, and storage medium. The method includes: in response to an inference task request, determining an execution server among several servers to execute the inference task request; wherein multiple service models of inference services awaiting GPU allocation are deployed in the memory of the several servers, each server deploys a GPU management unit for allocating GPUs to the inference service, and the execution server deploys a target inference service required for executing the inference task; and allocating GPUs to the target inference service through the GPU management unit. This method avoids latency caused by thread start-up and shutdown by pre-deploying the service models of the inference service in the server's memory, placing them in a state of being started but awaiting activation. The method of this invention can significantly reduce the response time of inference services, providing a better user experience.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of computer processors, and more specifically, to GPU resource allocation methods, apparatus, electronic devices, and storage media. Background Technology

[0002] This section is intended to provide background or context for embodiments of the invention set forth in the claims. The description herein may include concepts that may be explored, but not necessarily concepts that have been previously conceived or explored. Therefore, unless otherwise stated, what is described in this section is not prior art for the purposes of this application's specification and claims, and is not acknowledged as prior art simply by virtue of its inclusion in this section.

[0003] When deploying online deep learning inference services, it is typically necessary to deploy several different inference services on a server cluster. When a particular inference service generates a task requirement, the server needs to allocate the corresponding GPU to it so that it can obtain the right to use the GPU and the corresponding computing space and model space. Since the demand of each inference service changes in real time, configuring resources for each inference service according to the maximum possible traffic would result in a significant waste of resources.

[0004] In existing technologies, some methods have emerged that monitor traffic and add or remove containers to configure GPUs for inference services. These methods abstract the server into several containers, each allocated a preset number of GPUs for use by an inference service. When traffic increases, a new container can be started to increase capacity; when traffic decreases, some containers can be shut down to free up server resources.

[0005] However, since each container is allocated a fixed number of GPUs, and each container is used by one inference service, once a GPU is assigned to a specific inference service, it cannot be temporarily reassigned to other inference services. If there are no available GPUs on the server and a new task is generated, it is necessary to shut down a certain inference service based on traffic to release the GPUs it occupies before allocating those GPUs to the inference service corresponding to the new task. This process requires shutting down the original inference service's service thread and re-establishing the service thread for the new inference service, i.e., the service process needs to be restarted and initialized. This process is time-consuming, usually taking seconds, and cannot meet the requirement of starting or stopping an inference service in real time based on traffic, resulting in lag and high latency in resource configuration adjustments. Summary of the Invention

[0006] Because the demand for inference services changes in real time, existing technologies employ methods such as monitoring traffic and adding or removing containers to configure GPUs for inference services.

[0007] However, in the existing technology, this process requires the original inference service thread to be shut down and a new service thread to be re-established for the new inference service, which means that the service process needs to be restarted and initialized, which takes a long time.

[0008] Therefore, there is a great need for an improved GPU resource allocation scheme to increase the speed of GPU allocation and the response speed of inference services.

[0009] In this context, embodiments of the present invention are intended to provide a GPU resource configuration method, apparatus, electronic device, and storage medium.

[0010] In a first aspect of the present invention, a GPU resource allocation method is provided, comprising: in response to an inference task request, determining an execution server among a plurality of servers to execute the inference task request; wherein the plurality of servers have service models of multiple inference services to be allocated GPUs deployed in their memory, each server has a GPU management unit for allocating GPUs for the inference service, and the execution server has a target inference service required to execute the inference task; and allocating GPUs to the target inference service through the GPU management unit.

[0011] In one embodiment of the invention, each server stores a service model of part or all of the inference service to be allocated GPUs in its memory, so that each inference service has at least one execution server.

[0012] In one embodiment of the present invention, allocating a GPU to a target inference service via a GPU management unit includes: determining a target GPU among the GPUs of the execution server; and allocating the target GPU to the target inference service via the GPU management unit.

[0013] In one embodiment of the present invention, the GPU of the execution server is deployed with a service model for inference services for some or all of the GPUs to be allocated in the execution server.

[0014] In one embodiment of the present invention, before determining the execution server to execute the inference task request among a plurality of servers in response to the inference task request, the method further includes: deploying at least one capacity unit and at least one hot standby unit for each inference service in at least one server; wherein the capacity unit of the inference service is a GPU whose computing space is occupied by the inference service, and the hot standby unit of the inference service is a GPU whose service model of the inference service is stored only in the model space.

[0015] In one embodiment of the present invention, if the inference service has a hot standby unit deployed in the current server, the service model of the inference service is stored in the hot standby unit, and the service model image of the inference service is stored in the memory of the current server; if the inference service does not have a hot standby unit deployed in the current server, the service model image of the inference service is stored in the memory of the current server.

[0016] In one embodiment of the present invention, determining the target GPU in the GPU of the execution server includes: if the difference between the capacity of the capacity unit of the target inference service and the load of the task is less than a preset value, then activating the hot standby unit of the target inference service as a capacity unit, so that the target inference service occupies the computing space and usage rights of the hot standby unit.

[0017] In one embodiment of the present invention, each hot standby unit stores at least one service model of an inference service. If a hot standby unit stores multiple service models, the multiple inference services corresponding to the multiple service models share the hot standby unit. Each inference service has multiple hot standby units. Accordingly, activating the hot standby unit of the target inference service as a capacity unit includes: activating one or more hot standby units of the target inference service as a capacity unit; removing the activated hot standby unit from the hot standby unit set of the shared object; wherein the shared object is an inference service that shares a hot standby unit with the target inference service.

[0018] In one embodiment of the present invention, activating one or more hot standby units of the target inference service as capacity units includes: determining a hot standby unit to be activated from among multiple hot standby units of the target inference service based on maximizing the total capacity expansion of the hot standby units; activating the hot standby unit to be activated as a capacity unit; wherein the capacity expansion of a hot standby unit is negatively correlated with the number of existing hot standby units in its to which it belongs.

[0019] In one embodiment of the present invention, determining the hot standby unit to be activated among multiple hot standby units of the target inference service based on maximizing the total expansion of the hot standby unit includes: traversing multiple hot standby units of the target inference service, calculating the loss value of the total expansion of the hot standby unit after each hot standby unit is activated, and taking the hot standby unit corresponding to the minimum loss value as the hot standby unit to be activated.

[0020] In one embodiment of the present invention, the calculation of the total expansion value of the hot standby unit after each hot standby unit is activated includes: summing the expansion value of the activated hot standby unit for each shared object to obtain the total expansion value of the hot standby unit after the hot standby unit is activated.

[0021] In one embodiment of the present invention, the GPU resource allocation method further includes: in response to a capacity release signal, determining a capacity unit to be released among a plurality of capacity units of the inference service corresponding to the capacity release signal; releasing the computing space and usage rights of the capacity unit to be released, so that the capacity unit to be released is restored as a hot standby unit; wherein, the capacity release signal is used to indicate that the difference between the capacity of the capacity unit of the corresponding inference service and the load of the corresponding inference service is greater than a preset value.

[0022] In one embodiment of the present invention, determining the capacity unit to be released among multiple capacity units of the inference service corresponding to the capacity release signal includes: determining the capacity unit to be released among multiple capacity units based on maximizing the total expansion of hot standby units; wherein the expansion of a hot standby unit is negatively correlated with the number of existing hot standby units in its to-belonging inference service.

[0023] In one embodiment of the present invention, determining the capacity unit to be released among multiple capacity units based on maximizing the total capacity expansion of the hot standby unit includes: traversing multiple capacity units corresponding to the inference service of the capacity release signal, calculating the output value of the total capacity expansion of the hot standby unit after the recovery of each capacity unit, and taking the capacity unit corresponding to the largest output value as the capacity unit to be released.

[0024] In one embodiment of the present invention, the calculation of the total capacity expansion of the hot standby unit after the capacity unit is restored includes: summing the capacity expansion of the restored hot standby unit for each shared object to obtain the total capacity expansion of the hot standby unit after the capacity unit is restored.

[0025] In one embodiment of the present invention, the GPU resource allocation method further includes: in response to a hot-cold conversion signal, performing a hot-cold conversion process on some hot standby units of the inference service to be converted to cold, and adding hot standby units for the inference service to be converted to hot; wherein, the hot-cold conversion signal is used to indicate that the number of hot standby units of the inference service is less than a preset number; the hot-cold conversion process includes: removing the service model of the inference service to be converted to cold from the hot standby units.

[0026] In one embodiment of the present invention, the cooling process further includes: after removing the service model to be cooled inference service, moving the service model retained in the hot standby unit so that the service models retained in the hot standby unit are arranged continuously.

[0027] In one embodiment of the present invention, the process of converting some hot standby units of the inference service to be converted to cold status into cold status and adding hot standby units for the inference service to be converted to hot status includes: determining the inference service to be converted to cold status, the hot standby units to be converted to cold status, the inference service to be converted to hot status, and the added hot standby units based on maximizing the total expansion capacity of the hot standby units; performing cold status conversion on the hot standby units to be converted to cold status; and adding hot standby units for the inference service to be converted to hot status; wherein the expansion capacity of a hot standby unit is negatively correlated with the number of existing hot standby units of its to which it belongs.

[0028] In one embodiment of the present invention, the determination of the hot standby service to be converted to cold inference, the hot standby unit to be converted to cold processing, the hot inference service to be converted to hot inference, and the added hot standby unit based on maximizing the total expansion of the hot standby unit includes: traversing each combination of the hot standby unit to be converted to cold processing and the added hot standby unit, calculating the loss value and output value of the total expansion of the hot standby unit under each combination, calculating the total value-added under each combination based on the loss value and output value of the total expansion of the hot standby unit, and selecting the combination corresponding to the maximum total value-added to perform the cold processing and hot standby unit addition steps.

[0029] In a second aspect of the present invention, a GPU resource allocation apparatus is provided, comprising: a task scheduling center and a plurality of servers; wherein each server has a plurality of GPUs and a GPU management unit for allocating GPUs for inference services; the memory of the plurality of servers contains service models of multiple inference services for which GPUs are to be allocated; wherein the memory of each server stores the service models of multiple inference services for which GPUs are to be allocated; the task scheduling center is used to respond to an inference task request, determine the execution server among the plurality of servers to execute the inference task request, and issue a GPU allocation instruction to the GPU management unit of the execution server; wherein the execution server is a server that has deployed the target inference service required for executing the task; the GPU management unit is used to allocate GPUs to the target inference service according to the GPU allocation instruction issued by the task scheduling center.

[0030] In a third aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing executable program instructions that, when executed by the processor, cause the electronic device to perform the method as described in any of the first aspects.

[0031] In a fourth aspect of the present invention, a computer-readable storage medium is provided having computer program instructions stored thereon, which, when executed by one or more processors, cause the processors to perform the method as described in any of the first aspects.

[0032] According to the GPU resource allocation method of this invention, since several inference service models are pre-deployed in the server's memory, these inference services are in an started state, but the server has not yet allocated GPUs to them. Therefore, each inference service deployed in the server is in a started but unactivated state. When a new task requirement arises, the existing GPU allocation method requires shutting down a certain inference service to release the GPU it occupies before allocating that GPU to the inference service corresponding to the new task, resulting in the shutdown of existing service threads and the startup of new service threads. However, the GPU resource allocation method of this invention only needs to schedule the task to the corresponding execution server and allocate the required GPU to the target inference service of the task through the GPU management unit to activate the target inference service. During this process, the server's memory always stores the service models of the inference services, avoiding the shutdown of existing service threads and the startup of new service threads, thus avoiding the delay caused by thread start-up and shutdown. This significantly improves the response speed of the inference service, reduces the user's response waiting time, and provides a better user experience.

[0033] In some embodiments of the present invention, the GPU resource configuration method provided by the embodiments is to deploy at least one capacity unit and at least one hot standby unit in at least one server for each inference service. When a new task request is generated, the hot standby unit is activated as a capacity unit to realize GPU resource configuration. Since the hot standby unit stores the service model of the target inference service, the target inference service can be activated to execute the task without waiting for the transmission time of model data from memory to GPU memory. The response speed is fast and can meet the real-time requirements of the inference service.

[0034] In some embodiments of the present invention, in the GPU resource configuration method provided by the embodiments, each inference service has multiple hot standby units, and the video memory of each hot standby unit can store multiple service models of inference services, so that the video memory of the hot standby unit can be shared by multiple inference services, and then the maximum expansion capability can be achieved in the limited video memory space through combination and adjustment.

[0035] In some embodiments of the present invention, the GPU resource allocation method provided by the embodiments aims to maximize the total expansion of hot standby units, selects hot standby units to be activated and / or capacity units to be released, and provides a quantifiable reference for the GPU allocation decision process by constructing the relationship between the expansion of hot standby units and inference services, thereby making GPU allocation more reasonable and reliable.

[0036] In some embodiments of the present invention, the GPU resource allocation method provided in the embodiments also releases GPU resources by performing a cooling process on some hot standby units of the service to be cooled, thereby adding hot standby units for the service to be heated. Through the above-mentioned cooling-hot conversion process, the number of hot standby units among inference services is reasonably allocated, effectively preventing the imbalance of too many hot standby units for some inference services and too few hot standby units for others, which is beneficial to the GPU resource allocation during subsequent task scheduling. Attached Figure Description

[0037] The above and other objects, features, and advantages of exemplary embodiments of the present invention will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of the invention are illustrated in the drawings by way of example and not limitation, wherein:

[0038] Figure 1 A block diagram schematically illustrates an exemplary computing system 100 suitable for implementing embodiments of the present invention;

[0039] Figure 2 A schematic diagram illustrating one method of deploying an inference service in the prior art is shown.

[0040] Figure 3 A schematic flowchart of a GPU resource allocation method according to an embodiment of the present invention is shown.

[0041] Figure 4 A schematic diagram illustrating a deployment method for an inference service according to an embodiment of the present invention is shown.

[0042] Figure 5 A schematic flowchart of a GPU allocation method according to some embodiments of the present invention is shown.

[0043] Figure 6 A schematic diagram illustrating the hot / cold conversion process of inference services according to some embodiments of the present invention is shown.

[0044] Figure 7 A schematic flowchart of a method for determining a target GPU based on maximizing the total capacity expansion of hot standby units according to some embodiments of the present invention is shown.

[0045] Figure 8 A schematic flowchart of a capacity unit release method based on maximizing the total expansion of hot standby units according to some embodiments of the present invention is shown.

[0046] Figure 9 A schematic diagram of a method for hot and cold switching of inference services based on maximizing the total capacity expansion of hot standby units, according to some embodiments of the present invention, is shown.

[0047] Figure 10 A schematic block diagram of a GPU resource configuration apparatus according to an embodiment of the present invention is shown.

[0048] Figure 11 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown.

[0049] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0050] The principles and spirit of the invention will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement the invention, and are not intended to limit the scope of the invention in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0051] Figure 1 A block diagram of an exemplary computing system 100 suitable for implementing embodiments of the present invention is shown. For example... Figure 1 As shown, the computing system 100 may include: a central processing unit (CPU) 101, random access memory (RAM) 102, read-only memory (ROM) 103, a system bus 104, a hard disk controller 105, a keyboard controller 106, a serial interface controller 107, a parallel interface controller 108, a display controller 109, a hard disk 110, a keyboard 111, a serial external device 112, a parallel external device 113, and a display 114. Among these devices, the CPU 101, RAM 102, ROM 103, hard disk controller 105, keyboard controller 106, serial controller 107, parallel controller 108, and display controller 109 are coupled to the system bus 104. The hard disk 110 is coupled to the hard disk controller 105, the keyboard 111 is coupled to the keyboard controller 106, the serial external device 112 is coupled to the serial interface controller 107, the parallel external device 113 is coupled to the parallel interface controller 108, and the display 114 is coupled to the display controller 109. It should be understood that... Figure 1 The structural block diagrams described are for illustrative purposes only and are not intended to limit the scope of the invention. In some cases, certain devices may be added or removed depending on the specific circumstances.

[0052] Those skilled in the art will recognize that embodiments of the present invention can be implemented as a system, method, or computer program product. Therefore, this disclosure can be specifically implemented as entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the present invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0053] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example,, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (not exhaustive) of a computer-readable storage medium may include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0054] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0055] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0056] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0057] Embodiments of the present invention will now be described with reference to flowchart illustrations and block diagrams of apparatus (or systems) according to embodiments of the invention. It should be understood that each block of the flowchart and / or block diagram, and combinations of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine that, when executed by the computer or other programmable data processing apparatus, creates means for implementing the functions / operations specified in the blocks of the flowchart and / or block diagram.

[0058] These computer program instructions may also be stored in a computer-readable medium that enables a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce a product comprising an instruction apparatus that implements the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0059] Computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, such that the instructions that execute on the computer or other programmable apparatus can provide a process for implementing the functions / operations specified in the boxes of a flowchart and / or block diagram.

[0060] According to an embodiment of the present invention, a GPU resource configuration method and device are proposed.

[0061] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0062] The principles and spirit of the present invention will be explained in detail below with reference to several representative embodiments. Invention Overview

[0064] The inventors have discovered that by pre-deploying potential inference services in the CPU's memory, the service model data can be transferred to the GPU's video memory at any time. These inference services are in an activated state, and the GPUs are centrally allocated by the GPU Management Unit. When a new task requirement arises, the GPU Management Unit allocates GPUs to the corresponding inference service, thereby changing the inference service from an activated but inactive state to an activated state. This activation process does not involve closing existing service threads or starting new service threads, avoiding the delays caused by thread start-up and shutdown, and achieving the effect of quickly starting inference services.

[0065] After introducing the basic principles of the present invention, various non-limiting embodiments of the present invention will be described in detail below.

[0066] Application Scenarios Overview

[0067] When deploying online deep learning inference services, it is usually necessary to deploy several different inference services on a server cluster. The demand for each inference service is dynamic. Therefore, if each inference service is configured with resources according to the maximum possible traffic, it will result in a large waste of resources.

[0068] First refer to Figure 2 , Figure 2 This diagram illustrates a prior art deployment method for inference services. To allocate GPUs according to the demand of the inference service, a prior art method provides monitoring traffic and adding / removing containers, which abstracts the server into several containers, each allocated a certain number of GPUs. If traffic is high, a new container is started to increase capacity; if traffic is low, some containers are shut down to free up GPUs.

[0069] However, when Figure 2 When a new task requirement arises as shown, one of the containers needs to be shut down first to release the GPU it occupies before a container can be allocated for the inference service corresponding to the new task. If the container of inference service 1 is shut down, the service thread of inference service 1 will end, and the service model of the new inference service needs to be re-initialized. The time required is usually in the second range. It is impossible to start or stop a service process in real time based on traffic, so there is a lag in the adjustment of resource volume.

[0070] Especially in scenarios requiring rapid expansion, this second-level adjustment time is insufficient to meet the requirements of rapid expansion.

[0071] Exemplary methods

[0072] It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present invention, and the embodiments of the present invention are not limited in any way. On the contrary, the embodiments of the present invention can be applied to any applicable scenario.

[0073] The acquisition, storage, and application of user personal information involved in the technical solution of this invention all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0074] To address the latency caused by thread start-stop in the prior art, especially in scenarios involving scaling up a high-demand inference service, this invention provides a GPU resource configuration method to achieve rapid scaling.

[0075] The embodiments of the present invention will now be described by way of example with reference to the accompanying drawings.

[0076] Figure 3 A schematic flowchart illustrating a GPU resource allocation method according to an embodiment of the present invention is shown. Figure 3 As shown, the GPU resource configuration method of this embodiment includes:

[0077] In step 301, in response to the inference task request, an execution server for executing the inference task request is determined from among several servers.

[0078] Before step 301 is executed, the memory of these servers contains service models for inference services with multiple GPUs to be allocated, and each server contains a GPU management unit for allocating GPUs to the inference services.

[0079] The execution server deploys the target inference service required to perform the inference task; that is, the execution server stores the service model of the target inference service in its memory.

[0080] In practical applications, several potential inference services can be pre-deployed on several servers. These potential inference services are those corresponding to tasks that the servers may execute. During deployment, these inference services can be distributed across servers; that is, each server stores a portion of the service model for the inference services to be allocated to GPUs in its memory. However, it is important to note that during deployment, it must be ensured that each inference service can be deployed on a server, so that each inference service has at least one execution server.

[0081] In other embodiments, a service model for inference services with all available GPUs can also be deployed on each server.

[0082] It is understandable that, compared to deploying all inference services on each server, by reasonably distributing potential inference services across several servers, flexible allocation of GPUs can be achieved without wasting memory space.

[0083] For ease of understanding, this invention combines... Figure 4 The schematic diagram shown illustrates a method for deploying inference services according to an embodiment of the present invention.

[0084] like Figure 4 As shown, inference services 1 and 2, which have higher demand, can be deployed on both server A and server B. The remaining inference services 3 through 10, which have lower demand, can be deployed across either server A or server B. In this deployment method, the execution servers for inference service 1 include both server A and server B, while the execution server for inference service 3 is server A.

[0085] Accordingly, the service models of inference service 1 and inference service 2 are stored in the memory of both server A and server B, while the service models of inference services 3 to 6 are stored in the memory of server A, and the service models of inference services 7 to 10 are stored in the memory of server B.

[0086] In step 302, a GPU is allocated to the target inference service through the GPU management unit.

[0087] In this embodiment of the invention, the GPUs in each server are centrally allocated by the GPU management unit. When the target inference service corresponding to the inference task in the inference task request is deployed in server A, the GPU management unit of server A allocates the computing space and usage rights of a portion of the GPUs in server A to the target inference service.

[0088] In the GPU resource allocation method provided above, since several inference service models are pre-deployed in the server's memory, but the server has not yet allocated GPUs to these inference services, each inference service deployed on the server is in a state of being started but awaiting activation. When a new task requirement arises, the GPU resource allocation method of this embodiment only needs to allocate the required GPUs to the target inference service of the task through the GPU management unit to activate the target inference service. During this process, the service thread of the inference service is always in a running state, which will not cause the closure of existing service threads or the starting of new service threads, avoiding the delay caused by thread start and stop, thereby significantly improving the speed of GPU allocation and reducing the user's response waiting time.

[0089] In some embodiments of the present invention, after scheduling the task to the corresponding execution server, it is necessary to determine the target GPU from a number of GPUs to allocate to the target inference service.

[0090] Some embodiments of the present invention also provide a GPU allocation method. Figure 5 A schematic flowchart illustrating a GPU allocation method according to an embodiment of the present invention is shown. Figure 5 As shown, some embodiments of the present invention provide a GPU allocation method that may include:

[0091] In step 501, the target GPU is determined in the GPU of the execution server.

[0092] In some embodiments, each execution server's GPU may contain a service model for inference services that includes some or all of the GPUs to be allocated in that execution server.

[0093] Still with Figure 4 Taking the deployment shown as an example, GPU1 in server A can be used to deploy one or more inference service models from inference service 1 to inference service 6. Similarly, GPU2 to GPU8 can also be used to deploy one or more inference service models from inference service 1 to inference service 6.

[0094] Assuming that inference service 1, inference service 2 and inference service 3 are deployed on GPU1, that is, the service models of inference service 1, inference service 2 and inference service 3 are stored in the video memory of GPU1, then GPU1 is considered to be a hot standby unit for inference service 1, inference service 2 and inference service 3.

[0095] It can be understood that the hot standby unit for the inference service is the GPU that only stores the service model of the inference service in the model space.

[0096] In some embodiments, the service model may be stored on the server in the following ways:

[0097] If the inference service has a hot standby unit deployed on the current server, the service model of the inference service is stored in the hot standby unit, and the service model image of the inference service is stored in the memory of the current server; if the inference service does not have a hot standby unit deployed on the current server, the service model image of the inference service is stored in the memory of the current server.

[0098] If GPU2 is currently occupied by inference service 1, that is, if GPU2's video memory stores the service model of inference service 1, and the right to use GPU2 belongs to inference service 1, then GPU1 is considered to be the capacity unit of inference service 1.

[0099] It can be understood that the capacity unit of an inference service is the GPU of computing space occupied by that inference service.

[0100] In this embodiment, when deploying inference services on a server, at least one capacity unit and at least one hot standby unit can be deployed in at least one server for each inference service, so that in step 501, a hot standby unit of the target inference service is determined as the target GPU.

[0101] Furthermore, when determining the target GPU, the workload of the task can be compared with the capacity of the target inference service's capacity unit. If the capacity of the target inference service's capacity unit is sufficient to handle the workload of the task, there is no need for expansion. If the difference between the capacity of the target inference service's capacity unit and the workload of the task is less than a preset value, it means that the workload of the task exceeds the current capacity of the target inference service's capacity unit. In this case, the operation of activating the hot standby unit of the target inference service as a capacity unit is performed to expand the capacity of the target inference service.

[0102] For each inference service, at least one capacity unit and at least one hot standby unit are deployed in at least one server. When a new task request is generated, the hot standby unit is activated as a capacity unit to realize GPU resource allocation. Since the service model of the target inference service is stored in the hot standby unit, the target inference service can be activated to execute the task without waiting for the transfer time of model data from memory to GPU memory. The response speed is fast and can meet the real-time requirements of the inference service.

[0103] In step 502, the target GPU is allocated to the target inference service via the GPU management unit.

[0104] In this embodiment, the GPU management unit allocates the usage rights and computing space of the target GPU to the target inference service.

[0105] As mentioned earlier, each execution server's GPU can deploy service models of some or all of the inference services to be allocated to the execution server. In other words, a GPU's video memory can store service models of multiple inference services.

[0106] This can be understood as multiple inference services sharing a single hot standby unit. Correspondingly, each inference service can also have multiple hot standby units.

[0107] Still with Figure 4Taking the deployment shown as an example, assume GPU1 is a hot standby unit for Inference Service 1, Inference Service 2, and Inference Service 3. In this case, Inference Service 2 and Inference Service 3 are considered shared objects of Inference Service 1. When GPU1 is identified as the target GPU for Inference Service 1, GPU1 is activated from a hot standby unit to a capacity unit. At this time, the computing space and usage rights of GPU1 are occupied by Inference Service 1. Correspondingly, GPU1 cannot be activated as a capacity unit for Inference Service 2 or Inference Service 3 when it is occupied. Therefore, after the GPU management unit allocates the usage rights and computing space of GPU1 to Inference Service 1, GPU1 also needs to be removed from the hot standby unit set of Inference Service 2 and Inference Service 3.

[0108] It is understandable that the steps for allocating the target GPU to the target inference service include:

[0109] The GPU management unit allocates the usage rights and computing space of the target GPU to the target inference service;

[0110] Remove the activated hot standby unit GPU1 from the hot standby unit set of the shared object; where the shared object is the inference service that shares a hot standby unit with the target inference service.

[0111] By enabling each inference service to have multiple hot standby units, and allowing each hot standby unit to store service models of various inference services in its video memory, the video memory of the hot standby unit can be shared by multiple inference services. Then, through combination and adjustment, the maximum expansion capability can be achieved in the limited video memory space.

[0112] Furthermore, in the process of GPU resource allocation, in addition to the situation where the load increases and needs to be expanded, there is also a scenario where the task load decreases. In this scenario, the capacity units of the inference service with decreased load can be released to ensure the number of hot standby units.

[0113] Specifically, some embodiments of the GPU resource configuration method provided by the present invention further include:

[0114] In response to a capacity release signal, determine the capacity unit to be released among multiple capacity units of the inference service corresponding to the capacity release signal;

[0115] Release the computing space and usage rights of the capacity unit to be released, so that the capacity unit to be released can be restored as a hot standby unit.

[0116] The capacity release signal is used to indicate that the difference between the capacity of the corresponding inference service's capacity unit and the load of the corresponding inference service is greater than a preset value.

[0117] Furthermore, since it is difficult to predict the changing trend of task load when setting up the initial service deployment plan, there may be a situation where the number of hot standby units between various services differs too much in the future.

[0118] To avoid an imbalance in the number of hot standby units among inference services, some embodiments of the present invention provide a method for switching between hot and cold standby inference services. This method can be applied to GPU resource allocation methods to convert some hot standby units to cold standby units and to add hot standby units to some inference services in a cold standby state. Here, a cold standby state refers to an inference service that does not currently have a hot standby unit on the server.

[0119] In a cold standby state, the inference service's service model is stored only in memory on the current server, not in the GPU's video memory. When the inference service in this cold standby state is activated, the service model in memory needs to be transferred to the video memory, which takes milliseconds to start. If the inference service is in a hot standby state, the service model is already stored in the video memory, and activating the inference service only takes microseconds.

[0120] Therefore, in order to improve the response speed of the inference service, switching the inference service from cold standby to hot standby in advance can effectively reduce latency.

[0121] Correspondingly, some embodiments of the present invention provide a method for switching between hot and cold inference services, which includes: in response to a hot / cold switching signal, performing a cold switching process on a portion of the hot standby units of the inference service to be switched to cold, and adding a hot standby unit for the inference service to be switched to hot.

[0122] The hot / cold switching signal indicates that the number of hot standby units for the inference service is less than a preset number. Optionally, the preset number can be set to 2.

[0123] It should be noted that the above numerical settings for the preset quantity are merely an example and do not constitute a limitation of the present invention.

[0124] In this embodiment, the cooling process includes removing the service model of the inference service to be cooled from the hot standby unit. The service model image of the inference service to be cooled is stored in the current server's memory, so that the service model can be moved back to the video memory during subsequent cooling-to-hot conversion. When the hot standby units for the inference service to be cooled are insufficient, additional hot standby units can be added to the current server, allowing the cooled inference service to return to a hot standby state.

[0125] Furthermore, in practical applications, to fully utilize the video memory space, the service models for inference services are stored starting from the end of the video memory, ensuring their space is arranged contiguously. However, when one of the inference services becomes less popular, the space occupied by its service model becomes free, potentially resulting in fragmented space.

[0126] To facilitate understanding, the following will be combined with... Figure 6 Explanation of the hot-cold conversion process.

[0127] like Figure 6 As shown, the service models of Inference Service 1 and Inference Service 2 are stored in the GPU's video memory, and their service model images are stored in memory; the service model of Inference Service 3 is stored in memory. Inference 1 and Inference Service 2 are in hot standby mode, while Inference Service 3 is in cold standby mode.

[0128] The service model of inference service 3 is stored in the GPU's video memory, which makes inference service 3 go from cold to hot. At this time, the service model image of inference service 3 is stored in memory.

[0129] When the demand for inference service 2 drops significantly, the service model of inference service 2 can be moved out of video memory, and inference service 2 will go from hot to cold. Since the service models of inference service 1, inference service 2 and inference service 3 are stored in video memory in a contiguous manner, moving the service model of inference service 2 out of video memory will create a fragmented space between inference service 1 and inference service 3.

[0130] After multiple temperature changes, the video memory will become too fragmented, which is not conducive to the full utilization of the video memory.

[0131] To address the aforementioned issues, some embodiments of the present invention further perform the following operation during the cooling process: after removing the service model of the inference service to be cooled, the service models retained in the hot standby unit are moved so that the service models retained in the hot standby unit are arranged consecutively.

[0132] by Figure 6 For example, by moving the service model of inference service 3, the service models of inference service 1 and inference service 3 are arranged consecutively starting from the end of the video memory.

[0133] As described above, in this invention, an inference service can have multiple hot standby units. When determining the target GPU of the target inference service, one or more of the multiple hot standby units of the target inference service need to be selected and activated as capacity units.

[0134] Based on this, the present invention also provides a search method based on maximizing the total expansion of hot standby units, which can be applied to the GPU resource configuration method of the present invention to determine the target GPU from multiple hot standby units.

[0135] Furthermore, this search method based on maximizing the total capacity expansion of hot standby units can also be applied to the release process of capacity units and / or the hot-cold conversion process of inference services.

[0136] The following is combined Figure 7 The method for determining the target GPU based on maximizing the total capacity expansion of hot spare units is explained.

[0137] When the above search method based on maximizing the total expansion of hot standby units is applied to... Figure 5 In the method shown, the process of determining the target GPU is as follows:

[0138] In step 701, the hot standby unit to be activated is determined from multiple hot standby units of the target inference service based on maximizing the total expansion of the hot standby units.

[0139] The scalability of a hot standby unit is negatively correlated with the number of existing hot standby units in its inference service. In other words, the fewer existing hot standby units a inference service has, the higher the scalability of the hot standby units for that inference service.

[0140] Since a hot standby unit can belong to multiple inference services, the process of calculating the expansion limit of a hot standby unit can be as follows: sum the expansion limits of the hot standby unit to each of its respective inference services to obtain the expansion limit of the hot standby unit.

[0141] For example, the hot standby unit GPU1 is equipped with inference service 1 and inference service 2. The expansion capacity of GPU1 relative to inference service 1 is value1; the expansion capacity of GPU1 relative to inference service 2 is value2. Therefore, the expansion capacity of GPU1 is equal to value1 + value2.

[0142] Specifically, in step 701, it is necessary to traverse multiple hot standby units of the target inference service and calculate the loss value of the total expansion capacity of the hot standby units after each hot standby unit is activated, and take the hot standby unit corresponding to the minimum loss value as the hot standby unit to be activated.

[0143] In step 701, the expansion degree of the activated hot standby unit for each shared object is summed to obtain the loss value of the total expansion degree of the hot standby unit after activation.

[0144] In step 702, the hot standby unit to be activated is activated as a capacity unit.

[0145] In step 702, the right to use and computing space of the hot standby unit to be activated are allocated to the target inference service.

[0146] The following is combined Figure 8The application of the search method based on maximizing the total capacity expansion of hot standby units in the capacity release process is explained.

[0147] When the above search method based on maximizing the total capacity expansion of hot standby units is applied to the capacity unit release process, the capacity unit release process is as follows:

[0148] In step 801, in response to the capacity release signal, the capacity cell to be released is determined among multiple capacity cells based on maximizing the total capacity expansion of the hot standby cells.

[0149] Specifically, in step 801, it is necessary to traverse multiple capacity units of the inference service corresponding to the capacity release signal, calculate the output value of the total expansion of the hot standby unit after the recovery of each capacity unit, and take the capacity unit corresponding to the largest output value as the capacity unit to be released.

[0150] The calculation process for the total expansion value of the hot standby unit after each capacity unit is restored is as follows: sum the expansion values ​​of the restored hot standby unit for each shared object to obtain the total expansion value of the hot standby unit after the capacity unit is restored.

[0151] In step 802, the computing space and usage rights of the capacity unit to be released are released so that the capacity unit to be released is restored as a hot standby unit.

[0152] The following is combined Figure 9 The application of a search method based on maximizing the total expansion of hot standby units in the hot-to-cold transition process of inference services is explained.

[0153] When the search method based on maximizing the total capacity expansion of hot standby units is applied to the hot / cold transition process of inference services, the hot / cold transition process of inference services is as follows:

[0154] In step 901, in response to the hot / cold conversion signal, the hot standby service to be converted to cold inference, the hot standby unit to be converted to cold processing, the hot inference service to be converted to hot inference, and the added hot standby unit are determined based on maximizing the total expansion capacity of the hot standby unit.

[0155] Specifically, in step 901, each combination of hot standby units to be converted to cold processing and added hot standby units is traversed, the loss value and output value of the total expansion of the hot standby units under each combination are calculated, the total value-added under each combination is calculated based on the loss value and output value of the total expansion of the hot standby units, and the combination corresponding to the maximum total value-added is selected to execute step 902.

[0156] For example, the total added value of the heat-cooling conversion is equal to the difference between the absolute value of the output and the absolute value of the loss.

[0157] Similar to the previous example, the output value of the total expansion capacity of the hot standby unit can be obtained by summing the expansion capacity of the recovered hot standby unit for each shared object; the loss value of the total expansion capacity of the hot standby unit can be obtained by summing the expansion capacity of the activated hot standby unit for each shared object.

[0158] In step 902, the hot standby unit to be switched to cold processing is switched to cold processing, and a hot standby unit is added for the inference service to be switched to hot processing.

[0159] Similar to the previous example, the cooling process may include: removing the service model of the inference service to be cooled from the hot standby unit. Further, the cooling process may also include: after removing the service model of the inference service to be cooled, moving the service models retained in the hot standby unit so that the service models retained in the hot standby unit are arranged consecutively.

[0160] The search method based on maximizing the total expansion of hot standby units provided in the above embodiments aims to maximize the total expansion of hot standby units. It selects hot standby units to be activated and / or capacity units to be released. By constructing the relationship between the expansion of hot standby units and inference services during the target GPU determination process, the capacity unit release process, and the hot-cold conversion process of inference services, it provides a quantifiable reference for the GPU allocation decision process, thereby making GPU allocation more reasonable and reliable.

[0161] Exemplary device

[0162] After introducing the method of exemplary embodiments of the present invention, the following references are made. Figure 10 A GPU resource configuration apparatus according to an exemplary embodiment of the present invention will be described.

[0163] Figure 10 A schematic block diagram of a GPU resource configuration apparatus according to an embodiment of the present invention is shown.

[0164] The GPU resource allocation apparatus of this invention includes a task scheduling center and several servers. Each server contains several GPUs and a GPU management unit for allocating GPUs for inference services.

[0165] Several servers have service models for inference services with multiple GPUs to be allocated deployed in their memory; wherein, each server stores several service models for inference services with multiple GPUs to be allocated in its memory.

[0166] The task scheduling center is used to respond to inference task requests, determine the execution server among several servers to execute the inference task request, and issue a GPU allocation instruction to the GPU management unit of the execution server; wherein, the execution server is a server that has deployed the target inference service required to execute the task.

[0167] The GPU management unit is used to allocate GPUs to the target inference service according to the GPU allocation instructions issued by the task scheduling center.

[0168] Corresponding to the aforementioned functional embodiments, this invention also provides a method such as... Figure 11 The electronic device shown. Figure 11 A schematic block diagram of an electronic device according to an embodiment of the present invention is shown.

[0169] Figure 11 The illustrated electronic device 1100 includes: a processor 1110; and a memory 1120, which stores executable program instructions that, when executed by the processor 1110, cause the electronic device to perform any of the methods described above.

[0170] exist Figure 11 In the electronic device 1100, only the components relevant to this embodiment are shown. Therefore, it will be apparent to those skilled in the art that the electronic device 1100 may also include components related to... Figure 11 The following are common constituent elements with different constituent elements.

[0171] Processor 1110 can control the operation of electronic device 1100. For example, processor 1110 controls the operation of electronic device 1100 by executing a program stored in memory 1120 on electronic device 1100. Processor 1110 can be implemented by a central processing unit (CPU), application processor (AP), artificial intelligence processor chip (IPU), etc., provided in electronic device 1100. However, this disclosure is not limited thereto. In this embodiment, processor 1110 can be implemented in any suitable manner. For example, processor 1110 can take the form of, for example, a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers, etc.

[0172] The memory 1120 can be used to store various data and instructions processed in the electronic device 1100. For example, the memory 1120 can store processed data and data to be processed in the electronic device 1100. The memory 1120 can store datasets that have been processed or are to be processed by the processor 1110, such as user input data, cached data, and cache index information. In addition, the memory 1120 can store applications, drivers, etc., to be driven by the electronic device 1100. For example, the memory 1120 can store various programs related to task type identification, operator type identification, etc., to be executed by the processor 1110. The memory 1120 can be DRAM, but this disclosure is not limited thereto. The memory 1120 can include at least one of volatile memory or non-volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, memory 1120 may include at least one of hard disk drive (HDD), solid-state drive (SSD), high-density flash memory (CF), secure digital card (SD), micro-secure digital card (Micro-SD), mini-secure digital card (Mini-SD), extreme digital card (xD), cache, or memory stick.

[0173] In summary, the specific functions implemented by the memory 1120 and processor 1110 of the electronic device 1100 provided in this specification can be explained in comparison with the aforementioned embodiments in this specification, and can achieve the technical effects of the aforementioned embodiments. Therefore, they will not be repeated here.

[0174] Alternatively, this disclosure may be implemented as a non-transitory machine-readable storage medium (or computer-readable storage medium, or machine-readable storage medium) storing computer program instructions (or computer program, or computer instruction code) that, when executed by a processor of an electronic device (or electronic device, server, etc.), cause the processor to perform some or all of the steps of the methods described above according to this application.

[0175] It should be noted that although several devices or sub-devices of the electronic device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0176] It should be noted that although several devices or sub-devices of the device have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of the invention, the features and functions of two or more devices described above can be embodied in one device. Conversely, the features and functions of one device described above can be further divided and embodied by multiple devices.

[0177] Furthermore, although the operations of the method of the present invention are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Rather, the steps depicted in the flowchart may be performed in a different order. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0178] The use of the verbs "including" and "contains" and their inflections in the application documents does not preclude the existence of elements or steps other than those described in the application documents. The article "a" or "one" preceding an element does not preclude the existence of multiple such elements.

[0179] While the spirit and principles of the invention have been described with reference to several specific embodiments, it should be understood that the invention is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for ease of description. The invention is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is to be interpreted in the broadest sense, thereby encompassing all such modifications and equivalent structures and functions.

Claims

1. A GPU resource allocation method, characterized in that, include: In response to an inference task request, an execution server is determined from among several servers to perform the inference task request; The servers contain multiple service models for inference services deployed in their memory, and each server has a GPU management unit deployed for allocating GPUs for the inference services. Among them, for at least one of the multiple inference services, at least one hot standby unit is deployed for it in the plurality of servers, wherein the hot standby unit is a GPU that only stores the service model of the inference service in the model space; The execution server is a server that deploys the target inference service required to execute the inference task request, and the execution server has at least one hot standby unit, the model space of the at least one hot standby unit stores the service model of the target inference service; The GPU management unit of the execution server activates the hot standby unit storing the target inference service model as a capacity unit to allocate GPUs to the target inference service, wherein the capacity unit is the GPU whose computing space is occupied by the inference service.

2. The GPU resource allocation method according to claim 1, characterized in that, Each of the plurality of servers stores part or all of the service models of the plurality of inference services in its memory, such that each inference service has at least one execution server among the plurality of servers.

3. The GPU resource allocation method according to claim 1, characterized in that, Before determining the execution server among several servers to execute the inference task request in response to the inference task request, the method further includes: For each inference service, deploy at least one capacity unit and at least one hot standby unit in at least one server.

4. The GPU resource allocation method according to claim 3, characterized in that, If the inference service has a hot standby unit deployed on the current server, the service model of the inference service is stored in the hot standby unit, and the service model image of the inference service is stored in the memory of the current server. If the inference service does not have a hot standby unit deployed on the current server, the service model image of the inference service is stored in the memory of the current server.

5. The GPU resource allocation method according to claim 3, characterized in that, The hot standby unit storing the target inference service model is activated as a capacity unit through the GPU management unit of the execution server, including: If the difference between the capacity of the target inference service's capacity unit and the load required by the inference task request is less than a preset value, then the hot standby unit of the target inference service is activated as a capacity unit, so that the target inference service occupies the computing space and usage rights of the hot standby unit.

6. The GPU resource allocation method according to claim 5, characterized in that, Each hot standby unit stores at least one service model for inference services. If a hot standby unit stores multiple service models, then the multiple inference services corresponding to the multiple service models share the hot standby unit. Each inference service has multiple hot standby units; Accordingly, activating the hot standby unit of the target inference service as a capacity unit includes: Activate one or more hot standby units of the target inference service as capacity units; Remove the activated hot standby unit from the hot standby unit set of the shared object; The shared object is the reasoning service that shares a hot standby unit with the target reasoning service.

7. The GPU resource allocation method according to claim 6, characterized in that, Activating one or more hot standby units of the target inference service as capacity units includes: The hot standby unit to be activated is determined from multiple hot standby units of the target inference service based on maximizing the total expansion capacity of the hot standby unit. Activate the hot standby unit to be activated as a capacity unit; The expansion capacity of a hot standby unit is negatively correlated with the number of existing hot standby units in its inference service.

8. The GPU resource allocation method according to claim 7, characterized in that, The step of determining the hot standby unit to be activated from among the multiple hot standby units of the target inference service based on maximizing the total expansion capacity of the hot standby units includes: Iterate through multiple hot standby units of the target inference service and calculate the loss value of the total expansion of the hot standby units after each hot standby unit is activated. Take the hot standby unit corresponding to the minimum loss value as the hot standby unit to be activated.

9. The GPU resource allocation method according to claim 8, characterized in that, The calculation of the loss value of the total capacity expansion of the hot standby units after each hot standby unit is activated includes: The expansion degree of each shared object of the activated hot standby unit is summed to obtain the total expansion degree loss of the hot standby unit after activation.

10. The GPU resource allocation method according to claim 3, characterized in that, Also includes: In response to a capacity release signal, a capacity unit to be released is determined from among a plurality of capacity units of the inference service corresponding to the capacity release signal; Release the computing space and usage rights of the capacity unit to be released, so that the capacity unit to be released can be restored as a hot standby unit; The capacity release signal is used to indicate that the difference between the capacity of the corresponding inference service's capacity unit and the load of the corresponding inference service is greater than a preset value.

11. The GPU resource allocation method according to claim 10, characterized in that, The step of determining the capacity unit to be released among the multiple capacity units of the inference service corresponding to the capacity release signal includes: The capacity unit to be released is determined from the plurality of capacity units based on maximizing the total expansion of the hot standby unit. The expansion capacity of a hot standby unit is negatively correlated with the number of existing hot standby units in its inference service.

12. The GPU resource allocation method according to claim 11, characterized in that, The step of determining the capacity unit to be released from the plurality of capacity units based on maximizing the total capacity expansion of the hot standby unit includes: Iterate through multiple capacity units corresponding to the inference service of the capacity release signal, and calculate the output value of the total expansion of the hot standby unit after the recovery of each capacity unit. Take the capacity unit corresponding to the largest output value as the capacity unit to be released.

13. The GPU resource allocation method according to claim 12, characterized in that, The calculation of the output value of the total capacity expansion of the hot standby unit after the recovery of each capacity unit includes: The total capacity expansion of the hot standby units after recovery is summed for each shared object to obtain the output value of the total capacity expansion of the hot standby units after capacity unit recovery.

14. The GPU resource allocation method according to claim 3, characterized in that, Also includes: In response to the hot / cold conversion signal, some hot standby units of the inference service to be switched to cold are switched to cold, and hot standby units are added for the inference service to be switched to hot. The hot / cold conversion signal is used to indicate that the number of hot standby units for the inference service is less than a preset number; the cold conversion process includes removing the service model of the inference service to be converted to cold from the hot standby units.

15. The GPU resource allocation method according to claim 14, characterized in that, The cooling process further includes: after removing the service model of the inference service to be cooled, moving the service models retained in the hot standby unit so that the service models retained in the hot standby unit are arranged continuously.

16. The GPU resource allocation method according to claim 14, characterized in that, The process of converting some hot standby units of the inference service to cold mode and adding hot standby units for the inference service to hot mode includes: The hot standby service to be converted to cold inference service, the hot standby unit to be converted to cold processing, the hot inference service to be converted to hot inference service, and the newly added hot standby unit are determined based on maximizing the total expansion of the hot standby unit. Perform a cooling process on the hot standby unit to be switched to cold processing, and add a hot standby unit for the hot inference service to be switched to hot processing; The expansion capacity of a hot standby unit is negatively correlated with the number of existing hot standby units in its inference service.

17. The GPU resource allocation method according to claim 16, characterized in that, The method of determining the hot standby service to be converted to cold inference, the hot standby unit to be converted to cold processing, the hot inference service to be converted to hot inference, and the added hot standby unit based on maximizing the total expansion capacity of the hot standby unit includes: Iterate through each combination of hot standby units to be converted to cold processing and added hot standby units, calculate the loss value and output value of the total expansion of hot standby units under each combination, calculate the total value-added under each combination based on the loss value and output value of the total expansion of hot standby units, and select the combination corresponding to the maximum total value-added to execute the conversion to cold processing and the addition of hot standby units.

18. A GPU resource allocation device, characterized in that, include: The system includes a task scheduling center and several servers; each server contains several GPUs and a GPU management unit for allocating GPUs for inference services. The servers have multiple inference service models deployed in their memory; for at least one of the multiple inference services, at least one hot standby unit is deployed on the servers, and the hot standby unit is a GPU that stores only the service model of the inference service in its model space. The task scheduling center is used to respond to inference task requests, determine the execution server to execute the inference task request from among several servers, and issue a GPU allocation instruction to the GPU management unit of the execution server; wherein, the execution server is a server that has deployed the target inference service required to execute the inference task request, and the execution server has at least one hot standby unit, and the model space of the at least one hot standby unit stores the service model of the target inference service. The GPU management unit is used to activate the hot standby unit storing the target inference service model as a capacity unit according to the GPU allocation instruction issued by the task scheduling center, so as to allocate GPUs to the target inference service. The capacity unit is the GPU whose computing space is occupied by the inference service.

19. An electronic device, characterized in that, include: processor; as well as A memory that stores executable program instructions, which, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-17.

20. A computer-readable storage medium, characterized in that, It stores computer program instructions that, when executed by one or more processors, cause the processors to perform the method as described in any one of claims 1-17.

Images