A graphics processor resource management system, method and server

Abstract

The application relates to the technical field of computer processors, and discloses a graphics processor resource management system, a graphics processor resource management method and a server. The resource management system comprises an inference engine module and an inference service module. The graphics processor resource pool comprises a plurality of graphics processors, and the graphics processors are configured with first identifiers. The inference service module is used for generating a message request indicating the starting of a target inference instance, and is used for sending the message request to the inference engine module. The message request comprises a second identifier and a third identifier. The inference engine module is used for determining a target graphics processor from the plurality of graphics processors according to the first identifier, the second identifier, the third identifier and occupation state information of the plurality of graphics processors, and is used for sending a response message to the inference service module. The inference service module is further used for mounting the target inference instance to the target graphics processor according to the response message. The application can make a plurality of inference instances more reasonably occupy the resources of the same GPU, and improve the utilization rate of the GPU.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer processors, in particular to a graphics processor resource management system, method and server. BACKGROUND

[0002] With the rapid development of artificial intelligence (AI) and big data, using machine learning technology to train business models and using trained business models to realize intelligent processing of big data business has gradually become a general means in the big data industry. For a specific business model, after training the business model in a specified training environment, the business model is deployed as an online inference service, which can be used when a user uses the same running environment as the training environment.

[0003] The execution of the inference service depends on graphics processor (GPU) resources. At present, in order to improve the resource utilization of the GPU, multi-instance GPU (MIG) division is performed, and the plurality of "virtual GPUs" (GPU instances) obtained by the division are provided to a plurality of inference services. However, due to the mechanism of MIG, the MIG division is usually equal division, which makes it difficult to fully utilize the GPU resources. SUMMARY

[0004] Therefore, the present application provides a graphics processor resource management system, method and server to solve the problem that a plurality of inference services cannot fully utilize GPU resources.

[0005] In a first aspect, the present application provides a graphics processor resource management system, which includes an inference engine module and an inference service module. The inference engine module and the inference service module are in communication connection with a graphics processor resource pool. The graphics processor resource pool includes a plurality of graphics processors. The graphics processors are configured with a first identifier, which is used to indicate whether the graphics processors can be shared by a plurality of inference instances. The inference service module is used to generate a message request indicating the start of a target inference instance, and is used to send the message request to the inference engine module. The message request includes a second identifier and a third identifier. The second identifier is used to indicate whether the target inference instance occupies a shareable graphics processor. The third identifier is used to indicate the target memory value required by the target inference instance after starting. The inference engine module is used to determine a target graphics processor from the plurality of graphics processors according to the first identifier, the second identifier, the third identifier and the occupancy state information of the plurality of graphics processors, and is used to send a response message to the inference service module. The response message includes indication information of the target graphics processor. The inference service module is further used to mount the target inference instance to the target graphics processor according to the response message.

[0006] The inference engine module configures a first identifier for the GPU, and the inference service module adds a second identifier and a third identifier in a message request sent to the inference engine module, so that the inference engine module can select a more suitable GPU for the inference instance, multiple inference instances can more reasonably occupy the same GPU, and GPU resources are used in a refined manner to reduce costs. At the same time, the inference engine module and the inference service module cooperate with each other, which can also improve the GPU resource utilization efficiency and optimize the resource energy consumption.

[0007] In an optional implementation, the inference engine module is configured to, when the second identifier indicates that the target inference instance occupies a shareable GPU, select at least one first GPU from the plurality of GPUs according to the first identifier, the first GPU being a shareable GPU; the inference engine module is further configured to determine whether a remaining memory value of the at least one first GPU is greater than a target memory value according to the occupancy state information of the at least one first GPU; and the inference engine module is further configured to determine one of the at least one second GPU as the target GPU, the second GPU being the first GPU whose remaining memory value is greater than the target memory value.

[0008] In an optional implementation, the inference engine module is configured to determine the second GPU whose difference between the remaining memory value and the target memory value is greater than a preset memory value as the target GPU.

[0009] In this embodiment, the second GPU whose difference between the remaining memory value and the target memory value is greater than a preset memory value is determined as the target GPU, so that the target GPU still has a resource space after mounting the target inference instance, preventing memory overflow and GPU crash, and ensuring service stability and reliability.

[0010] In an optional implementation, in the case where there is no first GPU or there is no second GPU, the inference engine module is further configured to select at least one third GPU from the plurality of GPUs according to the occupancy state information of the plurality of GPUs and the first identifier, the third GPU being an unshareable and unoccupied GPU; and the inference engine module is further configured to determine one of the at least one third GPU as the target GPU.

[0011] In an optional implementation, the inference service module is further configured to update the occupancy state information of the target GPU after the target inference instance is successfully started, and to synchronize the updated occupancy state information of the target GPU and information generated by the target inference instance to the inference engine module.

[0012] In an optional implementation, the inference engine module is further configured to update the first identifier corresponding to the target graphics processor to change the target graphics processor to the sharable graphics processor after determining the target graphics processor.

[0013] In an optional implementation, the inference engine module is further configured to, when the second identifier indicates that the target inference instance occupies the unshareable graphics processor, select at least one third graphics processor from the plurality of graphics processors according to the occupation state information of the plurality of graphics processors and the first identifier; and determine, as the target graphics processor, the third graphics processor with a display memory value greater than the target display memory value.

[0014] In an optional implementation, the plurality of graphics processors are deployed on different servers.

[0015] In a second aspect, the present application provides a graphics processor resource management method, which is applied to the inference engine module of the graphics processor resource management system in the first aspect or any of the corresponding implementations, and includes: receiving a packet request from an inference service module, the packet request including a second identifier and a third identifier, the second identifier being used to indicate whether a target inference instance occupies a sharable graphics processor, and the third identifier being used to indicate a target display memory value required by the target inference instance after being started; determining a target graphics processor from a plurality of graphics processors according to a first identifier, the second identifier, the third identifier and occupation state information of the plurality of graphics processors, the first identifier being used to indicate whether a graphics processor can be shared by a plurality of inference instances; and sending a response message to the inference service module, the response message including indication information of the target graphics processor, so as to enable the target inference instance to be mounted to the target graphics processor.

[0016] The graphics processor resource management method provided in this embodiment increases three identifier fields for configuration when starting an inference instance, and the combination facilitates the new creation of inference instances and the business, so that a plurality of inference instances can share the same GPU, and the GPU resources are used more efficiently.

[0017] In a third aspect, the present application provides a server, which includes the graphics processor resource management system in the first aspect or any of the corresponding implementations. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the specific embodiments or related art, the following will briefly introduce the drawings needed to be used in the specific embodiments or related art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0019] Figure 1 Fig. 1 is a structural schematic diagram of a graphics processor resource management system and a graphics processor resource pool according to an embodiment of the present application;

[0020] Figure 2 Fig. 2 is a schematic diagram of an interaction process of a graphics processor resource management system according to an embodiment of the present application;

[0021] Figure 3 Fig. 3 is a flow schematic diagram of a graphics processor resource management method according to an embodiment of the present application;

[0022] Figure 4 Fig. 4 is a flow schematic diagram of another graphics processor resource management method according to an embodiment of the present application;

[0023] Figure 5 Fig. 5 is a structural schematic diagram of a server according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. According to the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0025] In order to facilitate understanding of the present application, before the technical solutions of the present application are described, the related terms involved in the present application are first introduced.

[0026] (1) GPU

[0027] GPU is a hardware device specially used for processing graphics and image calculations, and is widely used in fields such as graphics rendering, game development, video processing, scientific calculation, machine learning, etc. A GPU card usually contains an independent graphics processing chip, as well as related video memory and external interfaces. Due to the powerful parallel processing capability of GPU, compared with the central processing unit (CPU), the GPU card can simultaneously process a large amount of data and perform high-speed parallel calculation, and therefore is used to accelerate various computation-intensive tasks.

[0028] (2) Inference service

[0029] The process of generating information (Response) that a user wants to obtain using a trained AI large model and according to a prompt (Prompt) input by the user is called "model inference". That is, the inference service is the process of deploying a trained model online for real-time prediction.

[0030] With the rapid development of AI technology, the demand for computationally intensive tasks such as deep learning and neural networks has increased significantly. To meet these demands, special accelerators such as GPUs have been introduced in the design of computing hardware. However, how to efficiently utilize a single GPU for inference services of small models has become a challenge. Small models refer to AI models with small parameter quantities and low computational requirements, and the GPU memory usage of small models is generally below 10 GB.

[0031] As mentioned in the background, the GPU can be divided into MIG, enabling a single GPU to support inference services of multiple small models. However, the above method has the problems of being difficult to flexibly divide as needed, only supporting NVIDIA (N card) and MIG, and being inconvenient to extend before the model starts inference service.

[0032] Therefore, the present application provides a graphics processor resource management system, method and server, which can more reasonably occupy the resources of the same GPU for multiple inference instances by identifying and managing multiple inference instances and GPUs, thereby improving GPU utilization.

[0033] The graphics processor resource management system provided by the present application will be described in detail below with reference to the accompanying drawings.

[0034] As shown in Figure 1 The graphics processor resource management system 100 includes an inference engine module 110 and an inference service module 120, both of which are in communication connection with a graphics processor resource pool 200. The inference engine module 110 and the inference service module 120 are also in communication connection with each other.

[0035] The graphics processor resource pool 200 includes multiple GPUs, which can be deployed on different servers (Server) or on the same server. Specifically, the graphics processor resource pool 200 can include a server in which multiple GPUs are deployed; the graphics processor resource pool 200 can also include multiple servers, each server deploying one or more GPUs.

[0036] Figure 1 For example, the graphics processor resource pool 200 includes two servers (referred to as a first server 1 and a second server 2), but is not limited thereto. In Figure 1In the example shown in FIG. 1, each server includes 4 GPUs, the first server 1 includes GPUs 11-14, and the second server 2 includes GPUs 21-24.

[0037] In particular, each GPU is configured with a first identifier, which is used to indicate whether the GPU can be shared by multiple inference instances. For example, the first identifier can be a field Gpu Is Share. When the field Gpu Is Share is 1, it indicates that the GPU can be shared by multiple inference instances (inference models). When the field Gpu Is Share is 0, it indicates that the GPU cannot be shared by multiple inference instances and is generally exclusively used by a single inference instance.

[0038] The inference engine module 110 can configure the first identifier when initializing the GPUs of the server. The inference engine module 110 can determine whether a GPU can be shared according to the GPU memory value and / or business requirements. For example, the inference engine module 110 can determine that a GPU with a GPU memory value greater than a preset value (e.g., 10G) is a shareable GPU, and the inference engine module 110 can determine that all GPUs except for GPUs designated by a specific business are shareable GPUs.

[0039] The inference service module 120 is configured to generate a message request indicating the start of a target inference instance and to send the message request to the inference engine module 110. In particular, the inference service module 120 generates the message request in response to a user operation. The inference service module 120 includes multiple inference instances, and the target inference instance can be one of the inference instances selected by the user from the multiple inference instances.

[0040] The message request includes a second identifier and a third identifier. The second identifier is used to indicate whether the target inference instance occupies a shareable GPU, and the third identifier is used to indicate the required GPU memory size (target GPU memory value) after the target inference instance is started.

[0041] For example, the second identifier can be a field GPU Utlization Share Enabled, and the third identifier can be a field Gpu Memory Use. When the field GPU Utlization Share Enabled is 1, it indicates that a shareable GPU is preferentially selected to start the target inference instance. When the field GPU Utlization Share Enabled is 0, it indicates that an exclusive GPU is selected to start the target inference instance. The value (INT value) corresponding to the field Gpu Memory Use is the target GPU memory value. For example, the target GPU memory value can be 5G, 10G, or 20G, etc.

[0042] The inference engine module 110 is configured to determine a target graphics processor from the plurality of GPUs according to the first identifier, the second identifier, the third identifier and the occupation state information of the plurality of GPUs after receiving the message request. The occupation state information includes information about whether each GPU is occupied and a GPU occupied memory value. The inference engine module 110 can interact with the servers in the graphics processor resource pool 200 to collect the occupation state information of the plurality of GPUs. The target graphics processor is a GPU selected by the inference engine module 110 from the plurality of GPUs to meet the running requirements of the target inference instance.

[0043] For example, after obtaining the occupation state information of the plurality of GPUs, the inference engine module 110 can synchronize the occupation state information of the plurality of GPUs to the inference service module 120.

[0044] After determining the target graphics processor, the inference engine module 110 is further configured to send a response message to the inference service module 120, wherein the response message includes indication information of the target graphics processor. After receiving the response message, the inference service module 120 is further configured to mount the target inference instance to the target graphics processor according to the response message.

[0045] The graphics processor resource management system provided in the embodiment can configure the first identifier for the GPU by the inference engine module 110, and add the second identifier and the third identifier in the message request sent to the inference engine module 110 by the inference service module 120, so that the inference engine module 110 can select more suitable GPUs for the plurality of inference instances. The plurality of inference instances can more reasonably occupy the same GPU, finely use the GPU resources, and reduce the cost. At the same time, the inference engine module 110 and the inference service module can cooperate with each other to improve the GPU resource utilization process and optimize the resource energy consumption.

[0046] Specifically, when the second identifier indicates that the target inference instance occupies a shareable GPU, i.e., GPU_Utlization_Share_Enabled = 1, the inference engine module 110 is configured to first screen at least one first graphics processor from the plurality of GPUs according to the first identifier, then determine whether the remaining memory value of the at least one first graphics processor is greater than the target memory value according to the occupation state information of the at least one first graphics processor, and finally determine one of the at least one second graphics processor as the target graphics processor. The first graphics processor is a shareable graphics processor, and the second graphics processor is the first graphics processor with a remaining memory value greater than the target memory value.

[0047] For example, Figure 1GPU 11, GPU 13 and GPU 21 in the GPU 11, GPU 12, GPU 13, GPU 21, GPU 22, GPU 23 and GPU 24 are shareable GPUs, and GPU 12, GPU 14, GPU 22, GPU 23 and GPU 24 are non-shareable GPUs. On this basis, if the inference engine module 110 receives a message request and GPU_Utlization_Share_Enabled = 1, the remaining memory values of GPU 11, GPU 13 and GPU 21 are determined. If the remaining memory values of GPU 11 and GPU 21 are greater than the target memory value, GPU 11 or GPU 21 can be directly determined as the target graphics processor.

[0048] In one example, when the determined number of second graphics processors is multiple, the target graphics processor is preferentially selected from the second graphics processors in the occupied state.

[0049] In this embodiment, the occupied second graphics processor is preferentially selected as the target graphics processor, which can keep the occupied GPU at a high utilization rate and fully exert the computing performance of the occupied GPU.

[0050] Further, the inference engine module 110 is configured to determine the second graphics processor with a difference between the remaining memory value and the target memory value greater than a preset memory value as the target graphics processor. The preset memory value can be determined based on the total memory value of the GPU. For example, the preset memory value can be 5% of the total memory value, that is, the sum of the GPU memory occupancy rates of each inference instance needs to be less than 95%.

[0051] For example, after determining that the remaining memory values of GPU 11 and GPU 21 are greater than the target memory value, the difference between the remaining memory value and the target memory value of GPU 11 and the difference between the remaining memory value and the target memory value of GPU 21 are determined. If the difference between the remaining memory value and the target memory value of GPU 21 is greater than the preset memory value, GPU 21 is determined as the target graphics processor.

[0052] In this embodiment, the second graphics processor with a difference between the remaining memory value and the target memory value greater than the preset memory value is determined as the target graphics processor, which can still leave a resource space after the target graphics processor mounts the target inference instance, prevent memory overflow and GPU crash, and ensure service stability and reliability.

[0053] In other embodiments, when the second identifier indicates that the target inference instance occupies a shareable GPU, the inference engine module 110 can also be configured to first select, according to the occupancy state information of the plurality of GPUs, a GPU with a remaining memory value greater than the target memory value from the plurality of GPUs, then select, according to the first identifier, a shareable GPU from the GPU with the remaining memory value greater than the target memory value, and finally determine the shareable GPU as the target GPU.

[0054] Specifically, in the case where the first GPU does not exist or the second GPU does not exist, the inference engine module 110 is further configured to filter, according to the occupancy state information of the plurality of GPUs and the first identifier, at least one third GPU from the plurality of GPUs, and determine one of the at least one third GPU as the target GPU. The third GPU is an unshareable and unoccupied GPU.

[0055] That is, if the inference engine module 110 determines that neither the first GPU nor the second GPU exists, the inference engine module 110 determines an unoccupied and unshareable GPU from the plurality of GPUs as the target GPU.

[0056] Further, after the inference engine module 110 determines that neither the first GPU nor the second GPU exists and determines a third GPU as the target GPU, the inference engine module 110 is further configured to update the first identifier corresponding to the target GPU to change the target GPU to a shareable GPU.

[0057] For example, Figure 1 The GPUs 11 to 14 and the GPUs 22 to 24 in the example are unshareable GPUs, and the GPU 21 is a shareable GPU but the remaining memory value does not satisfy the condition (less than the target memory value). At this time, the unoccupied GPU 11 can be determined as the target GPU, and the first identifier in the GPU 11 can be updated after the determination, such as updating the value 0 in the first identifier to the value 1.

[0058] For example, the inference service module is further configured to update the occupancy state information of the target GPU after the target inference instance is successfully started, and synchronize the updated occupancy state information of the target GPU and information generated by the target inference instance to the inference engine module.

[0059] Specifically, in the case where the second identifier indicates that the target inference instance occupies an unshareable GPU, the inference engine module 110 is configured to filter, according to the occupancy state information of the plurality of GPUs and the first identifier, at least one third GPU from the plurality of GPUs, and determine the third GPU with a memory value greater than the target memory value as the target GPU.

[0060] For example, Figure 1 GPUs 11, 13, and 21 are shareable GPUs, while GPUs 12, 14, 22, 23, and 24 are non-shareable GPUs. Based on this, if the inference engine module 110 receives a message request and GPU_Utlization_Share_Enabled=0, it determines the occupancy status information of GPUs 12, 14, 22, 23, and 24. If GPUs 12, 14, and 22 are not occupied, it determines whether the video memory value of GPUs 12, 14, and 22 is greater than the target video memory value. If the video memory value of GPUs 12 and 22 is greater than the target video memory value, then GPU 12 or GPU 22 can be directly identified as the target graphics processor.

[0061] Specifically, the inference engine module 110 is also used to update the occupancy status information and inference service status of the target graphics processor after the target inference instance releases the target graphics processor.

[0062] The following is in conjunction with the appendix Figure 2 In the scenario where the first inference instance 1 is mounted on the non-shareable GPU 12 and the second inference instance 2 is mounted on the shareable GPU 21, the processing flow of the graphics processor resource management system after the third inference instance 3 is triggered is explained.

[0063] Specifically, when a single small model third inference instance3 is launched, it can support the use of shared GPU card resources, and the model's graphics card requires a size of AG of video memory resources.

[0064] When the inference service module 120 starts the third inference instance (instance3), it configures the parameters GPU_Utlization_Share_Enabled=1 and Gpu_Memory_Use=A. The inference service module 120 also generates message requests, which carry the GPU_Utlization_Share_Enabled and Gpu_Memory_Use field information. The inference service module 120 can effectively manage the inference service by using a stateful set to identify an inference instance service, ensuring the stability and availability of the inference service.

[0065] Meanwhile, the inference engine module 110 interacts with the graphics processor resource pool 200 to monitor the launched inference instances (such as the first inference instance instance1 and the second inference instance instance2) and the occupancy state information of the plurality of GPUs. After receiving the message request, the inference engine module 110 processes in different scenarios.

[0066] Scenario 1: There is no supported shared GPU card resource at present.

[0067] Specifically, after the inference engine module 110 receives the message request for starting the third inference instance instance3, it queries whether there is a GPU card that meets the requirements in the current environment. If there is no GPU card resource with Gpu Is Share = 1 in the current environment, the inference engine module 110 synchronizes the third inference instance instance3 that there is no sharable GPU card resource at present and actively synchronizes a GPU card that can support exclusive supply to the third inference instance instance3. After receiving the reply information from the inference engine module 110, the inference service module 120 actively mounts the third inference instance instance3 to the exclusive GPU resource provided by the inference engine module 110, and starts the third inference instance instance3. At the same time, the inference engine module 110 will change the state of the GPU card to a sharable state. After the third inference instance instance3 is successfully started, the inference service module 120 will synchronize the information of the third inference instance instance3 and the corresponding occupancy state information of the GPU card to the inference engine module 110.

[0068] Scenario 2: There is a shared GPU card resource that can be supported at present, but the computing power of the GPU card at present does not meet the demand.

[0069] Specifically, after receiving the message request of starting the third inference instance instance3, the inference engine module 110 queries whether there is a GPU card meeting the requirement in the current environment. The current environment has a GPU card resource with Gpu Is Share = 1. At this time, the inference engine module 110 judges whether the existing shared GPU resource is greater than the GPU Memory USE field (i.e. AG). If neither meets the requirement, the inference engine module 110 synchronizes to the third inference instance instance3 that although there is a shareable GPU card resource at present, the computing power cannot meet the requirement. Subsequently, the inference engine module 110 actively synchronizes to the inference service module 120 a GPU resource that can support exclusive supply for the third inference instance instance3. After receiving the reply information of the inference engine module 110, the inference service module 120 actively mounts the third inference instance instance3 to the exclusive GPU resource, and starts the third inference instance instance3. After the third inference instance instance3 is successfully started, the inference instance information and the corresponding occupation state information of the GPU card are synchronized to the inference engine module 110. Moreover, the inference engine module 110 changes the state of the GPU card to a shareable state.

[0070] Scenario 3: There is a shareable GPU card resource at present, and the computing power meets the starting requirement.

[0071] Specifically, after receiving the message request of starting the third inference instance instance3, the inference engine module 110 queries whether there is a GPU card meeting the requirement in the current environment. The current environment has a GPU card resource with Gpu Is Share = 1. At this time, the inference engine module 110 compares whether the existing shared GPU resource is greater than the GPU Memory USE field. If the current computing power meets the starting requirement of the third inference instance instance3, the inference engine module 110 synchronizes to the third inference instance instance3 that there is a shareable GPU card resource and computing power at present. After receiving the reply information of the inference engine module 110, the inference service module 120 actively mounts the third inference instance instance3 to the shareable GPU selected by the inference engine module 110, and starts the third inference instance instance3, Figure 2 Taking mounting the third inference instance instance3 to the GPU 21 as an example. After the third inference instance instance3 is successfully started, the inference service module 120 synchronizes the inference instance information and the corresponding occupation state information of the GPU card to the inference engine module 110.

[0072] It should be noted that the same strategy is used to process the subsequent inference instance startup process. At the same time, when multiple inference instances share the same GPU, the sum of the GPU memory occupancy rates of each inference service is less than 95%, and some resources are reserved to avoid other problems. The inference engine module 110 needs to count the state information and specific information of multiple inference instances sharing a single GPU card.

[0073] After the inference instance releases the GPU card and the computing power of the GPU card, the inference engine module 110 refreshes the state and computing power of the current GPU card and the inference service at the same time.

[0074] In this embodiment, a graphics processor resource management method is also provided, which can be used for the inference engine module described above, Figure 3 is a flowchart of a graphics processor resource management method according to an embodiment of the application, as shown in Figure 3 The method comprises the following steps:

[0075] Step S301, receiving a packet request for indicating the startup of a target inference instance from an inference service module.

[0076] The packet request comprises a second identifier and a third identifier, the second identifier is used to indicate whether the target inference instance occupies a shareable graphics processor, and the third identifier is used to indicate a target memory value required by the target inference instance after startup.

[0077] Step S302, determining a target graphics processor from a plurality of graphics processors according to the first identifier, the second identifier, the third identifier and the occupancy state information of the plurality of graphics processors.

[0078] The first identifier is used to indicate whether the graphics processor can be shared by multiple inference instances.

[0079] Step S303, sending a response message to the inference service module.

[0080] The response message comprises indication information of the target graphics processor, so that the target inference instance is mounted to the target graphics processor.

[0081] Correspondingly, the inference service module receives the response message, and mounts the target inference instance to the target graphics processor according to the response message.

[0082] The graphics processor resource management method provided in this embodiment increases three identifier fields for configuration when starting the inference instance, which is convenient for new inference instances and business, so that multiple inference instances can share the same GPU, and the GPU resources can be used more efficiently.

[0083] In the embodiment, another graphics processor resource management method is also provided, which can be used in the inference engine module, Figure 4 is a flowchart of another graphics processor resource management method according to an embodiment of the present application, as shown in the figure, the method comprises the following steps: Figure 4

[0084] Step S401, receiving a packet request from the inference service module for indicating the target inference instance startup.

[0085] Step S402, determining the target graphics processor from the plurality of graphics processors according to the first identifier, the second identifier, the third identifier and the occupation state information of the plurality of graphics processors.

[0086] Specifically, the above step S402 comprises:

[0087] Step S4021, when the second identifier indicates that the target inference instance occupies the sharable graphics processor, at least one first graphics processor is selected from the plurality of graphics processors according to the first identifier.

[0088] Among them, the first graphics processor is the sharable graphics processor.

[0089] Step S4022, determining whether the remaining video memory value of the at least one first graphics processor is greater than the target video memory value according to the occupation state information of the at least one first graphics processor.

[0090] Step S4023, determining one of the at least one second graphics processor as the target graphics processor.

[0091] Among them, the second graphics processor is the first graphics processor whose remaining video memory value is greater than the target video memory value.

[0092] In some embodiments, the above step S4023 can be specifically: determining the second graphics processor whose difference between the remaining video memory value and the target video memory value is greater than a preset video memory value as the target graphics processor.

[0093] Among them, the preset video memory value can be determined based on the total video memory value of the GPU, for example, the preset video memory value can be 5% of the total video memory value.

[0094] Exemplarily, in the case where there is no first graphics processor or no second graphics processor, the above step S402 further comprises steps a1 and a2:

[0095] Step a1, selecting at least one third graphics processor from the plurality of graphics processors according to the occupation state information of the plurality of graphics processors and the first identifier.

[0096] ​The third graphics processor is a non-shareable and unoccupied graphics processor.

[0097] Step a2, determining one of the at least one third graphics processor as a target graphics processor.

[0098] For example, when the second identifier indicates that the target inference instance occupies a non-shareable graphics processor, the step S402 further includes steps b1 and b2:

[0099] Step b1, screening at least one third graphics processor from the plurality of graphics processors according to the occupation state information of the plurality of graphics processors and the first identifier.

[0100] The step b1 is the same as the step a1.

[0101] Step b2, determining the third graphics processor with a video memory value greater than a target video memory value as the target graphics processor.

[0102] Step S403, sending a response message to the inference service module.

[0103] In the embodiment, the three added identifiers are used in combination, so that the inference engine module can more reasonably control multiple inference instances to share a single GPU card according to the requirements of the inference instances and the occupation state information of the GPU, and optimize the utilization rate of the GPU.

[0104] In the embodiment, a server is also provided, which includes the graphics processor resource management system provided in any of the above embodiments. In one example, as shown in Figure 5 The graphics processor resource pool 200 can be deployed in the same server 500 as the graphics processor resource management system 100.

[0105] The server can also include one or more processors, memories, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components communicate with each other using different buses, and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the server, including instructions stored in the memory or on the memory to display a GUI on an external input / output device, such as a display device coupled to the interface. In some optional embodiments, multiple processors and / or buses can be used with multiple memories and multiple storage devices, if necessary. Similarly, multiple servers can be connected, each providing part of the necessary operations.

[0106] The processor can be a central processing unit, a network processing unit, or a combination thereof. The processor can further include a hardware chip. The hardware chip can be an application specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device can be a complex programmable logic device, a field programmable logic gate array, a generic array logic, or any combination thereof.

[0107] The memory can include a program storage area and a data storage area. The program storage area can store an operating system and application programs required for at least one function. The data storage area can store data created according to the use of the server, etc. In addition, the memory can include a high-speed random access memory, and can further include a non-transitory memory such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state storage device.

[0108] The memory can include a volatile memory such as a random access memory, and can include a non-volatile memory such as a flash memory, a hard disk, or a solid state disk. The memory can include a combination of the above-mentioned types of memory.

[0109] It should be understood that various aspects of the application can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or a combination thereof, can be used: discrete logic circuitry having logic gates for implementing logic functions on data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

[0110] In the description of the present specification, the description of the terms "the present embodiment", "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, the person skilled in the art can combine and combine the different embodiments or examples described in the present specification and the features of the different embodiments or examples without contradiction.

[0111] Furthermore, the terms "first", "second", etc. are used only for descriptive purposes and do not connote or imply relative importance or a quantity of the indicated technical features. Thus, features defined with "first", "second" etc. can include at least one of the features implicitly or explicitly. In the description of the application, the meaning of "a plurality" is at least two, for example two, three or the like, unless explicitly specified otherwise.

[0112] Although embodiments of the present application have been described in conjunction with the drawings, various modifications and changes can be suggested by those skilled in the art, and it is intended that the appended claims encompass such modifications and changes as fall within the scope of the present application.

Claims

1. A graphics processing unit resource management system, characterized in that, The system includes an inference engine module and an inference service module. Both the inference engine module and the inference service module are communicatively connected to a graphics processor resource pool. The graphics processor resource pool includes multiple graphics processors. Each graphics processor is configured with a first identifier, which is used to indicate whether the graphics processor can be shared by multiple inference instances. The inference service module is used to generate a message request that instructs the target inference instance to start, and to send the message request to the inference engine module. The message request includes a second identifier and a third identifier. The second identifier is used to indicate whether the target inference instance occupies a shareable graphics processor, and the third identifier is used to indicate the target video memory value required by the target inference instance after it starts. The inference engine module is used to determine a target graphics processor from the plurality of graphics processors based on the first identifier, the second identifier, the third identifier and the occupancy status information of the plurality of graphics processors, and is used to send a response message to the inference service module, the response message including indication information of the target graphics processor; The inference service module is also used to mount the target inference instance to the target graphics processor according to the response message.

2. The system according to claim 1, characterized in that, The inference engine module is used to filter at least one first graphics processor from the plurality of graphics processors according to the first identifier when the second identifier indicates that the target inference instance occupies a shareable graphics processor. The first graphics processor is a shareable graphics processor. The inference engine module is further configured to determine, based on the occupancy status information of the at least one first graphics processor, whether the remaining video memory value of the at least one first graphics processor is greater than the target video memory value. The inference engine module is further configured to identify one of at least one second graphics processor as the target graphics processor, wherein the second graphics processor is a first graphics processor with a remaining video memory value greater than the target video memory value.

3. The system according to claim 2, characterized in that, The inference engine module is used to identify the second graphics processor whose difference between the remaining video memory value and the target video memory value is greater than a preset video memory value as the target graphics processor.

4. The system according to claim 2, characterized in that, In the absence of the first graphics processor or the second graphics processor, the inference engine module is further configured to filter out at least one third graphics processor from the plurality of graphics processors based on the occupancy status information of the plurality of graphics processors and the first identifier, wherein the third graphics processor is a non-shareable and unoccupied graphics processor. The inference engine module is also used to identify one of at least one third graphics processor as the target graphics processor.

5. The system according to claim 4, characterized in that, The inference service module is also used to update the occupancy status information of the target graphics processor after the target inference instance is successfully started, and to synchronize the updated occupancy status information of the target graphics processor and the information generated by the target inference instance to the inference engine module.

6. The system according to claim 4 or 5, characterized in that, The inference engine module is further configured to update the first identifier corresponding to the target graphics processor after determining one of the at least one third graphics processor as the target graphics processor, so as to change the target graphics processor into a shareable graphics processor.

7. The system according to claim 4 or 5, characterized in that, The inference engine module is further configured to, when the second identifier indicates that the target inference instance occupies a non-shareable graphics processor, select at least one third graphics processor from the plurality of graphics processors based on the occupancy status information of the plurality of graphics processors and the first identifier; The inference engine module is also used to identify a third graphics processor whose video memory value is greater than the target video memory value as the target graphics processor.

8. The system according to any one of claims 1 to 5, characterized in that, The multiple graphics processors are deployed on different servers.

9. A method for managing graphics processor resources, characterized in that, The method is applied to the inference engine module of the graphics processor resource management system according to any one of claims 1 to 8, and the method includes: Receive a message request from the inference service module to instruct the target inference instance to start. The message request includes a second identifier and a third identifier. The second identifier is used to indicate whether the target inference instance occupies a shareable graphics processor, and the third identifier is used to indicate the target video memory value required by the target inference instance after it starts. Based on the first identifier, the second identifier, the third identifier, and the occupancy status information of multiple graphics processors, a target graphics processor is determined from the multiple graphics processors, wherein the first identifier is used to indicate whether the graphics processor can be shared by multiple inference instances; A response message is sent to the inference service module, the response message including indication information of the target graphics processor, so that the target inference instance is mounted to the target graphics processor.

10. A server, characterized in that, The server includes the graphics processor resource management system as described in any one of claims 1 to 8.

Images