Gpu resource management method, device, system and readable storage medium

Abstract

The application discloses a GPU resource management method, device and system and a computer readable storage medium, and the method comprises the following steps: when a creation instruction of a first application is detected, a first virtual machine containing the first application is created, and GPU resources are allocated to the first virtual machine; when it is detected that the first virtual machine does not receive a service request within a preset period, the GPU resources occupied by the first virtual machine are released, so that the first virtual machine enters a hibernation state. According to the application, the GPU resources occupied by the virtual machine corresponding to the first application which does not receive a service request within a preset period are released, so that the virtual machine enters a hibernation state, and then the virtual machine of other applications can use the GPU resources, thereby improving the utilization rate of the GPU resources.

Description

Technical Field

[0001] This invention relates to the field of communication technology, and in particular to GPU resource management methods, apparatus, systems and computer-readable storage media. Background Technology

[0002] As applications become more widely used, the reliance on GPU (graphics processing unit) resources is also increasing. Currently, the industry mainly uses GPU resources through NVIDIA's native drivers and CUDA (Compute Unified Device Architecture, a computing platform launched by NVIDIA). Typically, one application uses one GPU resource exclusively, or multiple applications share one GPU resource. However, when an application is not working, the GPU resources it occupies cannot be used by other applications, resulting in low utilization of GPU resources.

[0003] Therefore, improving the utilization rate of GPU resources is an urgent problem to be solved. Summary of the Invention

[0004] The main objective of this invention is to provide a GPU resource management method, apparatus, system, and computer-readable storage medium, aiming to solve the problem of improving GPU resource utilization.

[0005] To achieve the above objectives, the present invention provides a GPU resource management method, which includes the following steps:

[0006] Upon detecting a creation instruction for the first application, a first virtual machine containing the first application is created, and GPU resources are allocated to the first virtual machine.

[0007] When it is detected that the first virtual machine has not received any service requests within a preset period, the GPU resources occupied by the first virtual machine are released, causing the first virtual machine to enter a hibernation state.

[0008] Furthermore, the step of releasing the GPU resources occupied by the first virtual machine and causing it to enter a hibernation state when it is detected that the first virtual machine has not received a service request within a preset period includes:

[0009] When it is detected that the first virtual machine has not received a service request within a preset period, the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine is transferred to release the GPU resources occupied by the first virtual machine in the first GPU host, so that the first virtual machine enters a hibernation state.

[0010] Furthermore, the step of transferring the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine includes:

[0011] Copy the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine to the memory and system cache of the first GPU host.

[0012] Furthermore, after the step of releasing the GPU resources occupied by the first virtual machine and causing the first virtual machine to enter a hibernation state when it is detected that the first virtual machine has not received a service request within a preset period, the GPU resource management method includes:

[0013] The first application contained in the first virtual machine is moved from the running queue to the suspended queue, and the GPU resources released by the first virtual machine are allocated to the second application in the waiting queue for use;

[0014] If a business request corresponding to the first application is received, the first application is transferred from the suspended queue to the waiting queue;

[0015] When the virtual machine corresponding to the first application is allocated the corresponding GPU resources, the first application is transferred from the waiting queue to the running queue so that the first application can process the business request.

[0016] Furthermore, after the step of releasing the GPU resources occupied by the first virtual machine and causing the first virtual machine to enter a hibernation state when it is detected that the first virtual machine has not received a service request within a preset period, the GPU resource management method further includes:

[0017] Obtain the application information of the first application, determine the second GPU host based on the application information, and create a second virtual machine containing the first application on the second GPU host;

[0018] When a business request corresponding to the first application is detected, the remaining information of the current GPU resources is obtained, and the remaining first GPU resources of the first GPU host corresponding to the first virtual machine and the remaining second GPU resources of the second GPU host corresponding to the second virtual machine are determined based on the remaining information of the current GPU resources.

[0019] Determine the GPU resources required by the first application, compare the remaining resources of the first GPU and the remaining resources of the second GPU with the GPU resources required by the first application to obtain the comparison results, and allocate the corresponding GPU resources to the first virtual machine or the second virtual machine according to the comparison results.

[0020] The business requests corresponding to the first application are processed through the first virtual machine or the second virtual machine.

[0021] Further, based on the comparison results, the step of allocating corresponding GPU resources to the first virtual machine or the second virtual machine includes:

[0022] If the comparison result shows that the remaining resources of the first GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the first virtual machine through the first GPU host corresponding to the first virtual machine.

[0023] If the comparison result shows that the remaining resources of the second GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the second virtual machine through the second GPU host corresponding to the second virtual machine.

[0024] Further, after the step of allocating corresponding GPU resources to the first virtual machine or the second virtual machine, the GPU resource management method includes:

[0025] Delete the first virtual machine or the second virtual machine, and record the processing timestamp of the last processing of the business request by the first virtual machine or the second virtual machine;

[0026] Obtain the current timestamp, and based on the current timestamp and the processing timestamp, determine whether to put the first virtual machine or the second virtual machine into a hibernation state.

[0027] Furthermore, to achieve the above objectives, the present invention also provides a GPU resource management device, the GPU resource management device comprising:

[0028] A creation module is used to create a first virtual machine containing the first application when a creation instruction for the first application is detected, and to allocate GPU resources to the first virtual machine.

[0029] The release module is used to release the GPU resources occupied by the first virtual machine when it is detected that the first virtual machine has not received a service request within a preset period, so that the first virtual machine enters a hibernation state.

[0030] Furthermore, the release module is also used for:

[0031] When it is detected that the first virtual machine has not received a service request within a preset period, the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine is transferred to release the GPU resources occupied by the first virtual machine in the first GPU host, so that the first virtual machine enters a hibernation state.

[0032] Furthermore, the release module is also used for:

[0033] Copy the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine to the memory and system cache of the first GPU host.

[0034] Furthermore, the release module also includes a transfer module, which is used for:

[0035] The first application contained in the first virtual machine is moved from the running queue to the suspended queue, and the GPU resources released by the first virtual machine are allocated to the second application in the waiting queue for use;

[0036] If a business request corresponding to the first application is received, the first application is transferred from the suspended queue to the waiting queue;

[0037] When the virtual machine corresponding to the first application is allocated the corresponding GPU resources, the first application is transferred from the waiting queue to the running queue so that the first application can process the business request.

[0038] Furthermore, the release module also includes an allocation module, which is used for:

[0039] Obtain the application information of the first application, determine the second GPU host based on the application information, and create a second virtual machine containing the first application on the second GPU host;

[0040] When a business request corresponding to the first application is detected, the remaining information of the current GPU resources is obtained, and the remaining first GPU resources of the first GPU host corresponding to the first virtual machine and the remaining second GPU resources of the second GPU host corresponding to the second virtual machine are determined based on the remaining information of the current GPU resources.

[0041] Determine the GPU resources required by the first application, compare the remaining resources of the first GPU and the remaining resources of the second GPU with the GPU resources required by the first application to obtain the comparison results, and allocate the corresponding GPU resources to the first virtual machine or the second virtual machine according to the comparison results.

[0042] The business requests corresponding to the first application are processed through the first virtual machine, the second virtual machine, or the third virtual machine.

[0043] Furthermore, the allocation module also includes a comparison module, which is used for:

[0044] If the comparison result shows that the remaining resources of the first GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the first virtual machine through the first GPU host corresponding to the first virtual machine.

[0045] If the comparison result shows that the remaining resources of the second GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the second virtual machine through the second GPU host corresponding to the second virtual machine.

[0046] Furthermore, the allocation module also includes a deletion module, which is used to:

[0047] Delete the first virtual machine or the second virtual machine, and record the processing timestamp of the last processing of the business request by the first virtual machine or the second virtual machine;

[0048] Obtain the current timestamp, and based on the current timestamp and the processing timestamp, determine whether to put the first virtual machine or the second virtual machine into a hibernation state.

[0049] In addition, to achieve the above objectives, the present invention also provides a GPU resource management system, which includes: a memory, a processor, and a GPU resource management program stored in the memory and executable on the processor. When the GPU resource management program is executed by the processor, it implements the steps of the GPU resource management method as described above.

[0050] In addition, to achieve the above objectives, the present invention also provides a readable storage medium, which is a computer-readable storage medium, and stores a GPU resource management program thereon. When the GPU resource management program is executed by a processor, it implements the steps of the GPU resource management method as described above.

[0051] The GPU resource management method proposed in this invention creates a first virtual machine containing the first application when a creation instruction for the first application is detected, and allocates GPU resources to the first virtual machine. When it is detected that the first virtual machine has not received any service requests within a preset period, the GPU resources occupied by the first virtual machine are released, causing the first virtual machine to enter a hibernation state. By releasing the GPU resources occupied by the virtual machine corresponding to the first application that has not received any service requests within a preset period, and causing the virtual machine to enter a hibernation state, this invention allows virtual machines of other applications to use GPU resources, thereby improving the utilization rate of GPU resources. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention;

[0053] Figure 2 This is a flowchart illustrating the first embodiment of the GPU resource management method of the present invention;

[0054] Figure 3 This is a flowchart illustrating the second embodiment of the GPU resource management method of the present invention;

[0055] Figure 4 This is a flowchart illustrating the third embodiment of the GPU resource management method of the present invention;

[0056] Figure 5 This is a schematic diagram of the functional modules of the GPU resource management device of the present invention;

[0057] Figure 6 This is a timing diagram illustrating the GPU resource management method of the present invention;

[0058] Figure 7 This is a schematic diagram of the GPU resource management system architecture of the present invention.

[0059] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0061] like Figure 1 As shown, Figure 1 This is a schematic diagram of the device structure of the hardware operating environment involved in the embodiments of the present invention.

[0062] The device in this embodiment of the invention can be a PC or a server.

[0063] like Figure 1 As shown, the device may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0064] Those skilled in the art will understand thatFigure 1 The device structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0065] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a GPU resource management program.

[0066] The operating system is a program that manages and controls portable storage devices and software resources, and supports the operation of the network communication module, user interface module, GPU resource management program, and other programs or software; the network communication module is used to manage and control the network interface 1002; and the user interface module is used to manage and control the user interface 1003.

[0067] exist Figure 1 In the storage device shown, the storage device calls the GPU resource management program stored in the memory 1005 through the processor 1001 and executes the operations in the various embodiments of the GPU resource management method described below.

[0068] Based on the above hardware structure, an embodiment of the GPU resource management method of the present invention is proposed.

[0069] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of the GPU resource management method of the present invention, the method comprising:

[0070] Step S10: When the creation instruction of the first application is detected, a first virtual machine containing the first application is created, and GPU resources are allocated to the first virtual machine;

[0071] Step S20: When it is detected that the first virtual machine has not received a service request within a preset period, the GPU resources occupied by the first virtual machine are released, so that the first virtual machine enters a hibernation state.

[0072] This embodiment of the GPU resource management method is applied to the GPU resource management system of a communication service provider. This GPU resource management system manages GPU resources based on Kubernetes applications, solving the pain point of low GPU resource utilization. For ease of description, the management system is used as an example. When a first application creation instruction is detected, the application information corresponding to the first application creation instruction is obtained. Based on the application information, a first GPU host is determined, and a first virtual machine containing the first application is created in the first GPU host. Based on the application information, the management system determines the GPU resources required by the first application and allocates the corresponding GPU resources to the first virtual machine in the first GPU host. When the management system detects that the first virtual machine has not received a service request within a preset period, it transfers the data of the first application corresponding to the first virtual machine stored in the first GPU host to release the GPU resources occupied by the first virtual machine in the first GPU host, so that the first virtual machine enters a hibernation state. It should be noted that Kubernetes is an open-source application used to manage containerized applications on multiple hosts in a cloud platform. The GPU resource management system is based on Kubernetes to implement the above steps. Virtual machines are pods, which are the basic building blocks of Kubernetes applications. A pod can contain multiple or a single container. When it contains multiple containers, these containers always run on the same worker node, because a pod will never span multiple worker nodes.

[0073] The GPU resource management method of this embodiment creates a first virtual machine containing the first application when a creation instruction for the first application is detected, and allocates GPU resources to the first virtual machine. When it is detected that the first virtual machine has not received any service requests within a preset period, the GPU resources occupied by the first virtual machine are released, causing the first virtual machine to enter a hibernation state. By releasing the GPU resources occupied by the virtual machine corresponding to the first application that has not received any service requests within a preset period, and causing the virtual machine to enter a hibernation state, the virtual machines of other applications can use GPU resources, thereby improving the utilization rate of GPU resources.

[0074] The following will provide a detailed explanation of each step:

[0075] Step S10: When the creation instruction of the first application is detected, a first virtual machine containing the first application is created, and GPU resources are allocated to the first virtual machine;

[0076] In this embodiment, when the management system detects the creation instruction of the first application, it creates a first virtual machine containing the first application and allocates GPU resources to the first virtual machine. It is understood that in daily use, different users create corresponding first applications on the cloud according to their own needs. These first applications all require the use of GPU resources. That is to say, the management system will detect one or more creation instructions of the first application at the same time. The management system, in accordance with the conventional method of Kubernetes applications, determines the corresponding GPU host through its own master control unit based on the creation instruction of the first application, creates the first virtual machine corresponding to the first application on the GPU host, and allocates GPU resources to the first virtual machine.

[0077] Specifically, when the management system detects the first application creation instruction, it obtains the application information corresponding to the first application creation instruction. Based on the application information, it determines the GPU resources required by the first application during runtime. Based on the GPU resources required by the first application during runtime, it selects a GPU host with more remaining GPU resources than the GPU resources required by the first application during runtime from the GPU hosts in the management system. This GPU host is then used as the first GPU host for the first application. A first virtual machine containing the first application is created on the first GPU host. The first virtual machine includes the corresponding first application and a hibernation program. This hibernation program is automatically generated by the management system and is used to monitor in real time whether a business request corresponding to the first application has been received, and to monitor the running, waiting, and suspended first applications. The management system determines the GPU resources required by the corresponding first application during runtime based on the application information, and allocates the corresponding GPU resources to the first virtual machine in the first GPU host. For example, if the management system determines that the GPU resources required by the corresponding first application are 50 MB of storage space in the video memory of the GPU host, the management system will allocate 50 MB of storage space to the first virtual machine in the video memory of the first GPU host. Preferably, in order to prevent special circumstances from occurring, the management system can allocate more than 50 MB of storage space to the first virtual machine in the video memory of the first GPU host, which can be set in advance by relevant personnel.

[0078] Step S20: When it is detected that the first virtual machine has not received a service request within a preset period, the GPU resources occupied by the first virtual machine are released, so that the first virtual machine enters a hibernation state.

[0079] In this embodiment, when the management system detects that the first virtual machine receives a service request within a preset period, it directly uses the corresponding GPU resources of the first virtual machine to process the service request; when the management system detects that the first virtual machine does not receive a service request within the preset period, it releases the GPU resources occupied by the first virtual machine, causing the first virtual machine to enter a hibernation state. The duration of the preset period can be set according to actual needs, and this embodiment does not impose a specific limitation on it.

[0080] Specifically, step S20 includes:

[0081] Step a: When it is detected that the first virtual machine has not received a service request within a preset period, the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine is transferred to release the GPU resources occupied by the first virtual machine in the first GPU host, so that the first virtual machine enters a hibernation state.

[0082] In this step, when the management system detects that the first virtual machine corresponding to a certain first application has not received a business request within a preset period, it transfers the data corresponding to the first application stored in the video memory of the first GPU host corresponding to the first virtual machine. At this time, there is no data in the storage location of the video memory of the first GPU host that stores the data corresponding to the first application, and this storage location can be used by other applications. Then, the GPU resources occupied by the first virtual machine in the first GPU host are released, and the first virtual machine enters a hibernation state.

[0083] Furthermore, the step of transferring the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine includes:

[0084] Step a1: Copy the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine to the memory and system cache of the first GPU host.

[0085] In this step, when the management system detects that the first application corresponding to the first virtual machine has not received a business request within a preset period, it determines the corresponding first GPU host based on the GPU resources occupied by the first virtual machine corresponding to the first application, and copies the data corresponding to the first application stored in the video memory of the first GPU host to the memory and system cache of the first GPU host, so as to release the GPU resources occupied by the first virtual machine. It should be noted that the data corresponding to the first application stored in the video memory of the first GPU host is non-lossable data such as the model and calculation results used by the first application corresponding to the first virtual machine; the system cache is located in the management system and can be accessed by all GPU hosts.

[0086] In this embodiment, when the management system detects a first application creation instruction, it obtains the application information corresponding to the instruction, determines a first GPU host based on the application information, and creates a first virtual machine containing the first application on the first GPU host. The management system then determines the GPU resources required by the first application based on the application information and allocates the corresponding GPU resources to the first virtual machine on the first GPU host. When the management system detects that the first virtual machine has not received any service requests within a preset period, it transfers the data of the first application corresponding to the first virtual machine stored on the first GPU host to release the GPU resources occupied by the first virtual machine, causing the first virtual machine to enter a hibernation state. By releasing the GPU resources occupied by the virtual machine corresponding to the first application that has not received any service requests within a preset period, the virtual machine enters a hibernation state, thereby enabling virtual machines of other applications to use GPU resources and improving GPU resource utilization.

[0087] Furthermore, such as Figure 3 As shown, based on the first embodiment of the GPU resource management method of the present invention, a second embodiment of the GPU resource management method of the present invention is proposed.

[0088] The second embodiment of the GPU resource management method differs from the first embodiment in that, after step S20, the GPU resource management method includes:

[0089] Step b: Move the first application contained in the first virtual machine from the running queue to the suspended queue, and allocate the GPU resources released by the first virtual machine to the second application in the waiting queue for use;

[0090] Step c: If a business request corresponding to the first application is received, the first application is transferred from the suspended queue to the waiting queue.

[0091] Step d: When the virtual machine corresponding to the first application is allocated to the corresponding GPU resources, the first application is transferred from the waiting queue to the running queue so that the first application can process the business request.

[0092] In this embodiment, the management system contains multiple GPU request queues, including a running queue (which records all applications in the management system that occupy GPU resources), a waiting queue (which records all applications in the management system that are waiting to use GPU resources), and a suspended queue (which records all applications in the management system that have entered a dormant state). The application flow is as follows: suspended queue, waiting queue, running queue, and suspended queue. When the first application corresponding to a virtual machine does not receive a business request within a preset period, the management system uses a hibernation procedure to move the first application to a suspended queue. The management system then instructs the master control unit to allocate the GPU resources released by the virtual machine corresponding to the first application to the virtual machine corresponding to the second application with higher priority in the waiting queue. For example, if the first application contained in the first virtual machine does not receive a business request within a preset period, the management system releases the GPU resources occupied by the first virtual machine corresponding to the first application, and the first virtual machine enters a hibernation state. The management system moves the first application contained in the first virtual machine from the running queue to the suspended queue, and designates the application with the longest waiting time or the application with priority in the waiting queue as the highest priority second application. The management system then allocates the GPU resources released by the first virtual machine to the virtual machine corresponding to the highest priority second application in the waiting queue. When the first application contained in the first virtual machine receives a business request, the management system moves the first application from the suspended queue to the waiting queue. When the virtual machine corresponding to the first application is allocated the corresponding GPU resources, the first application enters the running queue and processes the business request, and this cycle continues. It should be noted that after the first virtual machine corresponding to the first application enters a hibernation state, the management system will generate a second virtual machine on a GPU host other than the GPU host corresponding to the first virtual machine. This second virtual machine does not consume GPU resources. The virtual machine corresponding to the first application may be either the first virtual machine or the second virtual machine. When the first or second virtual machine corresponding to the first application is allocated the corresponding GPU resources, the first application enters the running queue and processes business requests simultaneously, and this process continues in a loop.

[0093] The management system in this embodiment controls the application to circulate in the suspend queue, waiting queue, and running queue through the hibernation program, which improves the management efficiency of applications and corresponding virtual machines, and further helps to improve the utilization of GPU resources.

[0094] Furthermore, such as Figure 4 As shown, based on the first and second embodiments of the GPU resource management method of the present invention, a third embodiment of the GPU resource management method of the present invention is proposed.

[0095] The difference between the third embodiment of the GPU resource management method and the first and second embodiments of the GPU resource management method is that, after step S20, the GPU resource management method further includes:

[0096] Step e: Obtain the application information of the first application, determine the second GPU host based on the application information, and create a second virtual machine containing the first application on the second GPU host;

[0097] Step f: When a business request corresponding to the first application is detected, the current GPU resource remaining information is obtained, and based on the current GPU resource remaining information, the first GPU remaining resources of the first GPU host corresponding to the first virtual machine and the second GPU remaining resources of the second GPU host corresponding to the second virtual machine are determined respectively.

[0098] Step g: Determine the GPU resources required by the first application, compare the remaining resources of the first GPU and the remaining resources of the second GPU with the GPU resources required by the first application, obtain the comparison results, and allocate the corresponding GPU resources to the first virtual machine or the second virtual machine according to the comparison results;

[0099] Step h involves processing the business requests corresponding to the first application through either the first virtual machine or the second virtual machine.

[0100] In this embodiment, after the management system puts the first virtual machine into a hibernation state, it obtains the application information of the first application contained in the first virtual machine. Based on the application information, it determines the GPU resources required by the first application during runtime. Then, based on the GPU resources required by the first application during runtime, it selects a GPU host from the management system's GPU hosts with more remaining GPU resources than the GPU resources required by the first application during runtime, as the second GPU host for the first application. The second GPU host and the first GPU host are not the same GPU host. The management system creates a second virtual machine containing the first application on the second GPU host. The second virtual machine does not occupy the GPU resources in the second GPU host. When the management system detects a business request corresponding to the first application, it obtains... The system retrieves the current remaining GPU resource information. Based on this information, it queries the remaining first GPU resources of the first GPU host and the remaining second GPU resources of the second GPU host, respectively. This allows the system to determine the remaining first GPU resources of the first GPU host corresponding to the first virtual machine and the remaining second GPU resources of the second GPU host corresponding to the second virtual machine. The system then determines the GPU resources required by the first application, compares the remaining first and second GPU resources with those required by the first application, obtains the comparison results, and allocates the corresponding GPU resources to the first or second virtual machine based on these results. The system then processes the business requests corresponding to the first application through the first or second virtual machine.

[0101] In this embodiment, after the first virtual machine enters a hibernation state, the management system obtains the application information of the first application contained in the first virtual machine, determines the second GPU host based on the application information, and creates a second virtual machine containing the first application on the second GPU host. When a service request corresponding to the first application contained in the first virtual machine is received, the system determines to allocate corresponding GPU resources to the first virtual machine or the second virtual machine based on the first GPU remaining resources of the first GPU host corresponding to the first virtual machine and the second GPU remaining resources of the second GPU host corresponding to the second virtual machine. The first virtual machine and the second virtual machine are then used to process the service request corresponding to the first application. Based on the remaining GPU resources, the system determines the virtual machine that will process the service request of the first application, thereby improving the utilization rate of GPU resources.

[0102] The following will provide a detailed explanation of each step:

[0103] Step e: Obtain the application information of the first application, determine the second GPU host based on the application information, and create a second virtual machine containing the first application on the second GPU host;

[0104] In this step, the management system puts the first virtual machine corresponding to a certain first application into a hibernation state, obtains the application information of the first application, and determines the second GPU host based on the application information. On the second GPU host, a second virtual machine containing the first application is created to prevent insufficient GPU resources of the first GPU host where the first virtual machine is located when the first application needs to process business requests. It should be noted that the second virtual machine does not occupy the GPU resources of the second GPU host. The second virtual machine also includes the corresponding first application and the hibernation program. The hibernation program is automatically generated by the management system and is used to monitor in real time whether a business request corresponding to the first application has been received, and to monitor the running, waiting and suspended first applications.

[0105] Furthermore, when the management system creates a second virtual machine containing the first application, it simultaneously creates a third virtual machine containing the first application within its own CPU host. While the first virtual machine is still in a dormant state, i.e., the first application is in a suspended or waiting queue, upon receiving a business request corresponding to the first application, the management system can process the business request through the third virtual machine containing the first application within the CPU host, ensuring that business requests are not lost and improving the user experience. The third virtual machine does not require GPU resources, only CPU resources.

[0106] Step f: When a business request corresponding to the first application is detected, the current GPU resource remaining information is obtained, and based on the current GPU resource remaining information, the first GPU remaining resources of the first GPU host corresponding to the first virtual machine and the second GPU remaining resources of the second GPU host corresponding to the second virtual machine are determined respectively.

[0107] In this step, after the management system puts the first virtual machine corresponding to a first application into a hibernation state, upon detecting a business request corresponding to the first application, it obtains its own current remaining GPU resource information. Based on this information, it determines the remaining first GPU resources of the first GPU host corresponding to the first virtual machine of the first application and the remaining second GPU resources of the second GPU host corresponding to the second virtual machine of the first application. It should be noted that each virtual machine can only be allocated GPU resources from its corresponding GPU host.

[0108] Step g: Determine the GPU resources required by the first application, compare the remaining resources of the first GPU and the remaining resources of the second GPU with the GPU resources required by the first application, obtain the comparison results, and allocate the corresponding GPU resources to the first virtual machine or the second virtual machine according to the comparison results.

[0109] In this step, the management system determines the GPU resources required by the first application based on the application information of the first application. It then compares the remaining resources of the first and second GPUs with the GPU resources required by the first application to obtain the comparison results. Based on the comparison results, it allocates the corresponding GPU resources to the first or second virtual machine. When it is determined that the business request corresponding to the first application will be processed by the first virtual machine, the first virtual machine can directly obtain the data corresponding to the first application stored in the memory of the first GPU host. When it is determined that the business request corresponding to the first application will be processed by the second virtual machine, the second virtual machine obtains the data corresponding to the first application stored in the system cache.

[0110] Further, based on the comparison results, the step of allocating corresponding GPU resources to the first virtual machine or the second virtual machine includes:

[0111] Step g1: If the comparison result shows that the remaining resources of the first GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the first virtual machine through the GPU host corresponding to the first virtual machine.

[0112] Step g2: If the comparison result shows that the remaining resources of the second GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the second virtual machine through the GPU host corresponding to the second virtual machine.

[0113] In steps f1 to f2, if the management system obtains a comparison result indicating that the remaining resources of the first GPU are greater than the GPU resources required by the first application, then the management system allocates the corresponding GPU resources to the first virtual machine through the first GPU host corresponding to the first virtual machine; if the management system obtains a comparison result indicating that the remaining resources of the second GPU are greater than the GPU resources required by the first application, then the management system allocates the corresponding GPU resources to the second virtual machine through the second GPU host corresponding to the second virtual machine. It can be understood that since the first virtual machine was previously in a dormant state and the second virtual machine did not occupy GPU resources, the GPU resources of the GPU hosts where the first and second virtual machines are located may have been occupied by the virtual machine of the second application, so there will be a situation where the remaining GPU resources are less than the GPU resources required by the first application.

[0114] Furthermore, after the management system allocates the corresponding GPU resources to the first virtual machine through the first GPU host corresponding to the first virtual machine, it deletes the second virtual machine on the second host and deletes the third virtual machine on the CPU host; after the management system allocates the corresponding GPU resources to the second virtual machine through the second GPU host corresponding to the second virtual machine, it deletes the first virtual machine on the first GPU host and deletes the third virtual machine on the CPU host.

[0115] Further, after the step of allocating corresponding GPU resources to the first virtual machine or the second virtual machine, the following steps are included:

[0116] Step g21: Delete the first virtual machine or the second virtual machine, and record the processing timestamp of the last processing of the service request by the first virtual machine or the second virtual machine;

[0117] In this step, after the management system allocates corresponding GPU resources to the first virtual machine or the second virtual machine and processes the business request through the first virtual machine or the second virtual machine, it deletes the first virtual machine or the second virtual machine and records the processing timestamp of the last business request processed by the first virtual machine or the second virtual machine. It can be understood that when the management system determines to allocate corresponding GPU resources to the first virtual machine and processes the business request through the first virtual machine, it deletes the second virtual machine and records the processing timestamp of the last business request processed by the first virtual machine. When the management system determines to allocate corresponding GPU resources to the second virtual machine and processes the business request through the second virtual machine, it deletes the first virtual machine and records the processing timestamp of the last business request processed by the second virtual machine.

[0118] Step g22: Obtain the current timestamp, and determine whether to put the first virtual machine or the second virtual machine into a hibernation state based on the current timestamp and the processing timestamp.

[0119] In this step, the management system obtains the current timestamp and determines whether to put the first or second virtual machine into a hibernation state based on the current timestamp and the processing timestamp. Understandably, when the management system calculates the time difference between the current timestamp and the processing timestamp, it compares this time difference with a preset period. If the time difference is not greater than the preset period, it is determined that the first or second virtual machine has received a service request within the preset period, and the first or second virtual machine continues to process the corresponding service request. If the time difference is greater than the preset period, it is determined that the first or second virtual machine has not received a service request within the preset period, and the first or second virtual machine is put into a hibernation state, releasing the GPU resources occupied by the first or second virtual machine.

[0120] Step h involves processing the business requests corresponding to the first application through either the first virtual machine or the second virtual machine.

[0121] In this step, when the management system allocates the corresponding GPU resources to the first virtual machine through the first GPU host corresponding to the first virtual machine, the business requests corresponding to the first application are processed through the first virtual machine; when the management system allocates the corresponding GPU resources to the second virtual machine through the second GPU host corresponding to the second virtual machine, the business requests corresponding to the first application are processed through the second virtual machine; in an extreme case, the remaining resources of the first GPU and the remaining resources of the second GPU are both less than the GPU resources required by the first application. In this case, the management system processes the business requests through the third virtual machine containing the first application in the CPU host.

[0122] In this embodiment, after the first virtual machine enters a hibernation state, the management system obtains the application information of the first application contained in the first virtual machine, determines the second GPU host and CPU host based on the application information, creates a second virtual machine containing the first application on the second GPU host, and creates a third virtual machine containing the first application on the CPU host. When a service request corresponding to the first application contained in the first virtual machine is received, the system determines to allocate corresponding GPU resources to the first virtual machine or the second virtual machine based on the remaining GPU resources of the first GPU host corresponding to the first virtual machine and the second GPU host corresponding to the second virtual machine. The service request corresponding to the first application is processed by the first virtual machine, the second virtual machine, or the third virtual machine. By determining the virtual machine to process the service request of the first application based on the remaining GPU resources, the system improves the utilization rate of GPU resources.

[0123] In specific implementation, such as Figure 7As shown, the GPU resource management system includes a service gateway for receiving user commands or sending information to users; a hibernation program for real-time monitoring of whether business requests corresponding to the first application have been received, and for monitoring the running, waiting, and suspended first applications; a Kubernetes Master (control unit) for controlling the system's hibernation program, service gateway, system cache, CPU hosts, and GPU hosts; a CPU host, including CPU and memory, and a third virtual machine built by the GPU resource management system; a first GPU host, including GPU resources, GPU memory, and memory, and a first virtual machine built by the GPU resource management system; a second GPU host, including GPU resources, GPU memory, and memory, and a second virtual machine built by the GPU resource management system; and a system cache for caching first application data. It should be noted that the GPU resource management system also includes other GPU hosts besides the first and second GPU hosts, which are not shown in the diagram.

[0124] like Figure 6As shown, when the Kubernetes Master in the GPU resource management system receives a user's command to create the first application, it creates the corresponding first application and sends a message to the user indicating successful creation. Based on the application information of the first application and its own GPU resources, the GPU resource management system determines the first GPU host and creates a first virtual machine containing the first application on the first GPU host, allocating GPU resources to the first virtual machine through the first GPU host. The GPU resource management system uses a hibernation process to check if it receives a service request from the user. If a service request is received within a preset period, it processes the service request through the first virtual machine. If no service request is received within the preset period, it copies the data of the first application from the GPU host's video memory to the first GPU host's main memory and system cache, causing the first virtual machine to enter hibernation mode. This releases the GPU resources occupied by the first virtual machine on the first GPU host. Then, a second virtual machine containing the first application is built on a second GPU host. At this time, the second virtual machine does not occupy the GPU resources of the second GPU host. A third virtual machine containing the first application is also built on a CPU host. After the first virtual machine enters hibernation mode, the Kubernetes... When the Master receives a service request from a user, it uses a hibernation procedure to call a third virtual machine in the CPU host to buffer the service request. The hibernation procedure then determines the remaining GPU resources of the first and second GPU hosts. If the remaining GPU resources of the first GPU host meet the requirements, it allocates GPU resources to the first virtual machine through the first GPU host, copies the data of the first application from memory to video memory, processes the service request through the first virtual machine, and deletes the second virtual machine in the second GPU host and the third virtual machine in the CPU host. If the remaining GPU resources of the second GPU host meet the requirements, it allocates GPU resources to the second virtual machine through the second GPU host, copies the data of the first application from the system cache to video memory, processes the service request through the second virtual machine, and deletes the first virtual machine in the first GPU host and the third virtual machine in the CPU host. In an extreme case, if the remaining GPU resources of neither the first nor the second GPU host meet the requirements, it copies the data of the first application from the system cache to the memory of the CPU host, and processes the service request through the third virtual machine in the CPU host.

[0125] like Figure 5 As shown, the present invention also provides a GPU resource management device. The GPU resource management device of the present invention includes:

[0126] A creation module is used to create a first virtual machine containing the first application when a creation instruction for the first application is detected, and to allocate GPU resources to the first virtual machine.

[0127] The release module is used to release the GPU resources occupied by the first virtual machine when it is detected that the first virtual machine has not received a service request within a preset period, so that the first virtual machine enters a hibernation state.

[0128] Furthermore, the release module is also used for:

[0129] When it is detected that the first virtual machine has not received a service request within a preset period, the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine is transferred to release the GPU resources occupied by the first virtual machine in the first GPU host, so that the first virtual machine enters a hibernation state.

[0130] Furthermore, the release module is also used for:

[0131] Copy the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine to the memory and system cache of the first GPU host.

[0132] Furthermore, the release module also includes a transfer module, which is used for:

[0133] The first application contained in the first virtual machine is moved from the running queue to the suspended queue, and the GPU resources released by the first virtual machine are allocated to the second application in the waiting queue for use;

[0134] If a business request corresponding to the first application is received, the first application is transferred from the suspended queue to the waiting queue;

[0135] When the virtual machine corresponding to the first application is allocated the corresponding GPU resources, the first application is transferred from the waiting queue to the running queue so that the first application can process the business request.

[0136] Furthermore, the release module also includes an allocation module, which is used for:

[0137] Obtain the application information of the first application, determine the second GPU host based on the application information, and create a second virtual machine containing the first application on the second GPU host;

[0138] When a business request corresponding to the first application is detected, the remaining information of the current GPU resources is obtained, and the remaining first GPU resources of the first GPU host corresponding to the first virtual machine and the remaining second GPU resources of the second GPU host corresponding to the second virtual machine are determined based on the remaining information of the current GPU resources.

[0139] Determine the GPU resources required by the first application, compare the remaining resources of the first GPU and the remaining resources of the second GPU with the GPU resources required by the first application to obtain the comparison results, and allocate the corresponding GPU resources to the first virtual machine or the second virtual machine according to the comparison results.

[0140] The business requests corresponding to the first application are processed through the first virtual machine or the second virtual machine.

[0141] Furthermore, the allocation module also includes a comparison module, which is used for:

[0142] If the comparison result shows that the remaining resources of the first GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the first virtual machine through the first GPU host corresponding to the first virtual machine.

[0143] If the comparison result shows that the remaining resources of the second GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the second virtual machine through the second GPU host corresponding to the second virtual machine.

[0144] Furthermore, the allocation module also includes a deletion module, which is used to:

[0145] Delete the first virtual machine or the second virtual machine, and record the processing timestamp of the last processing of the business request by the first virtual machine or the second virtual machine;

[0146] Obtain the current timestamp, and based on the current timestamp and the processing timestamp, determine whether to put the first virtual machine or the second virtual machine into a hibernation state.

[0147] The present invention also provides a GPU resource management system.

[0148] The GPU resource management system includes: a memory, a processor, and a GPU resource management program stored in the memory and executable on the processor. When the GPU resource management program is executed by the processor, it implements the steps of the GPU resource management method as described above.

[0149] The method implemented when the GPU resource management program running on the processor is executed can be referred to in various embodiments of the GPU resource management method of the present invention, and will not be repeated here.

[0150] The present invention also provides a readable storage medium.

[0151] The readable storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a GPU resource management program. When the GPU resource management program is executed by the processor, it implements the steps of the GPU resource management method as described above.

[0152] The method implemented when the GPU resource management program running on the processor is executed can be referred to in various embodiments of the GPU resource management method of the present invention, and will not be repeated here.

[0153] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0154] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0155] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0156] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A GPU resource management method, characterized in that, The GPU resource management method includes the following steps: Upon detecting a creation instruction for the first application, a first virtual machine containing the first application is created, and GPU resources are allocated to the first virtual machine. When it is detected that the first virtual machine has not received a service request within a preset period, the GPU resources occupied by the first virtual machine are released, causing the first virtual machine to enter a hibernation state. Obtain the application information of the first application, determine the second GPU host based on the application information, and create a second virtual machine containing the first application on the second GPU host; When a business request corresponding to the first application is detected, the remaining information of the current GPU resources is obtained, and the remaining first GPU resources of the first GPU host corresponding to the first virtual machine and the remaining second GPU resources of the second GPU host corresponding to the second virtual machine are determined according to the remaining information of the current GPU resources. Determine the GPU resources required by the first application, compare the remaining resources of the first GPU and the remaining resources of the second GPU with the GPU resources required by the first application to obtain the comparison results, and allocate the corresponding GPU resources to the first virtual machine or the second virtual machine according to the comparison results. The business requests corresponding to the first application are processed through the first virtual machine or the second virtual machine.

2. The GPU resource management method as described in claim 1, characterized in that, The step of releasing the GPU resources occupied by the first virtual machine and causing the first virtual machine to enter a hibernation state when it is detected that the first virtual machine has not received a service request within a preset period includes: When it is detected that the first virtual machine has not received a service request within a preset period, the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine is transferred to release the GPU resources occupied by the first virtual machine in the first GPU host, so that the first virtual machine enters a hibernation state.

3. The GPU resource management method as described in claim 2, characterized in that, The step of transferring the data corresponding to the first application stored in the first GPU host corresponding to the first virtual machine includes: Copy the data corresponding to the first application stored in the video memory of the first GPU host corresponding to the first virtual machine to the memory and system cache of the first GPU host.

4. The GPU resource management method as described in claim 1, characterized in that, After the step of releasing the GPU resources occupied by the first virtual machine and causing the first virtual machine to enter a hibernation state when it is detected that the first virtual machine has not received a service request within a preset period, the GPU resource management method includes: The first application contained in the first virtual machine is moved from the running queue to the suspended queue, and the GPU resources released by the first virtual machine are allocated to the second application in the waiting queue for use; If a business request corresponding to the first application is received, the first application is transferred from the suspended queue to the waiting queue; When the virtual machine corresponding to the first application is allocated the corresponding GPU resources, the first application is transferred from the waiting queue to the running queue so that the first application can process the business request.

5. The GPU resource management method as described in claim 1, characterized in that, The step of allocating corresponding GPU resources to the first virtual machine or the second virtual machine based on the comparison result includes: If the comparison result shows that the remaining resources of the first GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the first virtual machine through the first GPU host corresponding to the first virtual machine. If the comparison result shows that the remaining resources of the second GPU are greater than the GPU resources required by the first application, then the corresponding GPU resources are allocated to the second virtual machine through the second GPU host corresponding to the second virtual machine.

6. The GPU resource management method as described in claim 1, characterized in that, After the step of allocating corresponding GPU resources to the first virtual machine or the second virtual machine, the GPU resource management method includes: Delete the first virtual machine or the second virtual machine, and record the processing timestamp of the last processing of the business request by the first virtual machine or the second virtual machine; Obtain the current timestamp, and based on the current timestamp and the processing timestamp, determine whether to put the first virtual machine or the second virtual machine into a hibernation state.

7. A GPU resource management device, characterized in that, The GPU resource management device includes: A creation module is used to create a first virtual machine containing the first application when a creation instruction for the first application is detected, and to allocate GPU resources to the first virtual machine. The release module is used to release the GPU resources occupied by the first virtual machine when it is detected that the first virtual machine has not received a business request within a preset period, so that the first virtual machine enters a hibernation state. The resource allocation module is used to obtain application information of the first application, determine a second GPU host based on the application information, and create a second virtual machine containing the first application on the second GPU host; when a service request corresponding to the first application is detected, obtain the current remaining GPU resource information, and determine the first remaining GPU resource of the first GPU host corresponding to the first virtual machine and the second remaining GPU resource of the second GPU host corresponding to the second virtual machine based on the current remaining GPU resource information; determine the GPU resources required by the first application, compare the first remaining GPU resource and the second remaining GPU resource with the GPU resources required by the first application, obtain the comparison result, and allocate the corresponding GPU resources to the first virtual machine or the second virtual machine based on the comparison result; and process the service request corresponding to the first application through the first virtual machine or the second virtual machine.

8. A GPU resource management system, characterized in that, The GPU resource management system includes: a memory, a processor, and a GPU resource management program stored in the memory and executable on the processor. When the GPU resource management program is executed by the processor, it implements the steps of the GPU resource management method as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium is a computer-readable storage medium, and the computer-readable storage medium stores a GPU resource management program, which, when executed by a processor, implements the steps of the GPU resource management method as described in any one of claims 1 to 6.

Images