Elastic scheduling method, system, device and storage medium for GPU virtualization computing power

Abstract

This application relates to the elastic scheduling method, system, equipment and storage medium of GPU virtualization computing power, which belongs to the technical field of GPU virtualization. Indicators, automatically determine the computing power quota of the container; detect the real-time running indicators of the container in the business scenario; automatically adjust the number of container instances according to the real-time running indicators and the preset computing power elastic scheduling conditions. It realizes the automatic determination of the computing power quota allocated by the AI ​​reasoning container, improves the deployment efficiency of the AI ​​reasoning container, and greatly improves the utilization rate of GPU computing power.

Description

technical field [0001] The present application relates to the technical field of GPU virtualization, in particular to a flexible scheduling method, system, device and storage medium of GPU virtualized computing power. Background technique [0002] With the rapid development of AI technology, many algorithm developers provide the most basic guarantee for deep learning technology, and the development of cloud computing is becoming more and more mature, cloud computing provides computing power guarantee for the progress of AI technology. The gradual maturity of GPU virtualization technology and container technology has improved the resource utilization rate of online reasoning and the flexibility of business deployment, thereby promoting the vigorous development of business. Among them, containerization is a method of software development. In this method, the program, its dependent components and collection packages, and related environment variable configuration files will be ...

AI Technical Summary

Problems solved by technology

However, unlike AI training with a relatively fixed computing cycle and long running time, the call of the AI ​​reasoning container will fluctuate with the fluctuation of the business, and there are often periodic peaks and valleys

Therefore, in the case of large-scale and high-concurrency node requirements, the conventional deployment scheme obviously cannot meet such requirements

[0004] Before the existing AI inference container is deployed, the quota allocation for GPU virtualization computing power needs to go through a series of pressure testing operations and manual analysis of monitoring logs before it can be finally determined, and the usage method is relatively complicated.

In addition, after the AI ​​reasoning container is deployed, its computing power quota and the number of container instances are fixed. When there is a burst of traffic, it will not be able to cope. Mitigates the impact of traffic spikes that reduce business availability

[0005] Therefore, the current allocation of GPU virtualization computing power quotas for AI inference containers still relies on manual operations and related experience. Not only is the use method more complicated, but the deployment efficiency of AI inference containers is low. The peak-valley effect cannot make better use of computing power resources. For this, there is no corresponding solution in related technologies

Images