Server-free platform-oriented low-latency and SLO perception general GPU (Graphics Processing Unit) sharing system

Abstract

The invention discloses a low-delay and SLO perception general GPU sharing system for a server-free platform, and relates to the field of GPUs, the invention provides an FAASHARE and an efficient server-free GPU sharing framework, the FAASHARE is composed of an MSContainer and a GPU sharing strategy based on Bayesian optimization, and the MSContainer container can provide a preheating environment for a GPU-request function in advance and reduce cold start delay; the Bayesian optimization can dynamically adjust resource allocation to adapt to different task loads. According to the method, the cold start delay is reduced, and the GPU resource utilization rate is improved while the service level target is ensured.

Description

Technical Field

[0001] The present invention relates to the field of GPU, and in particular to a low-latency and SLO-aware general-purpose GPU sharing system for a serverless platform. Background Art

[0002] Serverless computing platform is an emerging cloud computing paradigm that aims to simplify the workload of developers by dividing computing tasks into the smallest execution unit - function, providing a more lightweight resource abstraction. Compared with traditional infrastructure as a service (IaaS) and platform as a service (PaaS), serverless computing automatically manages the scheduling and expansion of computing resources, allowing developers to focus on application logic without having to worry about underlying hardware resources. However, despite the huge potential of serverless computing, the current platform still faces some challenges in practical applications, especially high latency and GPU sharing. As the complexity of deep learning and high-performance computing tasks continues to increase, the introduction of GPU as a computing resource has become particularly important. GPU provides parallel computing capabilities far beyond traditional CPUs, which can significantly improve the execution efficiency of deep learning tasks, especially when processing large amounts of data and complex models.

[0003] On the one hand, cold start is a core challenge in serverless computing. Although serverless platforms have high elasticity and automatic scaling capabilities, the response time during task execution is significantly increased due to the cold start delay during function startup. Cold start delays are usually caused by the startup time of the container, the initialization of the CUDA context, and the loading of third-party libraries (such as cuDNN, PyTorch, etc.). Especially in GPU-accelerated tasks, this delay may be several times the execution time of the computing task. This problem is particularly prominent in delay-sensitive scenarios such as real-time reasoning and online services, which seriously affects the application effect of serverless computing.

[0004] On the other hand, GPU sharing also faces huge challenges on serverless computing platforms. With the concurrent execution of multiple tasks, how to reasonably allocate GPU resources to avoid resource contention and ensure efficient task execution is the key to achieving high GPU utilization. Existing GPU sharing technologies, such as NVIDIA's Multi-Process Service (MPS), can support multiple CUDA processes to share a single GPU, thereby improving throughput, especially in scenarios where multiple small-scale tasks are processed in parallel. However, the parallel mechanism of MPS also faces challenges in task scheduling and efficient utilization of GPU resources. Although MPS can support the parallel execution of multiple processes, the allocation and scheduling of GPU resources becomes more complicated in the case of high-concurrency task execution. Resource competition between tasks may lead to performance degradation, which in turn affects the overall task execution efficiency. Therefore, how to optimize the scheduling mechanism of MPS and reduce resource contention between tasks is still an important problem in current GPU sharing. Summary of the invention

[0005] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to reduce resource contention between tasks.

[0006] To achieve the above-mentioned object, the present invention provides a low-latency and SLO-aware general GPU sharing system for serverless platforms, characterized in that the present invention proposes FAASHARE, an efficient serverless-based GPU sharing framework, wherein the FAASHARE consists of MSContainer and a GPU sharing strategy based on Bayesian optimization, and the MSContainer container can provide a preheating environment for GPU-required functions in advance to reduce cold start delay; Bayesian optimization can dynamically adjust resource allocation to adapt to different task loads.

[0007] Furthermore, the MSContainer provides a preheating mechanism to significantly reduce cold start delay by creating a preheating environment in advance.

[0008] Furthermore, the MSContainer will start a preheating container before the function is executed, and load the required CUDA context and third-party libraries.

[0009] Furthermore, the preheated container will be kept in a running state until the serverless function needs to be executed and then injected into the preheated environment for execution.

[0010] Furthermore, the MSContainer supports serverless GPU functions in multiple programming languages.

[0011] Furthermore, the MSContainer will back up the state of the preheated container before each function execution, thereby ensuring that the program can be quickly restored when an error occurs.

[0012] Furthermore, the Bayesian optimization strategy automatically adjusts the granularity of GPU resources by analyzing the resource requirements of the task so that it can adapt to different workloads.

[0013] Furthermore, when multiple tasks are executed in parallel, the Bayesian optimization can quickly adjust resource allocation according to resource usage between tasks to reduce resource competition.

[0014] Furthermore, the Bayesian optimization strategy optimizes the balance between exploration and utilization in traditional Bayesian optimization, allowing the system to dynamically adjust the weight between exploring new resource configurations and optimizing existing configurations based on the performance data of the current task, thereby accelerating the search for the optimal configuration.

[0015] Furthermore, FAASHARE introduces an incremental update mechanism into the Bayesian optimization strategy, so that when a new task arrives, the system can quickly adjust resource allocation based on the existing configuration.

[0016] The present invention has the following technical effects:

[0017] (1) The FAASHARE framework significantly reduces the cold start latency in serverless computing by introducing MSContainer. MSContainer provides a preheating environment for serverless functions required by the GPU, effectively reducing the impact of multiple delay components such as CUDA context initialization and library loading, thereby significantly speeding up the initial execution time of the function. In practical applications, FAASHARE can achieve up to 60 times improvement in cold start latency, ensuring low latency support for real-time reasoning and online data processing tasks, and greatly improving task response speed. Compared with existing serverless frameworks, FAASHARE has shown significant advantages in achieving low latency goals, improving the utilization efficiency of GPU resources and task processing speed, especially in the case of high concurrency tasks, and can more effectively cope with low latency requirements.

[0018] (2) FAASHARE models the GPU resource sharing problem through Bayesian optimization and performs accurate resource allocation based on the dynamic requirements of tasks. By optimizing the exploration and exploitation strategies, FAASHARE can effectively allocate GPU resources in a serverless environment, reduce resource competition between tasks, and improve GPU concurrent utilization. In a large number of experiments, FAASHARE's resource configuration optimization accuracy exceeds 90%, indicating that this method can significantly improve GPU sharing performance and reduce the complexity and computational overhead brought by traditional manual configuration.

[0019] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a FAASHARE framework of a preferred embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0022] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is appropriately exaggerated in some places in the drawings.

[0023] The present invention proposes FAASHARE, which is an efficient serverless GPU sharing framework that aims to reduce cold start delays, ensure service level objectives and improve GPU resource utilization. FAASHARE consists of MSContainer and a GPU sharing strategy based on Bayesian optimization. The following is a specific embodiment of FAASHARE:

[0024] Example 1: MSContainer

[0025] MSContainer is a key component in FAASHARE designed to solve the cold start problem. The cold start problem refers to the long delay caused by the loading of containers and required libraries when a serverless function is executed for the first time. This is particularly serious for latency-sensitive tasks such as online reasoning and real-time data processing. To solve this problem, MSContainer provides a preheating mechanism that significantly reduces cold start latency by creating a preheating environment in advance. Specifically, MSContainer starts a preheating container before the function is executed and loads the required CUDA context, third-party libraries (such as cuDNN, PyTorch, TensorFlow, etc.), and other resources. The preheating container will be kept running until the serverless function needs to be executed and then injected into the preheating environment for execution. In addition, in order to deal with the problem of abnormal container exit, MSContainer will also back up the state of the preheater before each function execution, so that the program can be quickly restored when an error occurs. In this way, MSContainer significantly reduces cold start latency and provides a scalable solution that supports serverless GPU functions in multiple programming languages ​​(such as Python and Java).

[0026] Example 2: Bayesian Optimization and Resource Allocation

[0027] In order to optimize the sharing of GPU resources in serverless environments, FAASHARE introduces a GPU sharing strategy based on Bayesian optimization. In a high-concurrency and resource-competing environment, multiple tasks may share the GPU at the same time, which can easily lead to resource contention and performance degradation. To solve this problem, we designed a GPU resource configuration algorithm based on Bayesian optimization, which aims to dynamically adjust the GPU resource allocation according to the characteristics of the task, thereby improving GPU utilization and reducing resource competition between tasks.

[0028] First, FAASHARE's Bayesian optimization strategy automatically adjusts the granularity of GPU resources by analyzing the resource requirements of the task so that it can adapt to different workloads. When multiple tasks are executed in parallel, Bayesian optimization will quickly adjust resource allocation based on the resource usage between tasks to reduce resource contention. Secondly, the Bayesian optimization strategy optimizes the balance between exploration and utilization in traditional Bayesian optimization, allowing the system to dynamically adjust the weight between exploration (exploring new resource configurations) and utilization (optimizing existing configurations) based on the performance data of the current task, thereby accelerating the search for the optimal configuration. Third, FAASHARE also introduces an incremental update mechanism in the Bayesian optimization strategy, so that when a new task arrives, the system can quickly adjust resource allocation based on the existing configuration, avoiding the high computational cost and long configuration search in traditional Bayesian optimization.

[0029] Through these optimizations, FAASHARE can achieve efficient GPU resource sharing in the serverless computing platform computing environment, reduce task execution latency, and significantly improve system performance.

[0030] The preferred specific embodiments of the present invention are described in detail above. It should be understood that ordinary technicians in the field can make many modifications and changes based on the concept of the present invention without creative work. Therefore, all technical solutions that can be obtained by technicians in the technical field based on the concept of the present invention through logical analysis, reasoning or limited experiments on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms, characterized in that: The present invention proposes FAASHARE, an efficient serverless GPU sharing framework. The FAASHARE consists of MSContainer and a GPU sharing strategy based on Bayesian optimization. The MSContainer can provide a preheating environment for GPU-required functions in advance to reduce cold start delay; Bayesian optimization can dynamically adjust resource allocation to adapt to different task loads.

2. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms according to claim 1, characterized in that: The MSContainer provides a preheating mechanism to significantly reduce the cold start delay by creating a preheating environment in advance.

3. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms as claimed in claim 2, characterized in that: The MSContainer will start a preheating container before executing the function and load the required CUDA context and third-party libraries.

4. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms as claimed in claim 3, characterized in that: The preheated container will be kept in a running state until the serverless function needs to be executed and then injected into the preheated environment for execution.

5. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms as claimed in claim 4, characterized in that: The MSContainer supports serverless GPU functions in multiple programming languages.

6. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms as claimed in claim 5, characterized in that: The MSContainer will back up the state of the preheated container before each function execution, thereby ensuring that the program can be quickly restored when an error occurs.

7. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms as claimed in claim 6, characterized in that: The Bayesian optimization strategy automatically adjusts the granularity of GPU resources by analyzing the resource requirements of the tasks so that it can adapt to different workloads.

8. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms as claimed in claim 7, characterized in that: When multiple tasks are executed in parallel, the Bayesian optimization will quickly adjust resource allocation according to the resource usage between tasks to reduce resource competition.

9. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms as claimed in claim 8, characterized in that: The Bayesian optimization strategy optimizes the balance between exploration and utilization in traditional Bayesian optimization, allowing the system to dynamically adjust the weight between exploring new resource configurations and optimizing existing configurations based on the performance data of the current task, thereby accelerating the search for the optimal configuration.

10. The low-latency and SLO-aware general-purpose GPU sharing system for serverless platforms according to claim 9, characterized in that: The FAASHARE introduces an incremental update mechanism into the Bayesian optimization strategy, so that when a new task arrives, the system can quickly adjust resource allocation based on the existing configuration.

Images