Reasing platform based on reasoning instance snapshot

Abstract

The invention discloses a reasoning platform based on reasoning instance snapshots in the technical field of AI reasoning platforms. The reasoning platform is a system composed of a gateway, a scheduler, a node agent, reasoning container instances, container runtime and snapshot object storage. The method comprises the following steps: S1, storing CPU and GPU states of an initialized inference container as files capable of being quickly recovered; s2, the GPU state is stored in a memory or a high-speed object storage, when the GPU state is imported into a video memory, the performance exceeding the bandwidth of a file system of an operating system is obtained, the reasoning platform based on the reasoning instance snapshot is reasonable in structural design, the reasoning instance does not occupy GPU and CPU computing resources in a standby state, the reasoning instance can be started in a low-delay mode, and the reasoning efficiency is improved. The InferX reasoning platform can effectively improve the GPU resource utilization rate in the following scenes: when a reasoning instance breaks down, a system can quickly restart a new instance to ensure service continuity; the new instance may be dynamically extended based on real-time load to meet burst request requirements while avoiding GPU resource idling.

Description

Technical Field

[0001] The present invention relates to the technical field of AI reasoning platforms, and in particular to a reasoning platform based on reasoning instance snapshots. Background Art

[0002] background.

[0003] The main technical defects of current AI inference platforms 1. Low-latency inference requirements Background requirements: AI inference platforms need to be able to respond quickly to user requests, typically requiring the response latency to be as short as possible and no more than 5 seconds. However, in scenarios with high concurrency or bursty traffic, existing platforms struggle to meet this stringent requirement.

[0004] 2. Cold start delay is too high Technical challenges: The current platform experiences significant latency when cold-starting GPU inference instances. For example, for a 12-byte model, the cold-start time typically exceeds 10 seconds, far exceeding the ideal response latency of less than 5 seconds. This high latency not only severely impacts the user experience but also reduces overall system efficiency.

[0005] 3. Limitations of the Pre-launch Inference Instance Strategy To reduce cold start latency, existing platforms typically use a pre-warmed instance strategy. However, this solution has the following issues: Idle instance resource waste: Since the arrival time of inference requests is unpredictable, pre-launched inference instances are idle when there are no requests, resulting in a large amount of GPU resources being wasted.

[0006] Redundant deployment leads to reduced GPU utilization: To ensure high availability, platforms often deploy backup instances for each AI model and switch to the primary instance when it fails. However, this redundant deployment further reduces the overall GPU utilization.

[0007] Limited elastic expansion: In high-concurrency scenarios, the system needs to dynamically scale inference instances based on traffic (scale-out) to handle additional requests. To meet low-latency requirements, the platform often reserves multiple backup instances, resulting in long-term inefficient utilization of GPU resources and limiting the system's elastic scalability.

[0008] 4. Caused Problem: High Inference Cost Cost pressure: Due to the high cost of GPU hardware itself, low resource utilization directly drives up inference costs. Furthermore, pre-startup redundancy and maintenance of idle instances further exacerbate the cost burden, impacting overall economic efficiency. In summary, while pursuing low latency and high availability, existing AI inference platforms face key challenges such as long cold start delays, low resource utilization, limited elastic scalability, and high costs. These issues urgently need to be addressed through technological optimization and innovation to improve GPU resource utilization and reduce inference costs, thereby driving the development of AI inference platforms towards greater efficiency and cost-effectiveness. To this end, we propose an inference platform based on inference instance snapshots. Summary of the Invention

[0009] The purpose of the present invention is to provide a reasoning platform based on reasoning instance snapshots to solve the problems raised in the above background technology.

[0010] To achieve the above-mentioned object, the present invention provides the following technical solutions: an inference platform based on inference instance snapshots, comprising a system consisting of a gateway, a scheduler, a node agent, an inference container instance, a container runtime, and a snapshot object storage; S1, saves the CPU and GPU states of the initialized inference container into a file that can be quickly restored; S2, by storing GPU state in memory or high-speed object storage, achieves performance exceeding the operating system file system bandwidth when importing GPU state into video memory; Among them, the inference instance snapshot includes the following parts: S101, container runtime operating system process startup: Based on the secure container runtime developed by our company, its essence is the user-mode virtual machine operating system kernel running on Linux; S102, container image mounting, Linux Namespace initialization, and container network initialization: Similar to starting a common container, completes container image mounting, Linux Namespace initialization, and container network initialization; S103, importing user process CPU state and GPU state metadata: The CPU state and GPU state are divided into two parts and saved as metadata and complete image data. Importing metadata requires less CPU memory, so at this stage, the container only occupies a small amount of CPU memory and does not occupy any GPU resources. S104, GPU state data recovery import: After this stage is completed, the container occupies all GPU resources, including video memory and computing resources; S105, user process CPU state data recovery import: The CPU state is completely imported into the CPU memory. After this stage is completed, the container occupies the entire CPU memory; Among them, snapshot-based rapid recovery includes the following parts: S201, snapshot generation: After the user registers the model container image, the platform first starts the first container according to the normal container startup process, and then generates a container snapshot based on this container instance. This is a one-time operation, and the generated snapshot can be deployed on multiple nodes in the cluster; S202, snapshot-based quick start: After the snapshot is generated, all subsequent starts of the container by the platform are based on this snapshot. Compared with the general container startup process (i.e., startup based only on the container image), the snapshot-based quick start process adds a snapshot generation step.

[0011] Preferably, the gateway is responsible for receiving user inference requests, routing the requests to the corresponding model inference instances, and returning the inference results to the user; Scheduler: When a user request arrives, if the target AI model lacks pre-warmed WARM inference instances, the scheduler will search for and reclaim idle WARM instances to free up GPU resources, and simultaneously start the standby inference instance of the model. The node agent runs on the GPU computing node and is responsible for quickly starting and retrieving inference container instances, managing the GPU vRAM resources of the node, and performing read and write operations on snapshot data in the snapshot object store. Inference container instance, the actual running instance of the user's inference model, based on the platform's proprietary InferX container runtime, runs the user-provided Docker container image; Container runtime: This platform provides an efficient container runtime that supports snapshot creation of inference container instances and enables quick startup. Snapshot object storage is responsible for storing and quickly accessing inference instance snapshots. When a standby container needs to be quickly started, the InferX node agent loads data from this storage and quickly imports it into GPU vRAM and CPU memory, enabling extremely fast startup of the inference container. This storage can be based on multiple local SSDs or remote SSDs.

[0012] Preferably, the inference container is pre-started: before the user inference request arrives, the inference platform will pre-start the inference container in advance and complete the above three steps S101~S103. Since these three steps only require importing metadata into the CPU memory, they only occupy a small amount (200~400MB) of CPU memory. When the pre-start is completed, the container enters the standby state. At this time, the platform will deprive the container of the CPU computing right so that it does not occupy any CPU resources. When the user inference request arrives, the platform will complete the remaining two steps S104 and S105 of the standby container of the model, and then start to respond to the user request, thereby avoiding the delay of the first three steps.

[0013] Preferably, the memory node agent pre-allocates the management and inference container memory sharing: To complete step S104, the platform needs to import the GPU state into the memory. If this step is completed by the container runtime, the GPU memory needs to be allocated first, which may cause a delay of 100 to 500 milliseconds. This delay can be avoided by pre-allocating the memory node agent. The node agent pre-allocates all GPU memory of this node at the global level. When a user request arrives, the node agent is responsible for importing the memory data into the memory, and then shares the memory allocated by the node agent with the runtime of a specific container through GPU memory sharing (for example, the GPU shares memory through CUDA's cudaIpcMemHandle).

[0014] Preferably, CPU memory node agent pre-allocation management and inference container CPU memory sharing based on 2MB large pages: To complete step S105, the platform needs to import the CPU state into the memory, similar to GPU video memory management. The node agent is responsible for global memory pre-allocation and importing CPU memory data. When a request arrives, the node agent shares this part of the memory with the container runtime, using 2MB large page management. Compared with 4K small pages, the container runtime can reduce the delay of 100~300 milliseconds when using this part of the memory.

[0015] Optimally, delayed eviction: When a user request arrives, the platform evicts (i.e., terminates) idle WARM containers of other models to free up GPU resources, and then starts the container of the requested model. The operating system terminating the container will cause a delay of hundreds of milliseconds. Since the platform uses graphics memory node agent pre-allocation and CPU memory node agent pre-allocation based on 2MB large pages, the platform can deprive the evicted container of its CPU usage before it is terminated, and reallocate its graphics memory and memory resources to the newly started container. When the new container is fully started, the eviction operation is completed, thereby saving 50 to 100 milliseconds of delay.

[0016] Preferably, a container snapshot runs multiple containers at the same time. Recovery copy technology: The snapshot of the present invention supports restoring and running multiple instances from the same snapshot at the same time. Multiple recovery instances can run on the same computing node and different computing nodes and have different container network IP addresses.

[0017] Optimized, optional GPU / CPU recovery based on files, high-speed object storage, and CPU memory: When restoring a container, the platform supports saving the CPU / GPU state in storage media such as operating system files, high-speed object storage, or CPU memory. Users can choose different storage media to achieve different performance based on cost and latency requirements.

[0018] Preferably, snapshot-based fast recovery includes two parts: Local object storage: consists of multiple local NVMeSSD disks; Remote object storage: Relies on the RDMA network card on the GPU node. Through the RDMA network card and GPUDirectStorage, data from other nodes is directly read into the local GPU memory and CPU memory. Local object storage: Consists of multiple local NVMeSSD disks and has the following features: Large block partitions (16-64MB): Instance snapshots are partitioned into large blocks (16-64MB) and stored on multiple SSDs. This large block design fully utilizes the sequential read bandwidth of SSDs, thereby maximizing the read bandwidth of all SSDs. User-mode driver: InferX object storage does not rely on the operating system file system, but is based on the SPDK user-mode driver, which can achieve greater read bandwidth and scalability.

[0019] Preferably, multi-tenant NVIDIA GPU NCCL network security isolation based on secure container runtime and GPU virtualization: NVIDIA GPU provides an inter-GPU NCCL high-speed communication network, whose physical network includes NVLink within the node and RDMA network across nodes.

[0020] Compared with the prior art, the present invention has the following advantages: the inference platform based on inference instance snapshots can achieve efficient GPU resource utilization while meeting the following conditions: 1. When there is no inference request: The inference instance does not occupy GPU and CPU computing resources and only occupies a small amount of CPU memory.

[0021] 2. When an inference request arrives: The inference instance can be started with low latency and occupy GPU resources to respond to the inference request.

[0022] 3. Based on this fast startup technology, the InferX inference platform can effectively improve GPU resource utilization and provide low-cost inference services in the following scenarios: 4. On-demand inference: Instances are launched only when inference requests are received, rather than occupying GPU resources for a long time, thereby improving GPU computing utilization.

[0023] 5. On-demand fault recovery: There is no need to reserve backup inference instances for high availability. When an inference instance fails, the system can quickly restart a new instance to ensure service continuity.

[0024] 6. On-demand elastic expansion: Traditional inference platforms require pre-reserved backup instances when the system load exceeds the upper limit. This platform can dynamically expand new instances based on real-time load to meet sudden request demands while avoiding idle GPU resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1A flowchart of an inference instance snapshot and snapshot rapid recovery based on an inference instance snapshot in an inference platform according to an embodiment of the present invention; Figure 2 A flowchart of starting an inference instance snapshot based on an inference instance snapshot of an inference platform according to an embodiment of the present invention; Figure 3 A flowchart of a quick startup of an inference platform snapshot based on an inference instance snapshot provided by an embodiment of the present invention; Figure 4 A functional system diagram of each module of an inference platform based on inference instance snapshots according to an embodiment of the present invention; Figure 5 A system diagram of snapshot generation of an inference platform based on inference instance snapshots provided by an embodiment of the present invention; Figure 6 A system diagram of network security isolation of an inference platform based on inference instance snapshots proposed in an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0027] Example 1: See also Figure 1-6 ,The present invention provides a technical solution: an inference platform based on inference instance snapshots, including a system consisting of a gateway, a scheduler, a node agent, an inference container instance, a container runtime, and a snapshot object storage; The gateway is responsible for receiving user inference requests, routing the requests to the corresponding model inference instances, and returning the inference results to the user; Scheduler: When a user request arrives, if the target AI model lacks pre-warmed WARM inference instances, the scheduler will search for and reclaim idle WARM instances to free up GPU resources, and simultaneously start the standby inference instance of the model. The node agent runs on the GPU computing node and is responsible for quickly starting and retrieving inference container instances, managing the GPU vRAM resources of the node, and performing read and write operations on snapshot data in the snapshot object store. Inference container instance, the actual running instance of the user's inference model, based on the platform's proprietary InferX container runtime, runs the user-provided Docker container image; Container runtime: This platform provides an efficient container runtime that supports snapshot creation of inference container instances and enables quick startup. Snapshot object storage, responsible for storing and quickly reading inference instance snapshots. When a standby container needs to be quickly started, the InferX node agent loads data from this storage and quickly imports it into GPU vRAM and CPU memory, enabling extremely fast startup of the inference container. This storage can be based on multiple local SSDs or remote SSDs. S1, saves the CPU and GPU states of the initialized inference container into a file that can be quickly restored; Among them, the inference instance snapshot includes the following parts: S101, container runtime operating system process startup: Based on the secure container runtime developed by our company, its essence is the user-mode virtual machine operating system kernel running on Linux; S102, container image mounting, Linux Namespace initialization, and container network initialization: Similar to starting a common container, completes container image mounting, Linux Namespace initialization, and container network initialization; S103, importing user process CPU state and GPU state metadata: The CPU state and GPU state are divided into two parts and saved as metadata and complete image data. Importing metadata requires less CPU memory, so at this stage, the container only occupies a small amount of CPU memory and does not occupy any GPU resources. S104, GPU state data recovery import: After this stage is completed, the container occupies all GPU resources, including video memory and computing resources; S105, user process CPU state data recovery import: the CPU state is completely imported into the CPU memory. After this stage is completed, the container occupies the entire CPU memory.

[0028] Inference container instance, the actual running instance of the user's inference model, based on the platform's proprietary InferX container runtime, runs the user-provided Docker container image; Container runtime: This platform provides an efficient container runtime that supports snapshot creation of inference container instances and enables quick startup. Snapshot object storage is responsible for storing and quickly accessing inference instance snapshots. When a standby container needs to be quickly started, the InferX node agent loads data from this storage and quickly imports it into GPU vRAM and CPU memory, enabling extremely fast startup of the inference container. This storage can be based on multiple local SSDs or remote SSDs.

[0029] Pre-starting the inference container: Before a user inference request arrives, the inference platform will pre-start the inference container and complete the above three steps S101 to S103. Since these three steps only require importing metadata into the CPU memory, they only occupy a small amount of CPU memory (200-400MB). After the pre-start is completed, the container enters the standby state. At this time, the platform will deprive the container of the CPU computing right so that it does not occupy any CPU resources. When a user inference request arrives, the platform will complete the remaining two steps S104 and S105 for the standby container of the model and can start responding to the user request, thus avoiding the delay of the first three steps.

[0030] Memory node agent pre-allocation management and inference container memory sharing: To complete step S104, the platform needs to import the GPU state into the memory. If this step is completed by the container runtime, the GPU memory needs to be allocated first, which may cause a delay of 100 to 500 milliseconds. This delay can be avoided by pre-allocation of the memory node agent. The node agent pre-allocates all GPU memory of the node at the global level. When a user request arrives, the node agent is responsible for importing the memory data into the memory, and then shares the memory allocated by the node agent with the runtime of a specific container through GPU memory sharing (for example, GPU shares memory through CUDA's cudaIpcMemHandle).

[0031] CPU memory node agent pre-allocation management and inference container CPU memory sharing based on 2MB large pages: To complete step S105, the platform needs to import the CPU state into the memory, similar to GPU memory management. The node agent is responsible for global memory pre-allocation and importing CPU memory data. When a request arrives, the node agent shares this part of memory with the container runtime, using 2MB large page management. Compared with 4K small pages, the container runtime can reduce the delay of 100-300 milliseconds when using this part of memory.

[0032] Delayed eviction: When a user request arrives, the platform evicts (terminates) idle WARM containers of other models to free up GPU resources, and then starts the container of the requested model. The operating system's termination of the container causes hundreds of milliseconds of delay. Because the platform uses proxy pre-allocation of graphics memory nodes and CPU memory nodes based on 2MB large pages, the platform can deprive the evicted container of its CPU usage before it is terminated and reallocate its graphics memory and memory resources to the newly started container. After the new container is fully started, the eviction operation is completed, saving 50 to 100 milliseconds of delay.

[0033] One container snapshot can run multiple containers at the same time to restore the copy technology: The snapshot of the present invention supports restoring and running multiple instances at the same time from the same snapshot. Multiple restored instances can run on the same computing node and different computing nodes and have different container network IP addresses.

[0034] Optional GPU / CPU recovery based on files, high-speed object storage, and CPU memory: When restoring a container, the platform supports saving the CPU / GPU state in storage media such as operating system files, high-speed object storage, or CPU memory. Users can choose different storage media to achieve different performance based on cost and latency requirements.

[0035] Example 2: See also Figure 1-6 ,The present invention provides a technical solution: an inference platform based on inference instance snapshots, including a system consisting of a gateway, a scheduler, a node agent, an inference container instance, a container runtime, and a snapshot object storage; S2, by storing GPU state in memory or high-speed object storage, achieves performance exceeding the operating system file system bandwidth when importing GPU state into video memory; Among them, snapshot-based rapid recovery includes the following parts: S201, snapshot generation: After the user registers the model container image, the platform first starts the first container according to the normal container startup process, and then generates a container snapshot based on this container instance. This is a one-time operation, and the generated snapshot can be deployed on multiple nodes in the cluster; S202, snapshot-based quick start: After the snapshot is generated, all subsequent starts of the container by the platform are based on this snapshot. Compared with the general container startup process (i.e., startup based only on the container image), the snapshot-based quick start process adds a snapshot generation step.

[0036] Snapshot-based fast recovery: consists of two parts: Local object storage: consists of multiple local NVMe SSD disks; Remote object storage: Relies on the RDMA network card on the GPU node. Through the RDMA network card and GPUDirectStorage, data from other nodes is directly read into the local GPU memory and CPU memory.

[0037] Local object storage: Consists of multiple local NVMe SSD disks and has the following features: Large block partitions (16-64MB): Instance snapshots are partitioned into large blocks (16-64MB) and stored on multiple SSDs. This large block design fully utilizes the sequential read bandwidth of SSDs, thereby maximizing the read bandwidth of all SSDs. User-mode driver: InferX object storage does not rely on the operating system file system, but is based on the SPDK user-mode driver, which can achieve greater read bandwidth and scalability.

[0038] Example 3: See also Figure 1-6 The present invention provides a technical solution: an inference platform based on inference instance snapshots, model inference instance startup delay and limitations of existing optimization methods 1. The inference instance startup delay mainly includes three aspects: Container startup delay The time required to initialize the container environment and start the process.

[0039] Inference framework initialization delay Inference frameworks (such as vLLM) require additional initialization during startup (for example, CUDAGraph initialization typically takes more than 6 seconds).

[0040] Model data loading delay The time required to load the model weights from the storage medium and transfer them to the GPU memory.

[0041] 2. Existing Optimization Methods and Their Limitations Currently, the industry mainly reduces the latency of inference instance startup by accelerating model data loading. Common measures include: Download the model data from the network storage to local high-speed storage (such as NVMe SSD) in advance.

[0042] When an inference request arrives, quickly import the model data into the GPU memory.

[0043] However, this approach has the following limitations, making it difficult to meet the requirement of completing inference within 5 seconds for large models of 12B+: Unable to resolve container startup and inference framework initialization delays Even if the model data is loaded quickly enough, the startup of the container and the initialization of the inference framework (such as vLLM) will still incur additional delays, and the overall response time cannot meet real-time requirements.

[0044] Limited by bus and operating system file read rate Even with RAID storage, file read rates are still limited by the bandwidth of a single PCI bus and the scalability of the operating system's file system read bandwidth. When hundreds of GB of model data need to be read in a very short time, performance bottlenecks are inevitable.

[0045] 3. Conclusion In summary, existing technologies mainly focus on optimizing the model data loading process, but fail to effectively solve the delay problem during the startup of containers and inference frameworks, resulting in a long delay in the startup of the overall inference instance, making it difficult to meet the requirements of on-demand startup of real-time inference.

[0046] Multi-tenant NVIDIA GPU NCCL network security isolation based on secure container runtime and GPU virtualization: NVIDIA GPUs provide a high-speed NCCL communication network between GPUs. Its physical network includes intra-node NVLink and cross-node RDMA networks. Currently, there is no multi-tenant network isolation solution for NCCL in the industry. This paper proposes a multi-tenant NVIDIA GPU NCCL network isolation solution based on secure container runtime and GPU virtualization. Specifically, the secure container runtime runs a virtual machine (VirtualMachine) in Linux user space, consisting of the InferX container runtime guest kernel and the InferX container runtime hostVMM (VirtualMachineMonitor). GPU virtualization uses the InferX NCCL call interception library running in guest user space to hijack NCCL calls from the user inference program process via LD_PRELOAD. The InferX container runtime guest kernel then forwards these calls to the NCCL call security check code running in the InferX container runtime hostVMM for security checks. Since the NCCL call security check code runs in the VMM, a security domain that cannot be attacked by user code, secure isolation of the NCCL network is achieved.

[0047] It should be noted that the GPU in the application can be replaced with an AI accelerator (AI accelerator), including but not limited to Nvidia GPU, TPU (Tensor Processing Unit) and NPU (Neural Processing Unit).

[0048] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus.

[0049] Although the present invention has been described above with reference to exemplary embodiments, various modifications may be made thereto and equivalent components may be substituted without departing from the scope of the present invention. In particular, the various features of the embodiments disclosed herein may be combined with one another in any manner, provided no structural conflicts exist. The omission of an exhaustive description of these combinations in this specification is solely for the sake of space and resource conservation. Therefore, the present invention is not limited to the specific embodiments disclosed herein, but encompasses all technical solutions within the scope of the claims.

Claims

1. A reasoning platform based on reasoning instance snapshots, characterized by: Includes a system consisting of a gateway, scheduler, node agent, inference container instance, container runtime, and snapshot object storage; S1, saves the CPU and GPU states of the initialized inference container into a file that can be quickly restored; S2, by storing GPU state in memory or high-speed object storage, and then importing the GPU state into video memory, achieves performance that exceeds the operating system file system bandwidth; Among them, the inference instance snapshot includes the following parts: S101, container runtime operating system process startup: Based on the secure container runtime developed by our company, its essence is the user-mode virtual machine operating system kernel running on Linux; S102, container image mounting, Linux Namespace initialization, and container network initialization: Similar to starting a common container, completes container image mounting, Linux Namespace initialization, and container network initialization; S103, importing user process CPU state and GPU state metadata: The CPU state and GPU state are divided into two parts and saved as metadata and complete image data. Importing metadata requires less CPU memory, so at this stage, the container only occupies a small amount of CPU memory and does not occupy any GPU resources. S104, GPU state data recovery import: After this stage is completed, the container occupies all GPU resources, including video memory and computing resources; S105, user process CPU state data recovery import: The CPU state is completely imported into the CPU memory. After this stage is completed, the container occupies the entire CPU memory; Among them, snapshot-based rapid recovery includes the following parts: S201, snapshot generation: After the user registers the model container image, the platform first starts the first container according to the normal container startup process, and then generates a container snapshot based on this container instance. This is a one-time operation, and the generated snapshot can be deployed on multiple nodes in the cluster; S202, snapshot-based quick start: After the snapshot is generated, all subsequent starts of the container by the platform are based on this snapshot. Compared with the general container startup process (i.e., startup based only on the container image), the snapshot-based quick start process adds a snapshot generation step.

2. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: The gateway is responsible for receiving user inference requests, routing the requests to the corresponding model inference instances, and returning the inference results to the user; Scheduler: When a user request arrives, if the target AI model lacks pre-warmed WARM inference instances, the scheduler will search for and reclaim idle WARM instances to free up GPU resources, and simultaneously start the standby inference instance of the model. The node agent runs on the GPU computing node and is responsible for quickly starting and retrieving inference container instances, managing the GPU vRAM resources of the node, and performing read and write operations on snapshot data in the snapshot object store. Inference container instance, the actual running instance of the user's inference model, based on the platform's proprietary InferX container runtime, runs the user-provided Docker container image; Container runtime: This platform provides an efficient container runtime that supports snapshot creation of inference container instances and enables quick startup. Snapshot object storage is responsible for storing and quickly accessing inference instance snapshots. When a standby container needs to be quickly started, the InferX node agent loads data from this storage and quickly imports it into GPU vRAM and CPU memory, enabling extremely fast startup of the inference container. This storage can be based on multiple local SSDs or remote SSDs.

3. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: Pre-starting the inference container: Before a user inference request arrives, the inference platform will pre-start the inference container and complete the above three steps S101 to S103. Since these three steps only require importing metadata into the CPU memory, they only occupy a small amount of CPU memory (200-400MB). After the pre-start is completed, the container enters the standby state. At this time, the platform will deprive the container of the CPU computing right so that it does not occupy any CPU resources. When a user inference request arrives, the platform will complete the remaining two steps S104 and S105 for the standby container of the model and can start responding to the user request, thus avoiding the delay of the first three steps.

4. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: Memory node agent pre-allocation management and inference container memory sharing: To complete step S104, the platform needs to import the GPU state into the memory. If this step is completed by the container runtime, the GPU memory needs to be allocated first, which may cause a delay of 100 to 500 milliseconds. This delay can be avoided by pre-allocation of the memory node agent. The node agent pre-allocates all GPU memory of the node at the global level. When a user request arrives, the node agent is responsible for importing the memory data into the memory, and then shares the memory allocated by the node agent with the runtime of a specific container through GPU memory sharing (for example, GPU shares memory through CUDA's cudaIpcMemHandle).

5. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: CPU memory node agent pre-allocation management and inference container CPU memory sharing based on 2MB large pages: To complete step S105, the platform needs to import the CPU state into the memory, similar to GPU memory management. The node agent is responsible for global memory pre-allocation and importing CPU memory data. When a request arrives, the node agent shares this part of memory with the container runtime, using 2MB large page management. Compared with 4K small pages, the container runtime can reduce the delay of 100-300 milliseconds when using this part of memory.

6. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: Delayed eviction: When a user request arrives, the platform evicts (terminates) idle WARM containers of other models to free up GPU resources, and then starts the container of the requested model. The operating system's termination of the container causes hundreds of milliseconds of delay. Because the platform uses proxy pre-allocation of graphics memory nodes and CPU memory nodes based on 2MB large pages, the platform can deprive the evicted container of its CPU usage before it is terminated and reallocate its graphics memory and memory resources to the newly started container. After the new container is fully started, the eviction operation is completed, saving 50 to 100 milliseconds of delay.

7. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: One container snapshot can run multiple containers at the same time to restore the copy technology: The snapshot of the present invention supports restoring and running multiple instances at the same time from the same snapshot. Multiple restored instances can run on the same computing node and different computing nodes and have different container network IP addresses.

8. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: Optional GPU / CPU recovery based on files, high-speed object storage, and CPU memory: When restoring a container, the platform supports saving the CPU / GPU state in storage media such as operating system files, high-speed object storage, or CPU memory. Users can choose different storage media to achieve different performance based on cost and latency requirements.

9. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: Snapshot-based fast recovery: consists of two parts: Local object storage: consists of multiple local NVMe SSD disks; Remote object storage: Relies on the RDMA network card on the GPU node. Through the RDMA network card and GPUDirectStorage, data from other nodes is directly read into the local GPU memory and CPU memory. Local object storage: Consists of multiple local NVMeSSD disks and has the following features: Large block partitions (16-64MB): Instance snapshots are partitioned into large blocks (16-64MB) and stored on multiple SSDs. This large block design fully utilizes the sequential read bandwidth of SSDs, thereby maximizing the read bandwidth of all SSDs. User-mode driver: InferX object storage does not rely on the operating system file system, but is based on the SPDK user-mode driver, which can achieve greater read bandwidth and scalability.

10. The reasoning platform based on reasoning instance snapshots according to claim 1, characterized in that: Multi-tenant NVIDIA GPU NCCL network security isolation based on secure container runtime and GPU virtualization: NVIDIA GPU provides an inter-GPU NCCL high-speed communication network, whose physical network includes NVLink within the node and RDMA network across nodes.

Images