A method and system for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling

Abstract

This application discloses a method and apparatus for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling. The method includes: after a user submits a declarative configuration of the inference resource pool, the inference pool controller constructs a preheating resource pool base and establishes lock-free pooling state management rules based on metadata tags; after a user submits the inference service configuration, the inference service controller matches and locks the target workload through a multi-dimensional pooling scheduling framework, completing activation and service deployment; when the service terminates, the two controllers collaboratively execute differentiated resource reclamation. This invention eliminates the need for an external database, reduces the cold start time of the inference service to the second level, adapts to business tidal scenarios to achieve efficient reuse of computing power, and meets the complex scheduling requirements of a pre-filling and decoding separation architecture.

Description

Technical Field

[0001] This application relates to the field of data processing technology, specifically to a method and system for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling. Background Technology

[0002] With the evolution of artificial intelligence technology, large language models have become the core infrastructure of Model-as-a-Service (MaaS) platforms. However, the parameter scale of large models is often as high as tens or even hundreds of billions, which brings extremely severe challenges to the deployment and resource scheduling of the underlying inference services:

[0003] Extreme cold start latency bottleneck: Unlike traditional stateless network services, large model inference services require not only initializing a complex deep learning computing environment upon startup, but also reading and loading massive model weight files (typically tens to hundreds of gigabytes) from persistent storage into GPU memory. This complete cold start process usually takes tens of seconds to several minutes. This latency is unacceptable for business applications when faced with sudden surges in concurrent requests.

[0004] The tidal nature of workloads and the extremely high demand for resource reuse: In real-world enterprise applications, computing resources are extremely expensive, making it impossible to equip all models with dedicated, resident GPUs. Furthermore, business traffic often exhibits significant tidal characteristics and Service Level Objectives (SLOs). For example, in enterprise AI platforms:

[0005] Peak daytime hours: Primarily online inference tasks such as real-time dialogue, customer service Q&A, and code completion. These tasks have extremely high service level requirements for first-letter response time (TTFT), demanding that underlying computing resources be available on demand and respond instantly.

[0006] Nighttime off-peak hours: Online traffic drops sharply, requiring the system to reclaim a large amount of idle computing power for offline asynchronous tasks such as summarizing long context documents, batch translation, and data distillation. These tasks are extremely time-consuming, but have a high tolerance for latency in service level objectives.

[0007] Sudden surge in traffic during peak hours or nighttime, requiring the system to immediately pause or release the computing power of low-priority batch tasks and switch to a large-scale online dialogue model within seconds. Without underlying pooling technology, such rapid switching across models and scenarios is simply impossible.

[0008] The evolution of complex distributed inference architectures: To overcome the physical limitations of a single GPU's memory and computing power, modern large-model inference commonly employs tensor parallelism, distributing the computational tasks of a single model across multiple GPUs or even multiple physical machines for collaborative execution. Simultaneously, in pursuit of ultimate throughput and first-word response time, the industry has widely adopted a "pre-filling and decoding separation" architecture. Under this architecture, a complete inference task is no longer a single process, but rather a cluster of interdependent microservices comprised of a "pre-filling workload" responsible for handling long-context semantic understanding and a "decoding workload" responsible for character-by-character generation.

[0009] State synchronization and scheduling coordination under high concurrency: In the complex architecture described above, the underlying computing nodes need to frequently report their status and hand over tasks. How to accurately grasp the real-time status of massive heterogeneous computing resources (such as idle GPUs and different types of accelerator cards) in a high-concurrency cluster environment, and match the optimal physical node for specific multi-role inference tasks, has become a core engineering challenge that MaaS platforms urgently need to solve.

[0010] Currently, the most similar existing implementation in the industry for managing and scheduling large-scale model inference services typically adopts a traditional microservice resource scheduling architecture consisting of imperative script orchestration and an external state database. The specific implementation logic is as follows:

[0011] At the workload lifecycle and switching level (lacking pooling elasticity): existing solutions employ static binding or hard rebuilding. When faced with the daytime dialogue model / nighttime batch model switching mentioned earlier, traditional workflow scripts directly issue instructions to destroy nighttime nodes at the underlying level, and then restart the dialogue nodes required for the daytime. This process of destroying and rebuilding simultaneously involves lengthy image fetching, framework initialization, and weight loading, resulting in a resource vacuum period of several minutes during the switch, severely compromising the service level objectives of high-priority dialogue tasks.

[0012] At the state management and persistence level: Existing technologies typically introduce a separate external database to act as a global resource state registry. The scheduler records the state of each compute node—whether it is idle, loading, or in service—by sending read and write requests to the database. In distributed concurrent scheduling, to prevent multiple tasks from simultaneously preempting the same idle node, the scheduler must explicitly implement a complex distributed locking mechanism in the business logic code.

[0013] At the resource monitoring and status synchronization level: the existing solution heavily relies on an active polling mechanism. The scheduling center starts a background scheduled task to send heartbeat probe requests to each physical node every few seconds to query its model loading progress or health status.

[0014] At the workload lifecycle management level: resource startup and destruction are typically handled by imperative process scripts (executed sequentially). When a deployment command is received, the script triggers node creation, followed by a lengthy cold start (pulling images, loading weights). When a service needs to be taken offline, the system issues a destruction command and waits for node feedback.

[0015] At the scheduling strategy level: most existing schedulers follow the traditional cloud computing resource scheduling logic, mainly performing simple resource matching based on the remaining CPU, memory, and GPU quantities of physical machines. They treat multi-replica or multi-role inference tasks as independent ordinary computing units and distribute them accordingly.

[0016] The existing technology has the following disadvantages:

[0017] State persistence and synchronization heavily rely on external databases to store state, resulting in a bloated architecture; proactive polling of resource state has a time lag, and decisions are easily based on expired data.

[0018] Inference service workloads require a cold start and have a long readiness time.

[0019] Distributed coordination is challenging because when dealing with concurrent requests, a significant amount of code needs to be written to handle issues such as concurrent locks and request consistency.

[0020] The lifecycle is unreliable, the creation and destruction processes of resources are fragmented, and zombie nodes and resource leaks are very likely to occur when services are abnormally interrupted.

[0021] The lack of workload coordination makes it impossible to bind multiple cooperating nodes such as pre-filling and decoding into a single atomic business unit for management.

[0022] The scheduling strategy is too simplistic, and the scheduler cannot perceive the constraints of node roles. This can easily lead to incompatible tasks being scheduled to the same physical machine, causing competition for computing power. Summary of the Invention

[0023] The purpose of this invention is to provide a method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, which at least solves one of the aforementioned technical problems.

[0024] One aspect of the present invention provides a method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, comprising:

[0025] Based on the inference resource pool configuration submitted by the user, a custom resource instance of the inference resource pool that is globally effective across the cluster is generated; the inference resource pool configuration submitted by the user includes a warm-up mode, target node range, concurrent deployment rate limiting strategy, and workload preset templates under different tensor parallel scales, wherein the warm-up mode includes two types: empty shell mode and memory resident mode;

[0026] Generate an inference resource pool base with available standby workloads based on custom resource instances in the inference resource pool, and generate globally effective pooling state management rules.

[0027] Obtain the deployment requirements for inference services submitted by users;

[0028] Generate custom resource instances for the inference service that are active within the namespace based on the deployment requirements submitted by the user.

[0029] Generate a pre-built set of workloads with exclusive lock based on custom resource instances from the inference service;

[0030] Activate the pre-configured set of workloads. After activation, update the lifecycle status of the workloads to ready. Group all ready workloads belonging to this inference service into a complete business service unit by node role. After verifying that all workloads in the group are ready, update the lifecycle status of all workloads to active in batches, thus obtaining an active inference service cluster.

[0031] Optionally, the step of generating an inference resource pool base with available standby workloads based on custom resource instances of the inference resource pool and generating globally effective pooling state management rules includes:

[0032] Based on the configuration parameters within the custom resource instance of the inference resource pool, a backup inference workload is created within the target node range to complete the workload preheating process, thereby obtaining an inference resource pool base with available standby workloads. Using the workload's own metadata tags as the sole source of state facts, a lifecycle state machine and atomic state transition rules for the pooled workload are established, thereby obtaining globally effective pooled state management rules. The lifecycle states include six types: idle, preheated, pre-configured, ready, active, and releasing. The mutual exclusion of tag updates is ensured through atomic update operations at the cluster level.

[0033] Optionally, generating a custom resource instance for the inference service within the namespace based on the user-submitted inference service deployment requirements includes:

[0034] Based on the inference resource pool base with available standby workloads, the business configuration parameters of the inference service to be deployed are submitted through a namespace-level declarative API, thereby generating a custom resource instance of the inference service. The custom resource instance of the inference service includes all business configuration parameters submitted by the user. The business configuration parameters include the associated inference resource pool, the target large model name, the inference service topology configuration, and the hardware acceleration configuration. The inference service topology configuration includes node roles, tensor parallel scale, and number of replicas. The node roles include three types: pre-filled, decoded, and regular.

[0035] Optionally, generating a pre-built set of workloads with exclusive locks based on the custom resource instance of the inference service includes:

[0036] After obtaining the custom resource instance of the inference service, a scheduling request is constructed based on the business configuration parameters within the instance. The multi-dimensional pooling scheduling framework is invoked to filter out standby workloads that meet all hard constraints from the generated inference resource pool base, and then the optimal matching workload is selected through weighted scoring.

[0037] The metadata tags of the selected workload are updated atomically, its lifecycle status is updated to preset, and the unique identifier and topology group name of the associated inference service are written, thereby obtaining a set of preset workloads that are exclusively locked and can only be used for this inference service.

[0038] Optionally, after obtaining the active inference service cluster, the method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling further includes:

[0039] When a user deletes the custom resource instance of the inference service corresponding to the generated active inference service cluster, after listening to the deletion command, a differentiated resource reclamation process is executed on all workloads associated with the service according to the warm-up mode in the custom resource instance of the inference resource pool. This restores the workload to the standby state and replenishes it to the inference resource pool base, completing the cyclic reuse of pooled resources and the entire process closed loop. In the empty shell mode, the corresponding workload is destroyed and the inference pool controller replenishes new standby workloads to the resource pool. In the memory resident mode, the workload is retained and reset to the standby state.

[0040] Optionally, the empty shell mode includes:

[0041] Based on the workload preset template in the custom resource instance of the inference resource pool, the workload is pre-started and the underlying computing environment of the Python runtime and deep learning framework is fully initialized without loading any large model weights. After initialization, the life cycle state of the workload is set to idle, thereby generating a general standby empty shell workload with no model binding and prefabricated according to tensor parallel specifications.

[0042] The memory-resident modes include:

[0043] Based on the workload preset template in the custom resource instance of the inference resource pool, the workload is pre-started and the underlying computing environment is initialized. The weight file of the target large model is loaded into the graphics processor's video memory. Then, the unloading action is performed to transfer the weight data without loss and cache it in the host machine's host memory. After completion, the life cycle status of the workload is set to pre-warmed, thereby completing the deep pre-warming of model preloading-unloading and generating a pre-warmed workload in a high-priority standby state.

[0044] Optionally, the globally effective pooling state management rules include:

[0045] The rule for changing from idle state to preset state is used for workloads in empty shell mode. After the scheduling and selection of idle state workloads is completed, the life cycle state of the workload is changed from idle state to preset state by atomically updating the workload metadata tag. At the same time, the unique identification of the associated inference service is written to complete this state change.

[0046] Change rules from preheated state to preset state: The change rules from preheated state to preset state are used for workloads in memory-resident mode. After the scheduling selection of preheated state workloads is completed, the life cycle state of the workload is changed from preheated state to preset state by atomically updating the workload metadata tag. At the same time, the unique identification of the associated inference service is written to complete this state change.

[0047] Change rules from preset state to ready state: The change rules from preset state to ready state are used by the inference service controller to change the life cycle state of the workload from preset state to ready state by atomically updating the workload metadata tags after the model loading or weight restoration activation operation of the preset state workload is completed.

[0048] The rules for changing from ready to active state: After completing the role grouping and aggregation management of all ready workloads belonging to the same inference service, the rules for changing the lifecycle state of the corresponding workload from ready to active state are changed by atomically updating the workload metadata tags.

[0049] Change rules from active state to release state and then to preheated state: The change rules from active state to release state and then to preheated state apply to workloads in memory-resident mode. When a user deletes a custom resource instance of the inference service, the inference service controller first changes the lifecycle state of the corresponding active workload from active state to release state. After the model weight release operation is completed, the lifecycle state is changed from release state to preheated state, so that it returns to the inference resource pool base and enters the standby loop, thus completing this state change.

[0050] Closed-loop rule from active state to destruction, and then to idle state replenishment: The closed-loop rule from active state to destruction, and then to idle state replenishment applies to workloads in empty shell mode. When a user deletes a custom resource instance of the inference service, the corresponding active workload is destroyed, the inference pool controller automatically creates a new workload, sets the lifecycle state of the new workload to idle state, and replenishes it to the inference resource pool base to maintain the preset replica level of the resource pool, thus completing this state closed loop.

[0051] Optionally, after obtaining the custom resource instance of the inference service, a scheduling request is constructed based on the business configuration parameters within the instance. A multi-dimensional pooled scheduling framework is invoked to filter out standby workloads that meet all hard constraints from the generated inference resource pool base. The optimal matching workload is then selected through a weighted scoring system, including:

[0052] The serial filtering chain is executed in sequence: resource pool type filtering, workload availability filtering, capacity filtering, and role anti-affinity filtering. If any filtering rule fails, the corresponding physical node is directly eliminated. Nodes that pass all rules enter the scoring stage. Among them, role anti-affinity filtering requires that the workloads of the three types of roles (pre-filled, decoded, and regular) must be deployed on different physical nodes, and the result is a set of candidate physical nodes that meet all hard constraints.

[0053] The parallel scorer performs multi-dimensional scoring on the filtered candidate nodes in parallel. The scores of each scorer are accumulated and sorted from high to low according to the total score. The result is the best-matched physical node and the corresponding set of standby workloads.

[0054] Optionally, upon detecting a deletion command, the process of performing a differentiated resource reclamation procedure for all workloads associated with the service, based on the preheating mode of the custom resource instance in the inference resource pool, includes:

[0055] Empty shell mode recycling process: After a user deletes a custom resource instance of the inference service, the inference service controller destroys all workloads associated with the service. At the same time, the inference pool controller automatically creates new idle workloads to supplement the inference resource pool base, resulting in an idle inference resource pool base that maintains a preset replica level.

[0056] Memory-resident mode recycling process: After a user deletes a custom resource instance of the inference service, the inference service controller retains all workloads associated with the service without destroying them, calls the workload activation module to execute the model weight release instruction, and pushes the weights in the GPU memory back down to the host memory cache. First, the lifecycle status of the workload is updated to "releasing", and after the release is completed, it is updated to "preheated". The result is a preheated inference resource pool base that returns to the high-priority standby state.

[0057] This application also provides a device for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, the device comprising:

[0058] The inference pool controller generates a custom resource instance of the inference resource pool that is globally effective across the cluster based on the inference resource pool configuration submitted by the user. The inference resource pool configuration submitted by the user includes a warm-up mode, a target node range, a concurrent deployment rate limiting strategy, and a preset workload template for different tensor parallel scales. The warm-up mode includes two types: empty shell mode and memory-resident mode.

[0059] The inference pool controller generates an inference resource pool base with available standby workloads based on custom resource instances in the inference resource pool, and generates globally effective pooling state control rules.

[0060] The inference service controller obtains the inference service deployment requirements submitted by the user;

[0061] The inference service controller generates custom resource instances of the inference service that are active within the namespace based on the inference service deployment requirements submitted by the user.

[0062] The inference service controller generates a pre-built set of workloads that are exclusively locked based on the custom resource instance of the inference service;

[0063] The inference service controller performs an activation operation on the pre-configured set of workloads. After activation, it updates the lifecycle status of the workloads to ready. All ready workloads belonging to this inference service are grouped and aggregated into a complete business service unit according to node roles. After verifying that all workloads in the group are ready, the lifecycle status of all workloads is updated to active in batches, thus obtaining an active inference service cluster.

[0064] This application has the following advantages:

[0065] This application defines workload pools through declarative APIs, pre-builds a resource base in standby state, and adopts a workload pooling mechanism to completely separate the workload warm-up and activation phases. This solves the core defect of traditional large model inference services, which require complete environment initialization and weight loading when starting from scratch, often taking tens of seconds or even more than a minute. It reduces the cold start time when requests are issued to the second level, and significantly accelerates the startup and deployment time of inference service workloads.

[0066] This application achieves dynamic pooling scheduling through a dual automated controller collaborative architecture of the inference pool controller and the inference service controller. It solves the problem that the traditional static allocation or complete destruction and reconstruction mechanism cannot complete the resource state switching in seconds. Without violating the service level objective (SLO) of high priority tasks, it realizes the rapid flow and efficient reuse of computing resources between different models and different business scenarios, and significantly improves the utilization rate of expensive large model computing resources. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating a method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, according to an embodiment of this application. Detailed Implementation

[0068] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be described in more detail below with reference to the accompanying drawings. In the drawings, the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The described embodiments are some, but not all, embodiments of this application. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0069] like Figure 1 The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, as shown, defines workload pools through a declarative application programming interface (API) and pre-builds a computing resource base in a "standby state." Relying on a multi-dimensional scheduling framework designed specifically for pooling scenarios, it accurately matches target workloads under complex multi-dimensional constraints. Combined with a customized full lifecycle state transition mechanism, it realizes a closed-loop process of automatic preheating, second-level activation, and secure recycling of pooled resources.

[0070] To achieve the above-mentioned inventive concept, this invention provides a method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, as detailed below:

[0071] Step 1: The user submits the declarative configuration of the inference resource pool, and the inference pool controller receives the configuration and completes the definition of the resource pool rules.

[0072] The main actors in this step are the user and the inference pool controller. The user submits the configuration parameters of the inference resource pool through a cluster-level declarative API. The configuration parameters are persisted through the cluster's underlying distributed key-value store, without the need for an external database. The inference pool controller listens for and receives the configuration parameters submitted by the user in real time through an event-driven listening mechanism. After receiving the parameters, it completes the rule definition for the inference resource pool and generates a globally effective custom resource instance for the inference resource pool, which serves as the sole basis for subsequent pooling resource construction.

[0073] The configuration parameters include at least the preheating mode, target node range, concurrent deployment rate limiting strategy, and preset workload templates for different tensor parallel scales. The preheating mode includes two types: empty shell mode and memory-resident mode, which are adapted to the startup requirements of different business scenarios.

[0074] This step replaces traditional imperative scripts with declarative configuration, enabling one-time definition and automated execution of resource pool rules. At the same time, it relies on the cluster's native distributed key-value storage to achieve natural persistence of configuration and state, fundamentally solving the core defects of the existing technical architecture, such as bloated architecture and strong reliance on external components for state persistence.

[0075] Step 2: The inference pool controller builds a preheating resource pool base and simultaneously establishes pooled workload status management rules.

[0076] The execution entity for this step is the inference pool controller. Based on the custom resource instances of the inference resource pool generated in step 1, the inference pool controller creates a corresponding number of standby inference workloads within the target node scope, completes the workload preheating process, and constructs an inference resource pool foundation with available standby workloads. Simultaneously, the inference pool controller uses the workload's own metadata tags as the sole source of state facts to establish a lifecycle state machine and atomic state change rules for the pooled workloads, generating globally effective pooled state management rules. The lifecycle states include six types: idle, preheated, provisioning, ready, active, and releasing. The mutual exclusion of tag updates is ensured through atomic update operations at the cluster's underlying layer, achieving state synchronization and mutual exclusion operations without the need for additional distributed locks.

[0077] This step specifically includes:

[0078] Dual-mode pooling preheating mechanism: The inference pool controller performs differentiated preheating processes for the two preheating modes:

[0079] Empty Shell Mode Warm-up: The inference pool controller pre-starts workloads based on workload templates, completing the loading and initialization of time-consuming underlying computing environments such as the Python runtime and deep learning frameworks in advance. No model weights are loaded. After initialization, corresponding metadata tags are set for the workloads, and their status is marked as idle, putting them into a general standby state. When the service goes offline, the corresponding workload is destroyed, and the inference pool controller automatically creates new empty shell instances to replenish the resource pool, maintaining a stable replica level. This mode eliminates the application environment initialization time and is suitable for inference scenarios with low-frequency creation and relatively tolerant requirements for ultra-fast, second-level startup.

[0080] Memory-resident mode warm-up: The inference pool controller pre-starts the workload based on the workload template and completes the initialization of the underlying computing environment. The weight file of the target large model is pre-loaded into the GPU memory. Then, the workload activation module performs an unloading action, transferring the weight data losslessly and caching it in the host machine's memory. After completion, corresponding metadata tags are set for the workload, marking its status as pre-warmed, and putting it into a high-priority standby state. When the service goes offline, the workload is not destroyed and can be returned to the pre-warmed state for reuse. This mode completely skips the extremely time-consuming weight reading process. During deployment, it only takes about 5.5 seconds to restore the weights from the host memory to the GPU memory, which is more than ten times faster than a complete cold start, achieving true second-level activation. It is specifically designed for core online inference services with extremely stringent requirements for first-word response latency.

[0081] Based on metadata tags, the pooled workload state machine is fully defined by the inference pool controller in this step.

[0082] Core metadata tags: Set complete identity and state tags for each pooled workload, including pod-pool (name of the inference pool to which it belongs), state (lifecycle state of the workload), request-id (unique identifier associated with the inference service instance, i.e., universal unique identifier UUID), tp (tensor parallel scale, with values ​​of 1 / 2 / 4 / 8), model (name of the preloaded model, applicable to memory-resident mode), role (the role of the workload in the inference business, including pre-filling, decoding, and regular), and topology-name (topology group name identifier), which serve as the unique source of facts for the state machine operation.

[0083] Lifecycle state definition: Six lifecycle states covering the entire pooling process are fully defined, including Idle (workload is in the resource pool waiting for scheduling), Pre-setting (workload has been locked by the scheduler and is performing activation operations such as model loading), Ready (model activation is complete and waiting to be grouped according to role), Active (already connected to the business network and is providing large model inference services to the outside world), Warmed up (only applicable to memory-resident mode, the model has been pre-loaded and swapped out to host memory), and Releasing (only applicable to memory-resident mode, the business is offline and the workload is being cleared from memory to reset back to the idle state).

[0084] Atomic state change rules: Mutual exclusion of label updates is guaranteed through atomic update operations at the cluster level, without the need for additional distributed locks. The state transition rules are as follows:

[0085] (1) Idle state changed to preset state: After the inference service controller completes the scheduling selection of idle state workloads in empty shell mode, it changes the life cycle state of the workload from idle state to preset state by atomically updating the workload metadata tag, and writes the unique identification of the associated inference service to complete this state change. This change means that the workload enters the pooled allocation state from the pooled standby state. After locking, other inference services cannot schedule the workload.

[0086] (2) Change from preheated state to preset state: This only applies to memory-resident mode. After the inference service controller completes the scheduling selection of the preheated state workload, it changes the life cycle state of the workload from the preheated state to the preset state by atomically updating the workload metadata tag. At the same time, it writes the unique identifier of the associated inference service to complete this state change. This change means that the workload directly enters the pooled allocation state from the pooled preheating state without repeating the preheating.

[0087] (3) Change from preset state to ready state: After the workload activation module completes the model loading or weight restoration activation operation for the preset state workload, the inference service controller changes the life cycle state of the workload from the preset state to the ready state by atomically updating the workload metadata tag, thus completing this state change. This change indicates that the workload has completed pooling activation and has the basic ability to provide inference services.

[0088] (4) Change from ready state to active state: After the inference service controller completes the role grouping and aggregation management of all ready state workloads belonging to the same inference service, it changes the life cycle state of the corresponding workload from ready state to active state by atomically updating the workload metadata tag, and completes this state change. This change indicates that the workload has entered the pooled service state and can officially provide large model inference capabilities to the outside world.

[0089] (5) Change the active state to the release state and then to the preheated state: This only applies to memory-resident mode. When a user deletes the corresponding inference service instance, the inference service controller first changes the lifecycle state of the corresponding active workload from the active state to the release state. After the model weight release operation is completed, it changes the lifecycle state from the release state to the preheated state, so that it returns to the inference resource pool base and enters the standby loop, thus completing this state change. This change realizes the internal loop reuse of resources in memory-resident mode.

[0090] (6) Closed-loop rule from active state to destruction and then to idle state replenishment: This only applies to the empty shell mode. When a user deletes the corresponding inference service instance, the corresponding active workload is destroyed. The inference pool controller automatically creates a new workload, sets the lifecycle state of the new workload to the idle state, and replenishes it to the inference resource pool base to maintain the preset replica level of the resource pool, thus completing this state closed loop.

[0091] The inference pool controller's pooling lifecycle management mechanism involves continuously performing full lifecycle maintenance of the resource pool during this step and subsequent operations.

[0092] Pool creation process: 1) Parse the specifications defined in the declarative configuration of the inference pool to obtain configurations such as pool type, node selector, and workload template; 2) Filter target nodes according to the node selector; 3) Control the creation rate of workloads on each node according to the node concurrent deployment rate limiting configuration; 4) Create workloads on the target nodes.

[0093] a. Empty shell mode: Create the corresponding number of workloads based on the workload templates of each tensor parallel scale (TP=1 / 2 / 4 / 8);

[0094] b. Memory-resident mode: Based on the workload templates for each model, role, and tensor parallel scale, create the corresponding number of workloads and bind them to the specified GPU card index;

[0095] 5) Set labels for newly created workloads (state=idle, pod-pool, tp / model / role, etc.); In this embodiment, workload labels include:

[0096] pod-pool: The name of the inference pool to which it belongs;

[0097] state: The lifecycle state of the workload;

[0098] request-id: A unique identifier (UUID) for the associated reasoning service instance;

[0099] tp: Tensor parallel scale (1 / 2 / 4 / 8);

[0100] model: Preloaded model name (memory saver mode);

[0101] role: The role of the workload in the inference business (prefill, decode, regular).

[0102] topology-name: In inference scenarios, the workload of a group of roles is grouped and aggregated for management. This tag is the name identifier of this topology group.

[0103] This application innovatively utilizes the workload's own metadata label as the sole source of state facts, defining the following state and state change rules. The state machine design revolves around workload pooling technology, ensuring that the workload's state within the pool is traceable and manageable.

[0104] 6) Set the inference pool instance as the owner of the workload, and establish a cascading ownership relationship between the pool and the workload;

[0105] Pool preheating process (memory-resident mode)

[0106] 1) The inference pool controller detected that a newly created workload has entered the running state;

[0107] 2) Poll the workload status and wait for the workload to complete loading the model weights;

[0108] 3) After the model is loaded, the workload activation module unloads the model weights from the GPU memory to the host's DRAM;

[0109] Wait for the unloading to complete, update the workload label to "preheated", and the workload enters the pooling preheating state, waiting for rapid activation.

[0110] Pool expansion process: When a user modifies the resource pool configuration to increase the number of replicas, calculate the number of missing workloads, create the corresponding workloads on the target node and complete the preheating, and update the resource pool status;

[0111] Pool shrinkage process: When a user modifies the resource pool configuration to reduce the number of replicas, calculate the number of workloads that need to be deleted, prioritize deleting unassigned idle / pre-warmed workloads, and wait for active workloads to be released before deleting them if there are no available idle workloads, and update the resource pool status.

[0112] Pool destruction process: When the resource pool instance is detected to be deleted, all associated workloads are automatically deleted in a cascading manner to complete a full cleanup of resources;

[0113] Node status monitoring: Real-time monitoring of physical machine additions or failures, automatic replenishment or cleanup of invalid workloads, and constant maintenance of the available replica level in the resource pool.

[0114] This step completely separates the workload warm-up phase from the subsequent business activation phase, pre-builds a ready-to-use standby resource base, and fundamentally solves the problem of excessively long cold start time in traditional solutions. At the same time, the tag-based lockless state machine design solves the shortcomings of existing technologies that require the additional development of distributed locks and have great difficulty in handling concurrent conflicts, and achieves millisecond-level precise synchronization of states, avoiding the state delay and decision failure problems caused by polling mechanisms.

[0115] Step 3: The user submits the declarative configuration for the inference service, and the inference service controller receives the configuration and completes the definition of business requirements.

[0116] The main actors in this step are the user and the inference service controller. Based on the available inference resource pool foundation built by the inference pool controller in step 2, the user submits the business configuration parameters for the inference service to be deployed via a namespace-level declarative API. These business configuration parameters are persisted through the cluster's underlying distributed key-value storage. The inference service controller, through an event-driven listening mechanism, listens in real-time and, upon receiving the user's submitted business configuration parameters, completes the definition of the inference service's business requirements and generates a custom resource instance of the inference service that is effective within the namespace, serving as the sole business basis for subsequent scheduling and deployment.

[0117] The business configuration parameters include at least the associated inference resource pool, the target large model name, the inference service topology configuration, and the hardware acceleration configuration. The inference service topology configuration includes node roles, tensor parallel scale, and the number of replicas. The node roles include three types: pre-filling, decoding, and regular, fully adapting to the business requirements of the pre-filling and decoding separation architecture.

[0118] This step decouples the inference service from the underlying resource pool through declarative business requirement definition. Users only need to define business goals without having to worry about the details of underlying resource scheduling. At the same time, the clear definition of node roles provides basic support for the deployment of the pre-filling and decoding separation architecture, solving the shortcomings of existing technologies in coordinating multi-role collaborative workloads.

[0119] Step 4: The inference service controller matches and locks the target workload, obtaining a pre-set set of exclusively locked workloads.

[0120] The execution entity for this step is the Inference Service Controller. Based on the custom resource instance of the Inference Service generated in Step 3, the Inference Service Controller constructs a scheduling request. This request includes the roles, tensor parallel scale, and number of replicas required for each topology group. It then invokes a multi-dimensional pooling scheduling framework to filter standby workloads that meet all hard constraints from the inference resource pool foundation built in Step 2. The best-matched workload is then selected through a weighted scoring system. Subsequently, the Inference Service Controller atomically updates the metadata tags of the selected workloads via the cluster API, updating their lifecycle status to "in progress" and writing the unique identifier of the associated inference service and the topology group name. This results in a set of pre-locked workloads that are exclusively locked and can only be used for this inference service. Once locked, other inference services cannot schedule this workload, completely avoiding concurrency preemption issues.

[0121] In this embodiment, the inference service controller manages pooled resources.

[0122] During the scheduling phase, suitable workloads are selected from the resource pool.

[0123] 1) The inference service controller constructs a scheduling request (including the roles required for each topology group, tensor parallelism scale, and number of replicas) based on the topology configuration in the inference service instance, and calls the workload scheduling framework to perform the selection:

[0124] 2) List all “idle” or “warm-up” workloads from the inference pool, construct a host node candidate information structure, and record the “idle” or “warm-up” workload groupings of each host node (by tensor parallel scale or by model-role-tensor parallel scale).

[0125] 3) Execute the filtering chain sequentially on each host node, eliminating host nodes that do not meet the conditions;

[0126] 4) Perform parallel scoring on the filtered host nodes and calculate the scheduling score for each host node;

[0127] 5) Select the target workload from the host node with the highest score, update the workload label atomically, update the status to preset, and write request-id and topology-name to indicate that the workload has been selected and entered the preset state.

[0128] In the preset, activate pooled workloads:

[0129] This step is divided into two sub-processes that are executed sequentially:

[0130] 1) Subprocess: Workload Activation:

[0131] The workload activation module is invoked to initiate activation calls for each selected workload. Upon successful activation, the workload's status is updated to "Ready".

[0132] 2) Sub-processes: Grouped and managed by role:

[0133] All workloads that meet the requirements of an inference service business are grouped and aggregated by role for management, forming a complete inference service and providing it to the outside world. The status of all workloads in this service is updated to "active".

[0134] The multi-dimensional pooling scheduling framework in this step is a configurable filtering-scoring-optimization evaluation system specifically designed for workload pooling scenarios in this application. The specific execution logic is as follows:

[0135] Serial Filter Chain: The filters are executed serially. If any filter returns a failure result, the corresponding host node is directly eliminated. Only host nodes that pass all filters can enter the subsequent scoring stage, ensuring that all business hard constraints are strictly met.

[0136] Built-in filters include:

[0137] Resource pool type filter, used to filter host nodes that match the resource pool type;

[0138] A workload availability filter is used to filter out idle workloads that meet the tensor parallel scaling requirements.

[0139] Capacity filters are used to filter host nodes that meet the replication requirements;

[0140] The role anti-affinity filter is used to determine whether the three types of workload roles—pre-filling, decoding, and regular—are deployed on different host nodes. This fundamentally avoids the competition for computing power between tasks of different roles on the same node, ensuring the correct deployment of the pre-filling and decoding separation architecture.

[0141] Parallel scorer: The scorer adopts a parallel execution mode. The scores output by each scorer are accumulated to obtain the comprehensive score of the host node. The comprehensive scores are sorted from high to low to select the best matching node among the nodes that meet the hard constraints, while taking into account the soft preferences of the business.

[0142] The built-in rating system includes:

[0143] Capacity scorer for bin packing optimization to reduce fragmented resources;

[0144] Locality scorers are used to aggregate workloads with the same model onto the same host node, maximizing cache utilization.

[0145] Load balancing scorer, used to evenly distribute load;

[0146] The idle capacity scorer prioritizes host nodes with a high amount of idle workload.

[0147] Differentiated scorer combination strategy: For different preheating modes, adopt a differentiated scorer combination adapted to the business scenario:

[0148] The default combination in the empty shell mode is: idle capacity scorer + load balancer scorer, which prioritizes load distribution and adapts to the scheduling requirements of offline batch tasks.

[0149] The default combination for memory-resident mode is: capacity scorer + locality scorer + load balancer, which takes into account bin packing optimization, cache locality and cluster load balancing, and adapts to the scheduling requirements of online high-priority tasks.

[0150] This step achieves exclusive locking of the workload through atomic tag updates, avoiding the problem of multiple services concurrently competing for the same resource without the need for additional distributed locks. At the same time, through a multi-dimensional scheduling framework designed specifically for pooling scenarios, it achieves accurate resource matching under complex constraints, solving the shortcomings of existing technologies such as single scheduling strategies, inability to meet multi-dimensional constraints, and easy occurrence of computing power contention.

[0151] In this embodiment, the multi-dimensional scheduling framework is a dynamically configurable "filter-scoring-optimization" evaluation system. It supports multiple filtering rules for strict screening and elimination, and executes multiple sets of scoring in parallel.

[0152] Step 5: The inference service controller completes workload activation and service launch, resulting in an active inference service cluster that can provide external capabilities.

[0153] The inference service controller is the main entity executing this step. The inference service controller calls the workload activation module to perform an activation operation on the locked workload set obtained in step 4:

[0154] In shell mode, the model weights are loaded from persistent storage to GPU memory. In memory-resident mode, the weights in host memory are rapidly restored to GPU memory. After activation, the lifecycle status of the workload is updated to ready. Then, the inference service controller groups all ready workloads belonging to this inference service into a complete business service unit by node role. After verifying that all workloads in the group are ready, the lifecycle status of all workloads is updated to active in batches, resulting in an active inference service cluster that is ready for all roles and can provide large model inference capabilities.

[0155] In this embodiment, the lifecycle states of the workload are defined as follows:

[0156] Idle: The workload is in the resource pool waiting to be scheduled;

[0157] Provisioning: The workload has been locked by the scheduler and is performing activation operations such as model loading.

[0158] Ready: The model is activated and awaits grouping by role;

[0159] Active: It has been connected to the business network and is providing large model inference services to the outside world;

[0160] Warmed: (For memory-resident mode only) The model has been preloaded and swapped out to host memory;

[0161] Releasing: (Only for memory-resident mode) The service is offline, and the workload is clearing memory to reset it back to an idle state.

[0162] This step includes:

[0163] Service delivery mechanism: When all workloads in the inference service are in an "active" state, the inference router automatically discovers these workloads through the label selector and accurately routes user inference requests to the workloads of the corresponding roles for processing, providing complete large model inference services to the outside world; once all workloads in the inference service are active, the inference router can be arranged to automatically discover these workloads through the label selector and route inference requests to the workloads of the corresponding roles.

[0164] Anomaly Handling Mechanism: If anomalies such as workload activation failure or role-based grouping timeout occur, the corresponding workload will be triggered to enter the error phase. The system supports automatic retry recovery, or manual intervention can be selected to avoid abnormal workloads from affecting the stability of business services.

[0165] This step, through an atomic deployment mechanism grouped by role, ensures that the workloads of collaborative roles such as pre-filling and decoding are all ready before providing services to the outside world. This enables overall control of multi-role inference services and solves the problem that existing technologies cannot bind collaborative nodes as atomic business units, which can easily lead to some instances being ready while others are not, causing request failures. At the same time, relying on pre-warmed pooled resources, the inference service is deployed in seconds, fully meeting the stringent requirements of high-priority online tasks for first-word response time.

[0166] Step 6: The user initiates a service termination command, and the inference service controller and inference pool controller work together to complete differentiated resource reclamation.

[0167] The entities performing this step are the user, the inference service controller, and the inference pool controller.

[0168] Users initiate a termination command to delete the custom resource instance of the inference service corresponding to step 5 via a declarative API. The termination command is persisted through the underlying distributed key-value storage of the cluster. The inference service controller listens for the termination command in real time through an event-driven listening mechanism. After receiving the termination command, it performs a differentiated resource reclamation process for all workloads associated with the service according to the warm-up mode in the inference resource pool configuration parameters submitted by the user in step 1. The inference pool controller cooperates to replenish and reset the resource pool replica level, restore the workload to the standby state, and replenish it to the inference resource pool base, thus completing the cyclic reuse of pooled resources and the closed loop of the entire process.

[0169] The differentiated resource recycling process for this step is as follows:

[0170] Empty shell mode recycling process: 1) The workload is destroyed as the inference service instance is deleted, and the inference pool controller automatically replenishes the new workload; specifically, the inference service controller destroys the corresponding workload, and the inference pool controller automatically creates new idle workloads to replenish the resource pool and maintain the preset replica level.

[0171] Memory-resident mode recycling process: 2) Memory-resident mode: retain the workload, use the workload activation module to release the model weights of the workload, and update the status to "releasing"; wait for the model weights to be released, and then update its status to "idle"; specifically, the inference service controller retains the corresponding workload without destroying it, calls the workload activation module to execute the model weight release instruction, and sinks the weights in the GPU memory back to the host memory cache. First, update the workload status to "releasing", and after the release is completed, update it to the preheated state, so that it returns to the inference resource pool base to enter the standby loop, realizing the internal loop reuse of resources.

[0172] In this embodiment, the workload activation module is used to encapsulate the calling logic such as asynchronous activation instructions and progress status confirmation for the inference service workload.

[0173] This step achieves the cyclical reuse of pooled resources through a differentiated resource recycling mechanism. For tidal scenarios, resource status switching can be completed in seconds. When there is a sudden surge in high-priority online traffic at night or during off-peak hours, the computing power of low-priority batch tasks can be released immediately and quickly switched to online inference services. This completely solves the resource vacuum period problem caused by the "destroy and rebuild" approach of traditional solutions. It achieves efficient reuse of expensive computing resources without violating the service level objectives of high-priority tasks. At the same time, the lifecycle terminator mechanism ensures the automatic and thorough recycling of resources when services go offline, avoiding the problems of zombie nodes and resource leaks when services are abnormally interrupted.

[0174] This application also provides a device for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, the device comprising:

[0175] The inference pool controller generates a custom resource instance of the inference resource pool that is globally effective across the cluster based on the inference resource pool configuration submitted by the user. The inference resource pool configuration submitted by the user includes a warm-up mode, a target node range, a concurrent deployment rate limiting strategy, and a preset workload template for different tensor parallel scales. The warm-up mode includes two types: empty shell mode and memory-resident mode.

[0176] The inference pool controller generates an inference resource pool base with available standby workloads based on custom resource instances in the inference resource pool, and generates globally effective pooling state control rules.

[0177] The inference service controller obtains the inference service deployment requirements submitted by the user;

[0178] The inference service controller generates custom resource instances of the inference service that are active within the namespace based on the inference service deployment requirements submitted by the user.

[0179] The inference service controller generates a pre-built set of workloads that are exclusively locked based on the custom resource instance of the inference service;

[0180] The inference service controller performs an activation operation on the pre-configured set of workloads. After activation, it updates the lifecycle status of the workloads to ready. All ready workloads belonging to this inference service are grouped and aggregated into a complete business service unit according to node roles. After verifying that all workloads in the group are ready, the lifecycle status of all workloads is updated to active in batches, thus obtaining an active inference service cluster.

[0181] In this embodiment, the system takes the user as the business initiator, the inference pool controller as the core execution entity for resource pool management, and the inference service controller as the core execution entity for the entire lifecycle management of business services. The three entities work together to complete the entire process of rapid deployment and elastic scheduling of inference services. The system responsibilities and built-in functional units of each entity are as follows:

[0182] 1. Initiating Entity: User

[0183] The user is the sole initiator of all business rules, resource configurations, and service requests in this system, and is also the ultimate user of the system's capabilities. Their core responsibilities within the system are:

[0184] Submit the declarative configuration of the inference resource pool, complete the definition of resource pool rules, and generate a custom resource instance of the inference resource pool that is globally effective in the system. The configuration parameters include the warm-up mode (empty shell mode or memory resident mode), the target node range, the concurrent deployment rate limiting strategy, and the workload preset templates under different tensor parallel scales.

[0185] Based on the already effective inference resource pool, submit the declarative configuration of the inference service, complete the definition of business requirements, and generate a custom resource instance of the inference service that is effective within the system namespace. The configuration parameters include the associated inference resource pool, the target large model name, the inference service topology configuration (node ​​role, tensor parallel scale, number of replicas), and the hardware acceleration configuration. The node roles include three types: pre-filled, decoded, and regular.

[0186] Initiate commands to expand, shrink, or destroy the inference resource pool, as well as commands to terminate and take the inference service offline, driving the system to complete the corresponding resource changes and recycling.

[0187] In this embodiment, the pool expansion process is as follows:

[0188] 1) Users modify the inference pool instance configuration to increase the number of workload replicas for a certain host node or a certain TP size;

[0189] 2) The inference pool controller detects a specification change, calculates the number of missing workloads, creates the missing workloads on the target host node, and performs a warm-up process (if applicable).

[0190] 3) Update the inference pool status to reflect the number of replicas after expansion.

[0191] 4.3.4 Pool Reduction Process

[0192] 1) Users modify the inference pool instance configuration to reduce the number of workload replicas on a certain host node or at a certain TP scale;

[0193] 2) The inference pool controller detects a specification change and calculates the number of workloads that need to be deleted.

[0194] 3) Prioritize deleting workloads that are in an "idle" or "warm-up" state (i.e., unassigned and inactive);

[0195] 4) If there are no "idle" workloads in the pool, wait for "active" workloads to be released before deleting them;

[0196] 5) Update the inference pool status to reflect the number of replicas after scaling down.

[0197] 4.3.5 Pool Destruction Process

[0198] 1) After an inference pool instance is deleted, the inference pool controller will automatically delete all associated workloads;

[0199] 2. Inference Pool Controller:

[0200] The inference pool controller is the cluster-level background automated daemon of this system. It is the sole management and execution entity of the resource pool base, responsible for the full lifecycle management of the inference resource pool from creation to destruction, ensuring that there is always a sufficient number of standby workloads in the pool. Its system responsibilities and built-in functions are as follows:

[0201] Resource pool rule parsing and execution: Real-time monitoring of user-submitted custom resource instances in the inference resource pool, parsing configuration parameters, filtering target nodes, creating a corresponding number of backup inference workloads on the target nodes according to the concurrent deployment rate limiting policy, and establishing the cascading affiliation relationship between inference pool instances and workloads;

[0202] Dual-mode workload warm-up execution: The built-in workload activation module performs differentiated warm-up processes for two different warm-up modes.

[0203] Empty shell mode: Pre-start the workload, complete the initialization of the underlying computing environment such as Python runtime and deep learning framework, set the corresponding metadata tags for the workload, mark the status as idle, and maintain the stability of the replica level in the pool;

[0204] Memory-resident mode: The workload is pre-started and the underlying environment is initialized. The built-in workload activation module completes the loading of model weights and the unloading of GPU memory to host memory. The corresponding metadata tags are set for the workload, and the status is marked as pre-warmed, so that it enters a high-priority standby state.

[0205] Specifically, 1) The inference pool controller detects that a newly created workload has entered the running state;

[0206] 2) Poll the workload status and wait for the workload to complete loading the model weights;

[0207] 3) After the model is loaded, the workload activation module unloads the model weights from the GPU memory to the host's DRAM.

[0208] 4) Wait for the unloading to complete, update the workload label to "preheated", and the workload enters the pooling preheating state, waiting for rapid activation.

[0209] Resource pool full lifecycle maintenance: Real-time monitoring of new and fault events of physical machines in the cluster, automatically replenishing or cleaning up invalid workloads; After listening to user-initiated expansion or reduction commands, automatically completing the creation, replenishment or safe deletion of workloads; After listening to user-initiated resource pool destruction commands, automatically cascading the deletion of all associated workloads and completing a full cleanup of resources;

[0210] Pooled state machine management: Using the workload's own metadata tags as the sole source of state facts, the lifecycle state machine and atomic state change rules of the pooled workload are established and maintained. The mutual exclusion of tag updates is guaranteed through the atomic update operation at the cluster level, and precise synchronization and mutual exclusion management of workload state can be achieved without additional distributed locks.

[0211] 3. Execution Entity Two: Inference Service Controller

[0212] The Inference Service Controller is a namespace-level background automated daemon of this system. It is the sole control and execution entity for the entire inference service workflow, responsible for the full lifecycle management of the inference service from deployment to termination. Its system responsibilities and built-in functions are as follows:

[0213] Business requirement analysis and scheduling execution: Real-time monitoring of user-submitted custom resource instances for inference services, parsing of business configuration parameters, construction of scheduling requests, invocation of the built-in multi-dimensional scheduling framework, and selection and matching of the optimal standby workload from the associated inference resource pool;

[0214] Multi-dimensional scheduling decision: It has a built-in multi-dimensional scheduling framework designed specifically for workload pooling scenarios. This framework is a dynamically configurable "filter-score-optimize" evaluation system. It performs hard constraint verification through a serial filter chain and soft preference matching through a parallel scorer, and finally selects the optimal set of workloads for the inference service. The filter chain includes a resource pool type filter, a workload availability filter, a capacity filter, and a role anti-affinity filter. The scorer includes a capacity scorer, a locality scorer, a load balancing scorer, and an idle capacity scorer. It supports dynamic adjustment of the scorer combination according to the warm-up mode.

[0215] In this embodiment, the filters are executed sequentially. If any filter returns false, it means that the host node is eliminated. If all filters pass, the host node enters the scoring stage.

[0216] Built-in filter:

[0217] Resource pool type filter, used to filter host nodes that match the resource pool type;

[0218] A workload availability filter is used to filter out idle workloads that meet the tensor parallel scaling requirements.

[0219] Capacity filters are used to filter host nodes that meet the replication requirements;

[0220] The role anti-affinity filter is used to determine whether the three types of workload roles—pre-filled, decoded, and regular—are deployed on different host nodes.

[0221] In this embodiment, the scorers are executed in parallel, and the scores output by each scorer are accumulated and then sorted.

[0222] Built-in rating system:

[0223] Capacity scorer for bin packing optimization to reduce fragmented resources;

[0224] Locality scorers are used to aggregate workloads with the same model onto the same host node, maximizing cache utilization.

[0225] Load balancing scorer, used to evenly distribute load;

[0226] The idle capacity scorer prioritizes host nodes with more idle workloads.

[0227] Scorer combination:

[0228] Empty shell mode default: Idle capacity scorer + load balancer scorer (prioritizing distribution);

[0229] The default mainstream memory mode is: capacity scorer + locality scorer + load balancing scorer (boxing + locality + balancing).

[0230] Workload Locking and Activation: Update the metadata tag of the selected workload through cluster atomic operations, lock it to the preset state, and exclusively use it for this inference service; call the workload activation module to complete the workload activation operation (load model weights in empty shell mode, restore the weights in host memory to GPU memory in memory-resident mode), and update the workload status to ready after activation.

[0231] Service Deployment and Traffic Access: The ready workloads belonging to this inference service are grouped and aggregated into complete business service units according to pre-filling, decoding, and regular roles. After verifying that all workloads are ready, their status is updated to active in batches, driving the inference router to discover active workloads through the label selector and achieve accurate routing and forwarding of inference requests.

[0232] Exception handling and service reclamation: For abnormal workloads such as activation failure and group timeout, an exception handling process is triggered, supporting automatic recovery or manual intervention; after listening for a service termination command initiated by a user, a differentiated resource reclamation process is executed according to the preheating mode: in the empty shell mode, the corresponding workload is destroyed and the inference pool controller is notified to replenish new idle workloads; in the memory-resident mode, the workload activation module is called to complete the weight release, the workload is reset to the preheated state, and returned to the resource pool to enter the standby loop, realizing the internal loop reuse of resources.

[0233] This application has the following advantages:

[0234] Compared with the traditional architecture of imperative script orchestration + external state database, which is the closest to existing technology, the present invention adopts the above technical solution and has the following significant advantages:

[0235] The system architecture is extremely simple and does not require the introduction of an external database: This invention reuses the distributed key-value storage device at the bottom layer of the cluster to achieve natural persistence of configuration and state, completely eliminating the external state database required by existing technologies, greatly simplifying the system architecture, reducing development and maintenance costs, and solving the core defect of the bloated architecture of existing technologies.

[0236] Completely solves the industry pain point of excessively long cold start time for large model inference services: This invention completely separates the workload warm-up phase from the business activation phase through a workload pooling mechanism, and pre-maintains a standby inference workload resource pool, reducing the traditional cold start time of tens of seconds to several minutes to the second level. Among them, the memory-resident mode can achieve an ultra-fast activation of about 5.5 seconds, which is more than ten times faster than a full cold start, fully meeting the stringent requirements of online inference services for first-word response time.

[0237] Perfectly adapted to business tidal scenarios, enabling rapid flow and efficient reuse of computing resources: This invention, through dynamic pooling scheduling and differentiated resource recycling mechanisms, can complete the switching of resource states between different models and different business scenarios in seconds. When there is a sudden surge in high-priority online traffic at night or during off-peak hours, the computing power of low-priority batch tasks can be released immediately and quickly switched to online inference services. This completely solves the resource vacuum period problem caused by the traditional solution of destroying and rebuilding simultaneously. Without violating the service level target of high-priority tasks, it significantly improves the utilization rate of expensive GPU computing resources.

[0238] This invention achieves lightweight and highly reliable distributed state management, solving the problem of concurrent coordination: It innovatively designs a lock-free pooled state machine based on metadata tags, and uses the atomic update operation of the cluster's underlying layer to ensure the mutual exclusion of state changes. Without the need to develop additional distributed lock logic, it can realize the prevention of concurrent conflicts and accurate state tracking in distributed scenarios. At the same time, it adopts an event-driven listening mechanism to replace the traditional polling, achieving millisecond-level accurate synchronization of resource states, avoiding erroneous scheduling decisions based on expired data, and solving the defects of high difficulty in distributed coordination and delayed state synchronization in existing technologies.

[0239] A multi-dimensional configurable scheduling process fully meets the complex scheduling needs of large model-as-a-service platforms: This invention constructs a two-layer scheduling framework of serial filter chain + parallel scorer, with built-in role anti-affinity filter, which ensures the physical correctness of the pre-filling and decoding separation architecture deployment from the scheduling level and avoids the problem of computing power contention on the same node; at the same time, it supports dynamic adjustment of the filter and scorer combination, which can meet the complex scheduling needs of multiple dimensions such as pool type matching, model cache locality, and multi-replica node scattering, and solves the defects of existing technologies in that the scheduling strategy is single and cannot adapt to the complex scenarios of large model inference.

[0240] This invention achieves reliable management and control of the entire workload lifecycle, preventing resource leaks: Relying on the collaborative mechanism of dual automated controllers, it realizes closed-loop management and control of the entire lifecycle of pooled resources, from creation, preheating, allocation, activation, service to recycling. It introduces a lifecycle terminator mechanism to ensure the automated and thorough recycling of associated resources when services are taken offline, avoiding the problems of zombie nodes and resource leaks when services are abnormally interrupted. At the same time, through custom grouped resource objects, it achieves overall atomic management and control of multi-role inference business, ensuring the integrity and availability of service launch.

[0241] In an alternative embodiment, this application further includes the following technical solution:

[0242] In step 4, when the inference service controller invokes the multi-dimensional pooling scheduling framework to perform workload matching, it performs role-based collaborative matching and scheduling of pooled workloads for the pre-filling and decoding separation architecture. Specifically, this includes:

[0243] (1) Define the role collaboration matching degree for the workload of the two strongly coupled roles of pre-filling and decoding in the inference service instance. The calculation formula is:

[0244] ;

[0245] In the formula, The physical distance level of the nodes where the pre-filled role and the decoding role are matched is set. The value is 0 for nodes on the same machine, 1 for nodes in the same rack, 2 for nodes across racks, and 3 for nodes across availability zones. The maximum node distance threshold allowed for business service level objectives; Tensor parallel scale for pre-filled role workloads To decode the tensor parallel scale of the role workload, The optimal ratio coefficient for the parallel scale of pre-filling and decoding tensors is 1 / 2, 1, or 2; The measured communication latency between the nodes containing the two sets of workloads to be matched is in milliseconds.

[0246] (2) The inference service controller first selects candidate workload groups of pre-filled and decoded roles that meet the single role hard constraints from the inference resource pool, and then iterates through all candidate groups to calculate Select The workload group corresponding to the maximum value is used as the final matching result;

[0247] (3) After the matching is completed, the inference service controller writes the same topology group name label and associated inference service unique identifier to the pre-filled and decoding workloads in the same group through atomic operations, and synchronously updates the life cycle status of the two workloads to the preset, thus completing the collaborative locking.

[0248] One of the core scenarios of this solution is a distributed inference architecture that separates pre-filling and decoding. However, existing general scheduling frameworks match the workloads of pre-filling and decoding roles independently, completely ignoring the strong coupling between the two roles: the context processing results of pre-filling need to be transmitted to the decoding node with low latency. The tensor parallel scale and node deployment location of the two directly determine the first word response time and throughput of the inference service. Independent matching is prone to problems such as pre-filling and decoding nodes being deployed across availability zones and mismatched tensor parallel scales, resulting in a surge in communication latency and serious overrun of service level targets, completely negating the second-level startup advantage brought by pooling technology.

[0249] Existing technologies for scheduling in a split architecture can only achieve simple same-node affinity scheduling, and cannot quantitatively evaluate the impact of multi-dimensional coupling factors on inference performance. Furthermore, they lack a role coordination and matching mechanism for standby workloads in pooled scenarios, and cannot find the optimal workload combination that balances communication latency, tensor parallelism ratio, and deployment level among massive pooled standby resources.

[0250] For the first time, this application constructs a quantification formula for the collaborative matching of pre-filling and decoding roles in the pooling scenario of this solution. The formula integrates and calculates three core coupling factors: node deployment distance, tensor parallel scale ratio, and node communication latency. It can accurately quantify the collaborative adaptability of different candidate workload groups and ensure the end-to-end performance of the inference service of the separate architecture from the scheduling root.

[0251] It breaks the inherent logic of independent scheduling for single roles in existing technologies. By atomically coordinating and locking tags in the same topology group, it ensures complete synchronization of the entire process of scheduling, activation, and deployment of strongly coupled pre-filling and decoding workloads. This avoids service gaps caused by one role being ready while another is not, further enhancing the rapid deployment advantage of pooling technology.

[0252] In the existing technology, there is no pre-filled-decoder separation architecture role collaborative scheduling mechanism for pooled standby workload scenarios, nor is there a multi-dimensional coupling collaborative matching metric model designed for this scenario. This solution breaks through the technical bias of independent scheduling of single roles in the existing technology. For the unique pooling base architecture of this invention, a dedicated collaborative scheduling mechanism is designed, which solves the performance loss problem when the separation architecture is combined with the pooling technology from the root.

[0253] Although the present invention has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to it, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present invention fall within the scope of protection claimed by the present invention.

Claims

1. A method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, characterized in that, The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling includes: Based on the inference resource pool configuration submitted by the user, a custom resource instance of the inference resource pool that is globally effective across the cluster is generated; the inference resource pool configuration submitted by the user includes a warm-up mode, target node range, concurrent deployment rate limiting strategy, and workload preset templates under different tensor parallel scales, wherein the warm-up mode includes two types: empty shell mode and memory resident mode; Generate an inference resource pool base with available standby workloads based on custom resource instances in the inference resource pool, and generate globally effective pooling state management rules. Obtain the deployment requirements for inference services submitted by users; Generate custom resource instances for the inference service that are active within the namespace based on the deployment requirements submitted by the user. Generate a pre-built set of workloads with exclusive lock based on custom resource instances from the inference service; Activate the pre-configured set of workloads. After activation, update the lifecycle status of the workloads to ready. Group all ready workloads belonging to this inference service into a complete business service unit by node role. After verifying that all workloads in the group are ready, update the lifecycle status of all workloads to active in batches, thus obtaining an active inference service cluster.

2. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 1, characterized in that, The process of generating an inference resource pool base with available standby workloads based on custom resource instances of the inference resource pool and generating globally effective pooling state management rules includes: Based on the configuration parameters within the custom resource instance of the inference resource pool, a backup inference workload is created within the target node range to complete the workload preheating process, thereby obtaining an inference resource pool base with available standby workloads. Using the workload's own metadata tags as the sole source of state facts, a lifecycle state machine and atomic state transition rules for the pooled workload are established, thereby obtaining globally effective pooled state management rules. The lifecycle states include six types: idle, preheated, pre-configured, ready, active, and releasing. The mutual exclusion of tag updates is ensured through atomic update operations at the cluster level.

3. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 2, characterized in that, The process of generating a custom resource instance for the inference service within the namespace based on the user-submitted inference service deployment requirements includes: Based on the inference resource pool base with available standby workloads, the business configuration parameters of the inference service to be deployed are submitted through a namespace-level declarative API, thereby generating a custom resource instance of the inference service. The custom resource instance of the inference service includes all business configuration parameters submitted by the user. The business configuration parameters include the associated inference resource pool, the target large model name, the inference service topology configuration, and the hardware acceleration configuration. The inference service topology configuration includes node roles, tensor parallel scale, and number of replicas. The node roles include three types: pre-filled, decoded, and regular.

4. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 3, characterized in that, The process of generating a pre-built set of exclusively locked workloads based on custom resource instances of the inference service includes: After obtaining the custom resource instance of the inference service, a scheduling request is constructed based on the business configuration parameters within the instance. The multi-dimensional pooling scheduling framework is invoked to filter out standby workloads that meet all hard constraints from the generated inference resource pool base, and then the optimal matching workload is selected through weighted scoring. The metadata tags of the selected workload are updated atomically, its lifecycle status is updated to preset, and the unique identifier and topology group name of the associated inference service are written, thereby obtaining a set of preset workloads that are exclusively locked and can only be used for this inference service.

5. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 4, characterized in that, After obtaining the active inference service cluster, the method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling further includes: When a user deletes the custom resource instance of the inference service corresponding to the generated active inference service cluster, after listening to the deletion command, a differentiated resource reclamation process is executed on all workloads associated with the service according to the warm-up mode in the custom resource instance of the inference resource pool. This restores the workload to the standby state and replenishes it to the inference resource pool base, completing the cyclic reuse of pooled resources and the entire process closed loop. In the empty shell mode, the corresponding workload is destroyed and the inference pool controller replenishes new standby workloads to the resource pool. In the memory resident mode, the workload is retained and reset to the standby state.

6. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 5, characterized in that, The empty shell mode includes: Based on the workload preset template in the custom resource instance of the inference resource pool, the workload is pre-started and the underlying computing environment of the Python runtime and deep learning framework is fully initialized without loading any large model weights. After initialization, the life cycle state of the workload is set to idle, thereby generating a general standby empty shell workload with no model binding and prefabricated according to tensor parallel specifications. The memory-resident modes include: Based on the workload preset template in the custom resource instance of the inference resource pool, the workload is pre-started and the underlying computing environment is initialized. The weight file of the target large model is loaded into the graphics processor's video memory. Then, the unloading action is performed to transfer the weight data without loss and cache it in the host machine's host memory. After completion, the life cycle status of the workload is set to pre-warmed, thereby completing the deep pre-warming of model preloading-unloading and generating a pre-warmed workload in a high-priority standby state.

7. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 5, characterized in that, The globally effective pooling state management rules include: The rule for changing from idle state to preset state is used for workloads in empty shell mode. After the scheduling and selection of idle state workloads is completed, the life cycle state of the workload is changed from idle state to preset state by atomically updating the workload metadata tag. At the same time, the unique identification of the associated inference service is written to complete this state change. Change rules from preheated state to preset state: The change rules from preheated state to preset state are used for workloads in memory-resident mode. After the scheduling selection of preheated state workloads is completed, the life cycle state of the workload is changed from preheated state to preset state by atomically updating the workload metadata tag. At the same time, the unique identification of the associated inference service is written to complete this state change. Change rules from preset state to ready state: The change rules from preset state to ready state are used by the inference service controller to change the life cycle state of the workload from preset state to ready state by atomically updating the workload metadata tags after the model loading or weight restoration activation operation of the preset state workload is completed. The rules for changing from ready to active state: After completing the role grouping and aggregation management of all ready workloads belonging to the same inference service, the rules for changing the lifecycle state of the corresponding workload from ready to active state are changed by atomically updating the workload metadata tags. Change rules from active state to release state and then to preheated state: The change rules from active state to release state and then to preheated state apply to workloads in memory-resident mode. When a user deletes a custom resource instance of the inference service, the inference service controller first changes the lifecycle state of the corresponding active workload from active state to release state. After the model weight release operation is completed, the lifecycle state is changed from release state to preheated state, so that it returns to the inference resource pool base and enters the standby loop, thus completing this state change. Closed-loop rule from active state to destruction, and then to idle state replenishment: The closed-loop rule from active state to destruction, and then to idle state replenishment applies to workloads in empty shell mode. When a user deletes a custom resource instance of the inference service, the corresponding active workload is destroyed, the inference pool controller automatically creates a new workload, sets the lifecycle state of the new workload to idle state, and replenishes it to the inference resource pool base to maintain the preset replica level of the resource pool, thus completing this state closed loop.

8. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 5, characterized in that, After obtaining the custom resource instance of the inference service, a scheduling request is constructed based on the business configuration parameters within the instance. A multi-dimensional pooled scheduling framework is invoked to filter standby workloads that meet all hard constraints from the generated inference resource pool base. Then, the optimal matching workload is selected through weighted scoring, including: The serial filtering chain is executed in sequence: resource pool type filtering, workload availability filtering, capacity filtering, and role anti-affinity filtering. If any filtering rule fails, the corresponding physical node is directly eliminated. Nodes that pass all rules enter the scoring stage. Among them, role anti-affinity filtering requires that the workloads of the three types of roles (pre-filled, decoded, and regular) must be deployed on different physical nodes, and the result is a set of candidate physical nodes that meet all hard constraints. The parallel scorer performs multi-dimensional scoring on the filtered candidate nodes in parallel. The scores of each scorer are accumulated and sorted from high to low according to the total score. The result is the best-matched physical node and the corresponding set of standby workloads.

9. The method for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling as described in claim 5, characterized in that, Upon detecting a deletion command, the process of performing a differentiated resource reclamation procedure for all workloads associated with the service, based on the preheating mode of the custom resource instance in the inference resource pool, includes: Empty shell mode recycling process: After a user deletes a custom resource instance of the inference service, the inference service controller destroys all workloads associated with the service. At the same time, the inference pool controller automatically creates new idle workloads to supplement the inference resource pool base, resulting in an idle inference resource pool base that maintains a preset replica level. Memory-resident mode recycling process: After a user deletes a custom resource instance of the inference service, the inference service controller retains all workloads associated with the service without destroying them, calls the workload activation module to execute the model weight release instruction, and pushes the weights in the GPU memory back down to the host memory cache. First, the lifecycle status of the workload is updated to "releasing", and after the release is completed, it is updated to "preheated". The result is a preheated inference resource pool base that returns to the high-priority standby state.

10. A device for rapid deployment and elastic scheduling of model inference services based on declarative workload pooling, characterized in that, The rapid deployment and elastic scheduling device for model inference services based on declarative workload pooling includes: The inference pool controller generates a custom resource instance of the inference resource pool that is globally effective across the cluster based on the inference resource pool configuration submitted by the user. The inference resource pool configuration submitted by the user includes a warm-up mode, a target node range, a concurrent deployment rate limiting strategy, and a preset workload template for different tensor parallel scales. The warm-up mode includes two types: empty shell mode and memory-resident mode. The inference pool controller generates an inference resource pool base with available standby workloads based on custom resource instances in the inference resource pool, and generates globally effective pooling state control rules. The inference service controller obtains the inference service deployment requirements submitted by the user; The inference service controller generates custom resource instances of the inference service that are active within the namespace based on the inference service deployment requirements submitted by the user. The inference service controller generates a pre-built set of workloads that are exclusively locked based on the custom resource instance of the inference service; The inference service controller performs an activation operation on the pre-configured set of workloads. After activation, it updates the lifecycle status of the workloads to ready. All ready workloads belonging to this inference service are grouped and aggregated into a complete business service unit according to node roles. After verifying that all workloads in the group are ready, the lifecycle status of all workloads is updated to active in batches, thus obtaining an active inference service cluster.

Images