Inference service replica pool management method, electronic device, and storage medium
By building active and dormant replica pools during the inference service creation phase, the problem of long scaling time for inference services was solved, enabling rapid response to user access requests.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INSPUR SUZHOU INTELLIGENT TECH CO LTD
- Filing Date
- 2026-04-30
- Publication Date
- 2026-07-31
AI Technical Summary
In existing technologies, scaling up inference services takes a long time and cannot process user access requests in a timely manner, mainly due to the long loading time of model files and container image downloads.
By launching multiple inference service replicas in both active and dormant states during the inference service creation phase to form a replica pool, and switching their states based on scaling metrics, model file loading and container image downloads are avoided every time the system is scaled up.
It reduced the time required to scale up the inference service, improved responsiveness, and decreased the frequency of inference service replica creation and destruction.
Smart Images
Figure CN122287909B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, and in particular to a method for managing a replica pool for inference services, electronic devices, and storage media. Background Technology
[0002] Most current inference platforms offer both offline and online inference services based on large models. Online inference services, due to the need to handle fluctuations in user access, typically provide elastic scaling capabilities. Among related technologies, the Horizontal Pod Autoscaler (HPA) capability of Kubernetes (k8s) enables scaling by monitoring target metrics and triggering expansion. This process sequentially completes resource scheduling, image download, container startup, and service access configuration, ultimately achieving the scaling of the inference service.
[0003] However, the scaling methods for inference services in related technologies require loading model files, downloading container images of inference service copies, and starting inference service copies each time scaling is performed, resulting in a long scaling time for inference services. Summary of the Invention
[0004] This application provides a method for managing a replica pool of an inference service, an electronic device, and a storage medium to at least solve the problem of long expansion time for inference services in related technologies.
[0005] This application provides a method for managing a replica pool for inference services, including: Obtain the scaling metrics for the inference service to be created; Based on scaling metrics, a pool of replicas of the inference service to be created is generated, and computing units are allocated to each replica of the inference service in the pool. For any inference service replica, determine the target node where the computing unit allocated to the inference service replica is located, download the container image of the inference service replica on the target node, start the inference engine process, load the model file corresponding to the inference service replica into the computing unit allocated to the inference service replica, and start the inference service replica according to the target configuration so that the inference service replica supports state switching. Based on the scaling metrics, the state of the first inference service replica in the replica pool is switched to the active state to handle inference requests; Switch the status of all inference service replicas in the replica pool except for the first inference service replica to dormant state; If the number of inference requests received by the target inference service within a preset time period exceeds the first request threshold, then the first target inference service replica in a dormant state is selected from the replica pool of the target inference service, and the state of the first target inference service replica is switched to an active state.
[0006] This application also provides an electronic device, including: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described inference service replica pool management methods.
[0007] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described inference service replica pool management methods.
[0008] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described inference service replica pool management methods.
[0009] This application obtains the scaling metrics of the inference service to be created; based on the scaling metrics, a replica pool of the inference service to be created is created, and computing units are allocated to each inference service replica in the replica pool; for any inference service replica, the target node where the computing unit allocated to the inference service replica is located is determined, the container image of the inference service replica is downloaded on the target node, the inference engine process is started, the model file corresponding to the inference service replica is loaded into the computing unit allocated to the inference service replica, and the inference service replica is started according to the target configuration so that the inference service replica supports state switching; based on the scaling metrics, the state of the first inference service replica in the replica pool is switched to the active state to handle inference requests; the states of other inference service replicas in the replica pool besides the first inference service replica are switched to the dormant state; if the number of inference requests received by the target inference service within a preset time period is greater than a first request threshold, the dormant first target inference service replica is selected from the replica pool of the target inference service, and the state of the first target inference service replica is switched to the active state. By launching multiple inference service replicas in both active and dormant states during the inference service creation phase, a replica pool of inference services is formed. Based on the state switching of the inference service replicas, the inference service can be expanded. This eliminates the need to download the container image of the inference service replica, load the model file, and launch the inference service replica each time expansion is performed. Therefore, it can solve the technical problem of long expansion time of inference services in related technologies and achieve the technical effect of reducing the expansion time of inference services. Attached Figure Description
[0010] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0011] Figure 1A schematic diagram of the structure of an inference service replica pool management system provided in this application embodiment; Figure 2 This is a schematic diagram of the structure of the inference service lifecycle management component provided in the embodiments of this application; Figure 3 A flowchart illustrating a method for managing a replica pool for an inference service, as provided in an embodiment of this application; Figure 4 A flowchart illustrating another inference service replica pool management method provided in an embodiment of this application; Figure 5 A flowchart for allocating graphics processors to inference service replicas in a replica pool is provided for embodiments of this application; Figure 6 A schematic diagram illustrating the state of a replica of an inference service in a replica pool provided in this application embodiment; Figure 7 A schematic diagram illustrating the switching of the state of the fourth inference service copy to an active state, provided for an embodiment of this application; Figure 8 A flowchart for scaling up or down the inference service provided in this application embodiment; Figure 9 Sequence diagram for creating inference services provided in embodiments of this application; Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0012] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0013] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0014] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0015] Inference service, short for Artificial Intelligence (AI) model inference service, is a service that deploys trained AI models (especially deep learning models) to the production environment to provide stable and efficient online prediction / inference capabilities.
[0016] Most current inference platforms offer inference services based on large models, including offline and online inference services. For online inference services, due to the volatility of user access, elastic scaling capabilities are typically provided. For example, elastic scaling of the inference service is achieved based on metrics such as Central Processing Unit (CPU), Queries Per Second (QPS), and Time To First Token (TTP), combined with preset maximum and minimum inference service replica counts. Specifically, when user access reaches a peak, the inference service is scaled up, increasing the number of inference service replicas (not exceeding the maximum number) to meet resource demands during peak periods. When user access reaches a trough, the inference service is scaled down, reducing the number of inference service replicas (not less than the minimum number) to release resources and maximize cluster resource utilization.
[0017] In related technologies, the scaling mechanism for large-model inference services and traditional application scaling mechanisms both employ the same approach. Specifically, based on Kubernetes' HPA capabilities, scaling is triggered by monitoring target metrics, sequentially completing processes such as resource scheduling, image download, container startup, and service access configuration, ultimately achieving the scaling of the inference service.
[0018] However, the scaling methods for inference services in related technologies require loading model files, downloading container images of inference service replicas, and starting the inference service replicas each time they are scaled up. Due to the unique characteristics of large model inference services, scaling up inference services takes a long time and cannot handle surges in user access requests in a timely manner.
[0019] The unique characteristics of large model inference services include: 1. Scaling up a large model inference service requires loading model files, which are typically very large, tens or hundreds of gigabytes in size, and usually reside in shared storage. Loading these files over the network takes tens of seconds or minutes, resulting in very slow scaling and an inability to process user access requests promptly. 2. Scaling up a large model inference service requires running a containerized copy of the inference service. This copy depends on very large container images, typically tens of gigabytes in size, which also need to be downloaded from an image repository, taking several minutes. 3. Scaling up a large model inference service requires starting the inference service copy. Starting the inference service copy involves multiple steps, such as VLLM inference engine initialization, Compute Unified Device Architecture (CUDA) context creation, memory allocation, and model file deserialization and parsing, taking tens of seconds to minutes.
[0020] The optimization schemes used in related technologies to reduce the scaling time of inference services include local caching of model files, optimization of model file loading, lazy loading of images, and optimization of inference engines. However, the scaling of inference services is still based on the scaling process in related technologies, and the scaling time is still in the minutes range, which is quite long.
[0021] To address the aforementioned technical problems, embodiments of this application provide a method for managing an inference service replica pool, an electronic device, and a storage medium. The method includes: obtaining scaling metrics for an inference service to be created; creating a replica pool for the inference service to be created based on the scaling metrics, and allocating computing units to each inference service replica in the replica pool; for any inference service replica, determining the target node where the computing unit allocated to that inference service replica is located, downloading the container image of the inference service replica on the target node, starting the inference engine process, loading the model file corresponding to the inference service replica into the computing unit allocated to that inference service replica, and starting the inference service replica according to the target configuration, so that the inference service replica supports state switching; switching the state of a first inference service replica in the replica pool to an active state based on the scaling metrics, for processing inference requests; switching the states of other inference service replicas in the replica pool besides the first inference service replica to a dormant state; if the number of inference requests received by the target inference service within a preset time period exceeds a first request threshold, then selecting a dormant first target inference service replica from the target inference service replica pool and switching the state of the first target inference service replica to an active state. The method provided by the above solution forms a replica pool of inference services by starting multiple inference service replicas in active and dormant states during the inference service creation phase. Based on the state switching of the inference service replicas, the inference service can be expanded. This eliminates the need to download the container image of the inference service replica, load the model file, and start the inference service replica each time expansion is performed. Therefore, it can solve the technical problem of long expansion time of inference services in related technologies and achieve the technical effect of reducing the expansion time of inference services.
[0022] The inference service replica pool management method, electronic device, and storage medium provided in this application are applicable to the management of inference service replica pools. Figure 1The diagram shown is a structural schematic of the inference service replica pool management system on which this application is based. The inference service replica pool management system includes a client and a cluster management node, which is the management node of the Kubernetes cluster. The cluster management node is equipped with an inference service lifecycle management component. This component receives inference service creation requests from clients and obtains the scaling metrics of the inference service to be created. Based on the scaling metrics, it creates a replica pool of the inference service to be created and allocates computing units to each inference service replica in the pool. For any inference service replica, it determines the target node where the computing units allocated to that replica are located, downloads the container image of the inference service replica on the target node, starts the inference engine process, loads the model file corresponding to the inference service replica into the computing units allocated to it, and starts the inference service replica according to the target configuration to enable it to support state switching. Based on the scaling metrics, it switches the state of the first inference service replica in the replica pool to the active state to handle inference requests. It switches the states of other inference service replicas in the replica pool to the dormant state. If the number of inference requests received by the target inference service within a preset time period exceeds a first request threshold, it selects the dormant first target inference service replica from the target inference service replica pool and switches its state to the active state.
[0023] This application describes embodiments using a graphics processing unit (GPU) as an example.
[0024] It should be noted that this embodiment of the application is based on a Kubernetes (k8s) managed graphics processor cluster. An inference service management platform is deployed on the k8s cluster. This inference service management platform has the capability to provide inference services for multiple types of models. GPU resources on the nodes are reused, with a reuse ratio of resuspension. num (A GPU can be used as a resource) num (The number of GPUs is used for resource scheduling).
[0025] Understandably, the inference service lifecycle management component is used to build an inference service replica pool, including the state management of active and dormant inference service replicas, resource scheduling based on GPU reuse and inference service replica state, load balancer (module) management, elastic scaling, etc.
[0026] Figure 2 This is a schematic diagram of the structure of the inference service lifecycle management component provided in the embodiments of this application, as shown below. Figure 2 As shown, the inference service lifecycle management component includes: a load balancing module, an elastic scaling module, and a graphics processor scheduling module.
[0027] The graphics processor scheduling module is used to allocate a graphics processor to each inference service replica in the replica pool to be created. A inference service replica is a service instance.
[0028] The elastic scaling module is used to increase / decrease the number of available replicas, that is, by setting the active / dormant state of service replicas. Setting the active / dormant state of service replicas requires increasing / decreases the access addresses of service instances in the load balancing module. Service replicas are inference service replicas.
[0029] The load balancing module is used to configure the access address (IP address) of the active inference service replica so that the active inference service replica can be used to process inference requests.
[0030] Embodiments of this application provide a method for managing an inference service replica pool, applied to the inference service lifecycle management component in the aforementioned cluster management node. Figure 3 A flowchart illustrating the inference service replica pool management method provided in this application embodiment is shown below. Figure 3 As shown, the inference service replica pool management method includes the following steps: Step S301: Obtain the scaling metrics for the inference service to be created.
[0031] The system receives a user's request to create an elastic inference service and obtains the scaling metrics of the elastic inference service configured by the user. In this embodiment, the inference service is an elastic inference service.
[0032] Scaling metrics include minimum number of replicas , the maximum number of copies Number of CPUs required ,Memory Number of GPUs The inference service lifecycle management component receives user requests to create elastic inference services, obtains the scaling metrics of the inference service to be created, and creates it on the inference service management platform. One inference service copy, of which Each inference service replica is marked as active. The number of compute units is equivalent to the number of GPUs.
[0033] The required number of CPUs refers to the number of CPUs needed to run one instance of the inference service. The required memory refers to the amount of memory needed to run one instance of the inference service. The required number of GPUs refers to the number of GPUs selected to run one instance of the inference service.
[0034] Step S302: Based on the scaling metrics, create a replica pool of the inference service to be created, and allocate computing units to each inference service replica in the replica pool.
[0035] Step S303: For any inference service replica, determine the target node where the computing unit allocated to the inference service replica is located, download the container image of the inference service replica on the target node, start the inference engine process, load the model file corresponding to the inference service replica into the computing unit allocated to the inference service replica, and start the inference service replica according to the target configuration so that the inference service replica supports state switching.
[0036] The target configuration is "vllm serve Qwen / Qwen3-0.6B --enable-sleep-mode --port8002". Starting the inference service replica according to this target configuration will enable state switching for that replica.
[0037] It should be noted that the inference service copy is in an active state by default after startup.
[0038] Understandably, after the inference service replica is successfully allocated a GPU, it is launched on the target node in a containerized manner, completing the download of the container image of the inference service replica, the startup of the VLLM inference engine process, the loading of model files on the GPU, and the startup process of the inference service replica. The target configuration is used when starting the inference service replica.
[0039] For any inference service replica, the node where the graphics processor assigned to that inference service replica resides is the same node.
[0040] Step S304: Based on the scaling metric, switch the state of the first inference service replica in the replica pool to the active state to handle inference requests.
[0041] Step S305: Switch the state of all inference service replicas in the replica pool except for the first inference service replica to dormant state.
[0042] Understandably, a dormant inference service replica cannot handle inference requests.
[0043] Step S306: If the number of inference requests received by the target inference service within a preset time period is greater than the first request threshold, then select the first target inference service replica in a dormant state from the replica pool of the target inference service and switch the state of the first target inference service replica to an active state.
[0044] The first request threshold is set by technical personnel and is not specifically limited here. If the number of inference requests received by the target inference service within a preset time period exceeds the first request threshold, it indicates that the target inference service needs to be expanded. In this case, a dormant first target inference service replica is selected from the replica pool of the target inference service, and its status is switched to active, thus completing the expansion of the target inference service.
[0045] The inference service replica pool management method provided in this embodiment constructs a replica pool of inference services by starting multiple inference service replicas in active and dormant states during the inference service creation phase. Based on the state switching of the inference service replicas, the inference service can be expanded. This eliminates the need to download the container image of the inference service replica, load the model file, and start the inference service replica each time expansion is performed. Therefore, it can solve the technical problem of long expansion time of inference services in related technologies, and achieve the technical effects of reducing the expansion time of inference services, reducing the frequency of inference service replica creation and destruction, and improving the responsiveness of inference services.
[0046] Embodiments of this application provide a method for managing an inference service replica pool, applied to the inference service lifecycle management component in the aforementioned cluster management node. Figure 4 A flowchart illustrating the inference service replica pool management method provided in this application embodiment is shown below. Figure 4 As shown, the inference service replica pool management method includes the following steps: Step S401: Obtain the scaling metrics for the inference service to be created. See details in [link to relevant documentation]. Figure 3 Step S301 of the illustrated embodiment will not be described again here.
[0047] Step S402: Based on the scaling metrics, create a replica pool of the inference service to be created, and allocate computing units to each inference service replica in the replica pool.
[0048] Specifically, step S402 includes: Step S4021: Based on the maximum number of replicas in the scaling metrics, create a replica pool for the inference service to be created. The number of inference service replicas included in the replica pool is the same as the maximum number of replicas.
[0049] Step S4022: Based on the number of computing units in the scaling metrics, allocate a corresponding number of computing units to each inference service replica in the replica pool.
[0050] Step S403: For any inference service replica, determine the target node where the computing unit allocated to that inference service replica is located, download the container image of that inference service replica on the target node, start the inference engine process, load the model file corresponding to that inference service replica into the computing unit allocated to that inference service replica, and start the inference service replica according to the target configuration to enable the inference service replica to support state switching. For details, please refer to [link to details]. Figure 3 Step S303 of the illustrated embodiment will not be described again here.
[0051] Step S404: Based on the scaling metric, switch the state of the first inference service replica in the replica pool to the active state to handle inference requests.
[0052] Specifically, step S404 includes: Step S4041: Determine the first inference service replica in the replica pool based on the minimum number of replicas in the scaling metrics.
[0053] Understandably, the number of first-inference service replicas is the same as the minimum number of replicas in the scaling metrics.
[0054] Step S4042: Obtain the address of the first inference service replica and configure the address of the first inference service replica to the load balancing module to switch the state of the first inference service replica to the active state.
[0055] The load balancing module, also known as the aforementioned load balancer, activates the first inference service replica by configuring its IP address. Inference requests received from users are then forwarded to the active inference service replica for processing. In other words, the active inference service replica can be used for inference request processing.
[0056] Step S405: Switch the state of all inference service replicas in the replica pool except for the first inference service replica to dormant state.
[0057] Specifically, step S405 includes: Step S4051: For any of the other inference service replicas, obtain the address of that inference service replica.
[0058] Step S4052: Based on the address of the inference service replica, call the hibernation interface of the inference engine to trigger the operation of unloading the computing unit resources corresponding to the inference service replica to the processor memory corresponding to the inference service replica, so that the state of the inference service replica is switched to hibernation state.
[0059] For any of the other inference service replicas, based on the IP address of that inference service replica, the sleep interface of the VLLM inference engine is called to trigger the unloading of the GPU memory resources corresponding to that inference service replica to the CPU memory corresponding to that inference service replica. The GPU resources occupied by that inference service replica are released and can be used by other inference service replicas.
[0060] Step S406: If the number of inference requests received by the target inference service within a preset time period exceeds a first request threshold, then a dormant first target inference service replica is selected from the target inference service replica pool, and the state of the first target inference service replica is switched to the active state. For details, please refer to... Figure 3 Step S306 of the illustrated embodiment will not be described again here.
[0061] The inference service replica pool management method provided in this embodiment pre-creates a replica pool based on the "maximum number of replicas" and statically allocates a specified number of GPUs to each replica. This ensures that the system can be immediately scaled up during peak traffic periods (without cold start waiting) while avoiding scheduling delays and resource contention issues caused by dynamic scaling. Core service replicas are precisely activated based on the minimum number of replicas, ensuring basic availability of the inference service and meeting minimum business access requirements. Only the addresses of activated core replicas are configured in the load balancing module, ensuring that user requests are accurately forwarded to available replicas, avoiding invalid requests and improving service response efficiency. All replicas beyond the minimum number of replicas undergo a unified hibernation operation, precisely releasing GPU memory core resources and improving overall GPU resource utilization.
[0062] In some optional implementations, step S4022 above includes: Step a1: For any inference service replica, obtain the status information of the computing units in each node of the target cluster.
[0063] The status information includes the number of times the device has been reused.
[0064] Step a2: Based on the status information of the computing units in each node of the target cluster, determine the first computing unit in each node whose reuse count is lower than the preset reuse count threshold.
[0065] The preset reuse threshold is the aforementioned reuse ratio.
[0066] Step a3: Select second computing units from the first computing units in each node that are not running active inference service replicas to determine the number of second computing units in each node.
[0067] Step a4: Based on the number of second computing units in each node, determine the nodes whose number of second computing units is not less than the number of computing units in the scaling index as the first nodes.
[0068] Step a5: Select the second node with the most second computing units from among the multiple first nodes.
[0069] Step a6: Sort the second computing units in the second node in order of increasing reuse count to obtain the sorting result.
[0070] Step a7: Based on the number of computing units and the sorting results in the scaling metrics, allocate computing units to the inference service replica.
[0071] It should be noted that computing units are allocated to the inference service replicas in the replica pool in sequence. The process for allocating computing units to each inference service replica is detailed in steps a1 to a7.
[0072] The inference service replica pool management method provided in this embodiment allocates computing units based on the number of GPU reuses, whether the GPU is running an active inference service replica, and the number of second computing units in the node, ensuring the rationality and accuracy of the computing units allocated to the inference service replica and avoiding resource conflicts.
[0073] Prioritize GPUs with low reuse counts and no active replicas to effectively control the resource consumption pressure of a single GPU and ensure the performance of the inference service. Select the node with the most second-highest number of computing units and allocate them according to their reuse count to improve the overall reuse efficiency of GPU resources in the cluster and reduce resource idleness.
[0074] In some alternative implementations, step a7 above includes: Step a71: Based on the sorting results, determine the preset number of second calculation units that rank first in the sorting. The preset number matches the number of calculation units in the scaling index.
[0075] Step a72: Allocate a preset number of second computing units that are ranked first to the inference service replica.
[0076] In some optional implementations, the above-described inference service replica pool management method further includes: Step b1: If the number of second computing units in each node is less than the number of computing units in the scaling index, then the inference service replica enters the allocation waiting state.
[0077] Step b2: At preset intervals, reassign computing units to the inference service replicas that are in the allocation waiting state.
[0078] The inference service replica pool management method provided in this embodiment triggers an allocation waiting mechanism for resource shortage scenarios, avoiding blind allocation that could lead to GPU resource overload or service startup failure, thus ensuring the stability of cluster resource allocation; it automatically retryes allocation according to a preset duration without manual intervention, achieving automated resource allocation and reducing operation and maintenance costs.
[0079] In some optional implementations, the above-described inference service replica pool management method further includes: Step c1: If the number of successfully started inference service replicas in the replica pool is less than the minimum number of replicas in the scaling metrics, then it is determined that the inference service cannot process inference requests.
[0080] Understandably, a successful startup includes successfully allocating computing units and successfully completing the startup process.
[0081] Step c2: If the number of successfully started inference service replicas in the replica pool is not less than the minimum number of replicas in the scaling metrics, then the inference service is determined to have the capability to process inference requests.
[0082] Step c3: If the inference service has the capability to make inference requests, based on the scaling metrics, switch the state of the first inference service replica in the replica pool to the active state to handle inference requests.
[0083] Understandably, when the inference service has the ability to make inference requests, the state of all inference service replicas in the replica pool except for the first inference service replica will be switched to a dormant state.
[0084] Figure 5 A flowchart for allocating graphics processors to inference service replicas in a replica pool, as provided in this application embodiment, is shown below. Figure 5 As shown, the process includes: Elastic inference services entering the scheduling queue. Scheduling begins for the minimum number of inference service replicas in the replica pool. It is necessary to ensure that the minimum number of inference service replicas can successfully allocate graphics processors. If not all of the minimum number of inference service replicas can be successfully allocated graphics processors, the inference service is in an allocation waiting state and cannot process external inference requests. At preset intervals, graphics processors are reassigned to the minimum number of inference service replicas in the replica pool that are in the allocation waiting state. This can be understood as reassigning graphics processors to the inference service replicas that failed to be allocated. Cards that meet the graphics processor reuse rate are selected, i.e., the first graphics processor in each node whose reuse count is lower than a preset reuse count threshold is determined. If at least one node has cards that meet the graphics processor reuse rate, cards without active service instances are selected, i.e., second graphics processors that are not running active inference service replicas are selected from the first graphics processors in at least one node. If no card meets the graphics processor reuse rate in each node, the elastic inference service corresponding to the inference service replica is determined to be in the allocation waiting state, and a periodic reallocation attempt is made. That is, at a preset interval, the graphics processor is reallocated to the smallest number of inference service replicas in the replica pool of the inference service in the allocation waiting state.
[0085] If at least one node contains a second graphics processor, then the node with a number of second graphics processors that is not less than the number of graphics processors in the scaling metric is designated as the first node. From the multiple first nodes, the second node with the largest number of second graphics processors is selected. The second graphics processors in the second node are sorted in ascending order of reuse count, and a predetermined number of the top-ranked second graphics processors are allocated to the currently assigned inference service replica. This reduces the probability of resource contention when other inference service replicas are activated.
[0086] If no second graphics processor exists in each node, the elastic inference service corresponding to the inference service replica is determined to be in an allocation waiting state, and a periodic reallocation attempt will be made.
[0087] The scheduling of the minimum replica and several inference service replicas is completed according to the above process.
[0088] The scheduling of remaining inference service replicas in the replica pool has begun. The number of remaining inference service replicas is [number missing]. These inference service replicas will enter a dormant state and will not exclusively occupy GPU resources. Cards that meet the GPU reuse rate are selected, i.e., the first GPU in each node whose reuse count is below a preset reuse count threshold is identified. If at least one node has a card that meets the GPU reuse rate, cards without active service instances are selected, i.e., a second GPU from the first GPU in at least one node that is not running an active inference service replica is selected. If no card in each node meets the GPU reuse rate, the currently pending inference service replica is determined to be in an allocation waiting state, and a periodic reallocation attempt is performed, i.e., at preset intervals, GPUs are reassigned to the inference service replicas in the allocation waiting state.
[0089] If at least one node contains a second graphics processor, then the node with a number of second graphics processors that is not less than the number of graphics processors in the scaling metric is designated as the first node. From the multiple first nodes, the second node with the largest number of second graphics processors is selected. The second graphics processors in the second node are sorted in ascending order of reuse count, and a predetermined number of the top-ranked second graphics processors are allocated to the currently assigned inference service replica. This reduces the probability of resource contention when other inference service replicas are activated.
[0090] If no second graphics processor exists in each node, the currently assigned inference service replica is determined to be in an allocation waiting state, and a periodic reallocation attempt is made.
[0091] Complete the scheduling of the remaining inference service replicas according to the above process, in order to complete the scheduling of all inference service replicas.
[0092] Figure 6 This application provides a schematic diagram illustrating the state of inference service replicas in a replica pool for an inference service, as shown in the embodiments of this application. Figure 6As shown, assuming each node includes 8 GPU cards, and each replica of inference services A and B requests 8 GPU cards, the minimum number of replicas for inference service A is 2, and the maximum number of replicas is 3, while the minimum number of replicas for inference service B is 1, and the maximum number of replicas is 3, this example illustrates the process. Node 1 runs the active replica 1 of inference service A, node 2 runs the active replica 1 of inference service B, node 3 runs the dormant replicas 2 of inference service A and 2 of inference service B, and node 4 runs the active replica 3 of inference service A and 3 of inference service B.
[0093] The inference service replica pool management method provided in this embodiment uses the minimum number of replicas as the core judgment threshold to accurately verify the basic operational capabilities of the inference service, ensuring that the service has the minimum computing power to handle requests and avoiding invalid service startup; it clarifies the dual judgment criteria for successful startup (GPU allocation completed + replica startup successful), improving the accuracy of service availability judgment and avoiding request processing failures caused by abnormal resource allocation from the source; it only activates core replicas to accept requests after the service is confirmed to have processing capabilities, ensuring the operational stability of the inference service and avoiding service crashes or abnormal responses caused by replicas not being ready to receive requests.
[0094] In some alternative implementations, step S306 includes: Step d1: If the number of inference requests received by the target inference service within a preset time period exceeds the first request threshold, then a dormant second inference service replica is selected from the replica pool of the target inference service.
[0095] The preset time period is set by technical personnel and is not specifically limited here. If the number of inference requests received by the target inference service within the preset time period exceeds the first request threshold, a second inference service replica that is in a dormant state is selected from the replica pool of the target inference service.
[0096] Step d2: Select the third inference service copy from the second inference service copy, which is a computing unit that is not running in an active state.
[0097] When multiple dormant inference service replicas exist, select the inference service replica assigned to the GPU card that does not have any other active inference service replicas.
[0098] Step d3: Determine the first target inference service copy from the third inference service copy, and switch the state of the first target inference service copy to the active state.
[0099] Specifically, one inference service copy is randomly selected from the third inference service copy as the first target inference service copy.
[0100] The inference service replica pool management method provided in this embodiment triggers expansion based on request volume thresholds, accurately responds to business traffic peaks, and ensures the request processing capacity and response efficiency of the inference service; it filters dormant replicas layer by layer, prioritizing the activation of replicas that are not active on GPUs, avoiding GPU resource contention and ensuring the running performance of the inference service after activation; it switches the state of the first target inference service replica to active state by calling the vLLM interface and registers the address of the first target inference service replica to the load balancing module, achieving second-level expansion.
[0101] In some alternative implementations, step d3 above includes: Step d31: Obtain the address of the first target inference service copy.
[0102] Step d32: Based on the address of the first target inference service replica, call the wake-up interface of the inference engine to trigger the operation of loading the resources of the processor memory corresponding to the first target inference service replica into the computing unit corresponding to the first target inference service replica.
[0103] Based on the IP address of the first target inference service replica, the VLLM inference engine's wake_up interface is invoked, triggering the loading of CPU memory resources corresponding to the first target inference service replica onto the GPU corresponding to the first target inference service replica. It can be understood that the GPU corresponding to the first target inference service replica is the GPU allocated to the first target inference service replica.
[0104] Step d33: Configure the address of the first target inference service replica to the load balancing module so that the state of the first target inference service replica is switched to the active state.
[0105] Configure the IP address of the first target inference service replica to the load balancing module so that the state of the first target inference service replica changes from dormant to active.
[0106] In some optional implementations, the above-described inference service replica pool management method further includes: Step e1: If a third inference service copy is not found from the second inference service copy, a fourth inference service copy is randomly selected from the second inference service copy.
[0107] In other words, if each second inference service copy corresponds to a GPU that is running an active inference service copy, resource preemption is triggered. A fourth inference service copy is randomly selected from the second inference service copies. The active inference service copy running in the computing unit allocated to the fourth inference service copy is switched to a dormant state. The fourth inference service copy is determined as the first target inference service copy. The state of the first target inference service copy is switched from a dormant state to an active state.
[0108] Step e2, switch the state of the active inference service replica running in the computing unit allocated to the fourth inference service replica to the dormant state.
[0109] Step e3: The fourth inference service copy is identified as the first target inference service copy, and the state of the first target inference service copy is switched to the active state.
[0110] Figure 7 A schematic diagram illustrating the switching of the state of the fourth inference service copy to an active state, provided for an embodiment of this application, is shown below. Figure 7 As shown, the fourth inference service replica is a dormant replica 2 of inference service B on node 3. It can be seen that an active inference service replica, i.e., active replica 2 of inference service A, is running on the GPU corresponding to the fourth inference service replica. To scale up inference service B and switch the state of the fourth inference service replica to active, it is necessary to preempt resources from inference service A, schedule active replica 2 of inference service A to node 4, and then switch the state of the fourth inference service replica to active. It is understandable that it is also possible to switch active replica 2 of inference service A to dormant state and switch the state of the fourth inference service replica to active state.
[0111] The inference service replica pool management method provided in this embodiment designs a fallback expansion strategy for resource consumption scenarios to ensure that there are always dormant replicas available for activation during traffic peaks, thus ensuring that the request handling capacity of the inference service is not interrupted.
[0112] In some optional implementations, the above-described inference service replica pool management method further includes: Step f1: If the number of inference requests received by the target inference service within a preset time period is less than the second request threshold, then select the active second target inference service replica from the replica pool of the target inference service and switch the state of the second target inference service replica to the dormant state.
[0113] The second request threshold is lower than the first request threshold. The second request threshold is set by technical personnel and is not specifically limited here.
[0114] Specifically, if the number of inference requests received by the target inference service within a preset time period is less than the second request threshold, an active second target inference service replica is randomly selected from the replica pool of the target inference service. The IP address of the second target inference service replica is removed from the load balancing module. Based on the IP address of the second target inference service replica, the sleep interface of the VLLM inference engine is called to trigger the operation of unloading the computing unit resources corresponding to the second target inference service replica to the processor memory corresponding to the second target inference service replica, so that the state of the second target inference service replica is switched to sleep state.
[0115] Furthermore, if the number of inference requests received by the target inference service within a preset time period is less than the second request threshold, then an active fifth inference service replica is selected from the target inference service replica pool. A sixth inference service replica is selected from the fifth inference service replicas to run other inference service replicas using allocated computing units; a second target inference service replica is randomly selected from the sixth inference service replicas, and its state is switched to a dormant state. Other inference services are those different from the target inference service.
[0116] Figure 8 A flowchart for scaling up and down the inference service provided in this application embodiment is shown below. Figure 8 As shown, the process for scaling up the inference service includes: triggering a scaling request; selecting a dormant replica; specifically, if the number of inference requests received by the target inference service within a preset time period exceeds a first request threshold, a dormant second inference service replica is selected from the target inference service's replica pool; determining whether a third inference service replica exists among the second inference service replicas; if so, determining the first target inference service replica from the third inference service replicas; and, based on the address of the first target inference service replica, calling the inference engine's wake-up interface to refresh the available replica configuration of the load balancing module, thereby switching the state of the first target inference service replica to the active state.
[0117] If it does not exist, resource preemption is performed, that is, a fourth inference service replica is randomly selected from the second inference service replicas; the active inference service replica running on the graphics processor allocated to the fourth inference service replica is switched to a dormant state; the fourth inference service replica is determined as the first target inference service replica, and based on the address of the first target inference service replica, the wake-up interface of the inference engine is called to refresh the available replica configuration of the load balancing module so that the state of the first target inference service replica is switched to an active state.
[0118] The process of scaling down the inference service includes: triggering a scaling-down request; selecting an active replica; specifically, if the number of inference requests received by the target inference service within a preset time period is less than a second request threshold, then selecting an active fifth inference service replica from the target inference service's replica pool; prioritizing the selection of graphics processors with replicas of other services from the fifth inference service replicas, i.e., selecting a sixth inference service replica from the fifth inference service replicas to run inference service replicas of other inference services; randomly determining a second target inference service replica from the sixth inference service replicas; and, based on the address of the second target inference service replica, calling the inference engine's sleep interface to refresh the available replica configuration of the load balancing module, so that the second target inference service replica switches to a sleep state.
[0119] The inference service replica pool management method provided in this embodiment distinguishes between traffic peaks and troughs based on dual request thresholds, triggering scaling-down actions to accurately match low-traffic scenarios and achieving closed-loop management of elastic scaling of the inference service. It proactively switches active replicas to dormant states under low traffic conditions, promptly releasing GPU memory and computing resources, improving the overall reuse efficiency of cluster GPU resources, and avoiding resource idleness. By calling the vLLM interface to switch the state of the second target inference service replica to dormant state and removing the address of the second target inference service replica in the load balancing module, it achieves second-level scaling-down.
[0120] The inference service replica pool management method provided in this embodiment constructs an instance pool for the inference service by activating multiple inference service replicas in both active and dormant states during the inference service creation phase. Based on the switching between active and dormant states of the service replicas, the method accelerates the scaling up and down of the inference service, reduces the frequency of inference service creation and destruction, lowers the latency of inference service scaling, and improves the responsiveness of the inference service. For service replicas in a dormant state, their GPU resources can be simultaneously used by other active inference services, improving the efficiency of computing resource utilization.
[0121] In some optional implementations, the above-described inference service replica pool management method further includes: Step g1: Monitor the running status of active inference service replicas in the inference service replica pool in real time.
[0122] Step g2: If at least one active inference service replica indicates that the active inference service replica is running abnormally, then restart the active inference service replica that is in an abnormal state.
[0123] Specifically, on the node where the computing unit corresponding to the active inference service copy in the abnormal state is located, the process of downloading the container image of the active inference service copy in the abnormal state is re-executed, starting the inference engine process, loading the model file corresponding to the active inference service copy in the abnormal state into the computing unit corresponding to the active inference service copy in the abnormal state, and starting the active inference service copy in the abnormal state according to the target configuration is carried out to restore the active inference service copy in the abnormal state to normal.
[0124] The inference service replica pool management method provided in this embodiment monitors the running status of active replicas in real time, enabling timely detection of anomalies and preventing service failures caused by abnormal replicas accepting requests from the source, thus ensuring the availability of the inference service. For abnormal active replicas, a local restart process is executed on the node, binding the originally allocated GPU resources to improve anomaly recovery efficiency. Restarting according to a standardized process ensures that the running status of abnormal replicas after recovery is consistent with that of normal replicas, guaranteeing service stability. It achieves automated detection and self-healing of active replica anomalies without manual intervention, reducing operation and maintenance costs while maintaining an effective active scale of the replica pool, ensuring inference request processing capacity.
[0125] To make the process of creating the reasoning service in this application clearer, combined with Figure 9 Please provide an explanation. Figure 9 A sequence diagram for creating inference services provided in the embodiments of this application, such as Figure 9 As shown, in response to a user's elastic inference service creation request, the inference service lifecycle management component creates the maximum number of service instances with replicas on the inference service management platform. These service instances enter the scheduling queue of the graphics processor scheduling module. The graphics processor scheduling module schedules the service instances with the minimum number of replicas, that is, it allocates graphics processors to the service instances with the minimum number of replicas. The graphics processor scheduling module schedules the remaining service instances with the remaining number of replicas, that is, it allocates graphics processors to the remaining service instances. All service replicas are started according to the startup process. That is, for any inference service replica, the target node where the graphics processor allocated to that inference service replica is located is determined, the container image of that inference service replica is downloaded on the target node, the inference engine process is started, the model file corresponding to that inference service replica is loaded onto the graphics processor allocated to that inference service replica, and the inference service replica is started according to the target configuration. The inference service lifecycle management component obtains the address information of all service replicas. Based on the address information of service replicas that are pre-set to a dormant state, it calls the dormant interface of the inference engine to trigger the graphics processor resource unloading, releasing the graphics processor resources so that the service replica is updated to a dormant service replica. The address information of the service replica that is pre-set to be in an active state is registered with the load balancing module so that the load balancing module can provide access addresses for the active service replica, and the service replica is updated to be in an active state.
[0126] If the number of active service replicas is greater than the minimum number of replicas, the inference service lifecycle management component determines that the inference service is running and can be used, i.e., it can handle inference requests.
[0127] Understandably, the graphics processor scheduling module periodically allocates graphics processors to service replicas that are in an allocation waiting state.
[0128] This application implements elastic scaling of the inference service replica pool, optimizing the existing AI platform's inference service scaling mechanism to accelerate rapid scaling of the inference service, reduce user request waiting time, and improve GPU resource utilization. Upon initial startup, an inference service with elastic scaling configuration starts the minimum number of active inference service replicas and multiple dormant inference service replicas, constructing a replica pool. When the inference service meets scaling requirements, the dormant service instances are activated, and their access addresses are configured in the load balancer. When the inference service meets scaling requirements, some inference service instances are put into dormancy, and the access addresses of the active service instances are removed from the load balancer. Simultaneously, based on GPU reuse scheduling, multiple inference service replicas are started on the same GPU card, with only one replica active and capable of processing inference requests, while the others are dormant and do not receive inference requests, thus improving cluster GPU resource utilization.
[0129] The inference service replica pool management method provided in this embodiment completes operations such as container image downloading, GPU loading of model files, CUDA initialization, and vLLM engine initialization for all replicas during the initial deployment phase of the elastic inference service. This advances the time-consuming operations of the scaling phase, thereby shortening the inference service scaling latency and improving the inference service's responsiveness under sudden load surges. After all replicas are loaded and running, based on the sleep mode of the vLLM inference engine, resources of service replicas other than the minimum number of replicas are released, including GPU resources. Simultaneously, based on GPU reuse scheduling capabilities, the released GPU resources are allocated to other inference service replicas. Based on this mechanism, the initialization results of the inference service can be fully reused, reducing the redundant overhead of model loading and inference engine initialization, achieving efficient management of inference service replicas, and improving the utilization rate of cluster GPU resources.
[0130] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0131] Embodiments of this application also provide an electronic device, such as... Figure 10 As shown, it includes a processor 1001 and a memory 1002, in which a computer program is stored. The processor 1001 is configured to run the computer program to perform the steps in any of the above-described embodiments of the inference service replica pool management method.
[0132] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above embodiments of the inference service replica pool management method when running.
[0133] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0134] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the inference service replica pool management method.
[0135] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above embodiments of the inference service replica pool management method.
[0136] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0137] The foregoing has provided a detailed description of the inference service replica pool management method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only intended to help understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A method for managing a replica pool for inference services, characterized in that, include: Obtain the scaling metrics for the inference service to be created; Based on the scaling metrics, a replica pool of the inference service to be created is established, and a computing unit is allocated to each inference service replica in the replica pool. For any inference service replica, determine the target node where the computing unit allocated to the inference service replica is located, download the container image of the inference service replica on the target node, start the inference engine process, load the model file corresponding to the inference service replica into the computing unit allocated to the inference service replica, and start the inference service replica according to the target configuration so that the inference service replica supports state switching. Based on the scaling metrics, the state of the first inference service replica in the replica pool is switched to the active state to handle inference requests. Switch the state of all inference service replicas in the replica pool, except for the first inference service replica, to a dormant state; If the number of inference requests received by the target inference service within a preset time period exceeds the first request threshold, then a dormant first target inference service replica is selected from the replica pool of the target inference service, and the state of the first target inference service replica is switched to the active state.
2. The method according to claim 1, characterized in that, The process of creating a replica pool of the inference service to be created based on the scaling metrics, and allocating computing units to each inference service replica in the replica pool, includes: Based on the maximum number of replicas in the scaling metrics, a replica pool for the inference service to be created is created, wherein the number of inference service replicas included in the replica pool is the same as the maximum number of replicas. Based on the number of computing units in the scaling metric, a corresponding number of computing units are allocated to each inference service replica in the replica pool.
3. The method according to claim 2, characterized in that, The allocation of a corresponding number of computing units to each inference service replica in the replica pool based on the number of computing units in the scaling metric includes: For any inference service replica, obtain the status information of the computing units in each node of the target cluster; Based on the status information of the computing units in each node of the target cluster, the first computing unit in each node whose reuse count is lower than a preset reuse count threshold is determined. The number of second computing units in each node is determined by filtering out the second computing units that are not running active inference service replicas from the first computing units in each node. Based on the number of second computing units in each node, the nodes in which the number of second computing units is not less than the number of computing units in the scaling index are determined as the first nodes; Select the second node with the largest number of second computing units from among the multiple first nodes; Sort the second computing units in the second node in ascending order of reuse frequency to obtain the sorting result; Based on the number of computing units in the scaling metrics and the sorting results, computing units are allocated to the inference service replica.
4. The method according to claim 3, characterized in that, The allocation of computing units to the inference service replica based on the number of computing units in the scaling metrics and the sorting results includes: Based on the sorting result, a preset number of second calculation units are determined to be ranked first, and the preset number matches the number of calculation units in the scaling index. The predetermined number of second computing units ranked first are allocated to the inference service copy.
5. The method according to claim 3, characterized in that, The method further includes: If the number of second computing units in each node is less than the number of computing units in the scaling index, then the inference service replica enters the allocation waiting state. At preset intervals, computing units are reallocated to inference service replicas that are in the allocation waiting state.
6. The method according to claim 1, characterized in that, The method further includes: If the number of successfully started inference service replicas in the replica pool is less than the minimum number of replicas in the scaling metric, then it is determined that the inference service cannot process inference requests. If the number of successfully started inference service replicas in the replica pool is not less than the minimum number of replicas in the scaling metric, then the inference service is determined to have the capability to process inference requests. When the inference service has the capability to make inference requests, based on the scaling metric, the state of the first inference service replica in the replica pool is switched to the active state to handle inference requests.
7. The method according to claim 1, characterized in that, The step of switching the state of the first inference service replica in the replica pool to an active state based on the scaling metric includes: The first inference service replica in the replica pool is determined based on the minimum number of replicas in the scaling metric. Obtain the address of the first inference service replica and configure the address of the first inference service replica into the load balancing module to switch the state of the first inference service replica to the active state.
8. The method according to claim 1, characterized in that, Switching the state of all inference service replicas in the replica pool, except for the first inference service replica, to a dormant state includes: For any of the other inference service replicas, obtain the address of that inference service replica; Based on the address of the inference service replica, the hibernation interface of the inference engine is invoked to trigger the operation of unloading the computing unit resources corresponding to the inference service replica to the processor memory corresponding to the inference service replica, so that the state of the inference service replica is switched to hibernation.
9. The method according to claim 1, characterized in that, If the number of inference requests received by the target inference service within a preset time period exceeds a first request threshold, then a dormant first target inference service replica is selected from the replica pool of the target inference service, and the state of the first target inference service replica is switched to an active state, including: If the number of inference requests received by the target inference service within a preset time period exceeds the first request threshold, a dormant second inference service replica is selected from the replica pool of the target inference service. The third inference service copy is selected from the second inference service copy and the computing units allocated are not running active inference service copies. The first target inference service copy is determined from the third inference service copy, and the state of the first target inference service copy is switched to the active state.
10. The method according to claim 9, characterized in that, Switching the state of the first target inference service copy to the active state includes: Obtain the address of the first target inference service copy; Based on the address of the first target inference service copy, the wake-up interface of the inference engine is called to trigger the operation of loading the resources of the processor memory corresponding to the first target inference service copy into the computing unit corresponding to the first target inference service copy; Configure the address of the first target inference service replica to the load balancing module so that the state of the first target inference service replica is switched to the active state.
11. The method according to claim 9, characterized in that, The method further includes: If the third inference service copy is not found from the second inference service copy, then a fourth inference service copy is randomly selected from the second inference service copy. The active inference service replica running in the computing unit allocated to the fourth inference service replica is switched to a dormant state; The fourth inference service copy is identified as the first target inference service copy, and the state of the first target inference service copy is switched to the active state.
12. The method according to claim 1, characterized in that, The method further includes: If the number of inference requests received by the target inference service within a preset time period is less than the second request threshold, then an active second target inference service replica is selected from the replica pool of the target inference service, and the state of the second target inference service replica is switched to a dormant state. The second request threshold is less than the first request threshold.
13. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the inference service replica pool management method as described in any one of claims 1 to 12 when executing the computer program.
14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it implements the steps of the inference service replica pool management method as described in any one of claims 1 to 12.
15. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the inference service replica pool management method as described in any one of claims 1 to 12.