Method and device for scaling inference service, server and storage medium
By monitoring and dynamically adjusting the number of inference service replicas at computing power sites, the problem of single-sided computing power site expansion failure was solved, and stable operation of servers under high load and resource optimization under low load were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA UNITED NETWORK COMM GRP CO LTD
- Filing Date
- 2026-03-04
- Publication Date
- 2026-06-09
Smart Images
Figure CN122173282A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of server technology, and in particular to a method, apparatus, server, and storage medium for scaling up and down inference services. Background Technology
[0002] As large-scale AI models are deployed, AI inference services are gradually becoming computationally intensive and subject to dramatic load fluctuations. For example, during major e-commerce promotions, the number of requests for recommendation models may surge from hundreds of queries per second to tens of thousands of queries per second, while during off-peak periods, a large number of processors will be idle, creating a core contradiction between high concurrency demands and low-cost control.
[0003] In existing technologies, traditional scaling methods for inference services mainly involve scaling down the inference service when the query rate per second decreases, and scaling up the inference service when the query rate per second increases, all through a single computing power site.
[0004] Furthermore, in the existing technology, the operation method of performing inference services through a single computing power site and scaling up the inference service according to the increase in the query rate per second is unable to quickly allocate sufficient computing power when dealing with the peak query rate per second, which easily leads to the failure of inference service scaling up and the server crash. Summary of the Invention
[0005] This application provides a method, apparatus, server, and storage medium for scaling up and down inference services to solve the problem that when inference services are executed through a single computing power site and the inference service is scaled up according to the increase in the query rate per second, it is difficult to quickly allocate sufficient computing power when dealing with peak query rates per second, which can easily lead to inference service scaling failure and server crashes.
[0006] Firstly, this application provides a method for scaling up and down an inference service, applied to the central site of a server, including:
[0007] Monitor the running status data of each inference instance on each pre-created computing power site to determine the number of replicas of the inference service for each computing power site, wherein the number of replicas is configured with a replica number field;
[0008] When the inference service enables scaling up or down, create custom resource data to declare the scaling up or down strategy, wherein the scaling up or down strategy includes at least a scaling up or down trigger threshold.
[0009] Based on the scaling-up / scaling strategy and the scaling-up / scaling trigger threshold, the number of replicas on each computing power site is modified to obtain the modified number of replicas.
[0010] Synchronize the corresponding replica count field based on the modified replica count;
[0011] After the number of replicas is synchronized, the inference service on each computing power site will be made available to the public.
[0012] When each inference service is opened to the outside world for users to access the corresponding inference service, the call address of the inference service of each computing power site is determined according to the number of replicas field.
[0013] Access the inference service according to the call address to determine whether the inference service of the computing power site has completed scaling up or down.
[0014] If it is determined that the scaling up or down of the inference service is not completed, the number of replicas on the remaining computing power sites is increased or decreased to obtain the total number of replicas for each computing power site, so as to complete the cross-site scaling up or down of the inference service for each computing power site.
[0015] In one possible design, monitoring the running status data of each inference instance on each pre-created computing power site to determine the number of replicas of the inference service for each computing power site includes: monitoring the running status data of each inference instance on each pre-created computing power site, identifying one or more target inference instances that are running; and determining the number of replicas of the inference service for each computing power site based on each target inference instance.
[0016] In one possible design, the creation process of each inference instance on each computing power site includes: generating resource configuration data in response to a selection operation on a display interface, the resource configuration data including one or more computing power site identifiers and the number of replicas corresponding to each computing power site identifier; determining the corresponding computing power site based on each computing power site identifier; allocating the resource configuration data to the resource scheduling unit of each computing power site to generate a resource allocation request based on the number of replicas corresponding to each computing power site; and creating one or more inference instances for the inference service of each computing power site based on the resource allocation request.
[0017] In one possible design, modifying the number of replicas on each computing power site according to the scaling-up / scaling-down strategy and the scaling-up / scaling-down trigger threshold to obtain the modified number of replicas includes: determining multiple relevant indicators corresponding to the scaling-up / scaling strategy; determining time-series data corresponding to each relevant indicator based on the running status data of each inference instance; calculating the corresponding scaling-up / scaling data based on each time-series data using a preset scaling-up / scaling calculation tool; comparing the scaling-up / scaling data with the scaling-up / scaling-down trigger threshold to determine whether each computing power site meets the scaling-up / scaling conditions; if one or more computing power sites are determined to meet the scaling-up / scaling conditions, then executing the scaling-up / scaling strategy to modify the number of replicas on each computing power site to obtain the modified number of replicas.
[0018] In one possible design, determining the inference service call address for each computing power site based on the replica quantity field includes: determining the replica quantity ratio for each computing power site based on the replica quantity field; determining the resolution weight for each computing power site based on the replica quantity ratio; determining the resolution script based on the resolution weight; and resolving the domain name corresponding to the inference service for each computing power site using a resolution tool that supports the resolution script, so as to obtain the inference service call address.
[0019] In one possible design, determining the parsing script based on each parsing weight includes: determining the domain name to be parsed and setting a random number sequence; determining the corresponding trigger probability based on each parsing weight; determining the call address corresponding to each trigger probability based on the domain name; determining the random array corresponding to each trigger probability based on the random number sequence; and determining the parsing script based on the domain name, each trigger probability, each call address, and each random array.
[0020] In one possible design, the custom resource data includes: a replica count range, monitoring and triggering rules, scaling-up / scaling trigger thresholds, and scaling-up / scaling logic; the replica count range includes a maximum replica count and a minimum replica count; the monitoring and triggering rules include monitoring targets and monitoring metrics; and the scaling-up / scaling logic includes scaling-up logic and scaling-down logic.
[0021] Secondly, this application provides a scaling device for inference services, including a central site for the server, comprising:
[0022] The monitoring module is used to monitor the running status data of each inference instance on each pre-created computing power site in order to determine the number of replicas of the inference service for each computing power site, wherein the number of replicas is configured with a replica number field;
[0023] The first creation module is used to create custom resource data when the inference service enables scaling up or down to declare the scaling up or down strategy, wherein the scaling up or down strategy includes at least a scaling up or down trigger threshold.
[0024] The modification module is used to modify the number of replicas on each computing power site according to the scaling-up and scaling-down strategy and the scaling-up and scaling-down trigger threshold, so as to obtain the modified number of replicas.
[0025] The synchronization module is used to synchronize the corresponding replica count field based on the modified replica count.
[0026] The open module is used to expose the inference service on each computing power site to the outside world after the number of replicas is synchronized.
[0027] The first determination module is used to determine the call address of the inference service of each computing power site based on the replica quantity field when each inference service is exposed to the outside world so that users can access the corresponding inference service;
[0028] The access module is used to access the inference service according to the call address in order to determine whether the inference service of the computing power site has completed the scaling up or down.
[0029] The adjustment module is used to control the number of replicas on the remaining computing power sites to increase or decrease if it is determined that the expansion or contraction of the inference service is not completed, so as to obtain the total number of replicas of each computing power site and complete the cross-site expansion and contraction of the inference service of each computing power site.
[0030] Thirdly, this application provides a server, including: at least one processor and a memory;
[0031] The memory stores computer-executed instructions;
[0032] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the scaling method for the inference service as described in the first aspect and various possible designs of the first aspect.
[0033] Fourthly, this application provides a computer storage medium storing computer execution instructions, which, when executed by a processor, implement the scaling method for inference services as described in the first aspect and various possible designs of the first aspect.
[0034] The scaling method, apparatus, server, and storage medium for inference services provided in this application determine the number of replicas of the inference service at each computing power site by monitoring the running status data of each inference instance on each pre-created computing power site. The number of replicas is configured with a replica count field. When scaling up or down the inference service, custom resource data is created to declare a scaling strategy, which includes at least a scaling trigger threshold. Based on the scaling strategy and the scaling trigger threshold, the number of replicas on each computing power site is modified to obtain the modified replica count. The corresponding replica count field is synchronized based on the modified replica count. After the replica count fields are synchronized, the inference service on each computing power site is made available to the outside world. When the inference service is made available to the outside world for users to access the corresponding inference service, the replica count field is used to determine the number of replicas for each computing power site. The system calls the address; it accesses the inference service based on the call address to determine whether the inference service of the computing power site has completed scaling up or down; if it determines that the inference service scaling up or down is not complete, it controls the number of replicas on the remaining computing power sites to increase or decrease, obtaining the total number of replicas for each computing power site, in order to complete the cross-site scaling up or down of the inference service of each computing power site. By modifying the number of replicas on each computing power site for scaling up or down according to the scaling up or down strategy and scaling up or down trigger threshold, it continues to determine whether the inference service of the computing power site has completed scaling up or down. If the inference service scaling up or down is not complete, it controls the number of replicas on the remaining computing power sites to increase or decrease, obtaining the total number of replicas for each computing power site, in order to complete the cross-site scaling up or down of the inference service of each computing power site. This allows for the rapid cross-site scheduling of sufficient computing power when dealing with peak query rates per second, avoiding inference service scaling up failure, and thus ensuring the stable operation of the server. Attached Figure Description
[0035] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0036] Figure 1 A schematic diagram illustrating an application scenario of the scaling up / down method for inference services provided in this application embodiment;
[0037] Figure 2 Flowchart of the scaling method for inference services provided in the embodiments of this application Figure 1 ;
[0038] Figure 3 Flowchart of the scaling method for inference services provided in the embodiments of this application Figure 2 ;
[0039] Figure 4 A schematic diagram of the expansion and contraction device for the inference service provided in the embodiments of this application;
[0040] Figure 5 This is a schematic diagram of the hardware structure of the server provided in an embodiment of this application. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0042] As large-scale AI models are deployed, AI inference services are increasingly characterized by computational intensity and drastic load fluctuations. For example, during peak e-commerce promotions, the request rate for recommendation models can surge from hundreds of queries per second to tens of thousands per second, while during off-peak periods, many processors idle, creating a core contradiction between high concurrency demands and low-cost control. Current technologies primarily use a single computing power station to scale inference services. When the query rate per second decreases, the inference service is scaled down, and when the query rate per second increases, it is scaled up. However, this current approach, where inference services are executed by a single computing power station and scaled up based on the increase in query rate per second, cannot quickly allocate sufficient computing power to handle peak query rates, easily leading to inference service scaling failures and server crashes.
[0043] To address the aforementioned technical problems, this application proposes the following technical concept: Considering the number of replicas for the inference service at each computing power site, when the inference service is scaled up or down, the inventors modify the number of replicas on each computing power site based on the scaling up / down strategy and scaling up / down trigger threshold declared in the custom resource data declaration to obtain the modified number of replicas. The modified number of replicas is then used to synchronize the corresponding replica number fields. After the synchronization of the replica number fields is complete, the call address for the inference service at each computing power site is determined based on the replica number fields. The inference service is accessed using the call address. If it is determined that the inference service scaling up or down is not complete, the number of replicas on the remaining computing power sites is increased or decreased to obtain the total number of replicas at each computing power site. This completes the cross-site scaling up and down of the inference service at each computing power site, ensuring stable server operation.
[0044] Figure 1 This is a schematic diagram illustrating an application scenario of the scaling up and down method for inference services provided in this application embodiment.
[0045] like Figure 1 As shown, this scenario includes: client 101 and server 102.
[0046] Among them, the user terminal 101 can be a display screen or a personal computer or other terminal.
[0047] Server 102 can be a standalone server or a cluster of multiple servers.
[0048] Server 102 monitors the running status data of each inference instance on each pre-created computing power site to determine the number of replicas of the inference service on each computing power site, where the number of replicas is configured with a replica count field. When the inference service is scalable, it creates custom resource data to declare the scaling strategy, which includes at least a scaling trigger threshold. Based on the scaling strategy and the scaling trigger threshold, it modifies the number of replicas on each computing power site to obtain the modified number of replicas. It synchronizes the corresponding replica count field based on the modified replica count. After the replica count field is synchronized, it exposes the inference service on each computing power site to the outside world. When the inference service is exposed to the outside world so that client 101 can access the corresponding inference service, it determines the calling address of the inference service on each computing power site based on the replica count field. It accesses the inference service based on the calling address to determine whether the inference service of the computing power site has completed scaling. If it is determined that the inference service scaling is not completed, it controls the number of replicas on the remaining computing power sites to increase or decrease, obtaining the total number of replicas on each computing power site, so as to complete the cross-site scaling of the inference service of each computing power site. The following detailed embodiments will be used to illustrate the point.
[0049] Figure 2 Flowchart of the scaling method for inference services provided in the embodiments of this application Figure 1 The execution entity in this embodiment can be Figure 1 The server in the illustrated embodiment is not specifically limited in this embodiment. Figure 2 As shown, the method includes:
[0050] S201: Monitor the running status data of each inference instance on each pre-created computing power site to determine the number of replicas of the inference service for each computing power site, where the number of replicas is configured with a replica number field.
[0051] In this embodiment, the replica count field is the status.replicas field of the `sevingruntime`.
[0052] S202: When the inference service enables scaling up or down, create custom resource data to declare the scaling up or down strategy, wherein the scaling up or down strategy includes at least the scaling up or down trigger threshold.
[0053] Specifically, when the inference service enables scaling, the modelmesh-controller creates custom resource data based on preset resource definition annotations to declare the scaling strategy.
[0054] Among them, modelmesh-controller is a core component of the ModelMesh project, mainly used to manage and coordinate the lifecycle of model services.
[0055] The default resource definition annotation is the keda.io / scale-object annotation added to the corresponding serving runtime.
[0056] The custom resource data is called scaledobject.
[0057] The scaling strategy consists of corresponding triggers, including subfields such as indicator calculation formulas and trigger thresholds.
[0058] In this embodiment, the custom resource data includes: replica count range, monitoring and triggering rules, scaling trigger thresholds, and scaling logic.
[0059] For example, the scaling threshold is 2, meaning that scaling is triggered when the average number of requests exceeds 2.
[0060] The range of replica counts includes the maximum and minimum replica counts.
[0061] The maximum number of replicas is maxReplicaCount; the minimum number of replicas is minReplicaCount.
[0062] For example, the maximum number of copies: 5; the minimum number of copies: 1.
[0063] Monitoring and triggering rules, including monitoring targets and monitoring metrics.
[0064] For example, the monitoring target is the llama-test deployment, i.e., scaleTargetRef: name: llama-test.
[0065] For example, the monitoring metric is requests_processing, which is the average number of requests over 5 minutes. The Prometheus query statement is: calculate the average number of requests processed over all time windows (5 minutes) in the default.llama-test deployment, and sum these averages.
[0066] The expansion and contraction logic includes expansion logic and contraction logic.
[0067] For example, the scaling logic is to increase the number of replicas (up to a maximum of 5) when the average number of requests is greater than 2.
[0068] For example, the scaling-down logic is to reduce the number of replicas (to a minimum of 1) when the average number of requests is less than 2.
[0069] In addition, it also includes the target object for scaling, namely scaleTargetRef.
[0070] S203: Based on the scaling-up / scaling strategy and the scaling-up / scaling trigger threshold, modify the number of replicas on each computing power site to obtain the modified number of replicas.
[0071] Specifically, step S203 includes steps a~e:
[0072] Step a: Determine the relevant metrics corresponding to the scaling up / down strategy.
[0073] In this embodiment, several relevant metrics include GPU / CPU utilization, video memory usage, QPS, inference latency, and request processing time.
[0074] Step b: Based on the running status data of each inference instance, determine the time series data corresponding to each relevant indicator.
[0075] Specifically, using keda-operator, time-series data corresponding to various relevant metrics are queried from Prometheus based on the runtime status data of each inference instance.
[0076] KEDA's controller component, keda-operator, manages the scaling logic within the cluster. It monitors Scaledobject resources and creates or deletes HPA resources based on defined triggers, thereby achieving automatic scaling.
[0077] Prometheus is an open-source monitoring and alerting toolkit specifically designed for collecting and querying time-series data, helping you monitor the real-time operational status of your systems and applications.
[0078] Step c: Calculate the corresponding scaling data based on each time series data using a preset scaling calculation tool.
[0079] In this embodiment, the preset scaling calculation tool is the index calculation formula in the triggers field.
[0080] Step d: Compare the scaling data with the scaling trigger threshold to determine whether each computing site meets the scaling conditions.
[0081] Step e: If one or more computing power sites are determined to meet the scaling conditions, then the scaling strategy is executed to modify the number of replicas on each computing power site to obtain the modified number of replicas.
[0082] Specifically, if one or more computing power sites are determined to meet the scaling conditions, the scaling strategy is executed. Through keda-operator, the deployment specified by scaledobject is scaled up or down to modify the number of replicas on each computing power site, so as to obtain the modified number of replicas.
[0083] S204: Synchronize the corresponding replica quantity field according to each modified replica quantity.
[0084] Specifically, the replica count field is synchronized based on the modified replica count using the modelmesh-controller.
[0085] In addition, the value of the replica count field (the actual number of replicas running) is synchronized to the central site by go-client.
[0086] S205: After the number of replicas is synchronized, the inference service on each computing power site will be made available to the outside world.
[0087] Specifically, after the number of replicas is synchronized, the inference services on each computing power site are made available to the public through a preset display tool.
[0088] The default display tool is a load-balance type Service.
[0089] S206: When each inference service is opened to the outside world so that users can access the corresponding inference service, the call address of the inference service of each computing power site is determined according to the number of replicas field.
[0090] Specifically, step S206 includes steps a~c:
[0091] Step a: Determine the replica quantity ratio for each computing power site based on the replica quantity field.
[0092] For example, each computing power station consists of two computing power stations, with a corresponding replica ratio of 7:3.
[0093] Step b: Determine the parsing weight of each computing site based on the proportion of each replica.
[0094] For example, the resolution weights of each computing site determined according to the 7:3 ratio are 0.7 and 0.3.
[0095] Step c: Determine the parsing script based on each parsing weight, and resolve the domain name corresponding to the inference service of each computing power site according to the parsing tool that supports the parsing script, so as to obtain the calling address of the inference service.
[0096] In this embodiment, the DNS server that supports the resolution script is PowerDNS.
[0097] PowerDNS is an open-source DNS service suite whose core functions are providing authoritative domain name resolution, recursive resolution, and load balancing, and it supports multiple databases and high-performance configurations.
[0098] Specifically, step c, determining the parsing script based on each parsing weight, includes steps c1 to c5:
[0099] Step c1: Determine the domain name to be resolved and set the random number sequence.
[0100] For example, the domain name to be resolved is www.example.com; the random number sequence is 1~100.
[0101] Step c2: Determine the corresponding trigger probability based on each parsing weight.
[0102] For example, the corresponding trigger probabilities are determined to be 70% and 30% based on the parsing weights of 0.7 and 0.3.
[0103] Step c3: Determine the calling address corresponding to each trigger probability based on the domain name.
[0104] For example, 70% corresponds to the call address IP1 (192.168.1.10); 30% corresponds to the call address IP2 (192.168.1.11).
[0105] Step c4: Determine the random array corresponding to each trigger probability based on the random number sequence.
[0106] For example, the random array corresponding to 70% is [1,70]; the random array corresponding to 30% is (70,100).
[0107] Step c5: Determine the parsing script based on the domain name, each trigger probability, each calling address, and each random array.
[0108] For example, the logic for parsing the script is as follows:
[0109] Triggering conditions: The parsing logic is executed only when the requested domain name is www.example.com; otherwise, it returns false. Random probability allocation: A random number between 1 and 100 is generated. If the random number is ≤70 (70% probability), the parsing result 192.168.1.10 is added to the request object; otherwise, 192.168.1.11 is added. The TTL for both is 300 seconds. Return result: Returns true if the parsing is successful, and false if it fails.
[0110] S207: Access the inference service based on the call address to determine whether the inference service of the computing power site has completed scaling up or down.
[0111] S208: If it is determined that the expansion or contraction of the inference service is not completed, then control the number of replicas on the remaining computing power sites to increase or decrease, so as to obtain the total number of replicas of each computing power site, so as to complete the cross-site expansion and contraction of the inference service of each computing power site.
[0112] In this embodiment, "incomplete scaling of inference service" means that a newly created pod cannot reach the Running state within a certain period of time and the status.errMsg field of servingruntime needs to be updated.
[0113] For example, the number of replicas on the remaining computing power sites is increased by increasing the spec.relicas field value of the corresponding servingruntime on other sites to increase the total number of replicas of the inference service instance.
[0114] In summary, the scaling method for inference services provided in this embodiment determines the number of replicas of the inference service at each computing power site by monitoring the running status data of each inference instance on each pre-created computing power site. The number of replicas is configured with a replica count field. When scaling up or down the inference service, custom resource data is created to declare the scaling strategy, which includes at least a scaling trigger threshold. Based on the scaling strategy and the scaling trigger threshold, the number of replicas on each computing power site is modified to obtain the modified replica count. The corresponding replica count field is synchronized based on the modified replica count. After the replica count fields are synchronized, the inference service on each computing power site is made available externally using a preset display tool. When the inference service is made available externally for users to access the corresponding inference service, the replica count field of each computing power site is used to determine the number of replicas. The system retrieves the call address of the inference service and accesses the inference service based on this address to determine if the inference service of the computing power site has completed scaling up or down. If the scaling up or down is incomplete, the system increases or decreases the number of replicas on the remaining computing power sites to obtain the total number of replicas for each computing power site. This allows for cross-site scaling up or down of the inference service across computing power sites. By modifying the number of replicas on each computing power site according to the scaling up / down strategy and trigger threshold, the system continues to determine if the scaling up or down of the inference service of the computing power site is complete. If the scaling up or down is incomplete, the system increases or decreases the number of replicas on the remaining computing power sites to obtain the total number of replicas for each computing power site. This ensures cross-site scaling up or down of the inference service across computing power sites, enabling rapid cross-site scheduling of sufficient computing power to handle peak query rates per second, preventing inference service scaling up failures and ensuring stable server operation. Furthermore, when handling low query rates per second, the system can quickly reduce the total number of replicas on each computing power site to minimize server resource waste.
[0115] In addition, the scaling up and down method for the inference service provided in this embodiment determines the proportion of replicas for each computing power site based on the replica quantity field; determines the resolution weight of each computing power site based on the replica quantity proportion; determines the resolution script based on the resolution weight; and resolves the domain name corresponding to the inference service of each computing power site based on the resolution tool that supports the resolution script, so as to obtain the inference service call address. This realizes the automatic adjustment of access traffic allocation based on the distribution ratio of replica quantity in each cloud pool.
[0116] Figure 3 Flowchart of the scaling method for inference services provided in the embodiments of this application Figure 2 In the embodiments of this application, in Figure 2Based on the provided embodiments, a detailed implementation method is provided for determining the number of replicas of the inference service for each computing power site by monitoring the running status data of each inference instance on each pre-created computing power site in S201. For example... Figure 3 As shown, the method includes:
[0117] S301: Monitor the running status data of each inference instance on each pre-created computing power site to identify one or more target inference instances that are in running status.
[0118] Specifically, the modelmesh-controller monitors the running status data of each inference instance on each pre-created computing power site to identify one or more target inference instances that are in running status.
[0119] Among them, modelmesh-controller is a core component of the ModelMesh project, mainly used to manage and coordinate the lifecycle of model services.
[0120] The running status is "Running".
[0121] In this embodiment, the creation process of each inference instance on each computing power station in S301 specifically includes steps a~d:
[0122] Step a: In response to the selection operation on the display interface, generate resource configuration data, which includes one or more computing site identifiers and the number of replicas corresponding to each computing site identifier.
[0123] Specifically, in response to the selection operation of site information and replicas on the display interface of the preset language client, resource configuration data is generated. The resource configuration data includes one or more computing site identifiers and the number of replicas corresponding to each computing site identifier.
[0124] In this embodiment, the default language client is the Go language client, i.e., go-client.
[0125] Among them, go-client refers to the client that uses the k8s client certificate of the computing power site to initialize a client that can operate the k8s cluster.
[0126] The display interface is the resource distribution display interface.
[0127] The site information includes the site area, the site itself, and the operator.
[0128] In this embodiment, the resource configuration data can be the servingruntime resource created by the client on the computing power site, or it can be other resource configuration data.
[0129] Among them, servingruntime is a type of custom resource, or CR. This CR carries the information required to run an AI inference service, and its custom resource has fields including but not limited to: spec.replicas, image, command, and resources.
[0130] Here, spec.replicas is the desired number of replicas; image is the running image; command is the startup command; and resources are the required resources.
[0131] For example, the computing power site is identified by a site ID, such as 001.
[0132] For example, the resource configuration data includes: the site area is province A, the site is five districts of city B, there are 2 general zones, the operator is C, and the number of copies is 2; the site area is province D, the site is five districts of city E, there is 1 intelligent computing zone, the operator is F, and the number of copies is 2.
[0133] The replica count of 2 means that the spec.replicas field of the servingruntime instance always has a value of 2.
[0134] In addition, an inbound instruction is created based on the resource configuration data to be recorded in the database at the central site.
[0135] The example of an inbound command is an SQL update statement used to modify the resource_map field value of the record with id=1 in the infer_service table. Specifically, it updates the resource_map field of this record to a JSON array containing two objects, each specifying the expected resource count for sites 001 and 002 – both with an expectedCount of 2.
[0136] Step b: Determine the corresponding computing power station based on the identifier of each computing power station.
[0137] In this embodiment, the computing power site is a cloud site that has deployed a Kubernetes cluster.
[0138] Step c: Distribute the resource configuration data to the resource scheduling unit of each computing power site to generate a resource allocation request based on the number of replicas corresponding to each computing power site.
[0139] In this embodiment, the resource allocation request can be either to create a deployment component or to create an inference instance.
[0140] The deployment component, Deployment, is used to put the trained AI model into a practical application, enabling it to truly provide services to users, rather than just remaining in the development environment.
[0141] Inference instances are Pods, which are the smallest deployable units in Kubernetes. They consist of one or more containers and share network and storage resources.
[0142] Step d: Create one or more inference instances for the inference service of each computing power site based on the resource allocation request.
[0143] Specifically, through the modelmesh-controller, the corresponding deployment components are tuned according to the definition of the serving runtime and the resource allocation request mentioned above. The Kubernetes cluster then creates one or more inference instances for the inference service of each computing power site based on the deployment components.
[0144] S302: Determine the number of replicas of the inference service for each test site based on each target inference instance.
[0145] The number of replicas refers to the actual number of replicas running, which is the status.replicas of servingruntime.
[0146] In addition, the value of status.replicas of servingruntime is synchronized to the central site via go-client and recorded in the central site's database, as shown in the following example:
[0147] This UPDATE statement is used to update data in a table named `infer_service` in the database. The table name to be updated is specified as `infer_service`, and the value of the `resource_map` column is set to a JSON string provided later. This JSON string is an array containing two objects, each with attributes such as `siteID`, `expectedCount`, and `runningCount`. The `where id=1` filter condition applies, and the update operation is performed on rows where the `id` column value is 1.
[0148] In summary, the scaling up and down method for inference services provided in this embodiment determines one or more target inference instances that are in operation by monitoring the running status data of each inference instance on each pre-created computing power site; and determines the number of replicas of the inference service for each computing power site based on each target inference instance, making the determination of the number of replicas of the inference service for each computing power site more accurate.
[0149] Figure 4A schematic diagram of the scaling device for the inference service provided in this application embodiment. (See attached diagram.) Figure 4 As shown, the scaling device for the inference service includes: a monitoring module 401, a first creation module 402, a modification module 403, a synchronization module 404, an opening module 405, a first determination module 406, an access module 407, and an adjustment module 408.
[0150] The monitoring module 401 is used to monitor the running status data of each inference instance on each pre-created computing power site in order to determine the number of replicas of the inference service for each computing power site, wherein the number of replicas is configured with a replica number field;
[0151] The first creation module 402 is used to create custom resource data when the inference service starts scaling up or down to declare the scaling up or down strategy, wherein the scaling up or down strategy includes at least the scaling up or down trigger threshold.
[0152] Modification module 403 is used to modify the number of replicas on each computing power site according to the scaling strategy and scaling trigger threshold, so as to obtain the modified number of replicas.
[0153] Synchronization module 404 is used to synchronize the corresponding replica count field according to the modified replica count;
[0154] Open module 405 is used to open up the inference services on each computing power site to the outside world through a preset display tool after the synchronization of the replica quantity field is completed;
[0155] The first determining module 406 is used to determine the calling address of the inference service of each computing power site based on the replica quantity field when each inference service is exposed to the outside world so that the user terminal can access the corresponding inference service.
[0156] Access module 407 is used to access the inference service based on the call address in order to determine whether the inference service of the computing power site has completed the scaling up or down.
[0157] The adjustment module 408 is used to control the number of replicas on the remaining computing power sites to increase or decrease if it is determined that the expansion or contraction of the inference service is not completed, so as to obtain the total number of replicas of each computing power site and complete the cross-site expansion and contraction of the inference service of each computing power site.
[0158] In one possible implementation, the monitoring module 401 specifically includes:
[0159] The monitoring unit is used to monitor the running status data of each inference instance on each pre-created computing power site and identify one or more target inference instances that are in running status.
[0160] The determination unit is used to determine the number of replicas of the inference service for each test site based on each target inference instance.
[0161] In one possible implementation, the device further includes:
[0162] The generation module is used to generate resource configuration data in response to the selection operation on the display interface. The resource configuration data includes one or more computing site identifiers and the number of copies corresponding to each computing site identifier.
[0163] The second determination module is used to determine the corresponding computing power station based on the identifier of each computing power station.
[0164] The allocation module is used to allocate resource configuration data to the resource scheduling unit of each computing power site, and generate resource allocation requests based on the number of replicas corresponding to each computing power site.
[0165] The second creation module is used to create one or more inference instances for the inference service of each computing power site based on resource allocation requests.
[0166] In one possible implementation, module 403 is modified, specifically including:
[0167] The first determining unit is used to determine multiple relevant indicators corresponding to the expansion and contraction strategies.
[0168] The second determining unit is used to determine the time series data corresponding to each relevant indicator based on the running status data of each inference instance;
[0169] The calculation unit is used to calculate the corresponding scaling data based on each time series data using a preset scaling calculation tool.
[0170] The comparison unit is used to compare the scaling data with the scaling trigger threshold to determine whether each computing power site meets the scaling conditions.
[0171] The modification unit is used to execute a scaling-up / scaling strategy if it is determined that one or more computing power sites meet the scaling-up / scaling conditions, so as to modify the number of replicas on each computing power site to obtain the modified number of replicas.
[0172] In one possible implementation, the first determining module 406 specifically includes:
[0173] The first determining unit is used to determine the proportion of replicas corresponding to each computing power site based on each replica quantity field.
[0174] The second determining unit is used to determine the parsing weight of each computing power site based on the proportion of each replica.
[0175] The third determining unit is used to determine the parsing script based on each parsing weight, and to resolve the domain name corresponding to the inference service of each computing power site according to the parsing tool that supports the parsing script, so as to obtain the calling address of the inference service.
[0176] In one possible implementation, the third determining unit specifically includes:
[0177] The first determining unit is used to determine the domain name to be resolved and set the random number sequence;
[0178] The second determining unit is used to determine the corresponding trigger probability based on each parsing weight;
[0179] The third determining unit is used to determine the calling address corresponding to each trigger probability based on the domain name;
[0180] The fourth determining unit is used to determine the random array corresponding to each trigger probability based on the random number sequence;
[0181] The fifth determining unit is used to determine the parsing script based on the domain name, each trigger probability, each calling address, and each random array.
[0182] In one possible implementation, custom resource data is defined, including: replica number range, monitoring and triggering rules, scaling-up and scaling-down triggering thresholds, and scaling-up / scaling logic; replica number range, including the maximum replica number and the minimum replica number; monitoring and triggering rules, including monitoring targets and monitoring metrics; and scaling-up / scaling logic, including scaling-up logic and scaling-down logic.
[0183] The apparatus provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effects are similar, and will not be described again here.
[0184] Figure 5 This is a schematic diagram of the hardware structure of the server provided in an embodiment of this application. Figure 5 As shown, the server in this embodiment includes: a processor 501 and a memory 502; the memory stores computer execution instructions; at least one processor executes the computer execution instructions stored in the memory, causing at least one processor to execute the scaling method of the inference service described above.
[0185] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.
[0186] When the memory 502 is configured independently, the server also includes a bus 503 for connecting the memory 502 and the processor 501.
[0187] This application also provides a computer storage medium storing computer execution instructions. When the processor executes the computer execution instructions, the above-described method for scaling up and down the inference service is implemented.
[0188] This application also provides a computer program product, including a computer program, which, when executed by a processor, implements the above-described scaling method for inference services.
[0189] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.
[0190] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to implement the solution of this embodiment according to actual needs.
[0191] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The unit composed of the above modules can be implemented in hardware or in the form of hardware plus software functional units.
[0192] The integrated modules described above, implemented as software functional modules, can be stored in a computer-readable storage medium. These software functional modules, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods of the various embodiments of this application.
[0193] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. A general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.
[0194] The memory may include high-speed RAM, and may also include non-volatile storage (NVM), such as at least one disk storage device, and may also be a USB flash drive, external hard drive, read-only memory, disk or optical disc, etc.
[0195] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.
[0196] The aforementioned storage media can be implemented from any type of volatile or non-volatile storage device or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The storage media can be any available medium accessible to general-purpose or special-purpose computers.
[0197] An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Alternatively, the storage medium can be an integral part of the processor. Both the processor and the storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and storage medium can exist as discrete components in an electronic device or host device.
[0198] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.
[0199] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.
Claims
1. A method for scaling up and down an inference service, characterized in that, The central site used for the server includes: Monitor the running status data of each inference instance on each pre-created computing power site to determine the number of replicas of the inference service for each computing power site, wherein the number of replicas is configured with a replica number field; When the inference service enables scaling up or down, create custom resource data to declare the scaling up or down strategy, wherein the scaling up or down strategy includes at least a scaling up or down trigger threshold. Based on the scaling-up / scaling strategy and the scaling-up / scaling trigger threshold, the number of replicas on each computing power site is modified to obtain the modified number of replicas. Synchronize the corresponding replica count field based on the modified replica count; After the number of replicas is synchronized, the inference service on each computing power site will be made available to the public. When each inference service is opened to the outside world for users to access the corresponding inference service, the call address of the inference service of each computing power site is determined according to the number of replicas field. Access the inference service according to the call address to determine whether the inference service of the computing power site has completed scaling up or down. If it is determined that the scaling up or down of the inference service is not completed, the number of replicas on the remaining computing power sites is increased or decreased to obtain the total number of replicas for each computing power site, so as to complete the cross-site scaling up or down of the inference service for each computing power site.
2. The method according to claim 1, characterized in that, The monitoring of the runtime status data of each inference instance on each pre-created computing power site is used to determine the number of replicas of the inference service for each computing power site, including: Monitor the running status data of each inference instance on each pre-created computing power site to identify one or more target inference instances that are running. The number of replicas of the inference service for each test site is determined based on each target inference instance.
3. The method according to claim 2, characterized in that, The creation process of each inference instance on each computing power site includes: In response to a selection operation on the display interface, resource configuration data is generated, which includes one or more computing site identifiers and the number of replicas corresponding to each computing site identifier. The corresponding computing power site is determined based on the identifier of each computing power site. The resource configuration data is allocated to the resource scheduling unit of each computing power site to generate a resource allocation request based on the number of replicas corresponding to each computing power site. Based on the resource allocation request, one or more inference instances are created for the inference service of each computing power site.
4. The method according to claim 1, characterized in that, The step of modifying the number of replicas on each computing power site according to the scaling-up / scaling strategy and the scaling-up / scaling trigger threshold to obtain the modified number of replicas includes: Determine multiple relevant indicators corresponding to the scaling-up / shrinking strategy; Based on the running status data of each inference instance, determine the time series data corresponding to each relevant indicator; The corresponding expansion and contraction data is calculated based on each time series data using a preset expansion and contraction calculation tool. The scaling data is compared with the scaling trigger threshold to determine whether each computing power site meets the scaling conditions. If one or more computing power sites are determined to meet the scaling conditions, the scaling strategy is executed to modify the number of replicas on each computing power site to obtain the modified number of replicas.
5. The method according to claim 1, characterized in that, The step of determining the inference service call address for each computing power site based on the replica quantity field includes: Determine the replica quantity ratio for each computing power site based on the replica quantity field; The parsing weight of each computing site is determined based on the proportion of each replica. The parsing script is determined based on each parsing weight, and the domain name corresponding to the inference service of each computing power site is resolved according to the parsing tool that supports the parsing script, so as to obtain the calling address of the inference service.
6. The method according to claim 5, characterized in that, The step of determining the parsing script based on each parsing weight includes: Determine the domain name to be resolved and set the random number sequence; The corresponding trigger probability is determined based on each parsing weight; Based on the domain name, determine the call address corresponding to each trigger probability; Based on the random number sequence, determine the random array corresponding to each trigger probability; The parsing script is determined based on the domain name, each trigger probability, each calling address, and each random array.
7. The method according to claim 1, characterized in that, The custom resource data includes: replica count range, monitoring and triggering rules, scaling trigger thresholds, and scaling logic; The range of replica counts includes the maximum number of replicas and the minimum number of replicas; The monitoring and triggering rules include monitoring targets and monitoring metrics; The expansion and contraction logic includes expansion logic and contraction logic.
8. A scaling device for inference services, characterized in that, The central site used for the server includes: The monitoring module is used to monitor the running status data of each inference instance on each pre-created computing power site in order to determine the number of replicas of the inference service for each computing power site, wherein the number of replicas is configured with a replica number field; The first creation module is used to create custom resource data when the inference service enables scaling up or down to declare the scaling up or down strategy, wherein the scaling up or down strategy includes at least a scaling up or down trigger threshold. The modification module is used to modify the number of replicas on each computing power site according to the scaling-up and scaling-down strategy and the scaling-up and scaling-down trigger threshold, so as to obtain the modified number of replicas. The synchronization module is used to synchronize the corresponding replica count field based on the modified replica count. The open module is used to expose the inference service on each computing power site to the outside world after the number of replicas is synchronized. The first determination module is used to determine the call address of the inference service of each computing power site based on the replica quantity field when each inference service is exposed to the outside world so that users can access the corresponding inference service; The access module is used to access the inference service according to the call address in order to determine whether the inference service of the computing power site has completed the scaling up or down. The adjustment module is used to control the number of replicas on the remaining computing power sites to increase or decrease if it is determined that the expansion or contraction of the inference service is not completed, so as to obtain the total number of replicas of each computing power site and complete the cross-site expansion and contraction of the inference service of each computing power site.
9. A server, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the scaling up / down method of the inference service as described in any one of claims 1 to 7.
10. A computer storage medium, characterized in that, The computer storage medium stores computer execution instructions, and when the processor executes the computer execution instructions, it implements the scaling method for the inference service as described in any one of claims 1 to 7.