A reasoning service system

Abstract

The application provides a reasoning service system, a global controller is used for pre-dividing machine resources into an interactive resource pool and a batch processing resource pool; when a reasoning request is received, a service level target type of the reasoning request is determined, and resource expansion and contraction are performed on the interactive resource pool and the batch processing resource pool according to the service level target type; a cluster controller is used for pre-dividing the interactive resource pool and the batch processing resource pool into a prompt word resource pool and a token resource pool respectively; for a prompt word stage and a token stage of the reasoning request, corresponding prompt word resources and token resources are allocated from the prompt word resource pool and the token resource pool respectively, and resource expansion and contraction are performed on the prompt word resource pool and the token resource pool; and a machine controller is used for adjusting a batch processing size of processing the reasoning request according to the service level target type and a use state of a machine. Under the premise of meeting the service level target, the resource utilization rate and the throughput performance of the reasoning service system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of inference service, and in particular to an inference service system. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, inference service technology based on large language model (LLM) has emerged, which has powerful natural language understanding and generation capabilities and has been widely applied in many fields such as intelligent customer service, document processing, etc. LLM inference service is generally deployed on machine resources. In many LLM inference scenarios, there are usually characteristics such as large fluctuation in the number of user inference requests, high requirements for inference request delay, etc.

[0003] In the traditional technical solution, to solve these problems, cloud service providers usually use automatic scaling technology to handle inference requests. By monitoring and predicting key indicators such as resource utilization, request throughput, and inference delay of machine resources, the machine resources are dynamically adjusted. However, according to the current technical solution, it is still possible to cause violation of service level objective (SLO) (failure to meet service level objective) due to inference request burst peaks, affecting inference efficiency and resource utilization of machine resources.

[0004] Therefore, how to improve the inference efficiency, resource utilization, and throughput performance of the inference service system is a technical problem that needs to be solved by those skilled in the art at present. SUMMARY

[0005] The purpose of the present application is to provide an inference service system that can avoid the impact of resource allocation problems on the inference efficiency of the inference service system while meeting the service level objective, and improve the resource utilization and throughput performance of the inference service system.

[0006] In a first aspect, the present application provides an inference service system, which comprises a global controller, a cluster controller, and a plurality of machine controllers; the global controller is in communication connection with the cluster controller, and the cluster controller is in communication connection with each of the machine controllers;

[0007] The global controller is configured to pre-divide machine resources into an interactive resource pool and a batch processing resource pool; when an inference request is received, the service level objective type of the inference request is determined, and the interactive resource pool and the batch processing resource pool are scaled according to the service level objective type;

[0008] The cluster controller is configured to pre-divide the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool respectively; allocate corresponding prompt word resources and token resources from the prompt word resource pool and the token resource pool respectively for a prompt word stage and a token stage of the inference request, and perform resource expansion and contraction on the prompt word resource pool and the token resource pool.

[0009] The machine controller is configured to adjust a batch size for processing the inference request according to the service level objective type and a usage state of the machine.

[0010] In one of the embodiments, the global controller is configured to, when performing the pre-dividing of the machine resources into the interactive resource pool and the batch resource pool:

[0011] pre-divide the machine resources into the interactive resource pool, the batch resource pool and a hybrid resource pool;

[0012] The global controller is configured to, when performing the resource expansion and contraction on the interactive resource pool and the batch resource pool according to the service level objective type:

[0013] if it is determined according to the service level objective type that the inference request is an interactive request, perform resource expansion and contraction on the interactive resource pool by using the hybrid resource pool.

[0014] In one of the embodiments, the global controller is configured to, when performing the resource expansion and contraction on the interactive resource pool by using the hybrid resource pool if it is determined according to the service level objective type that the inference request is an interactive request:

[0015] determine a first resource amount of the interactive resource pool that is processing the interactive request;

[0016] determine a second resource amount of the hybrid resource pool that is processing the interactive request;

[0017] determine a resource utilization rate of the interactive resource pool according to a ratio of the first resource amount to a preset resource amount of the interactive resource pool;

[0018] determine a resource proportion of the first resource amount and the second resource amount;

[0019] perform resource expansion and contraction on the interactive resource according to the resource utilization rate and the resource proportion.

[0020] In one of the embodiments, the global controller is configured to, when performing the resource expansion and contraction on the interactive resource according to the resource utilization rate and the resource proportion:

[0021] if the resource utilization is greater than or equal to a first preset upper limit of utilization, then according to the resource proportion, part of resources in the mixed resource pool are allocated to the interactive resource pool;

[0022] if the resource utilization is greater than or equal to a second preset upper limit of utilization, then according to the resource proportion, part of resources in the batch processing resource pool are allocated to the interactive resource pool; the first preset upper limit of utilization is less than the second preset upper limit of utilization;

[0023] if the resource utilization is less than or equal to a first preset lower limit of utilization, then according to the resource proportion, a corresponding amount of resources in the interactive resource pool are allocated to the batch processing resource pool;

[0024] if the resource utilization is less than or equal to a second preset lower limit of utilization, then according to the resource proportion, a corresponding amount of resources in the interactive resource pool are allocated to the mixed resource pool; the first preset lower limit of utilization is greater than the second preset lower limit of utilization.

[0025] In one of the embodiments, the global controller is further configured to:

[0026] determine a third amount of resources in the batch processing resource pool that are processing batch processing requests, and determine a remaining amount of resources of the batch processing resource pool according to the third amount of resources;

[0027] based on the remaining amount of resources, perform resource scaling of the interactive resource pool.

[0028] In one of the embodiments, when performing the resource scaling of the interactive resource pool and the batch processing resource pool according to the service level objective type, the global controller is configured to:

[0029] if it is determined according to the service level objective type that the inference request is a batch processing request, group according to the information type of the batch processing request, and put the batch processing request into a batch processing request queue for waiting;

[0030] perform resource scaling of the batch processing resource pool according to a global queue state of the batch processing request queue.

[0031] In one of the embodiments, the global queue state of the batch processing request queue includes a number of batch processing requests and an estimated queue waiting duration.

[0032] In one of the embodiments, when performing the pre-division of the interactive resource pool and the batch processing resource pool into a prompt word resource pool and a token resource pool, the cluster controller is configured to:

[0033] pre-dividing the interactive resource pool and the batch resource pool into a prompt word resource pool, a token resource pool, and a stage mixed resource pool respectively;

[0034] The cluster controller is configured to:

[0035] determine the stage queue state corresponding to the prompt word resource pool and the token resource pool respectively;

[0036] According to the stage queue state, the stage mixed resource pool is used to scale the resource of the prompt word resource pool and the token resource pool.

[0037] In one embodiment, the cluster controller is configured to:

[0038] According to the estimated demand load and the input / output token distribution, the interactive resource pool and the batch resource pool are pre-divided into a prompt word resource pool, a token resource pool, and a stage mixed resource pool respectively.

[0039] In one embodiment, the machine controller is configured to:

[0040] monitor the usage state of the machine in real time;

[0041] According to the usage state, the machine resource type of the machine, and the service level target type, the batch size of the inference request is adjusted.

[0042] The inference service system provided by the application, in a multi-level automatic expansion and contraction framework, a global controller performs resource expansion and contraction on an interactive resource pool and a batch processing resource pool according to a service level target type of an inference request, to ensure that the inference request can obtain resource support conforming to the service level target type; a cluster controller allocates corresponding prompt word resources and token resources from a prompt word resource pool and a token resource pool for a prompt word stage and a token stage of the inference request, and performs resource expansion and contraction on the prompt word resource pool and the token resource pool to meet resource requirements of different processing stages of the inference request; and a machine controller adjusts a batch processing size of processing the inference request according to the service level target type and a use state of a machine belonging to the machine. According to the service level target type of the inference request and characteristics of different processing stages of the inference request, machine resources are reasonably allocated, and the batch processing size is set, so that, on the premise of meeting the service level target, the inference efficiency of the inference service system is prevented from being affected by resource allocation problems, and the resource utilization rate and the throughput performance of the inference service system are improved. BRIEF DESCRIPTION OF DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed to be used in the specific embodiments or prior art description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0044] Figure 1 A system architecture diagram of an inference service system provided by an embodiment of the present application;

[0045] Figure 2 An inference process schematic diagram of an inference service system provided by an embodiment of the present application;

[0046] Figure 3 A flowchart of a global controller performing resource expansion and contraction on an interactive resource pool provided by an embodiment of the present application;

[0047] Figure 4 A flowchart of a cluster controller performing resource expansion and contraction on a prompt word resource pool and a token resource pool provided by an embodiment of the present application. DETAILED DESCRIPTION

[0048] In the following description, specific details such as specific system structures, techniques, etc. are presented to thoroughly understand the embodiments of the present application. However, it should be clear to those skilled in the art that the present application can also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits and methods are omitted to avoid unnecessary details that hinder the description of the present application.

[0049] It should be understood that the term "includes" when used in the specification and the appended claims herein, specifies the presence of stated features, integers, steps, operations, elements, and / or components but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0050] It should also be understood that the term "and / or" when used in the specification and the appended claims herein, means any one or more of the associated listed items and includes all possible combinations of the associated listed items.

[0051] As used in the specification and the appended claims herein, the term "if' can be construed to mean "when" or "once" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be construed to mean "once it is determined" or "in response to determining" or "once [the described condition or event] is detected" or "in response to detecting [the described condition or event]," depending on the context.

[0052] In addition, the terms "first", "second", "third", etc. are used herein only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0053] Reference in the specification to "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in additional embodiments", etc. in various places in the specification are not necessarily all referring to the same embodiment, although they can. The terms "comprising", "including", "having" and their variants mean "including but not limited to", unless otherwise expressly specified. "Multiple" means "two or more".

[0054] Figure 1 A system architecture diagram of a reasoning service system provided for embodiments of the application is shown, only showing parts related to the embodiments for ease of description. The reasoning service system 100 provided by the embodiments includes a global controller 110, a cluster controller 120, and a plurality of machine controllers 130; the global controller 110 is in communication connection with the cluster controller 120, and the cluster controller 120 is in communication connection with each machine controller 130.

[0055] Specifically, the inference service system 100 is in communication connection with the user side 200, and the user side 200 sends an inference request to the inference service system 100 to require the inference service system 100 to process the given inference request and generate a corresponding output (processing result).

[0056] The inference service system 100 in the embodiment includes a plurality of machines to form a machine cluster, and a machine controller 130 is arranged on each machine; that is, the machine controller 130 runs on each machine; and the machine controller 130 monitors and manages the machine resources of the corresponding machine.

[0057] The machine resources refer to the resources in the machine for running the inference task. The machine cluster can be a machine cluster of a single machine resource type, or a machine cluster including a plurality of different machine resource types such as CPU (Central Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), TPU (Tensor Processing Unit), FPGA (Field-Programmable Gate Array), and the like.

[0058] Figure 2 An inference process schematic diagram of an inference service system provided by the embodiment of the present application is provided. In combination with FIG. 1, in the embodiment, the global controller 110 is a controller responsible for overall resource management and allocation, which is used to uniformly schedule and control all machine resources in the inference service system 100; and the cluster controller 120 is a controller responsible for managing the resource pool, which respectively allocates corresponding prompt word resources and token resources for different stages of the inference request. Figure 2

[0059] The global controller 110 is used to pre-divide the machine resources into an interactive resource pool and a batch processing resource pool; when receiving the inference request, the service level target type of the inference request is determined, and the interactive resource pool and the batch processing resource pool are expanded or shrunk according to the service level target type.

[0060] ​It should be noted that the performance evaluation indexes of the inference service system 100 mainly include end-to-end (E2E) delay, throughput, time to first token (TTFT), and time between tokens (TBT), etc., which can constitute different service level objective (SLO) targets. The inference requests of the LLM service can be divided into interactive requests and batch requests based on different SLO targets.

[0061] Among them, the interactive request refers to an inference request that needs to be responded in real time in a user-oriented application such as a chat robot, and the SLO target of the interactive request is usually in seconds, at which time TTFT and TBT are more important evaluation indexes. The interactive resource pool refers to a set of resources specially allocated to interactive requests.

[0062] Among them, the batch request refers to an inference request that can allow a long delay in a backend application such as document processing, and the SLO target of the batch request is generally in minutes or even hours, which pays more attention to the throughput of the request. The batch resource pool refers to a set of resources specially allocated to batch requests.

[0063] Specifically, the global controller 110 pre-divides the machine resources available in the machine cluster corresponding to the inference service system 100 into an interactive resource pool and a batch resource pool, so as to process different types of inference requests respectively.

[0064] When the user side 200 sends an inference request to the inference service system 100, the global controller 110 in the inference service system 100 determines the service level objective type corresponding to the inference request after receiving the inference request, that is, determines whether the inference request is an interactive inference request or a batch inference request, and then performs resource expansion and contraction on the interactive resource pool and the batch resource pool according to the service level objective type, that is, dynamically adjusts the amount of resources allocated in the interactive resource pool and the batch resource pool.

[0065] The cluster controller 120 is configured to pre-divide the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool, allocate corresponding prompt word resources and token resources from the prompt word resource pool and the token resource pool for the prompt word stage and the token stage of the inference request, and perform resource expansion and contraction on the prompt word resource pool and the token resource pool.

[0066] Nowadays, LLMs are mostly based on the Transformer architecture, which uses attention mechanisms and multi-layer perceptrons to understand and analyze input content, and then generates and outputs corresponding content. Due to the autoregressive mode adopted by LLMs, they can only generate one new token at a time, and the generation of each new token depends on all tokens in the previous token sequence, especially their key-value cache (KV Cache). The entire LLM inference phase can be divided into a prompt phase and a token phase. The former processes the input prompt sequence to generate the first output token and preserves the key-value cache, while the latter generates new tokens one by one based on the latest generated token and the previously preserved key-value cache.

[0067] The cluster controller 120 divides the interactive resource pool and the batch resource pool into a prompt resource pool and a token resource pool, respectively, in advance based on the prompt phase and the token phase of the inference request.

[0068] That is, the prompt resource pool is a subset of resources divided from the interactive resource pool and the batch resource pool, and is used exclusively for processing the prompt phase of the inference task; the token resource pool is a subset of resources divided from the interactive resource pool and the batch resource pool, and is used exclusively for processing the token phase of the inference task.

[0069] When the inference request arrives, the cluster controller 120 allocates corresponding prompt resources and token resources from the prompt resource pool and the token resource pool, respectively, for the prompt phase and the token phase of the inference request according to the service level objective type to which the inference request belongs. In addition, the cluster controller 120 adjusts the resource utilization of the prompt resource pool and the token resource pool, that is, dynamically adjusts the amount of resources allocated in the prompt resource pool and the token resource pool.

[0070] The machine controller 130 is configured to adjust the batch size for processing the inference request according to the service level objective type and the usage state of the machine.

[0071] The usage state of the machine includes the current resource utilization of the machine, including CPU utilization, memory utilization, and GPU memory utilization, etc. The specific content of the usage state is not limited in this embodiment.

[0072] The machine controller 130 monitors the usage state of the machine and adjusts the batch size for processing the inference request according to the service level objective type and the usage state of the machine.

[0073] According to the method of the embodiment, for the user side, after sending a reasoning request to the reasoning service system, the reasoning service system processes the reasoning request by using the multi-level automatic expansion and contraction framework, and returns the corresponding processing result.

[0074] The reasoning service system provided in the embodiment can ensure that the reasoning request can obtain resource support conforming to the service level target type thereof by expanding and contracting the interactive resource pool and the batch processing resource pool according to the service level target type of the reasoning request in the multi-level automatic expansion and contraction framework; the cluster controller can allocate corresponding prompt word resources and token resources from the prompt word resource pool and the token resource pool respectively according to the prompt word stage and the token stage of the reasoning request, and expand and contract the prompt word resource pool and the token resource pool to meet the resource requirements of different processing stages of the reasoning request; and the machine controller can adjust the batch processing size for processing the reasoning request according to the service level target type and the use state of the machine. The machine resources are reasonably allocated and the batch processing size is set according to the service level target type of the reasoning request and the characteristics of different processing stages of the reasoning request, so that the reasoning efficiency of the reasoning service system is avoided from being affected by the resource allocation problem on the premise of meeting the service level target, and the resource utilization rate and the throughput performance of the reasoning service system are improved.

[0075] On the basis of the above-mentioned embodiment, the technical solution is further described and optimized in the embodiment. Specifically, in the embodiment, the global controller is configured to perform the following when pre-dividing the machine resources into the interactive resource pool and the batch processing resource pool:

[0076] The machine resources are pre-divided into the interactive resource pool, the batch processing resource pool and a hybrid resource pool.

[0077] In the embodiment, the global controller is configured to pre-divide the machine resources into the interactive resource pool, the batch processing resource pool and the hybrid resource pool. The hybrid resource pool is established to avoid violating the strict SLO target of the interactive request due to insufficient resources as much as possible during the burst peak of the interactive request. The hybrid resource pool can also be used to process the batch processing request, so as to multiplex the two types of requests and further improve the resource utilization rate. That is, the hybrid resource pool is a flexible resource pool, and the machine resources in the hybrid resource pool can be allocated to the interactive resource pool or the batch processing resource pool according to the actual demand.

[0078] The global controller is configured to perform the following when expanding and contracting the interactive resource pool and the batch processing resource pool according to the service level target type:

[0079] If it is determined according to the service level target type that the reasoning request is an interactive request, the hybrid resource pool is used to expand and contract the interactive resource pool.

[0080] Specifically, if the inference request is determined to be an interactive request according to the service level objective type, when a burst peak of the interactive request occurs, the global controller preferentially allocates resources from the hybrid resource pool to the interactive resource pool to scale the resources of the interactive resource pool, so as to avoid violating the service level objective due to insufficient resources. When the number of interactive requests decreases, the machine resources allocated from the hybrid resource pool are returned to the interactive resource pool to maintain the resource amount of the hybrid resource pool.

[0081] According to the method of the embodiment, by dividing all machine resources into an interactive resource pool, a batch processing resource pool and a hybrid resource pool, and scaling the resources of the interactive resource pool by using the hybrid resource pool according to the service level objective type, the resource amount of the interactive resource pool can be adjusted to more conveniently meet the SLO target of the inference request.

[0082] On the basis of the above-mentioned embodiments, the technical solutions are further described and optimized in the embodiment. Specifically, in the embodiment, the global controller is configured to, when it is determined that the inference request is an interactive request according to the service level objective type, scale the resources of the interactive resource pool by using the hybrid resource pool, as follows:

[0083] determine a first resource amount of the interactive resource pool that is currently processing the interactive request;

[0084] determine a second resource amount of the hybrid resource pool that is currently processing the interactive request;

[0085] determine a resource utilization rate of the interactive resource pool according to a ratio of the first resource amount to a preset resource amount of the interactive resource pool;

[0086] determine a resource proportion of the first resource amount and the second resource amount;

[0087] scale the resources of the interactive resource pool according to the resource utilization rate and the resource proportion.

[0088] The embodiment provides an automatic scaling strategy of a global controller for an interactive request based on an interactive resource pool and a hybrid resource pool, which dynamically changes the resource amount of the interactive resource pool by monitoring the resource proportion of the interactive resource pool and the hybrid resource pool that is currently processing the interactive request.

[0089] Specifically, the global controller real-time counts the number of machines in the interactive resource pool that are currently processing the interactive request to obtain the first resource amount, that is, to determine the resource amount of the dedicated interactive resource pool that is allocated to process the interactive request; and counts the number of machines in the hybrid resource pool that are currently processing the interactive request to obtain the second resource amount, that is, to determine the resource amount of the hybrid resource pool that is originally allocated to process the interactive request.

[0090] Then, the resource ratio is determined according to the ratio of the first resource amount and the second resource amount; the resource utilization of the interactive resource pool is determined according to the ratio of the first resource amount and the preset resource amount of the interactive resource pool; and the interactive resource is scaled according to the resource ratio and the resource utilization.

[0091] Specifically, if the resource utilization is greater than or equal to the preset upper limit of the utilization, part of the resource will be dynamically allocated from the mixed resource pool to the interactive resource pool according to the resource ratio, so as to adjust the resource amount of the interactive resource pool. If the resource utilization is less than or equal to the preset lower limit of the utilization, the corresponding amount of resource will be dynamically allocated from the interactive resource pool to the mixed resource pool according to the resource ratio, so as to maintain the resource amount of the mixed resource pool.

[0092] For example, it is assumed that the inference service system has 100 machines, 60 of which are allocated to the interactive resource pool, 20 of which are allocated to the batch resource pool, and 20 of which are allocated to the mixed resource pool; and it is assumed that at the initial moment, 50 machines in the interactive resource pool are processing interactive requests, i.e., the first resource amount is 50, the resource utilization of the interactive resource pool is 50 / 60=83.3%, and no machine in the mixed resource pool is processing interactive requests, i.e., the second resource amount is 0, and the resource ratio is 50:0.

[0093] If the number of interactive requests gradually increases, the number of machines in the interactive resource pool processing interactive requests increases, and the utilization of the interactive resource increases; when the utilization of the interactive resource pool approaches saturation, for example, 60 machines in the interactive resource pool are processing interactive requests, i.e., the first resource amount is 60, and the resource ratio increases to 60:0; therefore, part of the resource needs to be allocated from the mixed resource pool to the interactive resource pool. For example, 10 machines in the mixed resource pool are temporarily allocated to the interactive resource pool, so that the number of machines in the interactive resource pool increases to 70, and the newly added 10 machines can start processing interactive requests, thereby ensuring that the interactive requests can be supported by sufficient resources; the number of machines in the mixed resource pool decreases to 10, and the resource ratio at this time is 60:10.

[0094] If the number of interactive requests gradually decreases, part of the machine resources in the interactive resource pool becomes idle. Assuming that 40 machines in the interactive resource pool are processing interactive requests, i.e., the first resource amount is 40, the resource utilization of the interactive resource pool is 40 / 60 = 66.7%; 10 machines originally belonging to the mixed resource pool are processing interactive requests, i.e., the second resource amount is 10, and the resource ratio at this time is 40:10; at this time, the 10 machines previously allocated from the mixed resource pool need to be returned to the mixed resource pool, so that the interactive resource pool recovers to 60 machines and the mixed resource recovers to 20 machines; thereby avoiding allocating too many resources to interactive requests, so that the mixed resource can be utilized by the batch processing request again, and the overall resource utilization is improved.

[0095] According to the method of the embodiment, the resource expansion and contraction of the interactive resource pool is performed according to the resource utilization and the resource ratio, and the resource amount of the interactive resource and the mixed resource is adjusted, which can efficiently relieve the pressure of the interactive resource pool, maximize the resource utilization and the throughput performance of the inference service system on the premise of meeting the service level objective.

[0096] In actual application, if all the resources in the mixed resource pool are allocated to the interactive resource pool, and the number of interactive requests is still increasing, at this time, resources can be allocated to the interactive resource pool from the batch processing resource pool. For this, the embodiment further describes and optimizes the technical solution on the basis of the above-mentioned embodiment. Specifically, in the embodiment, the global controller is configured to:

[0097] Case 1: If the resource utilization is greater than or equal to the first preset utilization upper limit value, part of the resources in the mixed resource pool is allocated to the interactive resource pool according to the resource ratio;

[0098] Case 2: If the resource utilization is greater than or equal to the second preset utilization upper limit value, part of the resources in the batch processing resource pool is allocated to the interactive resource pool according to the resource ratio; the first preset utilization upper limit value is less than the second preset utilization upper limit value;

[0099] Case 3: If the resource utilization is less than or equal to the first preset utilization lower limit value, a corresponding amount of resources in the interactive resource pool is allocated to the batch processing resource pool according to the resource ratio;

[0100] Case 4: If the resource utilization is less than or equal to the second preset utilization lower limit value, a corresponding amount of resources in the interactive resource pool is allocated to the mixed resource pool according to the resource ratio; the first preset utilization lower limit value is greater than the second preset utilization lower limit value.

[0101] In this embodiment, when a burst peak of interactive requests occurs, the interactive resource pool is expanded or shrunk by comprehensively using the mixed resource pool and the batch processing resource pool.

[0102] Specifically, if the resource utilization rate is greater than or equal to a first preset utilization rate upper limit value, part of the resources in the mixed resource pool will be dynamically allocated to the interactive resource pool according to the resource ratio; if the resource utilization rate is greater than or equal to a second preset utilization rate upper limit value, part of the resources in the batch processing resource pool will be allocated to the interactive resource pool; wherein the first preset utilization rate upper limit value is less than the second preset utilization rate upper limit value, indicating that part of the resources in the mixed resource pool is preferentially allocated to the interactive resource pool, and when the resources in the mixed resource pool are insufficient, the resources in the batch processing resource pool are allocated to the interactive resource pool.

[0103] If the resource utilization rate is less than or equal to a first preset utilization rate lower limit value, a corresponding amount of resources in the interactive resource pool will be dynamically allocated to the batch processing resource pool according to the resource ratio to maintain the resource amount of the batch processing resource pool; if the resource utilization rate is less than or equal to a second preset utilization rate lower limit value, a corresponding amount of resources in the interactive resource pool will be dynamically allocated to the mixed resource pool according to the resource ratio to maintain the resource amount of the mixed resource pool; wherein the second preset utilization rate lower limit value is greater than the second preset utilization rate lower limit value, indicating that the resources of the batch processing resource pool are preferentially returned, the resource amount of the batch processing resource pool is maintained first, and then the resources of the mixed resource pool are returned to maintain the resource amount of the mixed resource pool.

[0104] According to the method of this embodiment, all machine resources in the inference service system can be comprehensively reasoned for resource expansion and shrinkage, the resource utilization rate of the inference service system is maximized, and the throughput performance of the inference service system is improved.

[0105] In one specific embodiment, the global controller is further configured to:

[0106] determine a third resource amount in the batch processing resource pool that is processing batch processing requests, and determine a remaining resource amount of the batch processing resource pool according to the third resource amount;

[0107] expand or shrink the interactive resource pool based on the remaining resource amount.

[0108] Specifically, if the utilization rate of the interactive resource pool approaches saturation, and all machine resources in the hybrid resource pool have been allocated to the interactive resource pool; the number of interactive requests is still gradually increasing, the global controller determines a third resource amount of the batch processing resource pool that is processing batch processing requests, and determines a remaining resource amount of the batch processing resource pool according to a difference between the third resource amount and a preset resource amount of the batch processing resource pool; resource expansion and contraction is performed on the interactive resource pool according to the remaining resource amount; the resource expansion and contraction includes allocating corresponding machine resources from the batch processing resource pool to the interactive resource pool, thereby relieving the pressure of the interactive resource pool and ensuring that the delay requirement of the interactive request is met; and the machine resources allocated from the batch processing resource pool are returned to the batch processing resource pool to maintain the resource amount of the batch processing resource pool.

[0109] For example, assuming that the inference service system has a total of 100 machines, 60 of which are allocated to the interactive resource pool, 20 of which are allocated to the batch processing resource pool, and 20 of which are allocated to the hybrid resource pool; and assuming that at the current time, all 60 machines in the interactive resource pool are processing interactive requests, and the 20 machines in the hybrid resource pool are also allocated to the interactive resource pool to process interactive requests; at this time, the number of interactive requests is still increasing, so it can be determined that the third resource amount of the batch processing resource pool that is processing batch processing requests is 5 machines; according to the third resource amount, the remaining resource amount of the batch processing resource pool is 15 machines; part or all of the 15 machines can be allocated to the interactive resource pool to process interactive requests, thereby ensuring that the interactive requests can be supported by sufficient resources. When the number of interactive requests decreases and only 40 machines in the interactive resource pool are processing interactive requests, the machine resources allocated to the interactive resource pool can be returned to the batch processing resource pool to maintain the resource amount of the batch processing resource pool.

[0110] It can be seen that according to the method of the embodiment, machine resources in the inference service system can be fully utilized for resource expansion and contraction in the case of a burst peak of interactive requests, thereby avoiding violation of the SLO target and improving the resource utilization rate and throughput performance of the inference service system.

[0111] Figure 3 A flowchart of a global controller performing resource expansion and contraction on an interactive resource pool is provided in the embodiments of the present application. Based on the above embodiments, the technical solutions are further described and optimized in the embodiments. Specifically, in the embodiments, the global controller is configured to perform resource expansion and contraction on the interactive resource pool and the batch processing resource pool according to the service level objective type when performing resource expansion and contraction on the interactive resource pool and the batch processing resource pool according to the service level objective type.

[0112] S310: If it is determined according to the service level objective type that the inference request is a batch request, grouping is performed according to the information type of the batch request, and the batch request is placed in a batch request queue for waiting.

[0113] It can be understood that the SLO number level unit of the batch request is often a minute or even an hour, and it is not necessary to allocate resources every time a new request arrives as for the interactive request. Therefore, in this embodiment, if it is determined according to the service level objective type that the inference request is a batch request, the batch request is grouped according to the SLO target of the batch request, and is placed in the corresponding batch request queue for queuing and waiting, and then a corresponding resource expansion and contraction strategy is formulated according to the global queue state.

[0114] Specifically, the global controller groups the batch requests according to the information types of the document types, document sizes and priorities of the batch requests. The document types include PDF, Word, Excel, etc.; the document sizes are different size intervals such as 0-10MB, 10MB-100MB and greater than 100MB; and the priority represents the processing order or importance of the batch request.

[0115] After grouping the batch requests, the batch requests are placed in the corresponding batch request queue for waiting.

[0116] S320: Resource expansion and contraction is performed on the batch resource pool according to the global queue state of the batch request queue.

[0117] Specifically, the global queue state of each batch request queue is monitored, and the resource amount of the batch resource is adjusted according to the global queue state of the batch request queue.

[0118] In a specific example, when the queue length of the batch request queue is greater than the upper limit value of the batch request queue length, part or all of the machine resources in the hybrid resource pool can be allocated to the batch resource pool for processing the batch request; when the queue length of the batch request queue is less than the lower limit value of the batch request queue length, part or all of the machine resources allocated to the batch resource pool are returned to the hybrid resource pool to maintain the resource amount of the batch resource pool.

[0119] According to the method of the embodiment, the inference requests are grouped according to the information type of the inference request, and are placed in the batch request queue for queuing and waiting, and the batch resource pool is resource expanded and contracted according to the global queue state of the batch request queue, so that the SLO target is not violated, and the resource utilization rate of the inference service system is improved.

[0120] In one specific embodiment, the global queue state of the batch request queue includes the number of batch requests and the estimated queue waiting time.

[0121] The number of batch requests refers to the number of batch requests, and the estimated queue waiting time refers to the estimated time for processing each batch request in the batch request queue.

[0122] In one specific example, the number of batch requests in the batch request queue is obtained, and it is determined whether the number of batch requests exceeds a preset upper limit value. If the number of batch requests in the batch request queue exceeds the preset upper limit value, it indicates that the current batch resource pool will not be able to meet the SLO target during processing, and therefore the batch resource pool is scaled in terms of resources, specifically, some or all machine resources are allocated from the hybrid resource pool to the batch resource pool for processing batch requests.

[0123] In one specific example, the estimated processing time corresponding to each batch request in the batch request queue is determined, and the sum of the estimated processing times is used to determine the estimated queue waiting time. It is determined whether the estimated queue waiting time exceeds a preset upper limit value of the queue waiting time. If the estimated queue waiting time exceeds the preset upper limit value of the queue waiting time, it indicates that the current batch resource pool will not be able to meet the SLO target during processing, and therefore the batch resource pool is scaled in terms of resources, specifically, some or all machine resources are allocated from the hybrid resource pool to the batch resource pool for processing batch requests.

[0124] Conversely, if the number of batch requests in the batch request queue is less than a preset lower limit value, or the estimated queue waiting time of the batch request queue is less than a preset lower limit value of the queue waiting time, some or all machine resources allocated to the batch resource pool can be returned to the hybrid resource pool to release excess machine resources.

[0125] For example, if there are 100 batch requests in the first batch request queue, the estimated queue waiting time is determined to be 1.5 hours according to the estimated processing time corresponding to each batch request in the batch request queue, that is, it takes 1.5 hours to process all batch requests in the batch request queue according to the current resource amount of the batch resource pool, but this will exceed the 1-hour SLO target. Therefore, a corresponding number of machines are allocated from the hybrid resource pool to the batch resource pool for processing batch requests to improve processing speed and avoid violating the SLO target.

[0126] According to the method of the present embodiment, resource utilization and cost efficiency can be maximized while ensuring that the SLO of batch requests is met.

[0127] On the basis of the above-mentioned embodiments, the technical solutions are further described and optimized in the embodiments. Specifically, in the embodiments, the cluster controller is configured to:

[0128] previously divide the interactive resource pool and the batch processing resource pool into a prompt word resource pool, a token resource pool, and a stage mixed resource pool;

[0129] It should be noted that when processing the inference request, two stages are included, namely, a prompt word stage and a token stage. The prompt word stage is computationally intensive and requires sufficient resource computing power, while the token stage is more susceptible to resource storage capacity and bandwidth.

[0130] In the embodiments, the cluster controller subdivides the interactive resource pool and the batch processing resource pool into three subsets: a prompt word resource pool, a token resource pool, and a stage mixed resource pool.

[0131] The stage mixed resource pool is used to dynamically change the number of machine resources in the prompt word resource pool and the token resource pool to reduce resource fragmentation and ensure that the SLO target is not violated in the case of a sudden high workload.

[0132] Figure 4 A flowchart of a cluster controller performing resource expansion and contraction on a prompt word resource pool and a token resource pool is provided in the embodiments. The cluster controller performs the following steps:

[0133] S410: Determine the stage queue state corresponding to the prompt word resource pool and the token resource pool.

[0134] Specifically, after receiving an inference request, the cluster controller allocates a pair of resources, namely, a prompt word resource and a token resource, for the prompt word stage and the token stage of each inference request, respectively, to be responsible for the prompt word stage and the token stage. For the prompt word resource pool and the token resource pool, the corresponding requests are added to the respective stage queues, i.e., the stage queues are updated. The cluster controller obtains the stage queue state of the stage queue corresponding to the prompt word resource pool and the token resource pool.

[0135] The stage queue state includes the number of stage requests and the estimated stage waiting time. The number of stage requests is the number of inference requests in the stage queue corresponding to the prompt word resource pool or the token resource pool, and the estimated stage waiting time is the estimated time required by the prompt word resource pool or the token resource pool to process the inference requests in the stage queue corresponding to itself.

[0136] S420: according to the stage queue state, resource expansion and contraction are performed on the prompt word resource pool and the token resource pool by using the stage hybrid resource pool.

[0137] If the number of stage requests is greater than the preset upper limit value of the number of stage requests, or the estimated stage waiting time is greater than the preset upper limit value of the stage waiting time, the corresponding machine resources of the stage hybrid resource pool need to be allocated to the corresponding prompt word resource pool and token resource pool; if the number of stage requests is less than the preset lower limit value of the number of stage requests, or the estimated stage waiting time is less than the preset lower limit value of the stage waiting time, the corresponding machine resources of the stage hybrid resource pool need to be returned to the stage hybrid resource pool.

[0138] Suppose that in the initial stage, the prompt word resource pool has 50 machines, the token resource pool has 50 machines, and the hybrid resource pool has 20 machines. The cluster controller allocates prompt word resources for the prompt word stage of each inference request and token resources for the token stage. When the number of interactive requests increases, the estimated stage waiting time of the stage queue corresponding to the prompt word resource pool becomes longer; when the estimated stage waiting time is greater than the preset upper limit value of the stage waiting time, the cluster controller allocates 10 machines from the stage hybrid resource pool to the prompt word resource pool, so that the number of machines of the prompt word resource pool increases to 60, and the number of machines of the stage hybrid resource pool decreases to 10; thereby relieving the pressure of the prompt word resource pool, ensuring that the prompt word stage of the interactive request can be processed in time, and avoiding violation of SLO.

[0139] With the decrease in the number of interactive requests, the estimated stage waiting time of the stage queue corresponding to the prompt word resource pool becomes shorter, and the pressure of the prompt word resource pool gradually decreases. When the estimated stage waiting time is less than the preset lower limit value of the stage waiting time, the cluster controller returns the 10 machines previously allocated from the stage hybrid resource pool to the stage hybrid resource pool, so that the number of machines of the prompt word resource pool returns to 50, and the number of machines of the stage hybrid resource pool returns to 20.

[0140] In this embodiment, the cluster controller allocates prompt word resources and token resources from the prompt word resource pool and the token resource pool according to the different stage characteristics of the inference request, and then performs resource expansion and contraction on the prompt word resource pool and the token resource pool by using the stage hybrid resource pool according to the stage queue state; resource scheduling and automatic expansion and contraction are performed for the two stages of a single inference request.

[0141] In a specific embodiment, the cluster controller is configured to perform the following when performing the pre-division of the interactive resource pool and the batch resource pool into the prompt word resource pool, the token resource pool, and the stage hybrid resource pool:

[0142] According to the estimated demand load and the input / output token distribution, the interactive resource pool and the batch processing resource pool are respectively divided into a prompt word resource pool, a token resource pool and a stage mixed resource pool.

[0143] In this embodiment, the cluster controller divides the interactive resource pool and the batch processing resource pool into three subsets: a prompt word resource pool, a token resource pool and a stage mixed resource pool according to the estimated demand load and the input / output token distribution.

[0144] The estimated demand load refers to the total number of inference requests and resource requirements that the inference service system may receive in the future according to historical data, business scenarios and user behavior, etc. It should be noted that the larger the estimated demand load, the larger the capacity of the corresponding resource pool; otherwise, the smaller the estimated demand load, the smaller the capacity of the corresponding resource pool.

[0145] The input / output token distribution refers to the distribution characteristics of the input prompt word sequence length and the output token sequence length of the inference request. It can be understood that the prompt word sequence length affects the computational resource requirement of the prompt word stage, and the token sequence length affects the storage and bandwidth resource requirement of the token stage. The cluster controller can more accurately divide the size of the prompt word resource pool and the token resource pool according to the input / output token distribution to meet the resource requirements of different stages. For example, if the input token length is generally long, the size of the prompt word resource pool can be appropriately increased; if the output token length distribution is wide, the storage and bandwidth resources of the token resource pool can be appropriately increased.

[0146] In this embodiment, the cluster controller is configured to divide the prompt word resource pool, the token resource pool and the stage mixed resource pool according to the estimated demand load and the input / output token distribution, which can improve the rationality of the initial division of the prompt word resource pool, the token resource pool and the second mixed resource capacity.

[0147] On the basis of the above-mentioned embodiments, the technical solutions are further described and optimized in this embodiment. Specifically, in this embodiment, the machine controller is configured to adjust the batch size of processing the inference request according to the service level objective type and the usage state of the corresponding machine when executing the adjustment.

[0148] monitoring the usage state of the corresponding machine in real time;

[0149] adjusting the batch size of processing the inference request according to the usage state, the machine resource type of the corresponding machine and the service level objective type.

[0150] The usage status of the machine to which the machine controller belongs includes storage status and processing queue status. Storage status includes CPU utilization, memory utilization, and GPU memory utilization. Processing queue status refers to the status of the processing queue corresponding to the machine; it includes the processing queue length and the number of tasks in the processing queue.

[0151] The machine resource type refers to the type of hardware resources configured, such as CPU, GPU, NPU, TPU, and FPGA.

[0152] Batch size refers to the number of inference requests or the amount of data that a machine can process simultaneously using its own machine resources. For example... Figure 2 As shown, R1, R2, and R3 represent three different batch sizes determined by the machine controller.

[0153] In this embodiment, the machine controller monitors the machine's usage status in real time and determines the batch size based on the service level target type of the inference request, the usage status of the machine, and the machine resource type of the machine.

[0154] For example, for interactive requests with a SLO target of low latency, the batch size will be set to a smaller size to reduce processing time and avoid excessive latency; while for batch requests with an SLO target that allows for a certain amount of latency, the batch size can be set to a larger size to improve throughput performance and make full use of resources.

[0155] If the memory utilization rate is greater than the preset upper limit, it indicates that storage resources are tight, and the batch size needs to be reduced to avoid resource exhaustion and task backlog. If the memory utilization rate is less than the preset lower limit, it indicates that storage resources are sufficient, and the batch size can be increased to improve resource utilization.

[0156] In addition, different machine resource types have different performance characteristics and applicable scenarios when processing inference requests, so it is necessary to set the batch size reasonably according to the different characteristics of machine resource types.

[0157] When adjusting the batch size, the machine's maximum batch size should also be considered; that is, when the machine processes inference requests, its batch size cannot exceed its own maximum batch size.

[0158] The method described in this embodiment can optimize resource utilization efficiency and improve the throughput performance of the inference service system while meeting the SLO objective.

[0159] It should be noted that the information collection process (such as the face image collection process, the fingerprint information collection process, etc.) and the feature extraction process involved in the present application are executed with the user's knowledge and permission, that is, the information collection process / feature extraction process meets the legal requirements and does not belong to the act of obstructing public interests.

[0160] It should be noted that the information interaction, execution process, etc. between the above devices / units are based on the same concept as the method embodiments of the present application, and the specific functions and technical effects brought about can be referred to the method embodiment part for specific functions and technical effects brought about, which will not be described here.

[0161] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above-mentioned division of each functional unit and module is exemplified, and in actual application, the above-mentioned functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be realized in the form of hardware or software function unit. In addition, the specific name of each functional unit and module is only for easy distinction, and does not limit the protection scope of the present application. The specific working process of the unit and module in the above system can be referred to the corresponding process in the foregoing method embodiments, which will not be described here.

[0162] In the above embodiments, the description of each embodiment has its own emphasis, and the parts not described or recorded in a certain embodiment can be referred to the related description of other embodiments.

[0163] Those of ordinary skill in the art can realize that the devices and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0164] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, and the coupling or direct coupling or communication connection between the displayed or discussed each other can be through some interface, and the indirect coupling or communication connection between devices can be electrical, mechanical or other forms.

[0165] The above-described embodiments are only used to illustrate the technical solutions of the present application, but not limit them; although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. An inference service system, characterized by, The inference service system comprises a global controller, a cluster controller and a plurality of machine controllers; the global controller is in communication connection with the cluster controller, and the cluster controller is in communication connection with each of the machine controllers; The global controller is configured to pre-divide machine resources into an interactive resource pool, a batch processing resource pool and a hybrid resource pool; when an inference request is received, a service level objective type of the inference request is determined, and resource expansion and contraction of the interactive resource pool and the batch processing resource pool are performed according to the service level objective type; the inference request is divided into an interactive request and a batch processing request based on different service level objectives; the interactive request is an inference request requiring immediate response, and the batch processing request is an inference request allowing delay; the interactive resource pool is a resource set specially allocated to the interactive request, and the batch processing resource pool is a resource set specially allocated to the batch processing request; The global controller is configured to, when performing the resource expansion and contraction of the interactive resource pool and the batch processing resource pool according to the service level objective type: if it is determined according to the service level objective type that the inference request is the interactive request, determining a first resource amount of the interactive resource pool that is processing the interactive request; determining a second resource amount of the hybrid resource pool that is processing the interactive request; determining a resource utilization rate of the interactive resource pool according to a ratio of the first resource amount to a preset resource amount of the interactive resource pool; determining a resource proportion of the first resource amount and the second resource amount; and performing resource expansion and contraction of the interactive resource according to the resource utilization rate and the resource proportion; The cluster controller is configured to pre-divide the interactive resource pool and the batch processing resource pool into a prompt word resource pool and a token resource pool respectively; for a prompt word stage and a token stage of the inference request, corresponding prompt word resources and token resources are allocated from the prompt word resource pool and the token resource pool respectively, and resource expansion and contraction of the prompt word resource pool and the token resource pool are performed; The machine controller is configured to adjust a batch processing size for processing the inference request according to the service level objective type and a usage state of the machine.

2. The inference service system of claim 1, wherein, The global controller is configured to, when performing the resource expansion and contraction of the interactive resource according to the resource utilization rate and the resource proportion: if the resource utilization rate is greater than or equal to a first preset utilization rate upper limit value, then according to the resource proportion, part of resources in the hybrid resource pool are allocated to the interactive resource pool; if the resource utilization rate is greater than or equal to a second preset utilization rate upper limit value, then according to the resource proportion, part of resources in the batch processing resource pool are allocated to the interactive resource pool; The first preset utilization rate upper limit value is less than the second preset utilization rate upper limit value; if the resource utilization rate is less than or equal to a first preset utilization rate lower limit value, then according to the resource proportion, a corresponding number of resources in the interactive resource pool are allocated to the batch processing resource pool; If the resource utilization is less than or equal to a second preset utilization lower limit value, a corresponding number of resources in the interactive resource pool is allocated to the hybrid resource pool according to the resource ratio; the first preset utilization lower limit value is greater than the second preset utilization lower limit value.

3. The inference service system of claim 2, wherein, The global controller is further configured to: determine a third resource amount in the batch processing resource pool that is processing the batch processing request, and determine a remaining resource amount of the batch processing resource pool according to the third resource amount; based on the remaining resource amount, perform resource scaling on the interactive resource pool.

4. The inference service system of claim 1, wherein, When performing the resource scaling on the interactive resource pool and the batch processing resource pool according to the service level objective type, the global controller is configured to: if the inference request is determined to be the batch processing request according to the service level objective type, group the batch processing request according to the information type of the batch processing request, and put the batch processing request into a batch processing request queue to wait; perform resource scaling on the batch processing resource pool according to a global queue state of the batch processing request queue.

5. The inference service system of claim 4, wherein, The global queue state of the batch processing request queue includes a batch processing request quantity and an estimated queue waiting time.

6. The inference service system of any one of claims 1 to 5, wherein, When performing the pre-division of the interactive resource pool and the batch processing resource pool into a prompt word resource pool and a token resource pool, the cluster controller is configured to: pre-divide the interactive resource pool and the batch processing resource pool into a prompt word resource pool, a token resource pool, and a stage hybrid resource pool; When performing the resource scaling on the prompt word resource pool and the token resource pool, the cluster controller is configured to: determine the corresponding stage queue state of the prompt word resource pool and the token resource pool; according to the stage queue state, perform resource scaling on the prompt word resource pool and the token resource pool using the stage hybrid resource pool.

7. The inference service system of claim 6, wherein, When performing the pre-division of the interactive resource pool and the batch processing resource pool into a prompt word resource pool, a token resource pool, and a stage hybrid resource pool, the cluster controller is configured to: pre-divide the interactive resource pool and the batch processing resource pool into a prompt word resource pool, a token resource pool, and a stage hybrid resource pool according to an estimated demand load and an input / output token distribution.

8. The inference service system of any one of claims 1 to 5, wherein, When performing the adjustment of the batch processing size of the inference request according to the service level objective type and the usage state of the corresponding machine, the machine controller is configured to: monitor the usage state of the corresponding machine in real time; adjust the batch processing size of the inference request according to the usage state, the machine resource type of the corresponding machine, and the service level objective type.

Images