Inference service system

Abstract

The invention provides a reasoning service system. A global controller is used for dividing machine resources into an interactive resource pool and a batch processing type resource pool in advance; when a reasoning request is received, determining a service level target type of the reasoning request, and performing resource expansion and contraction on the interactive resource pool and the batch processing type resource pool according to the service level target type; the cluster controller is used for respectively dividing the interactive resource pool and the batch processing type resource pool into a cue word resource pool and a token resource pool in advance; aiming at a cue word stage and a token stage of the reasoning request, respectively distributing corresponding cue word resources and token resources from a cue word resource pool and a token resource pool, and carrying out resource expansion and contraction on the cue word resource pool and the token resource pool; and the machine controller is used for adjusting the batch processing size for processing the reasoning request according to the service level target type and the use state of the affiliated machine. And on the premise of meeting the service level target, the resource utilization rate and throughput performance of the inference service system are improved.

Description

Technical Field

[0001] The present application relates to the field of reasoning service technology, and in particular to a reasoning service system. Background Art

[0002] With the rapid development of artificial intelligence (AI), reasoning services based on large language models (LLMs) have emerged. These powerful natural language understanding and generation capabilities have been widely used in numerous fields, such as intelligent customer service and document processing. LLM reasoning services are typically deployed on machine resources. Many LLM reasoning scenarios often experience significant fluctuations in the number of user reasoning requests and high latency requirements.

[0003] Traditionally, cloud service providers have addressed these issues by using autoscaling to handle inference requests. These solutions dynamically adjust machine resources by monitoring and predicting key metrics such as resource utilization, request throughput, and inference latency. However, current solutions can still cause service level objectives (SLOs) to be violated (failed to meet them) due to sudden spikes in inference requests, impacting inference efficiency and machine resource utilization.

[0004] Therefore, how to improve the reasoning efficiency, resource utilization and throughput performance of the reasoning service system is a technical problem that those skilled in the art currently need to solve. Summary of the Invention

[0005] The purpose of this application is to provide an inference service system that can avoid the influence of resource allocation problems on the inference efficiency of the inference service system while meeting the service level goals, and improve the resource utilization and throughput performance of the inference service system.

[0006] In a first aspect, the present application provides an inference service system, the inference service system comprising a global controller, a cluster controller, and a plurality of machine controllers; the global controller is communicatively connected to the cluster controller, and the cluster controller is communicatively connected to each of the machine controllers; The global controller is used to pre-divide machine resources into an interactive resource pool and a batch resource pool; when receiving an inference request, determine the service level objective type of the inference request, and scale resources of the interactive resource pool and the batch resource pool according to the service level objective type; The cluster controller is configured to pre-divide the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool, respectively; allocate corresponding prompt word resources and token resources from the prompt word resource pool and the token resource pool, respectively, for the prompt word phase and the token phase of the inference request, and perform resource expansion and contraction on the prompt word resource pool and the token resource pool; The machine controller is configured to adjust a batch size for processing the inference request according to the service level objective type and a usage status of the machine.

[0007] In one embodiment, when performing the pre-division of machine resources into interactive resource pools and batch resource pools, the global controller is configured to: Dividing the machine resources into the interactive resource pool, the batch resource pool and the hybrid resource pool in advance; When performing resource scaling of the interactive resource pool and the batch resource pool according to the service level target type, the global controller is configured to: If the inference request is determined to be an interactive request according to the service level objective type, the hybrid resource pool is used to scale resources of the interactive resource pool.

[0008] In one embodiment, when the global controller determines that the inference request is an interactive request according to the service level objective type, scaling resources of the interactive resource pool by using the hybrid resource pool is performed, the global controller is configured to: determining a first amount of resources in the interactive resource pool that are processing the interactive request; determining a second amount of resources in the hybrid resource pool that are processing the interactive request; determining a resource utilization rate of the interactive resource pool according to a ratio of the first resource amount to a preset resource amount of the interactive resource pool; determining a resource ratio between the first resource amount and the second resource amount; The interactive resource is scaled up or down according to the resource utilization rate and the resource ratio.

[0009] In one embodiment, when performing resource scaling of the interactive resource according to the resource utilization and the resource ratio, the global controller is configured to: If the resource utilization rate is greater than or equal to a first preset utilization rate upper limit, allocating part of the resources from the hybrid resource pool to the interactive resource pool according to the resource ratio; If the resource utilization rate is greater than or equal to a second preset utilization rate upper limit, allocating some resources from the batch resource pool to the interactive resource pool according to the resource ratio; the first preset utilization rate upper limit is less than the second preset utilization rate upper limit; If the resource utilization rate is less than or equal to a first preset utilization rate lower limit, allocating a corresponding amount of resources from the interactive resource pool to the batch resource pool according to the resource ratio; If the resource utilization is less than or equal to a second preset utilization lower limit, a corresponding number of resources are allocated from the interactive resource pool to the hybrid resource pool according to the resource ratio; the first preset utilization lower limit is greater than the second preset utilization lower limit.

[0010] In one embodiment, the global controller is further configured to: Determining a third amount of resources in the batch resource pool that are currently processing batch requests, and determining a remaining amount of resources in the batch resource pool based on the third amount of resources; The interactive resource pool is scaled up or down based on the remaining resources.

[0011] In one embodiment, when performing resource scaling of the interactive resource pool and the batch resource pool according to the service level objective type, the global controller is configured to: If it is determined that the inference request is a batch request according to the service level objective type, grouping the batch requests according to their information types and placing the batch requests in a batch request queue for waiting; The batch resource pool is scaled up or down according to the global queue state of the batch request queue.

[0012] In one embodiment, the global queue status of the batch request queue includes the number of batch requests and the estimated queue waiting time.

[0013] In one embodiment, when performing the step of pre-dividing the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool, the cluster controller is configured to: The interactive resource pool and the batch resource pool are respectively divided into a prompt word resource pool, a token resource pool and a stage mixed resource pool in advance; When executing the resource expansion and contraction of the prompt word resource pool and the token resource pool, the cluster controller is configured to: Determine the stage queue states corresponding to the prompt word resource pool and the token resource pool respectively; According to the stage queue status, the stage hybrid resource pool is used to scale the prompt word resource pool and the token resource pool.

[0014] In one embodiment, when performing the step of pre-dividing the interactive resource pool and the batch resource pool into a prompt word resource pool, a token resource pool, and a stage-mixed resource pool, the cluster controller is configured to: The interactive resource pool and the batch resource pool are respectively divided into a prompt word resource pool, a token resource pool and a stage mixed resource pool according to the estimated demand load and the input / output token distribution.

[0015] In one embodiment, when adjusting the batch size of the inference request according to the service level objective type and the usage status of the machine, the machine controller is configured to: Real-time monitoring of the usage status of the machine; A batch size for processing the inference request is adjusted according to the usage status, the machine resource type of the belonging machine, and the service level target type.

[0016] The present application provides an inference service system. In a multi-level automatic scaling framework, a global controller scales up and down the interactive resource pool and the batch resource pool according to the service level target type of the inference request, ensuring that the inference request can obtain resource support that meets its service level target type. The cluster controller allocates corresponding prompt word resources and token resources from the prompt word resource pool and the token resource pool respectively for the prompt word stage and the token stage of the inference request, and scales up and down the prompt word resource pool and the token resource pool to meet the resource requirements of the different processing stages of the inference request. The machine controller adjusts the batch size for processing the inference request according to the service level target type and the usage status of the machine to which it belongs. Based on the service level target type of the inference request and the characteristics of the different processing stages of the inference request, the machine resources are reasonably allocated and the batch size is set. On the premise of meeting the service level target, the inference efficiency of the inference service system is avoided from being affected by resource allocation problems, thereby improving the resource utilization and throughput performance of the inference service system. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the specific implementation methods of the present application or the technical solutions in the prior art, the following is a brief introduction to the drawings required for use in the specific implementation methods or the description of the prior art. Obviously, the drawings described below are some implementation methods of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1A system architecture diagram of an inference service system provided in an embodiment of the present application; Figure 2 A schematic diagram of the reasoning process of a reasoning service system provided in an embodiment of the present application; Figure 3 A flowchart of a global controller providing an embodiment of the present application for scaling resources in an interactive resource pool; Figure 4 A flow chart of a cluster controller performing resource expansion and contraction on a prompt word resource pool and a token resource pool provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In the following description, specific details such as specific system structures and techniques are provided for purposes of illustration rather than limitation to facilitate a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to avoid obscuring the description of the present application with unnecessary detail.

[0020] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or collections thereof.

[0021] It will also be understood that the term "and / or" used in this specification and the appended claims refers to and includes any and all possible combinations of one or more of the associated listed items.

[0022] As used in this specification and the appended claims, the term "if" can be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0023] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0024] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of the present application include a particular feature, structure, or characteristic described in conjunction with that embodiment. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof all mean "including but not limited to," unless otherwise specifically emphasized. "Multiple" means "two or more."

[0025] Figure 1 This is a system architecture diagram of an inference service system provided in an embodiment of the present application. For ease of illustration, only the portions relevant to this embodiment are shown. The inference service system 100 provided in this embodiment includes a global controller 110, a cluster controller 120, and multiple machine controllers 130. The global controller 110 is in communication with the cluster controller 120, which is in communication with each machine controller 130.

[0026] Specifically, the inference service system 100 is in communication with the user side 200 , and the user side 200 sends an inference request to the inference service system 100 to request the inference service system 100 to process the given inference request and generate a corresponding output (processing result).

[0027] The reasoning service system 100 in this embodiment includes multiple machines to form a machine cluster, and each machine is provided with a machine controller 130; that is, the machine controller 130 runs on each machine to monitor and manage the machine resources of the machine.

[0028] Machine resources refer to the resources within a machine used to run inference tasks. A machine cluster can be a single machine resource type, or it can include multiple machine resource types, such as CPUs (Central Processing Units), GPUs (Graphics Processing Units), NPUs (Neural Network Processors), TPUs (Tensor Processing Units), and FPGAs (Field-Programmable Gate Arrays).

[0029] Figure 2 A schematic diagram of the reasoning process of a reasoning service system provided in an embodiment of the present application. Figure 2As shown, in this embodiment, the global controller 110 is the controller responsible for overall resource management and allocation, and is used to uniformly schedule and control all machine resources in the inference service system 100; the cluster controller 120 is the controller responsible for managing the resource pool, and allocates corresponding prompt word resources and token resources for different stages of the inference request.

[0030] The global controller 110 is used to pre-divide machine resources into interactive resource pools and batch resource pools; when receiving an inference request, it determines the service level target type of the inference request and scales the interactive resource pool and the batch resource pool according to the service level target type.

[0031] It should be noted that the performance evaluation metrics of the inference service system 100 mainly include end-to-end (E2E) latency, throughput, time to first token (TTFT), and time between tokens (TBT). These evaluation metrics can be combined into different service level objectives (SLOs). Inference requests for the LLM service can be divided into interactive requests and batch requests based on different SLOs.

[0032] Interactive requests refer to reasoning requests that require immediate responses in user-oriented applications such as chatbots. The SLO for interactive requests is often measured in seconds, making TTFT and TBT more important evaluation metrics. The interactive resource pool refers to the collection of resources specifically allocated to interactive requests.

[0033] Batch requests refer to inference requests in backend applications such as document processing that can tolerate long delays. The SLO targets for batch requests are typically measured in minutes or even hours, focusing on request throughput. A batch resource pool is a collection of resources specifically allocated to batch requests.

[0034] Specifically, the global controller 110 divides the available machine resources in the machine cluster corresponding to the inference service system 100 into an interactive resource pool and a batch resource pool in advance, so as to process different types of inference requests respectively.

[0035] When the user side 200 sends an inference request to the inference service system 100, the global controller 110 in the inference service system 100 determines the service level target type corresponding to the inference request after receiving the inference request, that is, determines whether the inference request is an interactive inference request or a batch inference request, and then scales the interactive resource pool and the batch resource pool according to the service level target type, that is, dynamically adjusts the amount of resources allocated in the interactive resource pool and the batch resource pool.

[0036] The cluster controller 120 is used to pre-divide the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool respectively; for the prompt word stage and the token stage of the inference request, the corresponding prompt word resources and token resources are allocated from the prompt word resource pool and the token resource pool respectively, and the resources of the prompt word resource pool and the token resource pool are scaled up or down.

[0037] Today's LLMs are generally based on the Transformer architecture, which uses an attention mechanism and multi-layer perceptrons to understand and analyze input content, thereby generating and outputting corresponding content. However, due to the autoregressive model used by LLM, it can only generate one new token at a time, and the generation of each new token depends on all tokens in the previous token sequence, especially their key-value cache (KV cache). The entire LLM reasoning phase can be generally divided into the prompt phase and the token phase. The former processes the input prompt sequence to generate the first output token and retains the key-value cache, while the latter generates new tokens one by one based on the newly generated token and the previously retained key-value cache.

[0038] The cluster controller 120 divides the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool respectively based on the prompt word stage and the token stage of the inference request in advance.

[0039] That is to say, the prompt word resource pool is a resource subset divided from the interactive resource pool and the batch resource pool, which is specifically used to process the prompt word stage of the reasoning task; the token resource pool is a resource subset divided from the interactive resource pool and the batch resource pool, which is specifically used to process the token stage of the reasoning task.

[0040] When an inference request arrives, the cluster controller 120 allocates corresponding prompt word resources and token resources from the prompt word resource pool and token resource pool, respectively, based on the service level objective type of the inference request. Furthermore, based on the resource utilization of the prompt word resource pool and token resource pool, the cluster controller 120 dynamically adjusts the amount of resources allocated to the prompt word resource pool and token resource pool.

[0041] The machine controller 130 is configured to adjust the batch size of inference requests according to the service level objective type and the usage status of the associated machine.

[0042] The usage status of the machine includes the current resource utilization of the machine, including CPU utilization, memory utilization, and GPU memory utilization, etc. This embodiment does not limit the specific content of the usage status.

[0043] The machine controller 130 monitors the usage status of its machines and adjusts the batch size of inference requests according to the service level objective type and the usage status of its machines.

[0044] According to the method of this embodiment, for the user side, after sending an inference request to the inference service system, the inference service system uses a multi-level automatic scaling framework to process the inference request and return a corresponding processing result.

[0045] An embodiment of the present application provides an inference service system. In a multi-level automatic scaling framework, a global controller scales resources for an interactive resource pool and a batch resource pool according to the service level target type of an inference request, ensuring that the inference request can obtain resource support that meets its service level target type. A cluster controller allocates corresponding prompt word resources and token resources from the prompt word resource pool and the token resource pool, respectively, for the prompt word stage and the token stage of the inference request, and scales resources for the prompt word resource pool and the token resource pool to meet the resource requirements of different processing stages of the inference request. A machine controller adjusts the batch size for processing inference requests according to the service level target type and the usage status of the machine to which it belongs. Based on the service level target type of the inference request and the characteristics of the different processing stages of the inference request, the machine resources are reasonably allocated and the batch size is set. On the premise of meeting the service level target, the inference efficiency of the inference service system is avoided from being affected by resource allocation problems, thereby improving the resource utilization and throughput performance of the inference service system.

[0046] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically, in this embodiment, when pre-dividing machine resources into interactive resource pools and batch resource pools, the global controller is configured as follows: Machine resources are pre-divided into interactive resource pools, batch resource pools, and hybrid resource pools.

[0047] In this embodiment, the global controller is used to pre-divide machine resources into interactive resource pools, batch resource pools, and hybrid resource pools. The hybrid resource pool is designed to minimize resource shortages and violations of strict SLOs during sudden peaks in interactive requests. The hybrid resource pool can also be used to process batch requests, thereby multiplexing the two types of requests and further improving resource utilization. In other words, the hybrid resource pool is a flexible resource pool that can allocate machine resources from the hybrid resource pool to either the interactive resource pool or the batch resource pool based on actual needs.

[0048] The global controller is configured to perform resource scaling for interactive and batch resource pools based on service level objective types: If the inference request is determined to be an interactive request based on the service level objective type, the hybrid resource pool is used to scale resources of the interactive resource pool.

[0049] Specifically, if an inference request is determined to be an interactive request based on the service level objective (SLO) type, and a sudden spike in interactive requests occurs, the global controller prioritizes allocating resources from the hybrid resource pool to the interactive resource pool, scaling the interactive resource pool to avoid violating the SLO due to insufficient resources. As the number of interactive requests decreases, machine resources allocated from the hybrid resource pool are returned to the interactive resource pool to maintain the resource capacity of the hybrid resource pool.

[0050] According to the method of this embodiment, by dividing all machine resources into interactive resource pools, batch resource pools and hybrid resource pools, and using hybrid resource pools to scale the interactive resource pool according to the service level target type, and adjusting the resource amount of the interactive resource pool, the SLO target of the inference request can be met more conveniently.

[0051] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically, in this embodiment, if the global controller determines that the inference request is an interactive request based on the service level objective type, the hybrid resource pool is used to scale the interactive resource pool as follows: determining a first amount of resources in an interactive resource pool that are processing an interactive request; determining a second resource amount in the hybrid resource pool that is processing the interactive request; determining a resource utilization rate of the interactive resource pool according to a ratio of the first resource amount to a preset resource amount of the interactive resource pool; determining a resource ratio between the first resource amount and the second resource amount; Scale interactive resources based on resource utilization and resource ratio.

[0052] This embodiment provides a global controller that implements an automatic scaling strategy for interactive requests based on an interactive resource pool and a hybrid resource pool. The global controller dynamically changes the resource amount of the interactive resource pool by monitoring the ratio of resources currently running interactive requests in the interactive resource pool and the hybrid resource pool.

[0053] Specifically, the global controller counts the number of machines in the interactive resource pool that are currently processing interactive requests in real time to obtain a first resource amount, that is, to determine the amount of resources of the dedicated interactive resource pool allocated to process interactive requests; and counts the number of machines in the hybrid resource pool that are currently processing interactive requests to obtain a second resource amount, that is, to determine the amount of resources originally belonging to the hybrid resource pool that are allocated to process interactive requests.

[0054] Then, the resource ratio is determined according to the ratio of the first resource amount to the second resource amount; the resource utilization rate of the interactive resource pool is determined according to the ratio of the first resource amount to the preset resource amount of the interactive resource pool; and the interactive resources are scaled according to the resource ratio and the resource utilization rate.

[0055] Specifically, if the resource utilization rate is greater than or equal to the preset upper utilization limit, some resources will be dynamically allocated from the hybrid resource pool to the interactive resource pool based on the resource ratio to adjust the resource quantity of the interactive resource pool. If the resource utilization rate is less than or equal to the preset lower utilization limit, the corresponding number of resources will be dynamically allocated from the interactive resource pool to the hybrid resource pool based on the resource ratio to maintain the resource quantity of the hybrid resource pool.

[0056] For example, suppose the inference service system has a total of 100 machines, 60 of which are allocated to the interactive resource pool, 20 to the batch resource pool, and 20 to the hybrid resource pool. Also, suppose that at the initial moment, 50 machines in the interactive resource pool are processing interactive requests, i.e., the first resource amount is 50, and the resource utilization rate of the interactive resource pool is 50 / 60 = 83.3%. However, no machine in the hybrid resource pool is processing interactive requests, i.e., the second resource amount is 0, and the resource ratio is 50:0.

[0057] As the number of interactive requests gradually increases, the number of machines in the interactive resource pool processing interactive requests increases, and the utilization of interactive resources increases. When the utilization of the interactive resource pool approaches saturation, for example, if all 60 machines in the interactive resource pool are processing interactive requests, that is, the first resource amount is 60, the resource ratio increases to 60:0. Therefore, it is necessary to allocate some resources from the hybrid resource pool to the interactive resource pool. For example, 10 machines in the hybrid resource pool can be temporarily allocated to the interactive resource pool, bringing the number of machines in the interactive resource pool to 70. The 10 additional machines can begin processing interactive requests, ensuring that interactive requests have sufficient resources to support them. The number of machines in the hybrid resource pool is reduced to 10, and the resource ratio is now 60:10.

[0058] If the number of interactive requests gradually decreases, some machine resources in the interactive resource pool become idle. For example, suppose 40 machines in the interactive resource pool are processing interactive requests, with a first resource amount of 40 and a resource utilization rate of 40 / 60 = 66.7%. Ten machines originally in the hybrid resource pool are processing interactive requests, with a second resource amount of 10, resulting in a resource ratio of 40:10. At this point, the 10 machines previously allocated from the hybrid resource pool need to be returned to the hybrid resource pool, restoring the interactive resource pool to 60 machines and the hybrid resource pool to 20 machines. This avoids allocating too many resources to interactive requests, allowing hybrid resources to be used by batch requests and improving overall resource utilization.

[0059] According to the method of this embodiment, the interactive resource pool is scaled based on resource utilization and resource ratio, and the resource amount of interactive resources and hybrid resources is adjusted, which can effectively alleviate the pressure on the interactive resource pool and maximize the resource utilization and throughput performance of the reasoning service system while meeting the service level objectives.

[0060] In actual applications, if all resources in the hybrid resource pool are allocated to the interactive resource pool, and the number of interactive requests continues to increase, resources can be allocated from the batch resource pool to the interactive resource pool. This embodiment further illustrates and optimizes the technical solution based on the above embodiment. Specifically, in this embodiment, when performing resource scaling of interactive resources based on resource utilization and resource ratio, the global controller is configured as follows: Case 1: If the resource utilization is greater than or equal to the first preset utilization upper limit, some resources are allocated from the hybrid resource pool to the interactive resource pool according to the resource ratio; Case 2: If the resource utilization is greater than or equal to the second preset utilization upper limit, some resources are allocated from the batch resource pool to the interactive resource pool based on the resource ratio; the first preset utilization upper limit is less than the second preset utilization upper limit; Case 3: If the resource utilization is less than or equal to the first preset utilization lower limit, a corresponding amount of resources are allocated from the interactive resource pool to the batch resource pool according to the resource ratio; Case 4: If the resource utilization is less than or equal to the second preset utilization lower limit, a corresponding number of resources are allocated from the interactive resource pool to the hybrid resource pool according to the resource ratio; the first preset utilization lower limit is greater than the second preset utilization lower limit.

[0061] In this embodiment, when a sudden peak occurs in interactive requests, the hybrid resource pool and the batch resource pool are comprehensively utilized to scale resources of the interactive resource pool.

[0062] Specifically, if the resource utilization rate is greater than or equal to the first preset utilization rate upper limit value, part of the resources will be dynamically allocated from the hybrid resource pool to the interactive resource pool according to the resource ratio; if the resource utilization rate is greater than or equal to the second preset utilization rate upper limit value, part of the resources will be allocated from the batch resource pool to the interactive resource pool; wherein, the first preset utilization rate upper limit value is less than the second preset utilization rate upper limit value, indicating that part of the resources will be allocated from the hybrid resource pool to the interactive resource pool first. When the resources in the hybrid resource pool are insufficient, resources will be allocated from the batch resource pool to the interactive resource pool.

[0063] If the resource utilization rate is less than or equal to the first preset utilization lower limit, a corresponding number of resources will be dynamically allocated from the interactive resource pool to the batch resource pool based on the resource ratio to maintain the resource volume of the batch resource pool; if the resource utilization rate is less than or equal to the second preset utilization lower limit, a corresponding number of resources will be dynamically allocated from the interactive resource pool to the hybrid resource pool based on the resource ratio to maintain the resource volume of the hybrid resource pool; wherein, the second preset utilization lower limit is greater than the second preset utilization lower limit, indicating that the resources of the batch resource pool are returned first to maintain the resource volume of the batch resource pool, and then the resources of the hybrid resource pool are returned to maintain the resource volume of the hybrid resource pool.

[0064] According to the method of this embodiment, all machine resources in the inference service system can be comprehensively used to perform resource expansion and contraction, thereby maximizing resource utilization of the inference service system and improving throughput performance of the inference service system.

[0065] In a specific embodiment, the global controller is further configured to: Determining a third resource amount in the batch resource pool that is processing the batch request, and determining a remaining resource amount in the batch resource pool based on the third resource amount; Scale resources in interactive resource pools based on remaining resources.

[0066] Specifically, if the utilization rate of the interactive resource pool is close to saturation, and all machine resources in the hybrid resource pool have been allocated to the interactive resource pool; and the number of interactive requests is still gradually increasing, the global controller determines the third resource amount in the batch resource pool that is processing batch requests, and determines the remaining resource amount of the batch resource pool based on the difference between the third resource amount and the preset resource amount of the batch resource pool; scales the resources of the interactive resource pool based on the remaining resource amount; resource scaling includes allocating corresponding machine resources from the batch resource pool to the interactive resource pool, thereby alleviating the pressure on the interactive resource pool and ensuring that the delay requirements of the interactive requests are met; and returning the machine resources allocated from the batch resource pool to the batch resource pool to maintain the resource amount of the batch resource pool.

[0067] For example, suppose the inference service system has 100 machines, 60 of which are allocated to the interactive resource pool, 20 to the batch resource pool, and 20 to the hybrid resource pool. Furthermore, assume that all 60 machines in the interactive resource pool are currently processing interactive requests, and all 20 machines in the hybrid resource pool are also deployed to the interactive resource pool to process interactive requests. At this point, the number of interactive requests continues to increase, so a third amount of resources (assuming 5 machines) in the batch resource pool currently processing batch requests can be determined. Based on this third amount of resources, the remaining resources in the batch resource pool, 15 machines, can be determined. Some or all of the 15 machines can be deployed to the interactive resource pool to process interactive requests, ensuring sufficient resources for interactive requests. When the number of interactive requests decreases and only 40 machines in the interactive resource pool are processing interactive requests, the machine resources deployed to the interactive resource pool can be returned to the batch resource pool to maintain the batch resource pool's resource capacity.

[0068] It can be seen that according to the method of this embodiment, when there is a sudden peak in interactive requests, the machine resources in the inference service system can be fully utilized to scale resources, avoid violating the SLO target, and improve the resource utilization and throughput performance of the inference service system.

[0069] Figure 3This embodiment of the present application provides a flowchart of a global controller performing resource scaling on an interactive resource pool. Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically, in this embodiment, when performing resource scaling on the interactive resource pool and batch resource pool according to the service level objective type, the global controller is configured to: S310: If the inference request is determined to be a batch request according to the service level target type, the batch requests are grouped according to their information types, and the batch requests are placed in a batch request queue for waiting.

[0070] It's understandable that the SLO for batch requests is often measured in minutes or even hours, and unlike interactive requests, resources don't need to be allocated every time a new request arrives. Therefore, in this embodiment, if an inference request is determined to be a batch request based on the service level objective type, the batch requests are grouped according to their SLO targets and placed in corresponding batch request queues. Resource scaling policies are then formulated based on the global queue status.

[0071] Specifically, the global controller groups batch requests based on their document type, size, and priority. Document types include PDF, Word, and Excel, while document sizes range from 0-10MB, 10MB-100MB, and greater than 100MB. Priority indicates the order or importance of processing batch requests.

[0072] After the batch requests are grouped, the batch requests are placed in a corresponding batch request queue to wait.

[0073] S320: scaling resources in the batch resource pool according to the global queue state of the batch request queue.

[0074] Specifically, the global queue status of each batch request queue is monitored, and the resource amount of the batch resource is adjusted according to the global queue status of the batch request queue.

[0075] In a specific example, when the queue length of the batch request queue is greater than the upper limit of the batch request queue length, some or all machine resources can be allocated from the hybrid resource pool to the batch resource pool to process batch requests; when the queue length of the batch request queue is less than the lower limit of the batch request queue length, some or all machine resources allocated to the batch resource pool will be returned to the hybrid resource pool to maintain the resource volume of the batch resource pool.

[0076] According to the method of this embodiment, the inference requests are grouped according to their information type, and the inference requests are placed in a batch request queue for waiting, and the batch resource pool is scaled based on the global queue status of the batch request queue to avoid violating the SLO target and improve the resource utilization of the inference service system.

[0077] In a specific embodiment, the global queue status of the batch request queue includes the number of batch requests and the estimated queue waiting time.

[0078] The number of batch requests refers to the number of batch requests; the estimated queue waiting time refers to the estimated time it takes to process each batch request in the batch request queue.

[0079] In a specific example, the number of batch requests in the batch request queue is obtained, and it is determined whether the number of batch requests exceeds a preset upper limit. If the number of batch requests in the batch request queue exceeds the preset upper limit, it means that the current batch resource pool will not be able to meet the SLO target during the processing process. Therefore, the batch resource pool is scaled up or down, specifically, some or all machine resources are allocated from the hybrid resource pool to the batch resource pool to process the batch requests.

[0080] In a specific example, the estimated processing time corresponding to each batch request in the batch request queue is determined, and the estimated queue waiting time is determined based on the sum of the estimated processing times; it is judged whether the estimated queue waiting time exceeds the preset queue waiting time upper limit; if the estimated queue waiting time exceeds the preset queue waiting time upper limit, it means that the current batch resource pool will not be able to meet the SLO target during the processing process, so the batch resource pool is scaled up or down, specifically, part or all of the machine resources are allocated from the hybrid resource pool to the batch resource pool to process the batch requests.

[0081] On the contrary, if the number of batch requests in the batch request queue is less than the preset lower limit, or the estimated queue waiting time of the batch request queue is less than the preset lower limit, some or all of the machine resources allocated to the batch resource pool can be returned to the hybrid resource pool to release excess machine resources.

[0082] For example, if there are 100 batch requests in the first batch request queue, the estimated queue wait time is 1.5 hours based on the estimated processing time for each batch request in the batch request queue. This means that based on the current batch resource pool resources, it will take 1.5 hours to process all the batch requests in the batch request queue, but this will exceed the SLO target of 1 hour. Therefore, a corresponding number of machines are allocated from the hybrid resource pool to the batch resource pool to process the batch requests, thereby increasing processing speed and avoiding SLO violations.

[0083] According to the method of this embodiment, resource utilization and cost-effectiveness can be maximized while ensuring that the SLO of batch requests is met.

[0084] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically, in this embodiment, when the interactive resource pool and the batch resource pool are pre-divided into the prompt word resource pool and the token resource pool, respectively, the cluster controller is configured as follows: The interactive resource pool and the batch resource pool are pre-divided into a prompt word resource pool, a token resource pool and a stage mixed resource pool respectively; It should be noted that when processing inference requests, there are two stages, namely the prompt word stage and the token stage; among them, the prompt word stage is computationally intensive and requires sufficient computing power support, while the token stage is more easily affected by resource storage capacity and bandwidth.

[0085] In this embodiment, the cluster controller subdivides the interactive resource pool and the batch resource pool into three subsets: a prompt resource pool, a token resource pool, and a stage mixed resource pool.

[0086] Among them, the stage hybrid resource pool is used to dynamically change the number of machine resources in the prompt word resource pool and token resource pool to reduce resource fragmentation and ensure that the SLO target is not violated in the case of sudden high workload.

[0087] Figure 4 This is a flow chart of a cluster controller scaling resources in a prompt word resource pool and a token resource pool provided in an embodiment of the present application. The cluster controller performs the following steps: S410: Determine the stage queue states corresponding to the prompt word resource pool and the token resource pool respectively.

[0088] Specifically, after receiving an inference request, the cluster controller allocates a pair of resources, namely, a prompt word resource and a token resource, for each prompt word phase and token phase of the inference request. For the prompt word resource pool and the token resource pool, the corresponding request is added to their respective phase queues, thereby updating the phase queues. The cluster controller then obtains the phase queue status of the phase queues corresponding to the prompt word resource pool and the token resource pool, respectively.

[0089] Among them, the stage queue status includes the number of stage requests and the estimated stage waiting time; the number of stage requests is the number of inference requests in the stage queue corresponding to the prompt word resource pool or token resource pool, and the estimated stage waiting time is the estimated time required for the prompt word resource pool or token resource pool to process the inference requests in the stage queue corresponding to itself.

[0090] S420: According to the stage queue status, the stage hybrid resource pool is used to scale resources of the prompt word resource pool and the token resource pool.

[0091] If the number of stage requests is greater than the preset upper limit of the number of stage requests, or the estimated stage waiting time is greater than the preset upper limit of the stage waiting time, the corresponding machine resources need to be allocated from the stage mixed resource pool to the corresponding prompt word resource pool and token resource pool; if the number of stage requests is less than the preset lower limit of the number of stage requests, or the estimated stage waiting time is less than the preset lower limit of the stage waiting time, the corresponding machine resources need to be allocated from the stage mixed resource pool and returned to the stage mixed resource pool.

[0092] Assume that in the initial stage, the prompt word resource pool has 50 machines, the token resource pool has 50 machines, and the hybrid resource pool has 20 machines. The cluster controller allocates prompt word resources to the prompt word phase of each inference request and token resources to the token phase. As the number of interactive requests increases, the estimated phase wait time of the phase queue corresponding to the prompt word resource pool increases. When the estimated phase wait time exceeds the preset phase wait time limit, the cluster controller allocates 10 machines from the hybrid phase resource pool to the prompt word resource pool, increasing the number of machines in the prompt word resource pool to 60 and reducing the number of machines in the hybrid phase resource pool to 10. This alleviates the pressure on the prompt word resource pool, ensures that the prompt word phase of interactive requests can be processed promptly, and avoids SLO violations.

[0093] As the number of interactive requests decreases, the estimated stage wait time of the stage queue corresponding to the prompt word resource pool decreases, gradually reducing the pressure on the prompt word resource pool. When the estimated stage wait time falls below the preset stage wait time lower limit, the cluster controller returns the 10 machines previously allocated from the stage mixed resource pool to the stage mixed resource pool, restoring the number of machines in the prompt word resource pool to 50 and the number of machines in the stage mixed resource pool to 20.

[0094] In this embodiment, the cluster controller allocates prompt word resources and token resources from the prompt word resource pool and token resource pool respectively according to the different stage characteristics of the inference request, and then uses the stage hybrid resource pool to scale the prompt word resource pool and token resource pool according to the stage queue status; resource scheduling, allocation and automatic scaling are performed for the two stages of a single inference request.

[0095] In a specific embodiment, the cluster controller is configured to: Based on the estimated demand load and input / output token distribution, the interactive resource pool and the batch resource pool are divided into a prompt word resource pool, a token resource pool and a stage mixed resource pool respectively.

[0096] In this embodiment, the cluster controller divides the interactive resource pool and the batch resource pool into three subsets respectively according to the estimated demand load and input / output token distribution: a prompt word resource pool, a token resource pool, and a stage-mixed resource pool.

[0097] The estimated demand load refers to the total number of inference requests and resource requirements that the inference service system is likely to receive over a period of time, predicted based on historical data, business scenarios, and user behavior. It should be noted that a larger estimated demand load increases the capacity of the corresponding resource pool; conversely, a smaller estimated demand load decreases the capacity of the corresponding resource pool.

[0098] The input / output token distribution refers to the distribution characteristics of the input prompt word sequence length and the output token sequence length for the inference request. It is understandable that the prompt word sequence length affects the computing resource requirements of the prompt word stage, and the token sequence length affects the storage and bandwidth resource requirements of the token stage. The cluster controller can more accurately divide the size of the prompt word resource pool and the token resource pool based on the input / output token distribution to meet the resource requirements of different stages. For example, if the input token length is generally long, the size of the prompt word resource pool can be appropriately increased; if the output token length distribution is wide, the storage and bandwidth resources of the token resource pool can be appropriately increased.

[0099] In this embodiment, the cluster controller is used to divide the prompt word resource pool, token resource pool and stage hybrid resource pool in advance according to the estimated demand load and input / output token distribution, which can improve the rationality of the initially divided prompt word resource pool, token resource pool and second hybrid resource capacity.

[0100] Based on the above embodiment, this embodiment further illustrates and optimizes the technical solution. Specifically, in this embodiment, when adjusting the batch size of inference requests based on the service level objective type and the usage status of the machine, the machine controller is configured to: Real-time monitoring of the usage status of the machines; Adjust the batch size for processing inference requests based on the usage status, machine resource type of the machine, and service level objective type.

[0101] The usage status of the machine controller includes storage status and processing queue status. Storage status includes CPU utilization, memory utilization, and GPU memory utilization. Processing queue status refers to the status of the processing queue corresponding to the machine, including the processing queue length and the number of tasks in the processing queue.

[0102] The machine resource type of the machine refers to the type of configured hardware resources, such as CPU, GPU, NPU, TPU, and FPGA.

[0103] The batch size refers to the number of inference requests or the amount of data that a machine can process simultaneously using its own machine resources. Figure 2 As shown, R1, R2 and R3 represent three different batch sizes determined by the machine controller.

[0104] In this embodiment, the machine controller monitors the usage status of the machine in real time and determines the batch size according to the service level target type of the inference request, the usage status of the machine, and the machine resource type of the machine.

[0105] For example, for interactive requests with a low latency SLO, the batch size is set smaller to reduce processing time and avoid excessive latency. For batch requests with a certain latency SLO, the batch size can be set larger to improve throughput and make full use of resources.

[0106] If the memory utilization is greater than the preset upper limit, it indicates that storage resources are tight, and you need to reduce the batch size to avoid resource exhaustion and task backlogs. If the memory utilization is less than the preset lower limit, it indicates that storage resources are sufficient, and you can increase the batch size to improve resource utilization.

[0107] In addition, different machine resource types have different performance characteristics and applicable scenarios when processing inference requests. Therefore, the batch size needs to be reasonably set according to the different characteristics of the machine resource type.

[0108] When adjusting the batch size, you should also consider the machine's batch size limit. That is, when a machine processes an inference request, its batch size cannot be larger than its own batch size limit.

[0109] According to the method of this embodiment, it is possible to optimize resource utilization efficiency and improve the throughput performance of the reasoning service system while meeting the SLO target.

[0110] It should be noted that the information collection process (such as the facial image collection process, fingerprint information collection process, etc.) / feature extraction process involved in this application is performed with the user's knowledge and permission, that is, the information collection process / feature extraction process complies with the requirements of laws and regulations and does not constitute an act that harms the public interest.

[0111] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of this application. Their specific functions and technical effects can be found in the method embodiment section and will not be repeated here.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated into one processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0113] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant description of other embodiments.

[0114] Those skilled in the art will appreciate that the devices and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0115] In the embodiments provided herein, it should be understood that the disclosed devices and methods may be implemented in other ways. For example, the device embodiments described above are merely illustrative. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interface, or the devices may be indirectly coupled or communicated in some manner, whether electrical, mechanical, or other.

[0116] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. An inference service system, characterized in that: The inference service system includes a global controller, a cluster controller and a plurality of machine controllers; the global controller is in communication with the cluster controller, and the cluster controller is in communication with each of the machine controllers respectively; The global controller is used to pre-divide machine resources into an interactive resource pool and a batch resource pool; when receiving an inference request, determine the service level objective type of the inference request, and scale resources of the interactive resource pool and the batch resource pool according to the service level objective type; The cluster controller is configured to pre-divide the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool, respectively; allocate corresponding prompt word resources and token resources from the prompt word resource pool and the token resource pool, respectively, for the prompt word phase and the token phase of the inference request, and perform resource expansion and contraction on the prompt word resource pool and the token resource pool; The machine controller is configured to adjust a batch size for processing the inference request according to the service level objective type and a usage status of the machine.

2. The reasoning service system according to claim 1, characterized in that When executing the pre-dividing of machine resources into interactive resource pools and batch resource pools, the global controller is configured to: Dividing the machine resources into the interactive resource pool, the batch resource pool and the hybrid resource pool in advance; When performing resource scaling of the interactive resource pool and the batch resource pool according to the service level target type, the global controller is configured to: If the inference request is determined to be an interactive request according to the service level objective type, the hybrid resource pool is used to scale resources of the interactive resource pool.

3. The reasoning service system according to claim 2, characterized in that: When the global controller determines, according to the service level objective type, that the inference request is an interactive request, and uses the hybrid resource pool to scale resources of the interactive resource pool, the global controller is configured to: determining a first amount of resources in the interactive resource pool that are processing the interactive request; determining a second amount of resources in the hybrid resource pool that are processing the interactive request; determining a resource utilization rate of the interactive resource pool according to a ratio of the first resource amount to a preset resource amount of the interactive resource pool; determining a resource ratio between the first resource amount and the second resource amount; The interactive resource is scaled up or down according to the resource utilization rate and the resource ratio.

4. The reasoning service system according to claim 3, characterized in that: When executing the resource scaling of the interactive resource according to the resource utilization and the resource ratio, the global controller is configured to: If the resource utilization rate is greater than or equal to a first preset utilization rate upper limit, allocating part of the resources from the hybrid resource pool to the interactive resource pool according to the resource ratio; If the resource utilization rate is greater than or equal to a second preset utilization rate upper limit, allocating part of the resources from the batch resource pool to the interactive resource pool according to the resource ratio; The first preset utilization rate upper limit value is less than the second preset utilization rate upper limit value; If the resource utilization rate is less than or equal to a first preset utilization rate lower limit, allocating a corresponding amount of resources from the interactive resource pool to the batch resource pool according to the resource ratio; If the resource utilization is less than or equal to a second preset utilization lower limit, a corresponding number of resources are allocated from the interactive resource pool to the hybrid resource pool according to the resource ratio; the first preset utilization lower limit is greater than the second preset utilization lower limit.

5. The reasoning service system according to claim 4, characterized in that: The global controller is also used to: Determining a third amount of resources in the batch resource pool that are currently processing batch requests, and determining a remaining amount of resources in the batch resource pool based on the third amount of resources; The interactive resource pool is scaled up or down based on the remaining resources.

6. The reasoning service system according to claim 1, characterized in that: When performing resource scaling of the interactive resource pool and the batch resource pool according to the service level target type, the global controller is configured to: If the inference request is determined to be a batch request according to the service level objective type, grouping the batch requests according to their information types and placing the batch requests in a batch request queue for waiting; The batch resource pool is scaled up or down according to the global queue state of the batch request queue.

7. The reasoning service system according to claim 6, characterized in that: The global queue status of the batch request queue includes the number of batch requests and the estimated queue waiting time.

8. The reasoning service system according to any one of claims 1 to 7, characterized in that: When executing the step of pre-dividing the interactive resource pool and the batch resource pool into a prompt word resource pool and a token resource pool, the cluster controller is configured to: The interactive resource pool and the batch resource pool are respectively divided into a prompt word resource pool, a token resource pool and a stage mixed resource pool in advance; When executing the resource expansion and contraction of the prompt word resource pool and the token resource pool, the cluster controller is configured to: Determine the stage queue states corresponding to the prompt word resource pool and the token resource pool respectively; According to the stage queue status, the stage hybrid resource pool is used to scale the prompt word resource pool and the token resource pool.

9. The reasoning service system according to claim 8, characterized in that: When the cluster controller performs the step of pre-dividing the interactive resource pool and the batch resource pool into a prompt word resource pool, a token resource pool, and a stage-mixed resource pool, the cluster controller is configured to: The interactive resource pool and the batch resource pool are respectively divided into a prompt word resource pool, a token resource pool and a stage mixed resource pool according to the estimated demand load and the input / output token distribution.

10. The reasoning service system according to any one of claims 1 to 7, characterized in that: When the machine controller adjusts the batch size of the inference request according to the service level target type and the usage status of the machine, the machine controller is configured to: Real-time monitoring of the usage status of the machine; A batch size for processing the inference request is adjusted according to the usage status, the machine resource type of the belonging machine, and the service level target type.

Images