Reasoning request scheduling method and related device

Abstract

A reasoning request scheduling method, comprising: a reasoning routing apparatus acquires a first reasoning request, and when the number of reasoning requests to be processed is greater than or equal to a threshold defined in a flow control policy, the reasoning routing apparatus checks the remaining resources of at least one reasoning server in a reasoning resource pool; the reasoning routing apparatus predicts consumed resources of the first reasoning request; and when the consumed resources are less than the remaining resources, the reasoning routing apparatus determines a target reasoning server corresponding to the first reasoning request, and schedules the first reasoning request to the target reasoning server, wherein the target reasoning server is one or more reasoning servers in the reasoning resource pool of which the remaining resources satisfy the first reasoning request, and a language model deployed in the target reasoning server matches the first reasoning request. In the method, when the number of requests is greater than a flow control threshold, reasoning requests are dynamically scheduled on the basis of the remaining resources of reasoning servers dynamically fed back by a reasoning resource pool side, so as to use reasoning resources to the greatest extent, thereby improving the utilization rate of the reasoning resources.

Description

A method for scheduling inference requests and related equipment

[0001] This application claims priority to Chinese Patent Application No. 202411471280.4, filed with the State Intellectual Property Office of China on October 21, 2024, entitled "A Reasoning Request Scheduling Method and Related Equipment", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a reasoning request scheduling method, scheduling system, computing device cluster, computer-readable storage medium, and computer program product. Background Technology

[0003] With the continuous development of artificial intelligence (AI) technology, especially the rapid development of large language models (LLMs), LLM-based inference services have been widely applied. For example, LLM-based inference services can be applied to chatbot scenarios to answer user questions. Here, LLMs can be language models (LMs) with a parameter scale of over 1 billion. LLMs greatly expand the data used for training and inference, and improve the model's capabilities.

[0004] LLM-based inference services typically rely on underlying inference resources to provide services to users. These inference resources can include underlying hardware resources, such as Graphics Processing Units (GPUs). Inference resources are often the ultimate bottleneck for LLM-based inference services. The larger the number of users supported, the greater the consumption of GPUs and other inference resources by the LLM-based inference service. In large-scale inference scenarios, how to better utilize existing inference resources to meet inference requests with varying traffic volumes, thus ensuring efficient utilization of inference resources, has become a hot research topic.

[0005] Currently, related technologies offer a solution for time-sharing GPU scheduling to maximize GPU utilization. However, this solution lacks integration with business scenarios and often fails to meet business requirements. For example, during peak traffic periods, a large number of users access the inference service to request inference tasks. When the number of requests exceeds the flow control threshold set in the flow control policy, some inference requests may be denied service or require long waiting times, thus affecting the availability of the inference service, especially during peak traffic periods. Summary of the Invention

[0006] This application provides an inference request scheduling method. This method constructs a scheduling scheme based on dynamic feedback of remaining resources to maximize the use of inference resources such as GPUs, increase the number of concurrent users during peak traffic periods, reduce the number of users experiencing service timeouts or unavailability, and improve service experience. This application also provides a scheduling system, computing device cluster, computer-readable storage medium, and computer program product corresponding to the above method.

[0007] Firstly, this application provides an inference request scheduling method. This method is applied to a scheduling system. The scheduling system includes an inference routing device and an inference resource pool. The inference resource pool includes multiple inference servers, and the inference resource pool can be a common resource area constructed by deploying different language models (such as different LLMs) to different inference servers. Each inference server deploys the same inference framework. The inference framework can be used for resource detection. The inference server includes multiple accelerators for running language models to provide inference services, or the inference server uses multiple accelerators for running language models to provide inference services.

[0008] Specifically, the inference routing device obtains a first inference request, which requests at least one inference server from the plurality of inference servers. When the number of pending inference requests is greater than or equal to a threshold defined in the traffic control policy, the inference routing device checks the remaining resources of at least one inference server in the inference resource pool. The inference routing device predicts the resource consumption of the first inference request. When the resource consumption is less than the remaining resources, the inference routing device determines the target inference server corresponding to the first inference request and schedules the first inference request to the target inference server. The target inference server is one or more inference servers in the inference resource pool whose remaining resources satisfy the first inference request, and the language model deployed on the target inference server matches the first inference request.

[0009] This method constructs a unified inference resource pool. When the number of pending inference requests exceeds the threshold in the flow control policy, it schedules inference requests based on the remaining resources of at least one inference server dynamically fed back from the inference resource pool, maximizing the use of inference resources such as GPUs. This avoids triggering flow control and prevents underutilization of inference resources, thus improving resource utilization. Furthermore, by fully utilizing inference resources such as GPUs, it increases the number of concurrent users during peak traffic periods, reduces the number of users experiencing service timeouts or unavailability, and improves the service experience.

[0010] In some possible implementations, the inference routing device adds a first label to the first inference request according to its task type. The inference routing device determines the similarity between the first inference request and a second inference request in the request scheduling queue that is tagged with the first label. When the similarity is greater than a threshold, the inference routing device removes the second inference request from the request scheduling queue.

[0011] This method achieves traffic shaping by canceling duplicate requests, avoiding resource waste caused by performing inference on duplicate inference requests, and improving the utilization rate of inference resources.

[0012] In some possible implementations, the task type includes code continuation or text continuation. When performing code continuation or text continuation, new tokens can be generated one by one based on the preceding part of the token sequence. The process of generating new tokens may include repeated inference requests. By marking the inference requests for code continuation or text continuation with a first tag, repeated inference requests can be canceled, avoiding the waste of resources caused by code continuation or text continuation.

[0013] In some possible implementations, the inference routing device can also determine the total remaining resources of the inference resource pool based on the remaining resources of the inference servers in the inference resource pool. The inference routing device can predict the inference traffic distribution for the next n periods based on the first inference request. The inference traffic distribution includes the distribution of the number of inference requests and / or the distribution of the number of input tokens for the inference requests. When the total remaining resources are less than a first value, and the inference traffic distribution meets set conditions, the inference routing device compresses the input of the inference requests in the request scheduling queue.

[0014] This method compresses the input of inference requests in the request scheduling queue when the total amount of remaining resources is small and a large number of inference requests are continuously arriving, thereby achieving traffic smoothing and ensuring overall performance.

[0015] In some possible implementations, the inference routing device can compress the input of at least one inference request in the request scheduling queue using a compression algorithm. Alternatively, the inference routing device can add a second tag to at least one inference request in the request scheduling queue and send the inference request tagged with the second tag to the agent. The second tag indicates that the input of the inference request tagged with the second tag is compressed; for example, the second tag can be a compression tag. Accordingly, the inference routing device receives the compressed inference request sent by the agent.

[0016] In this method, the inference routing device can compress inference requests in the request scheduling queue by itself using a compression algorithm, or add a second tag indicating compression to the inference request and route the inference request with the second tag to the agent for compression, so as to achieve traffic smoothing and high availability.

[0017] In some possible implementations, the inference routing device can compress the input of at least one inference request in the request scheduling queue whose task type is the target type using a compression algorithm. The target type includes code-related types or resource-intensive types. Alternatively, the inference routing device can add a second tag to at least one inference request in the request scheduling queue whose task type is the target type, said target type including code-related types or resource-intensive types.

[0018] In this method, reasoning requests of code-related types or resource-intensive reasoning requests (such as analysis requests) can consume a lot of reasoning resources. By specifically compressing the reasoning requests of the above-mentioned task types, the reasoning routing device can greatly reduce the consumption of reasoning resources by the reasoning requests of the above-mentioned task types and improve the utilization rate of reasoning resources.

[0019] In some possible implementations, the inference routing device checks the inference capacity of at least one inference server in the inference resource pool, whereby the inference capacity indicates the number of input tokens that can be used to support an inference request. Accordingly, the inference routing device determines the target inference server corresponding to the first inference request based on the resource consumption of the first inference request, the remaining resources of the at least one inference server in the inference resource pool, and the inference capacity.

[0020] This allows inference requests to be scheduled to appropriate inference servers based on inference capacity, remaining resources of the inference server, and resource consumption of the inference request. This maximizes the utilization of inference resources while ensuring business operations.

[0021] In some possible implementations, the inference routing device can also update the thresholds defined in the traffic control policy. This allows for a shift from quantitatively configuring inference capabilities to dynamically adjusting them, ensuring maximum utilization of inference resources such as GPUs and satisfying the inference requests of the largest number of multi-tenant users.

[0022] Secondly, this application provides a scheduling system. The scheduling system includes an inference routing device and an inference resource pool. The inference resource pool includes multiple inference servers, and each inference server includes multiple accelerators for running language models to provide inference services, or the inference server uses multiple accelerators for running language models to provide inference services.

[0023] The inference routing device is configured to acquire a first inference request, the first inference request being used to request at least one inference server among the plurality of inference servers; when the number of inference requests to be processed is greater than or equal to a threshold defined in the traffic control policy, check the remaining resources of at least one inference server in the inference resource pool; and predict the resource consumption of the first inference request.

[0024] The inference routing device is further configured to, when the resource consumption is less than the remaining resources, determine a target inference server corresponding to the first inference request, and schedule the first inference request to the target inference server. The target inference server is one or more inference servers in the inference resource pool whose remaining resources satisfy the first inference request, and the language model deployed on the target inference server matches the first inference request.

[0025] In some possible implementations, the inference routing device is also used for:

[0026] Add a first tag to the first inference request according to the task type;

[0027] Determine the similarity between the first inference request and the second inference request in the request scheduling queue that is tagged with the first label;

[0028] When the similarity is greater than a threshold, the second inference request is removed from the request scheduling queue.

[0029] In some possible implementations, the task type includes code continuation or text continuation.

[0030] In some possible implementations, the inference routing device is also used for:

[0031] The total amount of remaining resources in the inference resource pool is determined based on the remaining resources of the inference server in the inference resource pool.

[0032] Based on the first inference request, predict the inference traffic distribution for the next n periods, wherein the inference traffic distribution includes the distribution of the number of inference requests and / or the distribution of the number of input tokens for the inference requests;

[0033] When the total remaining resources are less than a first value and the inference traffic distribution meets the set conditions, the input of inference requests in the request scheduling queue is compressed.

[0034] In some possible implementations, the inference routing device is specifically used for:

[0035] The input of at least one inference request in the request scheduling queue is compressed using a compression algorithm; or,

[0036] A second tag is added to at least one inference request in the request scheduling queue, and the inference request marked with the second tag is sent to the agent. The second tag is used to indicate the input of the inference request marked with the second tag.

[0037] Receive the compressed inference request sent by the intelligent agent.

[0038] In some possible implementations, the inference routing device is specifically used for:

[0039] The input of at least one inference request with a task type of target type in the request scheduling queue is compressed using a compression algorithm. The target type includes code-related types or resource-intensive types; or...

[0040] The inference routing device is specifically used for:

[0041] Add a second tag to at least one inference request in the request scheduling queue whose task type is a target type, where the target type includes code-related types or resource-consuming types.

[0042] In some possible implementations, the inference routing device is also used for:

[0043] Check the inference capacity of at least one inference server in the inference resource pool, the inference capacity being used to indicate the number of input tokens that can be carried for inference requests;

[0044] The inference routing device is specifically used for:

[0045] The inference routing device determines the target inference server corresponding to the first inference request based on the resource consumption of the first inference request, the remaining resources of at least one inference server in the inference resource pool, and the inference capacity.

[0046] In some possible implementations, the inference routing device is also used for:

[0047] Update the threshold defined in the flow control policy.

[0048] Thirdly, this application provides a computing device cluster. The computing device cluster includes at least one computing device, and the at least one computing device includes at least one processor and at least one memory. The at least one processor and the at least one memory communicate with each other. The at least one processor is used to execute instructions stored in the at least one memory to cause the computing device or the computing device cluster to perform the inference request scheduling method as described in the first aspect or any implementation thereof.

[0049] Fourthly, this application provides a computer-readable storage medium storing instructions that instruct a computing device or a cluster of computing devices to execute the inference request scheduling method described in the first aspect or any implementation thereof.

[0050] Fifthly, this application provides a computer program product containing instructions that, when run on a computing device or a cluster of computing devices, causes the computing device or cluster of computing devices to execute the inference request scheduling method described in the first aspect or any implementation thereof.

[0051] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. Attached Figure Description

[0052] To more clearly illustrate the technical methods of this application, the accompanying drawings used will be briefly described below.

[0053] Figure 1 is a schematic diagram of a time-sharing resource allocation method provided in this application;

[0054] Figure 2 is a schematic diagram of the architecture of a scheduling system provided in this application;

[0055] Figure 3 is a diagram showing the relationship between an intelligent agent and a large model provided in this application;

[0056] Figure 4 is a schematic diagram of a research and development conversation scenario provided in this application;

[0057] Figure 5 is a flowchart of an inference request scheduling method provided in this application;

[0058] Figure 6 shows an application scenario of the inference request scheduling method provided in this application;

[0059] Figure 7 is a schematic diagram of the structure of a computing device provided in this application;

[0060] Figure 8 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0061] Figure 9 is a schematic diagram of another computing device cluster provided in this application;

[0062] Figure 10 is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0063] The terms "first" and "second" used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include one or more of that feature.

[0064] First, some technical terms involved in the embodiments of this application will be introduced.

[0065] Artificial intelligence (AI) is the ability to correctly interpret external data, learn knowledge from that data, and use that knowledge to achieve specific goals and tasks. Currently, a significant branch of AI is natural language processing (NLP). NLP uses algorithms such as machine learning (ML) and deep learning (DL) to build language models (LM), enabling computers to interpret, process, and understand human language.

[0066] A language model (or simply a model) is a probability distribution model of words in a natural language. The most common data in natural language processing is text data, such as a piece of natural language text, which can be viewed as a discrete time series. Assuming the words in a natural language text of length T are w1, w2, ..., wT, then in the discrete time series, wt (1 ≤ t ≤ T) can be seen as the output or label at time step t. Given a word sequence of length T, w1, w2, ..., wT, the language model will calculate the probability of this word sequence: P(w1, w2, ..., wT). Based on this, the language model can predict the next most likely word using the input context (e.g., several preceding context words).

[0067] Language models can also be categorized by parameter size into small language models and large language models (LLMs). Small language models can be simply referred to as small models, and large language models as large models. For ease of description, this application uses an LLM as an example. An LLM is a language model constructed from a deep neural network containing hundreds of billions of weights, which can be trained using self-supervised learning methods on a large amount of unlabeled text.

[0068] An intelligent agent is a computer program based on a language model, possessing planning and thinking abilities (also known as planning), memory capabilities, and the ability to use tool functions (or tool use), enabling it to autonomously complete a given task. Intelligent agents include LLM-based agents, also known as LLM agents. Planning ability can involve decomposing large tasks into subtasks, such as using Everything of Thought (XoT) to break down large tasks into subtasks and planning the execution process. Thinking ability allows for reflection and consideration of the task execution process, deciding whether to continue or terminate the task upon completion. Memory can include short-term and long-term memory. Short-term memory refers to the context generated and temporarily stored during task execution, which is cleared after task completion. Long-term memory refers to information retained for a long period, such as external knowledge bases, typically using vector databases for storage and retrieval. Tools refer to external resources, services, or application programming interfaces (APIs) that the intelligent agent can use to perform specific tasks or enhance functionality. The APIs can include, but are not limited to, tool APIs such as calculators, search tools, code executors, and database query tools. These tools can serve as supplementary components, extending the functionality of the intelligent agent beyond its inherent language generation capabilities.

[0069] Intelligent agents can come in different types. For example, intelligent agents can include, but are not limited to, conversational agents, task-oriented agents, creative agents, or collaborative agents. Conversational agents aim to engage in natural language conversations with users, providing information, answering questions, and assisting in completing various tasks. Conversational agents typically rely on language models such as LLMs to understand and generate human-like responses. Task-oriented agents focus on performing specific tasks or achieving predefined goals. They understand user needs by interacting with them and then perform actions to meet those needs. Examples of task-oriented agents include virtual assistants and task automation tools. Creative agents are used to create original and creative content, such as artwork, music, or writing. Collaborative agents are used to work collaboratively with humans to achieve common goals or tasks. LLMs can support collaborative agents by assisting in decision-making, generating reports, or providing insights.

[0070] Different types of agents perform different tasks internally, but all types of agents will process the prompts and forward them to the language model (such as LLM) to obtain more effective reasoning results.

[0071] Inference services built on language models or agents are typically deployed using underlying inference resources. Inference resources refer to the hardware resources that support the deployment of language models and agents to provide inference services. Inference resources can include, but are not limited to, Graphics Processing Units (GPUs), Neural Network Processing Units (NPUs), or Tensor Processing Units (TPUs). For ease of description, the following example uses GPUs as inference resources.

[0072] In large-scale inference scenarios, it is typically necessary to provide inference services to a large number of users. The more users supported, the greater the consumption of inference resources such as GPUs. GPUs and other inference resources become the bottleneck for inference services. How to better utilize existing inference resources to meet inference requests with varying traffic volumes, thus ensuring efficient utilization of inference resources, has become a hot research topic.

[0073] Related technologies provide a scheme for time-sharing GPU scheduling to maximize GPU utilization. As shown in Figure 1, each of the multiple computing devices 100 includes a processor 102, an accelerator 104, and a network interface card (NIC) 106. The processor 102 can be a central processing unit (CPU), the accelerator 104 can be a GPU, and the NIC 106 can be an InfiniBand (IB) NIC. The computing devices 100 are connected via the IB NIC for high-speed communication. The accelerators 104 in different computing devices 100 can share computing power; for example, the GPUs in different computing devices 100 can share computing power. Time-sharing based on this shared GPU computing power maximizes GPU utilization.

[0074] However, the above solutions do not consider business scenarios and often fail to meet business requirements. For example, language models such as LLMs and agents are typically deployed on a fixed number of GPUs. When an inference request arrives at an agent, the agent can look up the LLM corresponding to the request and perform inference. During peak traffic periods for inference requests, such as when a large number of users simultaneously access the inference service to request inference tasks, if the number of requests exceeds the flow control threshold or maximum concurrency set in the flow control policy, some inference requests cannot be processed. It is necessary to wait for some inference tasks in progress to complete their inference and release the corresponding GPUs before normal business operations can resume.

[0075] On the one hand, some inference requests may be denied service or require long waiting times, thus affecting the availability of the inference service, especially during peak periods. On the other hand, GPUs executing inference tasks may not be fully utilized because the proportion of computationally intensive inference tasks is relatively small.

[0076] In view of this, this application provides an inference request scheduling method. This method can be applied to a scheduling system. The scheduling system can be a software system. The software system can be a standalone software system, or integrated into other software systems as a plugin, component, app, or functional module. The software system can be distributed to customers as a software package for self-deployment. Alternatively, the software system can be deployed in a cloud environment and provided to users as a cloud service; when users subscribe to the cloud service, they can use the functions or capabilities of that cloud service. In some instances, the scheduling system can be a hardware system, such as a cluster of computing devices with inference request scheduling capabilities. When the computing device cluster is running, it executes the inference request scheduling method of this application.

[0077] The scheduling system includes an inference routing unit and an inference resource pool. The inference resource pool can be a shared resource area built by deploying different language models (such as different LLMs) across different inference servers, where each inference server deploys the same inference framework. The inference framework can be used for resource detection. The inference resource pool includes multiple inference servers. Each inference server includes multiple accelerators for running language models to provide inference services, or the inference server uses multiple accelerators for running language models to provide inference services. Similar to the scheduling system, the inference routing unit and the inference resource pool can be software or hardware.

[0078] In practice, the inference routing device obtains a first inference request, which requests at least one inference server from a plurality of inference servers. When the number of pending inference requests is greater than or equal to a threshold defined in the traffic control policy, the inference routing device checks the remaining resources of at least one inference server in the inference resource pool. Remaining resources include the amount or percentage of remaining resources of the accelerators in the inference server. Furthermore, the inference routing device also predicts the resource consumption of the first inference request. Similar to remaining resources including the amount or percentage of remaining resources, resource consumption can include the amount or rate of resource consumption. When resource consumption is less than the remaining resources, the inference routing device determines the target inference server corresponding to the first inference request and schedules the first inference request to the target inference server. The target inference server is one or more inference servers in the inference resource pool whose remaining resources satisfy the first inference request, and the language model deployed on the target inference server matches the first inference request.

[0079] This method constructs a unified inference resource pool. When the number of pending inference requests exceeds the threshold in the flow control policy, it schedules inference requests based on the remaining resources of at least one inference server dynamically fed back from the inference resource pool, maximizing the use of inference resources such as GPUs. This avoids triggering flow control and prevents underutilization of inference resources, thus improving resource utilization. Furthermore, by fully utilizing inference resources such as GPUs, it increases the number of concurrent users during peak traffic periods, reduces the number of users experiencing service timeouts or unavailability, and improves the service experience.

[0080] To make the technical solution of this application clearer and easier to understand, the system architecture of this application will be introduced below with reference to the accompanying drawings.

[0081] Referring to Figure 2, which illustrates the architecture of a scheduling system 20, the scheduling system 20 includes an inference routing device 202 and an inference resource pool 204. The inference resource pool 204 includes multiple inference servers 2042. Each inference server 2042 includes multiple accelerators for running language models to provide inference services. These accelerators may include, but are not limited to, GPUs or NPUs. Alternatively, the inference server 2042 may use multiple accelerators for running language models to provide inference services.

[0082] The inference routing unit 202 is used to obtain a first inference request. The first inference request requests at least one inference server 2042 from a plurality of inference servers 2042. The inference routing unit 202 is also used to check the remaining resources of at least one inference server 2042 in the inference resource pool 204 when the number of pending inference requests is greater than or equal to a threshold defined in the flow control policy. The remaining resources include the amount or proportion of remaining resources of the accelerators in the inference server 2042. In some examples, the remaining resources may include the remaining video memory of the GPU.

[0083] The inference routing device 202 is further configured to predict the resource consumption of the first inference request. Resource consumption may include the amount of resource consumed or the resource consumption rate. The inference routing device 202 is also configured to, when the resource consumption is less than the remaining resources, determine the target inference server corresponding to the first inference request and schedule the first inference request to the target inference server. Here, "resource consumption less than remaining resources" can mean that the amount of resource consumed is less than the amount of remaining resources, or that the resource consumption rate is less than the proportion of remaining resources. The target inference server is one or more inference servers 2042 in the inference resource pool 204 whose remaining resources satisfy the first inference request, and the language model deployed on the target inference server matches the first inference request. Further, the inference routing device 202 can also check the inference capacity of at least one inference server in the inference resource pool 204. Inference capacity indicates the number of input tokens that can be carried for the inference request. Accordingly, the inference routing device 202 can determine the target inference server corresponding to the first inference request based on the resource consumption of the first inference request, the remaining resources of at least one inference server 2042 in the inference resource pool 204, and the inference capacity, and schedule the first inference request to the target inference server.

[0084] In this system, inference servers 2042 within the inference resource pool 204 are deployed with the same inference framework. The inference framework enables resource detection and inference capacity calculation. Resource detection may include, but is not limited to, GPU memory detection. The inference routing device 202 can obtain the remaining resources of at least one inference server 2042 in the inference resource pool 204 through the inference framework. For example, the inference routing device 202 can call the resource detection interface of the inference framework to obtain the remaining resources of at least one inference server 2042 in the inference resource pool 204. Furthermore, the inference framework can also determine the inference capacity based on the mapping between tokens and resources and the remaining resources. In some examples, the mapping between tokens and resources can be the mapping between tokens and GPU memory. The inference routing device 202 can call the inference capacity acquisition interface of the inference framework to obtain the inference capacity of at least one inference server in the inference resource pool 204.

[0085] In some possible implementations, considering that some tasks may involve duplicate inference, the inference routing device 202 can also identify a second inference request that duplicates the first inference request and cancel the second inference request's call to inference resources to avoid resource waste. Specifically, the inference routing device 202 is also used to add a first tag to the first inference request according to the task type, determine the similarity between the first inference request and the second inference request in the request scheduling queue that is marked with the first tag, and remove the second inference request from the request scheduling queue when the similarity is greater than a threshold value. The task type may include, but is not limited to, code continuation or text continuation.

[0086] The inference routing device 202 includes a remaining resource aggregation module 2022, a traffic prediction module 2024, and a scheduling module 2026. The remaining resource aggregation module 2022 checks the remaining resources of at least one inference server 2042 in the inference resource pool 204. The traffic prediction module 2024 obtains a first inference request and predicts its resource consumption or resource consumption rate. The scheduling module 2026 schedules the first inference request to a target inference server when the resource consumption is less than the remaining resource amount or the resource consumption rate is less than the proportion of remaining resources.

[0087] In some possible implementations, the inference routing device 202 may further include a compression module 2028. The remaining resource aggregation module 2022 is further configured to determine the total remaining resources of the inference resource pool 204 based on the remaining resources of the inference server 2042 in the inference resource pool 204. The traffic prediction module 2024 is further configured to predict the inference traffic distribution for the next n periods based on the first inference request. The inference traffic distribution includes the distribution of the number of inference requests and / or the distribution of the number of input tokens for the inference requests. A token represents the smallest unit of meaning that a language model can understand and generate, and is the basic unit of the language model. Depending on the tokenization scheme used, a token can represent a word, a part of a word, or even just a character. Tokens are assigned numerical values ​​or identifiers, arranged in sequences or vectors, and input into or output from the language model; they are the language components of the language model. The compression module 2028 is configured to compress the input of the inference requests in the request scheduling queue when the total remaining resources are less than a first value and the inference traffic distribution meets set conditions.

[0088] Specifically, the compression module 2028 can compress the input of at least one inference request in the request scheduling queue using a compression algorithm. In concrete implementation, the compression module 2028 can compress the input of at least one inference request in the request scheduling queue whose task type is a target type. Target types include code-related types or resource-intensive types. Code-related types may include, but are not limited to, code continuation. Resource-intensive types may include computational resource-intensive types and storage resource-intensive types; typical examples of resource-intensive types include code analysis and text analysis.

[0089] As shown in Figure 2, the scheduling system 20 may also include an intelligent agent 206. In some examples, the intelligent agent 206 may be based on a general LLM. In other examples, the scheduling system 20 may include different intelligent agents 206 for different services, and the intelligent agents 206 for different services may use different LLMs as their base. As shown in Figure 3, in a software development scenario, the intelligent agent 206 may include a development intelligent agent, and the base of the development intelligent agent may be a large development model. Further, the large development model may include a design model, a development model (or code model), a testing model, an operation and maintenance model, and a collaboration model, and the development intelligent agent may include a development question-answering intelligent agent, a code intelligent agent, a testing intelligent agent, an operation and maintenance intelligent agent, and a collaboration intelligent agent. Among them, the development question-answering intelligent agent is based on the design model and realizes functions such as requirement decomposition, architecture design, development document generation, and interface specification generation. The code intelligent agent is based on the development model or the code model and realizes functions such as code generation, code translation, code explanation, code commenting, code debugging, code inspection, code review, code optimization, and unit test generation. The testing agent, based on a large testing model, implements functions such as test design, test script generation, test log recording, and test sequence generation. The operations agent, based on a large operations model, implements functions such as log tracking and issue ticket analysis. The collaborative agent, based on a large collaborative model, implements functions such as tool collaboration and code interpreter. Agent 206 is used to preprocess the first inference request to obtain the processed first inference request.

[0090] Preprocessing can also be referred to as preprocessing. In practice, the raw input data can be unstructured data, including but not limited to text, images, and speech. The purpose of data preprocessing is to transform the raw data, such as unstructured data like images, speech, and text, into tensor data suitable for network input. Furthermore, preprocessing can also include removing irrelevant information, recovering useful real-world information, enhancing the detectability of relevant information, and simplifying the data to the maximum extent possible, thereby improving the reliability of the model's feature extraction, image segmentation, matching, and recognition.

[0091] In some possible implementations, preprocessing may include feature encoding, data normalization, and / or outlier handling. Feature encoding involves encoding the raw data describing features into numbers for input into the model. Feature encoding can be achieved through discretization, ordinal encoding, one-hot encoding, binary encoding, etc. Data normalization modifies data values ​​to achieve a common scale without altering their correlation, thereby eliminating the influence of dimensional differences between data indicators. Data normalization may include, but is not limited to, Min-Max normalization and Z-score normalization. Min-Max normalization scales data to a set range, while Z-score normalization ensures data conforms to a normal distribution. Outliers are data points that are at a certain distance from other values ​​in the data. Outlier handling refers to removing outliers; appropriately removing outliers can improve the accuracy of the model.

[0092] Furthermore, the scheduling system 20 may also include a request routing device 208. The request routing device 208 is used to route the first inference request to the agent 206 corresponding to the first inference request. The request routing device 208 can identify the service type based on the first inference request, and determine the agent 206 corresponding to the first inference request based on the service type of the first inference request and the service type of the agent 206.

[0093] It should be noted that the scheduling system 20 can also compress the input of the compressed inference request through the agent 206. Specifically, the compression module 2028 adds a second tag to at least one inference request in the request scheduling queue and sends the inference request tagged with the second tag to the agent 206. The second tag indicates the input of the inference request tagged with the second tag. For example, the second tag can be a compression tag. Accordingly, the compression module 2028 can receive the compressed inference request sent by the agent 206.

[0094] Similar to the compression on the inference routing device 202 side, the inference routing device 202 or the compression module 2028 in the inference routing device 202 can add a second tag to at least one inference request in the request scheduling queue whose task type is the target type, so that the agent 206 can compress it according to the second tag. The target type includes code-related types or resource-consuming types.

[0095] It should be noted that the inference routing device 202 or the scheduling module 2026 within the inference routing device 202 schedules the first inference request to the target inference server. The target inference server can generate a reply using a language model and return the reply to the user. For ease of understanding, a research and development conversation scenario is used as an example. Referring to Figure 4, a schematic diagram of a research and development conversation scenario is shown. The user can input a chat message in a chat group or chat dialogue interface. The client can generate a first inference request based on the chat message. The first inference request can be preprocessed by the intelligent agent 206, which can process the first inference request into a query. Then, the inference routing device 202 can route the query to the target inference server. The target inference server can generate a reply using a fine-tuned large language model (LLM) and return the reply to the client. When the reply passes through the intelligent agent 206, the intelligent agent 206 can also perform post-processing on the reply. The client can present the reply or the post-processed reply to the user. In a specific implementation, the client can present the reply to the user in a chat group.

[0096] Post-processing can include continuous data discretization, data visualization, or widening the prediction range. Continuous data discretization refers to the process where models like LLM are actually used to predict discrete data. If the model is implemented using continuous data output, such as a regression model, then the output needs to be rounded, rounded up, or rounded down to obtain the actual result. For example, when a model is used to predict the quantity of goods, the result obtained through regression is a continuous value. Agent 206 can round this continuous value to obtain a discrete value. Data visualization can involve graphically or tabulating the data to find relationships between the data and thus determine the next analysis strategy. In some cases, regression models fail to predict very large or very small values, with results concentrated in the middle range. Therefore, the agent can multiply values ​​deviating from the normal range by a coefficient to amplify the data on both sides, obtaining a more accurate prediction result. For example, for hospital laboratory data, the agent can amplify values ​​deviating from the normal range to detect anomalies.

[0097] It should be noted that Figure 2 is only a schematic architecture of the scheduling system 20. In actual applications, the scheduling system 20 can also be divided in other ways, and this application does not limit this.

[0098] Based on the scheduling system 20 shown in Figure 2, this application also provides an inference request scheduling method. The inference request scheduling method of this application will be described in detail below with reference to the accompanying drawings.

[0099] Referring to Figure 5, a flowchart of an inference request scheduling method is shown. This method can be executed by a scheduling system 20. The scheduling system 20 includes an inference routing device 202 and an inference resource pool 204. The inference resource pool 204 includes multiple inference servers 2042. Each inference server 2042 includes multiple accelerators for running language models to provide inference services, or uses multiple accelerators for running language models to provide inference services. The method includes the following steps:

[0100] S502, the inference routing device 202 obtains the first inference request. When the number of inference requests to be processed is greater than or equal to the threshold defined in the flow control policy, S504 is executed.

[0101] The first inference request is used to request at least one inference server from a pool of multiple inference servers to perform inference. In different business scenarios, the first inference request is used to request the inference server to execute different inference tasks. These inference tasks can include, but are not limited to, R&D-related inference tasks such as code generation and code annotation, as well as inference tasks from other business scenarios.

[0102] The first inference request can be generated when the user enters text or code on the interactive interface to trigger an inference operation, or when the user clicks a relevant control to trigger an inference operation. When an inference operation is triggered, the user-side client can generate the first inference request based on the text, code, or trigger event of the relevant control entered by the user. The inference routing device 202 can receive the first inference request sent by the client.

[0103] Furthermore, when the scheduling system 20 includes an agent 206, the first inference request sent by the client can be routed to the agent 206 for preprocessing. The inference routing device 202 can receive the first inference request preprocessed by the agent 206.

[0104] The inference routing device 202 can count the number of pending inference requests. For example, the inference routing device 202 can add the first inference request to the request scheduling queue and obtain the number of pending inference requests through the request scheduling queue. It should be noted that pending inference requests refer to requests to be processed by the inference server 2042. The first inference request after preprocessing by the agent 206 can still be regarded as a pending inference request.

[0105] The flow control policy defines a threshold for triggering flow control, which can be the maximum concurrent number of inference requests. Typically, flow control is triggered when the number of pending inference requests is greater than or equal to the threshold. Considering that some inference requests consume fewer inference resources, triggering flow control could lead to resource waste. Therefore, the inference routing device 202 can execute S504 to check the remaining resources of at least one inference server 2042 in the inference resource pool 204. If sufficient resources remain, the threshold can be updated to avoid resource waste caused by triggering flow control.

[0106] S504, the inference routing device 202 checks the remaining resources of at least one inference server 2042 in the inference resource pool 204.

[0107] The remaining resources include the amount or proportion of remaining resources of the accelerators in the inference server 2042. Accelerators may include, but are not limited to, GPUs, NPUs, and TPUs; for ease of description, this application uses a GPU as an example. Accordingly, the remaining resources may include the remaining video memory of the GPU in the inference server 2042.

[0108] In specific implementation, each inference server 2042 in the inference resource pool 204 is deployed with an inference framework, which can detect the remaining resources of the inference server 2042. For example, the inference framework can detect the remaining video memory of the GPU in the inference server 2042. Specifically, when detecting resources, the inference framework can detect the remaining resources of each accelerator in the inference server 2042 separately, thus obtaining fine-grained remaining resource information. This fine-grained remaining resource information can be the remaining resource information of each accelerator in the inference server 2042. In this way, the inference routing device 202 can obtain the remaining resources of the inference server 2042 through the inference framework.

[0109] S506, the inference routing device 202 predicts the resource consumption of the first inference request. If the resource consumption is less than the remaining resources, S508 is executed.

[0110] Specifically, the inference routing device 202 can predict the resource consumption of the first inference request based on the mapping relationship between tokens and resources. Here, resource consumption can be either the amount of resource consumption or the resource consumption rate. Specifically, the inference routing device 202 can obtain the number of input tokens based on the first inference request, query the mapping relationship between tokens and resources based on the number of input tokens, and obtain the predicted result of the resource consumption amount or resource consumption rate of the first inference request. The predicted result can be the resource consumption amount or resource consumption rate matching the number of input tokens.

[0111] When the resource consumption is less than the remaining resource amount, or the resource consumption rate is less than the remaining resource ratio, it means that the inference resource pool 204 can meet the resource requirements of the first inference request, and the inference routing device 202 can execute S508 to schedule the first inference request.

[0112] It should be noted that checking the remaining resource information of at least one inference server 2042 in the inference resource pool 204 and predicting the resource consumption or resource consumption rate of the first inference request can be performed in parallel or sequentially according to a set order. This application does not impose any restrictions on this.

[0113] S508, the inference routing device 202 determines the target inference server corresponding to the first inference request.

[0114] S510, the inference routing device 202 schedules the first inference request to the target inference server.

[0115] The target inference server is one or more inference servers 2042 whose remaining resources in the inference resource pool 204 satisfy the first inference request, and the language model deployed on the target inference server matches the first inference request. For example, if the resource consumption rate of the first inference request is 10%, the remaining resource ratio of one inference server 2042 in the inference resource pool 204 is 12%, and the language model deployed on that inference server 2042 matches the first inference request (e.g., the language model is used to process the target business, and the first inference request is a request for the target business), then the inference routing device 202 can determine that the inference server 2042 is the target inference server corresponding to the first inference request, and schedule the first inference request to the target inference server to fully utilize the resources of the inference server 2042 as much as possible, thereby increasing the number of concurrent users.

[0116] The first inference request may include one or more inference requests. When scheduling, the inference routing device 202 can schedule a single inference request or combine multiple inference requests. For example, if the resource consumption rate of inference request A is 10%, the resource consumption rate of inference request B is 8%, and the remaining resource ratio of an inference server 2042 is 18%, then the inference routing device 202 can schedule inference request A and inference request B to the inference server 2042.

[0117] In some possible implementations, the inference routing device 202 can schedule inference requests by combining the language model corresponding to the service. Specifically, the inference routing device 202 can identify the service type of the first inference request and the service type of the language model deployed on at least one inference server 2042 in the inference resource pool 204. The inference routing device 202 can determine the target inference server whose remaining resources can satisfy the first inference request from the inference servers 2042 that match the service type of the first inference request. Based on this, the target inference server can be an inference server 2042 whose deployed language model's service type matches the service type of the first inference request and whose remaining resources can satisfy the first inference request.

[0118] The inference routing device 202 can also adaptively update the thresholds defined in the traffic control policy. For example, when resource consumption is less than the remaining resources, the inference routing device 202 can increase the thresholds defined in the traffic control policy to schedule the first inference request to the target inference server. This changes the quantitative configuration of inference capability to dynamic self-adjustment, ensuring maximum utilization of inference resources such as GPUs and satisfying the inference requests of the largest number of multi-tenant users.

[0119] In some possible implementations, the inference routing device 202 can also determine the total remaining resources of the inference resource pool 204 based on the remaining resources of the inference server 2042 in the inference resource pool 204. Furthermore, the inference routing device 202 can predict the inference traffic distribution for the next n periods based on the first inference request. This inference traffic distribution includes the distribution of the number of inference requests and / or the distribution of the number of input tokens for each inference request. The prediction of the inference traffic distribution for the next n periods can be made based on a modeled inference traffic distribution model. The traffic distribution model takes the inference traffic distribution of the current period as input and the inference traffic distribution for the next n periods as output. Here, n can be a positive integer. Alternatively, when the traffic distribution is periodic, the inference routing device 202 can obtain the inference traffic distribution for the current period based on the inference traffic distribution of historical periods corresponding to the current period. For example, if the inference traffic distribution has a daily cycle, the inference routing device 202 can predict the inference traffic distribution for today's 14:00-15:00 based on the inference traffic distribution from yesterday's 14:00-15:00.

[0120] When the total remaining resources are less than a first value and the inference traffic distribution meets the set conditions, the inference routing device 202 can compress the input of inference requests in the request scheduling queue to perform peak shaving. The set conditions can be that the inference traffic reaches a set value for m consecutive periods. m can be a positive integer. When the inference traffic reaches the set value for m consecutive periods, it indicates that a large number of inference requests are continuously being added to the request scheduling queue, allowing for traffic peak shaving to ensure overall performance. The compression process is described in detail below.

[0121] In some possible implementations, the inference routing device 202 compresses the input of at least one inference request in the request scheduling queue using a compression algorithm. Considering that some inference tasks are duplicated, the inference routing device 202 can cancel duplicate inference tasks to avoid resource waste. For example, the inference routing device 202 compresses the input of at least one inference request in the request scheduling queue whose task type is a target type, where the target type includes code-related types or resource-intensive types. For example, the target type may include code continuation.

[0122] In some possible implementations, the inference routing device 202 may add a second tag to at least one inference request in the request scheduling queue, and send the inference request tagged with the second tag to the agent 206. The second tag is used to indicate the input of the inference request tagged with the second tag. Then, the inference routing device 202 may receive the compressed inference request sent by the agent 206.

[0123] Furthermore, the inference routing device 202 can also check the inference capacity of at least one inference server 2042 in the inference resource pool 204. Inference capacity indicates the number of input tokens that can be carried for an inference request. Accordingly, the inference routing device 202 can determine the target inference server corresponding to the first inference request based on the resource consumption of the first inference request, the remaining resources of at least one inference server 2042 in the inference resource pool 204, and the inference capacity. For example, the target inference server can be an inference server 2042 whose remaining resources can satisfy the first inference request, whose deployed language model matches the first inference request, and whose inference capacity is greater than or equal to the number of input tokens for the first inference request. The inference routing device 202 can determine the target inference server from among the inference servers 2042 that match the service type of the first inference request, whose remaining resources satisfy the first inference request and whose inference capacity is greater than the number of input tokens.

[0124] As described above, the inference request scheduling method of this application constructs a scheduling scheme based on dynamic feedback of remaining resources to maximize the use of inference resources such as GPUs. This avoids triggering flow control, preventing underutilization of inference resources and improving resource utilization. Furthermore, by fully utilizing inference resources such as GPUs, the number of concurrent users during peak traffic periods can be increased, while the number of users experiencing service timeouts or unavailability can be reduced, thus improving the service experience. In addition, this method adapts the quantitative configuration of inference capacity to dynamic self-adjustment through adaptive threshold updates, ensuring maximum utilization of inference resources such as GPUs and satisfying the inference requests of the largest number of multi-tenant users.

[0125] To facilitate understanding, the inference request scheduling method of this application will be introduced below in conjunction with specific application scenarios.

[0126] Referring to Figure 6, which illustrates an application scenario of an inference request scheduling method, an inference resource pool 204 is pre-built. Inference servers 2042 within this pool deploy AI models. These servers can deploy different large models to meet diverse business needs. In the example in Figure 6, 50% of the inference servers in the pool deploy large model A, and the remaining 50% deploy large model B. Each inference server also deploys an inference framework. The inference framework can statistically analyze the remaining GPU memory of each inference server 2042 and return the results to the inference routing device 202 to assist in scheduling decisions. The inference routing device 202 includes a remaining resource aggregation module 2022, a traffic prediction module 2024, a scheduling module 2026, and a compression module 2028. The inference routing device 202 can predict the resource consumption or resource consumption rate of inference requests and schedule the requests to the target inference server based on the prediction results.

[0127] The scheduling process will be explained in detail below.

[0128] As shown in step 1 of Figure 6, the client responds to the user-triggered inference operation by generating an inference request. This inference request can be a code generation request or a code continuation request; this application does not impose any restrictions on this. The raw data in the inference request can be unstructured data. Then, referring to step 2 of Figure 6, the agent 206 obtains the inference request and preprocesses the raw data in the inference request into a format acceptable to the AI ​​model. The preprocessed data can be structured data. In this example, the structured data includes task, variable, context, and result check fields, as well as their corresponding field values. Then, as shown in step 3 of Figure 6, the agent 206 can send the processed inference request to the inference routing device 202.

[0129] Referring again to step 4 in Figure 6, the inference routing device 202 operates synchronously and checks each inference server 2042 in the inference resource pool 204 in real time, excluding inference servers 2042 that are under expansion or have failed, and monitoring the remaining GPU memory of other inference servers 2042. The inference routing device 202 calculates the GPU memory consumption rate of inference requests. When the GPU memory consumption rate is within an acceptable range, for example, if the GPU memory consumption rate is less than the remaining GPU memory of the inference server 2042, the inference routing device 202 can schedule the inference request to that inference server 2042.

[0130] Specifically, the inference routing unit 202 can add inference requests to a request scheduling queue and schedule multiple requests in the queue to utilize GPU resources as fully as possible based on the remaining GPU memory. For code continuation requests in the request scheduling queue, if the inference routing unit 202 detects inference requests with high similarity, it can cancel earlier inference requests to avoid resource waste.

[0131] It should be noted that the inference routing device 202 can also construct a water level model for the resources of each inference server 2042. When the water level model indicates that the remaining total GPU memory is less than 30%, and a large number of inference requests are continuously being added to the request scheduling queue, the inference routing device 202 initiates peak shaving. Specifically, the inference routing device 202 can add compression tags to code-related requests (such as code continuation requests) and resource-intensive inference requests (such as code analysis requests), and then return the inference requests to the agent 206. The agent 206 can segment or compress the inference requests according to the compression tags, reducing the number of tokens for prompts on large input models and reducing GPU resource consumption. Furthermore, when the remaining total GPU memory is greater than or equal to 30%, the inference routing device 202 can cancel peak shaving to ensure inference performance.

[0132] Based on the aforementioned reasoning request scheduling method, this application also provides a scheduling system 20. The structure of the scheduling system 20 will be described in detail below with reference to the accompanying drawings.

[0133] Referring to Figure 2, a schematic diagram of a scheduling system 20 is shown. The scheduling system 20 includes an inference routing device 202 and an inference resource pool 204. The inference resource pool 204 includes multiple inference servers 2042. Each inference server 2042 includes multiple accelerators for running language models to provide inference services, or the inference server 2042 uses multiple accelerators for running language models to provide inference services.

[0134] The inference routing device 202 is used to obtain a first inference request, which requests at least one inference server among a plurality of inference servers; when the number of inference requests to be processed is greater than or equal to a threshold defined in the flow control policy, it checks the remaining resources of at least one inference server in the inference resource pool; and predicts the resource consumption of the first inference request.

[0135] The inference routing device 202 is further configured to determine the target inference server corresponding to the first inference request when the resource consumption is less than the remaining resources, and schedule the first inference request to the target inference server; wherein, the target inference server is one or more inference servers in the inference resource pool 204 whose remaining resources satisfy the first inference request, and the language model deployed on the target inference server matches the first inference request.

[0136] For example, the aforementioned inference routing device 202 can be implemented in hardware or in software.

[0137] When implemented in software, the inference routing device 202 can be an application running on a computer device, such as a computing engine. This application can also be virtualized and provided to users as a virtualization service. Virtualization services can include virtual machine (VM) services, bare metal server (BMS) services, or container services. Specifically, a VM service can be a service that uses virtualization technology to create a pool of virtual machine (VM) resources on multiple physical hosts, providing VMs for users to use on demand. A BMS service is a service that uses virtualization technology to create a pool of BMS resources on multiple physical hosts, providing BMS for users to use on demand. A container service is a service that uses virtualization technology to create a pool of container resources on multiple physical hosts, providing containers for users to use on demand. A VM is a simulated virtual computer, that is, a logical computer. A BMS is a scalable, high-performance computing service with computing performance indistinguishable from traditional physical machines, featuring secure physical isolation. A container is a kernel virtualization technology that provides lightweight virtualization to isolate user space, processes, and resources. It should be understood that the VM service, BMS service, and container service mentioned above are merely specific examples. In practical applications, virtualization services can also include other lightweight or heavyweight virtualization services, which are not specifically limited here.

[0138] When implemented in hardware, the inference routing device 202 may include at least one computing device, such as a server. Alternatively, the inference routing device 202 may be implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0139] It should be noted that the inference resource pool 204 and the inference server 2042 within it can be implemented in hardware or software. When the inference server 2042 is implemented in hardware, it can be deployed as an inference service using computing devices such as servers. Correspondingly, the inference resource pool 204 can be a resource pool formed by multiple inference servers 2042 in hardware form. When the inference server 2042 is implemented in software, it can be server software that provides inference capabilities, which can be deployed on computing devices to provide inference services.

[0140] In some possible implementations, the inference routing device 202 is also used for:

[0141] Add a first tag to the first inference request based on the task type;

[0142] Determine the similarity between the first inference request and the second inference request in the request scheduling queue that is marked with the first label;

[0143] When the similarity is greater than a threshold, the second inference request is removed from the request scheduling queue.

[0144] In some possible implementations, the task type includes code continuation or text continuation.

[0145] In some possible implementations, the inference routing device 202 is also used for:

[0146] Determine the total amount of remaining resources in the inference resource pool based on the remaining resources of the inference servers in the inference resource pool;

[0147] Based on the first inference request, predict the inference traffic distribution for the next n periods. The inference traffic distribution includes the distribution of the number of inference requests and / or the distribution of the number of input tokens for the inference requests.

[0148] When the total remaining resources are less than the first value and the inference traffic distribution meets the set conditions, the input of inference requests in the request scheduling queue is compressed.

[0149] In some possible implementations, the inference routing device 202 is specifically used for:

[0150] The input of at least one inference request in the request scheduling queue is compressed using a compression algorithm; or,

[0151] A second tag is added to at least one inference request in the request scheduling queue, and the inference request marked with the second tag is sent to agent 206. The second tag is used to indicate the input of the inference request marked with the second tag.

[0152] Receive the compressed inference request sent by agent 206.

[0153] In some possible implementations, the inference routing device 202 is specifically used for:

[0154] The input of at least one inference request with a task type of target type in the request scheduling queue is compressed using a compression algorithm. The target type includes code-related types or resource-intensive types; or...

[0155] The inference routing device 202 is specifically used for:

[0156] Add a second tag to at least one inference request in the request scheduling queue whose task type is a target type, where the target type includes code-related types or resource-consuming types.

[0157] In some possible implementations, the inference routing device 202 is also used for:

[0158] Check the inference capacity of at least one inference server 2042 in the inference resource pool 204. The inference capacity is used to indicate the number of input tokens that can be used to handle inference requests.

[0159] The inference routing device 202 is specifically used for:

[0160] Based on the resource consumption of the first inference request, the remaining resources and inference capacity of at least one inference server 2042 in the inference resource pool 204, the target inference server corresponding to the first inference request is determined.

[0161] In some possible implementations, the inference routing device 202 is also used for:

[0162] Update the thresholds defined in the flow control policy.

[0163] Specifically, the inference routing device 202 can increase the threshold defined in the flow control policy when the number of pending inference requests exceeds the threshold defined in the flow control policy and the remaining resources in the inference resource pool exceed the resource consumption of the first inference request. In this way, the first inference request can be scheduled to the inference server that originally triggered flow control, thereby changing the quantitative configuration of inference capability to dynamic self-adjustment, ensuring maximum use of inference resources such as GPUs, and satisfying the inference requests of the maximum number of multi-tenant users.

[0164] This application also provides a computing device 700. As shown in FIG7, the computing device 700 includes: a bus 702, a processor 704, a memory 706, and a communication interface 708. The processor 704, the memory 706, and the communication interface 708 communicate with each other via the bus 702. The computing device 700 may be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0165] Bus 702 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one line is used in Figure 7, but this does not imply that there is only one bus or one type of bus. Bus 702 can include pathways for transmitting information between various components of computing device 700 (e.g., memory 706, processor 704, communication interface 708).

[0166] Processor 704 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0167] The memory 706 may include volatile memory, such as random access memory (RAM). The memory 706 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD). The memory 706 stores executable program code, which the processor 704 executes to implement the aforementioned inference request scheduling method. Specifically, the memory 706 stores instructions for the scheduling system 20 to execute the inference request scheduling method. For example, the memory 706 may store instructions for implementing the functions of the inference routing device 202 or the inference server 2042. The inference routing device 202 may include a residual resource aggregation module 2022, a traffic prediction module 2024, and a scheduling module; further, the inference routing device 702 may also include a compression module 2028. The instructions stored in the memory 706 for implementing the functions of the inference server 2042 may include a stored inference framework and a language model. It should be noted that the memory 706 can also store instructions for implementing the intelligent agent 206 and the request routing device 208.

[0168] The communication interface 708 uses transceiver modules, such as, but not limited to, network interface cards and transceivers, to enable communication between the computing device 700 and other devices or communication networks.

[0169] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0170] As shown in Figure 8, the computing device cluster includes at least one computing device 700. The memory 706 of one or more computing devices 700 in the computing device cluster may store the same scheduling system 20 for executing instructions of the inference request scheduling method.

[0171] In some possible implementations, one or more computing devices 700 in the computing device cluster can also be used to execute some of the instructions used by the scheduling system 20 to execute the inference request scheduling method. In other words, a combination of one or more computing devices 700 can jointly execute the instructions used by the scheduling system 20 to execute the inference request scheduling method.

[0172] It should be noted that the memory 706 in different computing devices 700 in the computing device cluster can store different instructions for executing some functions of the scheduling system 20.

[0173] Figure 9 illustrates one possible implementation. As shown in Figure 9, computing device 700A and multiple computing devices 700B are connected via communication interface 708. The memory in computing device 700A stores instructions for performing the functions of the inference routing device 202. The memory in computing device 700B stores instructions for performing the functions of the inference server 2042. For example, the memory in computing device 700B can store inference frameworks and language models. In other words, the memory 706 of computing devices 700A and 700B jointly stores instructions from the scheduling system 20 for performing inference request scheduling methods. Further, the computing device cluster may also include 700C and computing device 700D. The memory in computing device 700C stores instructions for performing the functions of the agent 206. The memory in computing device 700D stores instructions for performing the functions of the request routing device 208.

[0174] The connection method between the computing device clusters shown in Figure 9 can be considered because the inference request scheduling method provided in this application requires a lot of resources for routing inference requests. Therefore, it is considered that the functions implemented by the inference routing device 202 are executed by independent computing devices, such as computing device 700A, and the functions of the inference resource pool 204 are executed by multiple computing devices 700B.

[0175] It should be understood that the functions of computing device 700A shown in Figure 9 can also be performed by multiple computing devices 700. Similarly, the functions of computing devices 700B, 700C, and 700D can also be performed by multiple computing devices 700.

[0176] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 10 illustrates one possible implementation. As shown in Figure 10, computing device 700E and multiple computing devices 700F are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 706 in computing device 700E stores instructions for performing the functions of the inference routing device 202. Simultaneously, the memory in computing device 700F stores instructions for performing the functions of the inference server 2042. In some examples, the computing device cluster may also include 700G and computing device 700H. The memory in computing device 700G stores instructions for performing the functions of the agent 206. The memory in computing device 700H stores instructions for performing the functions of the request routing device 208.

[0177] The connection method between the computing device clusters shown in Figure 10 can be considered as follows: considering that the inference request scheduling method provided in this application requires a large amount of resources for analysis to determine which inference server(s)2042 to route the inference request, the function implemented by the inference routing device 202 is considered to be executed by an independent computing device, such as computing device 700E, and the function implemented by the inference resource pool 204 is executed by computing device 700F.

[0178] It should be understood that the functions of computing device 700E shown in Figure 10 can also be performed by multiple computing devices 700. Similarly, the functions of computing devices 700F, 700G, and 700H can also be performed by multiple computing devices 700.

[0179] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned inference request scheduling method applied to the scheduling system 20.

[0180] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product runs on at least one computing device, it causes the at least one computing device to execute the aforementioned inference request scheduling method.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for scheduling inference requests, characterized in that, The method, applied to a scheduling system including an inference routing device and an inference resource pool, wherein the inference resource pool includes multiple inference servers, and each inference server includes multiple accelerators for running language models to provide inference services, or the inference server uses multiple accelerators for running language models to provide inference services, comprises: The inference routing device obtains a first inference request, which is used to request at least one of the plurality of inference servers. When the number of pending inference requests is greater than or equal to the threshold defined in the traffic control policy, the inference routing device checks the remaining resources of at least one inference server in the inference resource pool. The inference routing device predicts the resource consumption of the first inference request; When the resource consumption is less than the remaining resources, the inference routing device determines the target inference server corresponding to the first inference request and schedules the first inference request to the target inference server. The target inference server is one or more inference servers whose remaining resources in the inference resource pool meet the first inference request, and the language model deployed on the target inference server matches the first inference request.

2. The method according to claim 1, characterized in that, The method further includes: The inference routing device adds a first tag to the first inference request according to the task type; The inference routing device determines the similarity between the first inference request and the second inference request in the request scheduling queue that is marked with the first tag; When the similarity is greater than a threshold, the inference routing device removes the second inference request from the request scheduling queue.

3. The method according to claim 2, characterized in that, The task types include code continuation or text continuation.

4. The method according to any one of claims 1 to 3, characterized in that, The method further includes: The inference routing device determines the total amount of remaining resources in the inference resource pool based on the remaining resources of the inference servers in the inference resource pool. The inference routing device predicts the inference traffic distribution for the next n periods based on the first inference request. The inference traffic distribution includes the distribution of the number of inference requests and / or the distribution of the number of input tokens for the inference requests. When the total remaining resources are less than a first value and the inference traffic distribution meets the set conditions, the inference routing device compresses the input of inference requests in the request scheduling queue.

5. The method according to claim 4, characterized in that, The inference routing device compresses the input of at least one inference request in the request scheduling queue, including: The inference routing device compresses the input of at least one inference request in the request scheduling queue using a compression algorithm; or, The inference routing device adds a second tag to at least one inference request in the request scheduling queue and sends the inference request tagged with the second tag to the agent. The second tag is used to indicate the input of the inference request tagged with the second tag. The inference routing device receives the compressed inference request sent by the agent.

6. The method according to claim 5, characterized in that, The inference routing device compresses the input of at least one inference request in the request scheduling queue using a compression algorithm, including: The inference routing device compresses the input of at least one inference request with a target type in the request scheduling queue using a compression algorithm. The target type includes code-related types or resource-intensive types. The inference routing device adds a second tag to at least one inference request in the request scheduling queue, including: The inference routing device adds a second tag to at least one inference request in the request scheduling queue whose task type is a target type. The target type includes code-related types or resource-consuming types.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: The inference routing device checks the inference capacity of at least one inference server in the inference resource pool, the inference capacity being used to indicate the number of input tokens that can be carried for inference requests; The inference routing device determines the target inference server corresponding to the first inference request, including: The inference routing device determines the target inference server corresponding to the first inference request based on the resource consumption of the first inference request, the remaining resources of at least one inference server in the inference resource pool, and the inference capacity.

8. The method according to any one of claims 1 to 7, characterized in that, The method further includes: The inference routing device updates the threshold defined in the flow control policy.

9. A scheduling system, characterized in that, The scheduling system includes an inference routing device and an inference resource pool. The inference resource pool includes multiple inference servers. Each inference server includes multiple accelerators for running language models to provide inference services, or the inference server uses multiple accelerators for running language models to provide inference services. The inference routing device is configured to acquire a first inference request, the first inference request being used to request at least one inference server among the plurality of inference servers; when the number of inference requests to be processed is greater than or equal to a threshold defined in the traffic control policy, check the remaining resources of at least one inference server in the inference resource pool; and predict the resource consumption of the first inference request. The inference routing device is further configured to, when the resource consumption is less than the remaining resources, determine a target inference server corresponding to the first inference request, and schedule the first inference request to the target inference server. The target inference server is one or more inference servers in the inference resource pool whose remaining resources satisfy the first inference request, and the language model deployed on the target inference server matches the first inference request.

10. The system according to claim 9, characterized in that, The inference routing device is also used for: Add a first tag to the first inference request according to the task type; Determine the similarity between the first inference request and the second inference request in the request scheduling queue that is tagged with the first label; When the similarity is greater than a threshold, the second inference request is removed from the request scheduling queue.

11. The system according to claim 10, characterized in that, The task types include code continuation or text continuation.

12. The system according to any one of claims 8 to 11, characterized in that, The inference routing device is also used for: The total amount of remaining resources in the inference resource pool is determined based on the remaining resources of the inference server in the inference resource pool. Based on the first inference request, predict the inference traffic distribution for the next n periods, wherein the inference traffic distribution includes the distribution of the number of inference requests and / or the distribution of the number of input tokens for the inference requests; When the total remaining resources are less than a first value and the inference traffic distribution meets the set conditions, the input of inference requests in the request scheduling queue is compressed.

13. The system according to claim 12, characterized in that, The inference routing device is specifically used for: The input of at least one inference request in the request scheduling queue is compressed using a compression algorithm; or, A second tag is added to at least one inference request in the request scheduling queue, and the inference request marked with the second tag is sent to the agent. The second tag is used to indicate the input of the inference request marked with the second tag. Receive the compressed inference request sent by the intelligent agent.

14. The system according to claim 13, characterized in that, The inference routing device is specifically used for: The input of at least one inference request with a task type of target type in the request scheduling queue is compressed using a compression algorithm. The target type includes code-related types or resource-intensive types; or... The inference routing device is specifically used for: Add a second tag to at least one inference request in the request scheduling queue whose task type is a target type, where the target type includes code-related types or resource-consuming types.

15. The system according to any one of claims 9 to 14, characterized in that, The inference routing device is also used for: Check the inference capacity of at least one inference server in the inference resource pool, the inference capacity being used to indicate the number of input tokens that can be carried for inference requests; The inference routing device is specifically used for: The inference routing device determines the target inference server corresponding to the first inference request based on the resource consumption of the first inference request, the remaining resources of at least one inference server in the inference resource pool, and the inference capacity.

16. The system according to any one of claims 9 to 15, characterized in that, The inference routing device is also used for: Update the threshold defined in the flow control policy.

17. A computing device cluster, characterized in that, The computing device cluster includes at least one computing device, the at least one computing device including at least one processor and at least one memory, the at least one memory storing computer-readable instructions; the at least one processor executes the computer-readable instructions to cause the computing device cluster to perform the inference request scheduling method as described in any one of claims 1 to 8.

18. A computer-readable storage medium, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the inference request scheduling method according to any one of claims 1 to 8.

19. A computer program product, characterized in that, Includes computer-readable instructions; the computer-readable instructions are used to implement the inference request scheduling method according to any one of claims 1 to 8.

Images