Flow control method and system and storage medium

Abstract

The embodiment of the invention provides a flow control method and system and a storage medium. The method is applied to a flow control system in communication connection with a large language model (LLM) processing engine, K authorization control units are deployed in the system, and each unit maintains a token bucket based on dynamically adjustable control parameters. In response to receiving the target reasoning request, the system authorizes the target reasoning request by consuming tokens from token buckets of K authorization control units. Under the condition that authorization succeeds, the target reasoning request is sent to an LLM processing engine for reasoning processing, an actual performance index corresponding to the target reasoning request is received from the LLM processing engine, and the actual performance index is used for representing the response delay condition of the LLM processing engine to the target reasoning request. And according to the actual performance index, adjusting the control parameters of at least part of the authorization control units in the K authorization control units so as to update the flow bearing capability of at least part of the authorization control units to the subsequent reasoning request.

Description

Technical Field

[0001] This specification relates to the field of Internet technology, and in particular to a flow control method, system and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely applied in various fields such as content generation, intelligent dialogue, and code assistance. In practical applications, users typically interact with the LLM processing engine by sending inference requests to obtain inference results. As the user base expands, the number of concurrent requests faced by the LLM processing engine increases daily, especially in high-concurrency scenarios. Maintaining the stability of the LLM processing engine service has become a key concern.

[0003] To avoid service unavailability due to request overload, current methods often employ rate limiting or circuit breaking strategies based on static rules to control traffic. For example, a fixed concurrency threshold or rate limit is preset. When the request traffic reaches the preset threshold, excess requests are directly rejected or circuit breaking protection is triggered to prevent system crashes.

[0004] The information in the background section is merely information known only to the inventor and does not imply that such information had entered the public domain before the date of this application, nor does it imply that it could be considered prior art in this disclosure. Summary of the Invention

[0005] This specification provides a flow control method, system, and storage medium that utilizes token buckets maintained in K authorized control units for flow control and dynamically adjusts the control parameters of the token buckets based on actual performance indicators fed back by the LLM processing engine, thereby achieving adaptive matching between the flow control strategy and the state of the LLM processing engine.

[0006] Firstly, this specification provides a flow control method applied to a flow control system communicatively connected to a Large Language Model (LLM) processing engine. The flow control system deploys K authorization control units, each maintaining a token bucket based on dynamically adjustable control parameters, where K is an integer greater than or equal to 1. The method includes: in response to receiving a target inference request, authorizing the target inference request by consuming tokens from the token buckets of the K authorization control units; if authorization is successful, sending the target inference request to the LLM processing engine for inference processing, and receiving an actual performance index corresponding to the target inference request from the LLM processing engine, the actual performance index characterizing the response latency of the LLM processing engine to the target inference request; and adjusting the control parameters of at least some of the K authorization control units according to the actual performance index to update the flow carrying capacity of the at least some authorization control units for subsequent inference requests.

[0007] In some embodiments, K is an integer greater than 1, different authorized control units are associated with different traffic priorities and correspond to different control parameters; the step of authorizing the target inference request by consuming tokens from the token buckets of the K authorized control units includes: obtaining the target traffic priority corresponding to the target inference request, determining the authorized control unit associated with the target traffic priority among the K authorized control units as the target authorized control unit, and authorizing the target inference request by consuming tokens from the token bucket of the target authorized control unit; and the step of adjusting the control parameters of at least some of the authorized control units among the K authorized control units according to the actual performance indicators includes: adjusting the control parameters of the target authorized control unit based on the actual performance indicators.

[0008] In some embodiments, the control parameters for each authorization control unit include at least one of the following: token replenishment rate, for controlling the replenishment rate of tokens in the token bucket; token bucket capacity, for controlling the upper limit of the number of tokens that can be accommodated in the token bucket; or queue length, for controlling the upper limit of the number of inference requests waiting in queue outside the token bucket.

[0009] In some embodiments, adjusting the control parameters of the target authorized control unit based on the actual performance index includes: obtaining a reference performance index corresponding to the target authorized control unit and a parameter adjustment strategy corresponding to the target authorized control unit, wherein the parameter adjustment strategy is used to indicate the magnitude of parameter adjustment; and adjusting the control parameters of the target authorized control unit based on the difference between the actual performance index and the reference performance index and the parameter adjustment strategy.

[0010] In some embodiments, different authorized control units among the K authorized control units correspond to different parameter adjustment strategies. For any first authorized control unit and second authorized control unit among the K authorized control units: if the traffic priority associated with the first authorized control unit is lower than the traffic priority associated with the second authorized control unit, then the parameter adjustment range indicated by the parameter adjustment strategy corresponding to the first authorized control unit is greater than the parameter adjustment range indicated by the parameter adjustment strategy corresponding to the second authorized control unit.

[0011] In some embodiments, when the response latency characterized by the actual performance metric is greater than the response latency characterized by the reference performance metric, the adjusted control parameters of the target authorization control unit satisfy at least one of the following: the token replenishment rate is reduced, the token bucket capacity is reduced, or the queue length is reduced; when the response latency characterized by the actual performance metric is less than the response latency characterized by the reference performance metric, the adjusted control parameters of the target authorization control unit satisfy at least one of the following: the token replenishment rate is increased, the token bucket capacity is increased, or the queue length is increased.

[0012] In some embodiments, each authorization control unit corresponds to a waiting queue. The step of authorizing the target inference request by consuming a token from the token bucket of the target authorization control unit includes: attempting to consume a token from the token bucket of the target authorization control unit; if the consumption is successful, the target inference request is determined to be successfully authorized; if the consumption fails, the target inference request is added to the waiting queue corresponding to the target authorization control unit.

[0013] In some embodiments, after adding the target inference request to the waiting queue corresponding to the target authorization control unit, the method further includes: if the waiting time of the target inference request in the waiting queue exceeds a preset time, then determining that the authorization of the target inference request has failed; and in response to the authorization failure, removing the target inference request from the waiting queue and returning a failure message to the initiator of the target inference request.

[0014] In some embodiments, the method further includes: counting the number of requests that are determined to be authorization failures by the target authorization control unit within a first time window; and controlling the target authorization control unit to enter a circuit breaker state when the number of requests exceeds a preset threshold, wherein, in the circuit breaker state, the waiting queue corresponding to the target authorization control unit does not accept new inference requests within a second time window.

[0015] In some embodiments, after controlling the target authorization control unit to enter the circuit breaker state, the method further includes: after the second time window, in response to receiving a new inference request corresponding to a traffic priority associated with the target authorization control unit, authorizing the new inference request by consuming a token from the token bucket of the target authorization control unit; if the authorization is successful, controlling the target authorization control unit to exit the circuit breaker state.

[0016] In some embodiments, the target inference request is a request initiated by a target user, and obtaining the target traffic priority corresponding to the target inference request includes: determining the target traffic priority corresponding to the target inference request based on at least one of the attribute information of the target user or the attribute information of the target inference request.

[0017] In some embodiments, the attribute information of the target user includes at least one of the following: the type of the target user, or the service level bound to the target user.

[0018] In some embodiments, the attribute information of the target inference request includes at least one of the following: inference request type, or inference request content.

[0019] In some embodiments, the target inference request is initiated by a target user, and the actual performance metrics include at least one of the following: First Character Delay Metric (TTFT), which represents the time interval between the target user initiating the target inference request and the target user receiving the first character of the inference response; or Inter-Character Delay Metric (TPOT), which represents the time interval between adjacent characters in the inference response received by the target user.

[0020] Secondly, this specification also provides a flow control system, comprising: at least one storage medium storing at least one instruction set for flow control; and at least one processor communicatively connected to the at least one storage medium, wherein, when the flow control system is running, the at least one processor reads the at least one instruction set and executes the flow control method as described in any one of the first aspects according to the instructions of the at least one instruction set.

[0021] Thirdly, this specification also provides a computer-readable non-transitory storage medium, wherein the computer-readable non-transitory storage medium stores at least one instruction set, which, when executed by at least one processor, implements the method as described in any one of the first aspects.

[0022] Other functions of the flow control methods, systems, and storage media provided in this specification will be partially listed in the following description. The inventive aspects of the flow control methods, systems, and storage media provided in this specification can be fully understood through practice or use of the methods, apparatus, and combinations described in the detailed examples below. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 A schematic diagram illustrating an application scenario of a flow control method provided according to an embodiment of this specification is shown. Figure 2 A hardware structure diagram of a computing system provided according to an embodiment of this specification is shown; Figure 3 A flowchart of a flow control method provided according to an embodiment of this specification is shown; Figure 4 A schematic diagram of a flow control method provided according to an embodiment of this specification is shown; and Figure 5 A schematic diagram of the processing flow of an authorized control unit provided according to an embodiment of this specification is shown. Detailed Implementation

[0025] The following description provides specific application scenarios and requirements for this specification, intended to enable those skilled in the art to make and use the contents of this specification. Various partial modifications to the disclosed embodiments will be apparent to those skilled in the art, and the general principles defined herein can be applied to other embodiments and applications without departing from the spirit and scope of this specification. Therefore, this specification is not limited to the embodiments shown, but rather to the widest scope consistent with the claims.

[0026] The terminology used herein is for the purpose of describing particular exemplary embodiments only and is not restrictive. For example, unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein may also include the plural forms. When used in this specification, the terms “comprising,” “including,” and / or “containing” mean that the associated integers, steps, operations, elements, and / or components are present, but do not exclude the presence of one or more other features, integers, steps, operations, elements, components, and / or groups, or that other features, integers, steps, operations, elements, components, and / or groups may be added to the system / method.

[0027] Considering the following description, these and other features of this specification, as well as the operation and function of the related components of the structure, and the economy of assembly and manufacture of the parts, can be significantly improved. All of these form part of this specification with reference to the accompanying drawings. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to limit the scope of this specification. It should also be understood that the drawings are not drawn to scale.

[0028] The flowcharts used in this specification illustrate operations implemented according to some embodiments of this specification. It should be clearly understood that the operations in the flowcharts may not be implemented in a sequential order. Instead, the operations may be implemented in reverse order or simultaneously. Furthermore, one or more additional operations may be added to the flowcharts. One or more operations may be removed from the flowcharts.

[0029] For ease of description, the terms that will appear later in this manual will be explained first.

[0030] Service Level Agreement (SLA): This refers to the service quality standards agreed upon between a service provider and a user, which define the standards for service quality, availability, response time, etc.

[0031] Time to First Token (TTFT): This measures the time it takes for the system to return the first available resource (such as a token or response) after a request has been initiated.

[0032] Inter-character latency metric (Time Per Output Token, TPOT): This metric measures the time required for the model to generate each output character (or token).

[0033] In this specification, the Large Language Model (LLM) may also be referred to simply as the Large Model. A Large Language Model is a natural language processing model based on deep learning techniques, typically with billions to hundreds of billions or even more parameters, possessing powerful language understanding and generation capabilities. Large Language Models can employ the Transformer architecture or its variants (such as GPT, BERT, etc.), which utilizes an attention mechanism to globally model sequential data, efficiently handling long-distance dependencies and thus performing excellently in natural language tasks. Large Language Models learn the statistical features and semantic relationships of language through pre-training on large-scale corpora, giving them excellent generalization capabilities. The core capabilities of Large Language Models include, but are not limited to: understanding contextual semantics, generating coherent and grammatically correct text, performing logical reasoning, and handling multi-task scenarios. Its usage typically includes two modes: direct inference and fine-tuning. In direct inference mode, the user guides the Large Language Model to generate specific outputs by designing prompts. Cue words can be task descriptions or instructions in text form, used to stimulate the semantic understanding and generation capabilities of large language models. In fine-tuning mode, large language models are further trained on small-scale datasets in specific domains to optimize their performance on specific tasks. The powerful generalization ability and flexibility of large language models make them an important tool in the field of artificial intelligence, providing efficient and accurate solutions for automated text generation and understanding.

[0034] In some embodiments, large language models can also understand and generate data from other modalities (such as visual and audio data). In this case, large language models can also be called multimodal large language models (MLLMs). MLLMs provide a richer and more natural interactive experience by integrating multiple types of input and output, such as text, images, and sound. The core advantage of MLLMs lies in their ability to process and understand information from different modalities and fuse this information to complete complex tasks. For example, MLLMs can analyze an image and generate descriptive text, or generate a corresponding image based on a text description. This cross-modal understanding and generation capability makes MLLMs widely applicable across multiple fields.

[0035] It should be noted that the key technologies of large language models can be found in the detailed description in the paper "A Survey of Large Language Models" (paper number: arXiv:2303.18223v16, published on March 11, 2025, public link: https: / / doi.org / 10.48550 / arXiv.2303.18223), and will not be repeated here.

[0036] The following section introduces the application scenarios of this manual.

[0037] With the rapid development of artificial intelligence technology, Large Language Models (LLMs) have been widely used in content generation, intelligent dialogue, and other fields. The expanding user base has led to a surge in concurrent requests for LLM inference engines, making it crucial to maintain service stability under high concurrency. Currently, the industry commonly employs static rule-based rate limiting or circuit breaking strategies to prevent overload. To maintain the stability of LLM services, this specification considers the following implementation methods: In some embodiments, a traditional fixed window counter can be used for rate limiting. This method counts the number of requests passing through within a certain time window, and rejects requests if the threshold is exceeded. However, this approach may have poor service resilience: it is difficult to cope with the sudden surge in requests common in LLM inference, and "traffic spikes" are prone to occur when switching windows, causing some requests to be mistakenly killed or protection to fail during peak periods.

[0038] In some embodiments, a leaky bucket algorithm can be used to force smoothing of traffic. By allowing requests to flow out at a constant rate, the downstream load is stabilized. However, in LLM inference scenarios, there may be an issue of increased response latency: this forced smoothing sacrifices the reasonable burst throughput potential of the inference task, significantly prolongs the first-word response wait time (TTFT) for users and the overall response cycle, and can easily cause latency-sensitive requests to time out due to long queues.

[0039] In some embodiments, a general application-layer rate limiting component can be used to dynamically adjust the allowance rate based on historical request rates. However, there may be a problem of adjustment lag due to a lack of perception capabilities: it cannot obtain the actual inference latency (such as TPOT) and system status (such as video memory usage) of the downstream LLM, and cannot respond in a timely manner when faced with scenarios with large variations in input length and drastic fluctuations in concurrency, making it difficult to guarantee the experience of high-priority users and easily causing extreme fluctuations in performance indicators.

[0040] In some embodiments, the LLM inference framework can employ a First-In, First-Out (FIFO) queue combined with static concurrency parameters for scheduling. While this treats all users equally, the lack of fine-grained safeguards leads to a severe degradation of the experience for high-priority users under high loads, failing to meet the needs of tiered service scenarios.

[0041] In summary, most of the above methods simply equate LLM inference services with ordinary API traffic, ignoring the non-linear and dynamic characteristics of latency and throughput. This static, one-size-fits-all strategy easily leads to wasted computing resources or service avalanches, making it difficult to balance resource utilization and user experience. Furthermore, these solutions lack awareness and flexibility mechanisms for core metrics such as TTFT or TPOT, and cannot perform tiered scheduling based on user latency tolerance. Due to the lack of an end-to-end dynamic feedback mechanism, the system struggles to continuously guarantee low latency and high availability in high-priority scenarios, failing to balance response speed and service quality.

[0042] To address this, this specification provides a flow control method that utilizes token buckets maintained in K authorization control units for flow control, and dynamically adjusts the control parameters of the token buckets based on actual performance indicators fed back by the LLM processing engine. This achieves adaptive matching between the flow control strategy and the state of the LLM processing engine, balancing the response speed and quality of service for inference requests.

[0043] Figure 1 A schematic diagram illustrating an application scenario of a flow control method provided according to an embodiment of this specification is shown. For example... Figure 1 As shown, the application scenario 100 may include a client 110, a flow control system 120, and an LLM processing engine 130.

[0044] Client 110 may be a terminal device or service node that sends inference requests to flow control system 120. In some embodiments, client 110 may provide an interactive interface to a user, who can trigger inference requests by inputting data such as text, code, or images. Client 110 may also be an upstream application server that automatically generates target inference requests according to preset logic.

[0045] The flow control system 120 is an intermediate system or gateway system deployed between the client 110 and the LLM processing engine 130, used to manage and schedule inference requests sent by the client 110. The flow control system 120 and the LLM processing engine 130 are connected via network communication. See also... Figure 1 The number of clients 110 can be multiple. The flow control system 120 can establish communication connections with multiple clients 110 simultaneously and concurrently receive inference requests from these multiple clients 110, thereby achieving unified access and control of concurrent traffic.

[0046] Combination Figure 1 The flow control system 120 is equipped with K authorized control units (e.g., authorized control unit 1, authorized control unit 2 to authorized control unit k in the figure). Each authorized control unit maintains an independently operating token bucket (e.g., token bucket 1, token bucket 2 to token bucket k) based on dynamically adjustable control parameters.

[0047] After receiving an inference request from the client 110, the flow control system 120 can authorize the inference request by consuming tokens from the token buckets of the K authorized control units.

[0048] Upon successful authorization, the flow control system 120 sends the inference request to the LLM processing engine 130. The LLM processing engine 130 performs inference processing on the inference request and generates corresponding actual performance metrics, which are then fed back to the flow control system 120. The flow control system 120 receives the actual performance metrics. In some embodiments, the flow control system 120 is equipped with an adjustment unit (or token bucket optimizer). The adjustment unit adjusts the control parameters of at least some of the K authorized control units based on the received actual performance metrics to update the flow carrying capacity of the at least some authorized control units for subsequent inference requests, thereby achieving flow control optimization.

[0049] The LLM processing engine 130 is a computing environment configured with a large language model for performing inference tasks. Upon receiving an inference request, the LLM processing engine 130 performs inference processing and generates the corresponding inference result. Simultaneously, the LLM processing engine 130 can return the actual performance metrics corresponding to the inference request to the flow control system 120. These actual performance metrics characterize the response latency of the LLM processing engine 130 to the inference request.

[0050] It is understood that the flow control system 120 and the LLM processing engine 130 may correspond to the same physical device or server cluster, or they may correspond to different physical devices or server clusters; this specification does not impose any restrictions on this. Furthermore, the communication connection can be implemented based on a wired network or a wireless network.

[0051] In some embodiments, the flow control system 120 can be integrated and deployed in the computing environment where the LLM processing engine 130 is located, serving as a front-end access module for the LLM processing engine 130, receiving inference requests from multiple clients 110, and performing flow control before the inference requests enter the inference computing stage.

[0052] In some embodiments, the traffic control system 120 can also be deployed independently between the client 110 and the LLM processing engine 130 as an independent gateway service or proxy node to uniformly manage the request traffic from multiple clients 110 and coordinate the data interaction between multiple clients 110 and the LLM processing engine 130.

[0053] In some embodiments, the flow control system 120 can also be deployed on the server corresponding to the client 110, running as a functional component of the server or an integrated software development kit, for unified access management and rate limiting of target inference requests from the client 110 on the server side, thereby reducing the pressure on the backend communication link before the request is forwarded to the LLM processing engine 130.

[0054] In some embodiments, the flow control system 120 can also be deployed on the server side of an application service system integrating the LLM processing engine 130. The application service system can be a service platform integrating the LLM processing engine 130. Users initiate consultation requests through applications on the client 110. The server side of this platform utilizes the inference capabilities of the integrated LLM processing engine 130 to respond to the user's consultation needs and executes the flow control method through the flow control system 120 deployed on the server side, thereby ensuring the response stability and high availability of the application service system. The client 110 can have one or more applications installed that interact with the application service system. These applications may include web browser applications, information search applications, chat applications, electronic trading applications, multimedia video applications, financial service applications, instant messaging tools, email clients, and social platform software, etc., which are not limited in the embodiments described in this specification.

[0055] It should be noted that all user data obtained in this manual has been authorized by the user and does not involve user privacy.

[0056] Figure 2 A hardware structure diagram of a computing system 200 provided according to an embodiment of this specification is shown. The computing system 200 can be used as... Figure 1 The flow control system 120 in the specification executes the flow control method described in this manual.

[0057] like Figure 2 As shown, the computing system 200 may include at least one storage medium 230 and at least one processor 220. In some embodiments, the computing system 200 may also include a communication port 250 and an internal communication bus 210. The computing system 200 may also include I / O components 260.

[0058] The internal communication bus 210 can connect to different system components. For example, the internal communication bus 210 can connect to storage medium 230, processor 220, communication port 250, and I / O component 260, etc.

[0059] I / O component 260 supports input / output between computing system 200 and other components.

[0060] Communication port 250 is used for data communication between computing system 200 and the outside world. For example, communication port 250 can be used for data communication between computing system 200 and network 140. Communication port 250 can be a wired communication port or a wireless communication port.

[0061] Storage medium 230 may include a data storage device. The data storage device may be a non-transitory storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 232, a read-only storage medium (ROM) 234, or a random access storage medium (RAM) 235. Storage medium 230 also includes at least one instruction set stored in the data storage device. The instruction set may include computer program code, which may include programs, routines, objects, components, data structures, procedures, modules, etc.

[0062] At least one processor 220 may be communicatively connected to at least one storage medium 230. When the computing system 200 is running, at least one processor 220 reads the at least one instruction set and executes the flow control method provided in this specification according to the instructions of the at least one instruction set. The processor 220 may perform the steps included in the flow control method. The processor 220 may be in the form of one or more processors. In some embodiments, the processor 220 may include one or more hardware processors, such as a microcontroller, microprocessor, reduced instruction set computer (RISC), application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), central processing unit (CPU), graphics processing unit (GPU), physical processing unit (PPU), microcontroller unit, digital signal processor (DSP), field-programmable gate array (FPGA), advanced RISC machine (ARM), programmable logic device (PLD), any circuit or processor capable of performing one or more functions, or any combination thereof.

[0063] For illustrative purposes only, the accompanying drawings show only one processor 220 for the computing system 200. However, it should be noted that the computing system 200 may also include multiple processors; therefore, the operations and / or method steps disclosed herein may be executed by one processor or by multiple processors in combination. For example, if the processor 220 of the computing system 200 described in this specification executes steps A and B, it should be understood that steps A and B may also be executed jointly or separately by two different processors 220 (e.g., a first processor executes step A, a second processor executes step B, or the first and second processors jointly execute steps A and B).

[0064] Figure 3 A flowchart of a flow control method P300 according to an embodiment of this specification is shown. As previously described, the flow control system can execute the flow control method P300 of this specification. Specifically, the processor in the flow control system can read an instruction set stored in its local storage medium and then execute the flow control method P300 of this specification according to the instructions in the instruction set. The flow control system is deployed with K authorized control units, each authorized control unit maintaining a token bucket based on dynamically adjustable control parameters, where K is an integer greater than or equal to 1. Figure 3 As shown, the flow control method P300 may include: S310: In response to receiving a target inference request, authorize the target inference request by consuming tokens from the token buckets of the K authorized control units.

[0065] The target inference request can be an instruction to instruct the LLM processing engine to perform a specific inference task. In some embodiments, the target inference request can be used to instruct the LLM processing engine to perform a natural language processing task. For example, the target inference request can be an instruction for text generation, such as writing an article or generating a summary; the target inference request can also be an instruction for code generation, i.e., generating code snippets in a specific programming language based on a natural language description; the target inference request can also be a request for logical reasoning, question-and-answer interaction, or multimodal data processing. The traffic control system is responsible for receiving inference requests initiated by users or clients.

[0066] In some embodiments, K can be 1, meaning the traffic control system deploys only one authorization control unit for global control. In this scenario, regardless of the source or type of the target inference request, token consumption and authorization processing are uniformly handled through this single authorization control unit. This scenario is suitable for situations where service levels do not need to be differentiated, and only the total traffic entering the LLM processing engine needs to be limited overall to prevent total overload.

[0067] In some embodiments, to achieve differentiated inference services, K is an integer greater than 1, different authorized control units are associated with different traffic priorities, and correspond to different control parameters. The traffic control system can first obtain the target traffic priority corresponding to the target inference request, determine the authorized control unit associated with the target traffic priority among the K authorized control units as the target authorized control unit, and authorize the target inference request by consuming tokens from the token bucket of the target authorized control unit.

[0068] In some embodiments, the traffic priority division can be pre-defined, for example, it can be preset to three levels: high, medium, and low. In some embodiments, the traffic priority division can also be dynamically determined based on the current traffic concurrency. For example, when the overall system traffic concurrency is detected to be low, only two priority levels can be divided to simplify the processing flow; while when the traffic concurrency rises sharply, the priority can be subdivided into more levels for more granular step-by-step control.

[0069] Please see Figure 4 The traffic control system may also include a gateway, which determines the target traffic priority corresponding to the target inference request. For example, when a target inference request initiated by a client arrives at the traffic control system, the gateway parses the request and determines the target traffic priority corresponding to the request based on a preset priority strategy (such as based on user attributes or request content). Assume that the target traffic priority is... Figure 4 If the gateway is associated with the authorized control unit 2 shown in the diagram, then the gateway will route the target inference request to the authorized control unit 2.

[0070] like Figure 4 As shown, the authorization control unit 2 maintains a token bucket (a grid sequence) based on dynamically adjustable control parameters. In the token bucket, gray-filled squares represent generated, available tokens, blank squares represent tokens consumed, and squares marked with an "X" represent tokens being consumed. The system attempts to consume tokens from the token bucket of the authorization control unit 2 to authorize the target inference request. If the token consumption is successful and authorization is granted, the flow control system sends the target inference request to the LLM processing engine for inference processing. Subsequently, the LLM processing engine feeds back the actual performance indicators generated from processing the request to the flow control system, and the adjustment unit adjusts the control parameters of the authorization control unit 2 accordingly, thereby achieving dynamic flow control.

[0071] In some embodiments, the mapping relationship between the traffic priority and the authorized control unit can be flexibly configured. For example, a traffic priority can be associated with one authorized control unit to achieve dedicated control. Alternatively, a traffic priority can be associated with multiple authorized control units. In this case, the traffic control system can randomly select one of the multiple authorized control units, or use load balancing strategies such as round-robin or hash algorithms to select one as the target authorized control unit, thereby supporting horizontal scaling under the same priority.

[0072] By routing target inference requests to the corresponding authorized control units (ECUs) based on their traffic priorities, resource isolation between different traffic levels can be achieved. Priority-based traffic control effectively prevents sudden fluctuations in low-priority traffic from causing resource contention for high-priority traffic, thus avoiding interference between requests of different priorities. For example, when a large number of low-priority requests flood in, they are confined to specific ECUs and do not consume the token resources of ECUs associated with high-priority requests, ensuring that high-priority requests receive stable computing resources and a fast response channel.

[0073] In some embodiments, the target inference request is a request initiated by a target user, and the target traffic priority corresponding to the target inference request can be based on at least one of the attribute information of the target user or the attribute information of the target inference request.

[0074] Traffic control systems can use static target user attribute information or dynamic target inference request attribute information as the basis for dividing traffic priorities, thereby constructing a multi-dimensional priority evaluation system and realizing refined service classification.

[0075] In some embodiments, the attribute information of the target user includes at least one of the following: the type of the target user, or the service level bound to the target user.

[0076] In some embodiments, the flow control system can perform mapping matching based on pre-stored user attribute information. For example, target users can be divided into enterprise users and individual users based on their type. The system can map requests initiated by enterprise users to high priority to meet the enterprise's requirements for high availability and low latency, and map requests initiated by individual users to normal priority.

[0077] Furthermore, based on the service level bound to the target user, users can be categorized into premium subscribers, basic users, and guest users. The traffic control system can map inference requests initiated by premium subscribers to high priority, inference requests initiated by basic users to medium priority, and inference requests initiated by guest users to low priority. In this way, when facing high traffic concurrency or resource constraints, the system can prioritize scheduling requests from premium subscribers, ensuring their request pass rate and response speed, thus reflecting differentiated service value.

[0078] In some embodiments, the service level bound to the target user can be determined based on an Service Agreement (SLA). The SLA typically specifies the service provider's commitment metrics to the user, such as availability percentage, maximum response time, etc. Different SLA levels correspond to different service priorities. Prioritizing based on SLAs ensures that the system prioritizes user requests with higher SLA levels, thereby maximizing the processing of high-SLA user requests under resource constraints.

[0079] In some embodiments, the attribute information of the target inference request includes at least one of the following: inference request type, or inference request content.

[0080] The inference request type may include information such as the version of the large model called by the inference request (e.g., a high-performance model or a lightweight model), or the interaction mode of the inference request (e.g., streaming output or non-streaming output).

[0081] In some embodiments, the traffic control system can determine the target traffic priority based on the inference request type. For example, requests that invoke a lightweight version of the model that consumes less computing resources can be given a higher priority to ensure the throughput of the LLM processing engine; or, considering that streaming output modes are typically used in real-time interactive scenarios and are more sensitive to response latency, streaming output type requests can be set to a high priority, while non-streaming output requests that have a higher tolerance for latency can be set to a low priority.

[0082] In some embodiments, the traffic control system can also determine target traffic priority by parsing the inference request content. For example, the traffic control system can determine the target traffic priority based on the length and complexity of the input text or the modality of the inference request (e.g., plain text, image generation, or code completion) according to the inference request content. The system can assign higher priority to short text requests with expected lower computational overhead to quickly free up connection resources and improve the overall throughput of the traffic control system and LLM processing engine. Alternatively, the traffic control system can identify inference requests for critical tasks containing specific keywords in the inference request content as urgent tasks, thereby increasing their priority and ensuring they are processed promptly.

[0083] In some embodiments, the traffic control system can also determine the target traffic priority by combining the inference request type and the inference request content. For example, the system can comprehensively determine the large model version invoked by the inference request and the modality of the inference request, marking online dialogue requests that invoke lightweight models and have high real-time requirements as high priority to ensure a smooth user experience; and marking background batch processing image generation requests that invoke high-performance models but are not sensitive to latency as low priority.

[0084] In some embodiments, the system can also determine the target traffic priority based on the attribute information of the target user or the attribute information of the target inference request. For example, when a high-level subscriber initiates an image generation request with extremely high computational complexity, although the user level is high, considering that the long-term resource occupation of the request may affect system performance, the system can allocate it to a medium-priority queue rather than a high-priority queue based on the attributes of the request content. Conversely, when a regular user initiates a very short text query request, the request can be elevated to a higher priority to improve the system's concurrent processing efficiency. By combining the static attribute information of the target user with the dynamic attribute information of the target inference request, the traffic control system can construct a flexible and multi-dimensional priority allocation mechanism to achieve a better traffic control method.

[0085] In some embodiments, the control parameters for each authorization control unit include at least one of the following: token replenishment rate, token bucket capacity, or queue length.

[0086] The token replenishment rate is used to control the replenishment rate of tokens in the token bucket, that is, the number of tokens added to the token bucket per unit time.

[0087] In some embodiments, the token replenishment rate of authorization control units associated with different traffic priorities can be different. To ensure service smoothness, the token replenishment rate of authorization control units associated with high traffic priorities can be greater than that of authorization control units associated with low traffic priorities. For example, the system can be configured to inject 10 tokens per second into authorization control units associated with high priorities to ensure the processing of high-priority traffic under high concurrency. It can be configured to inject 8 tokens per second into authorization control units associated with medium priorities to maintain the processing of medium-priority traffic. It can be configured to inject 5 tokens per second into authorization control units associated with low priorities to limit the resource consumption of low-priority traffic. This parameter affects the frequency at which requests are authorized. If the token replenishment rate is set higher, the number of requests allowed to pass per unit time is greater, and the throughput of the authorization control unit is also greater.

[0088] The token bucket capacity is used to control the upper limit of the number of tokens that can be held in the token bucket.

[0089] In some embodiments, the token bucket capacity of authorization control units associated with different traffic priorities can also be different. For example, for authorization control units associated with high priorities, the system can configure a larger token bucket capacity, such as 30. For authorization control units associated with low priorities, the system can configure a smaller token bucket capacity, such as 15. This parameter determines the authorization control unit's capacity to handle burst traffic. When no inference requests arrive for a period of time, tokens will continue to accumulate until the token bucket capacity limit is reached. The larger the token bucket capacity, the stronger its ability to handle burst traffic.

[0090] The queue length is used to control the upper limit of the number of inference requests waiting in line outside the token bucket.

[0091] In some embodiments, the queue lengths of authorization control units associated with different traffic priorities can also be different. For example, authorization control units associated with high priorities can be configured with longer queue lengths to minimize the probability of requests being directly rejected and improve service availability; while authorization control units associated with low priorities can be configured with shorter queue lengths to quickly discard requests when the system is congested, avoiding invalid waiting and resource waste. This parameter determines the size of requests that the authorization control unit is allowed to temporarily store and wait for processing when tokens are exhausted.

[0092] By jointly adjusting the above parameters, fine-grained management of traffic with different priorities can be achieved. High-priority requests are processed through an authorization control unit with a high replenishment rate and large-capacity token bucket, improving processing speed and reducing latency and rejection rate. Low-priority requests are processed through an authorization control unit with a low replenishment rate and small-capacity token bucket, proactively relinquishing computing resources when resources are scarce. Through differentiated configuration strategies, while ensuring a core user experience, it is possible to fully utilize computing power to process requests, improving system resource utilization efficiency and service quality stability.

[0093] In some embodiments, to provide a request buffering mechanism during peak traffic periods, each authorization control unit has a corresponding waiting queue. See also... Figure 5 In the process of authorizing the target inference request by consuming tokens from the token bucket of the target authorization control unit, the flow control system may attempt to consume tokens from the token bucket of the target authorization control unit. If the consumption is successful, the target inference request is determined to be authorized successfully. If the consumption fails, the target inference request is added to the waiting queue corresponding to the target authorization control unit.

[0094] In some embodiments, the number of tokens consumed can be fixed (e.g., 1 token per inference request) or dynamically calculated (e.g., determining the number of tokens to consume based on the length of the input data for the target inference request or the estimated computational load). If the current token balance in the token bucket is greater than or equal to the required number of tokens to be consumed, the consumption is successful, the flow control system determines that the target inference request has been successfully authorized, and sends the request to the downstream LLM processing engine for processing.

[0095] If the token bucket currently has insufficient tokens, the consumption fails. In this case, the flow control system does not immediately discard the request, but instead adds the target inference request to the waiting queue corresponding to the target authorization control unit for queuing. In some embodiments, before adding the request to the queue, the flow control system may also check whether the length of the current waiting queue has reached a preset queue length limit; if the limit has not been reached, queuing is allowed; if the limit has been reached, the request is directly rejected to prevent the waiting queue from overflowing.

[0096] In some embodiments, the waiting queues of different authorization control units can be configured with different queuing rules to adapt to the needs of different scenarios. For example, the waiting queue can be implemented based on a First-In, First-Out (FIFO) queuing rule to temporarily store inference requests that have not yet been authorized, ensuring fairness in request processing. The waiting queue can also be implemented based on a Last-In, First-Out (LIFO) rule. LIFO is suitable for authorization control units with high real-time requirements and the ability to discard old data, ensuring that the most recently arrived inference requests are processed first and avoiding the processing of expired, outdated requests.

[0097] In some embodiments, for a target inference request in the waiting queue, the flow control system can continuously or periodically monitor the token bucket status of the target authorization control unit. Once a new token is detected in the token bucket, and the current number of tokens meets the consumption requirements of the request at the head of the waiting queue, the system can retrieve the target inference request from the waiting queue according to a preset queuing rule (e.g., a first-in, first-out (FIFO) rule) and perform a token consumption operation. After successfully consuming the token, the system determines that the target inference request has been successfully authorized and then sends the target inference request to the LLM processing engine for inference processing.

[0098] Please continue reading Figure 5In some embodiments, to prevent inference requests from waiting indefinitely in the queue, leading to client timeouts or invalid use of system resources, this specification also introduces a fail-fast mechanism. Specifically, in some embodiments, if the target inference request waits in the waiting queue for more than a preset time, the authorization of the target inference request is determined to have failed. In response to the authorization failure, the traffic control system removes the target inference request from the waiting queue and returns a failure message to the initiator of the target inference request.

[0099] In some embodiments, the preset duration can be a globally uniformly configured fixed threshold (e.g., the preset duration of the waiting queue for each authorized control unit is consistent), or it can be a dynamically configured threshold differentiated according to the priority of the authorized control unit. For example, for authorized control units associated with low-priority traffic, a shorter preset duration can be set to cause them to fail quickly when resources are scarce. Conversely, for authorized control units associated with high-priority traffic, a longer preset duration can be set to increase the probability that high-priority traffic will eventually receive service.

[0100] In response to the authorization failure determination, the traffic control system will perform a cleanup and feedback operation, removing the target inference request from the waiting queue to release the memory resources occupied by the queue. Simultaneously, the system will return a failure message to the initiator of the target inference request. The failure message may include textual descriptions indicating request failure (such as "Server busy, please try again later" or "Request busy"), so that the target user can promptly understand the request processing status.

[0101] In some embodiments, this specification also provides a circuit breaker mechanism and a self-healing strategy. Specifically: the flow control system can count the number of requests that are determined to have failed authorization by the target authorization control unit within a first time window, and if the number of requests exceeds a preset threshold, control the target authorization control unit to enter a circuit breaker state. In the circuit breaker state, the waiting queue corresponding to the target authorization control unit will not accept new inference requests within a second time window.

[0102] In some embodiments, the size of the first time window may be a pre-set fixed duration (e.g., 20 seconds or 1 minute) or a variable duration that is dynamically adjusted according to the real-time load of the system or LLM processing engine.

[0103] In some embodiments, the number of authorization failure requests can be the number of authorization failures within a statistical period, or it can be the percentage of authorization failure requests in the total number of requests (i.e., the failure rate). If the number of requests or the failure rate exceeds a preset threshold, it indicates that the traffic admission rate of the authorization control unit corresponding to that priority is lagging, and the backend LLM processing engine may be under high load, preventing timely processing of inference requests. In this case, continuing to accept queued requests will lead to queue overflow or a large number of request timeouts, creating a vicious cycle. Therefore, the system can control the target authorization control unit to enter a circuit breaker state.

[0104] In the circuit breaker state, to allow recovery time for the downstream LLM processing engine, the waiting queue corresponding to the target authorization control unit will not accept new inference requests during the second time window (i.e., the circuit breaker cooldown period). During this time, for any newly arriving inference request corresponding to the target authorization control unit, the system will skip the token check and queuing steps, directly execute the rejection policy, and indicate a request failure message. By eliminating the need to read the token bucket state, enqueue operations, and subsequent polling and waiting processes, the CPU overhead and memory usage of the flow control system itself can be reduced, thereby minimizing the ineffective consumption of system resources.

[0105] In some embodiments, after controlling the target authorization control unit to enter the circuit breaker state, the flow control system also has a self-healing strategy for automatically restoring services. Specifically, after the second time window, in response to receiving a new inference request corresponding to the traffic priority associated with the target authorization control unit, the system authorizes the new inference request by consuming tokens from the token bucket of the target authorization control unit. If the authorization is successful, the system controls the target authorization control unit to exit the circuit breaker state.

[0106] If the authorization of the new inference request is successful, and the request is subsequently successfully forwarded and processed, it indicates that the token bucket of the target authorization control unit has accumulated enough tokens to handle the current traffic, and the processing capacity of the LLM processing engine has returned to normal. At this point, the traffic control system controls the target authorization control unit to exit the circuit breaker state and resume normal traffic processing.

[0107] If the new inference request still fails to grant authorization, or if the performance metrics reported by the backend still fail to meet the standards, it indicates that the overall system status has not yet recovered. In this case, the target authorization control unit will remain in a circuit breaker state.

[0108] S320: If authorization is successful, the target inference request is sent to the LLM processing engine for inference processing, and the actual performance index corresponding to the target inference request is received from the LLM processing engine. The actual performance index is used to characterize the response latency of the LLM processing engine to the target inference request.

[0109] In some embodiments, when performing inference tasks, the LLM processing engine records performance data during request processing and feeds this data back to the flow control system as actual performance indicators so that the flow control system can perceive the current processing capacity of the LLM processing engine.

[0110] In some embodiments, the target inference request is initiated by the target user, and the actual performance metrics include at least one of the following: first character delay metric TTFT or inter-character delay metric TPOT.

[0111] TTFT, or Time-To-Flight Response, characterizes the time interval between the target user initiating the target inference request and the target user receiving the first character (token) of the inference response. TTFT is a core metric for measuring the LLM (Low-Level Machine) user experience, determining how long a user needs to wait after issuing a command before seeing the first character of feedback—in other words, whether the user experiences "lag" or "waiting." TTFT typically encompasses network transmission time, request queuing time, Prompt prefill time, and the time to generate the first token. A higher TTFT often indicates a potential resource bottleneck in the LLM processing engine.

[0112] TPOT (Terminal Time Between Tokens) characterizes the time interval between adjacent characters received by the target user in the inference response. This metric reflects the average speed of the LLM processing engine during the generation phase. Since large language models typically generate tokens one by one using an autoregressive approach, TPOT determines the total time required to generate a complete response. A high TPOT generally indicates that the LLM processing engine is under excessive load and its parallel processing capability is reduced.

[0113] By using the two metrics, TTFT and TPOT, the flow control system can more comprehensively and accurately evaluate the performance of the LLM processing engine from two dimensions: response speed and generation efficiency.

[0114] S330: Based on the actual performance indicators, adjust the control parameters of at least some of the K authorized control units to update the traffic carrying capacity of the at least some authorized control units for subsequent inference requests.

[0115] In some embodiments, the flow control system may further include a token bucket optimizer. The token bucket optimizer is used to collect the actual performance metrics (such as TTFT and TPOT) from the LLM processing engine and dynamically adjust the control parameters of each authorized control unit based on the statistical results of these metrics. Based on the above metrics, embodiments of this specification construct a dynamic flow limiting feedback mechanism that incorporates the performance metrics of the LLM inference service into the flow control loop, establishing a real-time feedback link from backend inference performance to frontend traffic admission. This enables the flow control system to dynamically and adaptively adjust the token bucket parameters according to the fluctuation trends of TTFT and TPOT.

[0116] In some embodiments, the flow control system can adjust the control parameters of the target authorization control unit that processed the current request based on the actual performance indicators. Specifically, a reference performance indicator corresponding to the target authorization control unit and a parameter adjustment strategy corresponding to the target authorization control unit can be obtained. Then, based on the difference between the actual performance indicator and the reference performance indicator, and the parameter adjustment strategy, the control parameters of the target authorization control unit are adjusted, whereby the parameter adjustment strategy indicates the magnitude of the parameter adjustment.

[0117] The aforementioned targeted adjustment mechanism ensures that the system can accurately optimize the traffic carrying capacity of a specific priority channel based on the actual performance of that priority traffic, without having to blindly adjust all authorized control units.

[0118] The reference performance index can be preset, or it can be the historical average or a specific percentile value within a certain time window.

[0119] The adjustment period can be flexibly configured: it can be real-time, i.e., fine-tuning is performed after each inference request is processed and the metrics are fed back; it can also be adjusted within a specified period, such as aggregating metrics once per second for batch adjustment; or it can be a hybrid mode, such as using periodic adjustment to save computational overhead when the traffic load is low, while automatically switching to real-time adjustment to improve response accuracy when the traffic load is high.

[0120] In some embodiments, when the response latency characterized by the actual performance metric is greater than the response latency characterized by the reference performance metric, the adjusted control parameters of the target authorization control unit satisfy at least one of the following: the token replenishment rate is reduced, the token bucket capacity is reduced, or the queue length is reduced; when the response latency characterized by the actual performance metric is less than the response latency characterized by the reference performance metric, the adjusted control parameters of the target authorization control unit satisfy at least one of the following: the token replenishment rate is increased, the token bucket capacity is increased, or the queue length is increased.

[0121] Based on token generation rate ( Taking the adjustment of ) as an example, and combining it with formula (1), we will introduce how to adjust the control parameters of the target authorized control unit.

[0122] (1) in, This indicates the token generation rate adjusted based on TTFT. This indicates the token generation rate before adjustment. Indicates reference performance indicators, This represents the actual performance indicators (actual values). This represents the response coefficient, which characterizes the parameter adjustment strategy.

[0123] Similarly, the same logic can be applied to TPOT to obtain the adjusted token generation rate based on TPOT. The system can be based on and Determine the final adjusted token generation rate by taking at least one of the following (e.g., the minimum or weighted average of the two). .

[0124] When the response delay characterized by the actual performance metric is greater than the response delay characterized by the reference performance metric, for example, when the actual TTFT is greater than the target TTFT, A negative difference indicates a low token generation rate. The load on the LLM processing engine is reduced (the current limit is tightened).

[0125] When the response delay characterized by the actual performance metric is less than the response delay characterized by the reference performance metric, for example, when the actual TTFT is less than the target TTFT, A positive value indicates the token generation rate. It will automatically increase (relax the rate limit), allowing more inference requests to pass through and freeing up traffic.

[0126] In addition to the token generation rate, parameters such as the token bucket capacity and queue length can also be dynamically adjusted based on the above logic. It should be noted that the above formula is merely an exemplary adjustment formula; in practical applications, other control algorithms or nonlinear adjustment functions can also be used.

[0127] In some embodiments, different authorized control units among the K authorized control units correspond to different parameter adjustment strategies. For any first authorized control unit and second authorized control unit among the K authorized control units: if the traffic priority associated with the first authorized control unit is lower than the traffic priority associated with the second authorized control unit, then the parameter adjustment range indicated by the parameter adjustment strategy corresponding to the first authorized control unit is greater than the parameter adjustment range indicated by the parameter adjustment strategy corresponding to the second authorized control unit.

[0128] Specifically, in the formula above, the parameter adjustment range is the response coefficient. For low-priority (first authorized control unit), its corresponding The setting is relatively large. This means that low-priority buckets are more sensitive to changes in performance metrics, adjust faster, and exhibit greater resource elasticity. When system resources are scarce, low-priority buckets will tighten their rates more quickly, proactively relinquishing resources; when resources are released, they can quickly recover. For high-priority (second authorized control unit), its corresponding... The settings are relatively small. This means that the parameters of high-priority buckets are adjusted more smoothly and conservatively, avoiding drastic fluctuations in service quality due to short-term volatility, and prioritizing the stability of the experience for core users.

[0129] by Taking a value range of (0,1] as an example, for high-priority users, a smaller adjustment factor is set by default (e.g., =0.1). This means that when performance fluctuates, the tightening and loosening of the rate limiting threshold are relatively gradual, avoiding sudden service interruptions due to drastic rate limiting. For low-priority users, a larger adjustment factor is set by default (e.g., ...). = 0.8). This means that once the system detects resource shortages, it will significantly reduce the traffic of low-priority users, thereby quickly freeing up resources for high-priority users.

[0130] Furthermore, as the traffic in high-priority buckets decreases, the actual performance metrics (such as TTFT) of the LLM processing engine are optimized. At this point, the parameters of low-priority buckets are increased, leading to a higher volume of authorized traffic for low-priority requests. This flexibly achieves resource decentralization and effective utilization of idle resources. Simultaneously, during the dynamic changes based on priority requests, this mechanism ensures that low-priority traffic is automatically released while prioritizing high-priority requests, fully utilizing hardware computing power and avoiding idleness or resource contention. Through the hierarchical architecture and adaptive optimization parameter design of the k authorization control units in the embodiments of this specification, independent optimization of token buckets associated with different priorities can be supported, achieving more precise priority scheduling and tiered quality of service assurance.

[0131] In summary, the flow control method and system provided in this specification, through the deployment of K authorization control units maintaining token buckets within the flow control system, authorizes the target inference request by consuming tokens. This controls the rate at which requests enter the LLM processing engine, providing a certain level of flow buffer protection for the LLM processing engine and increasing its operational stability and reliability. Furthermore, the flow control system can utilize actual performance indicators representing response latency received from the LLM processing engine to perceive its responsiveness. It can also dynamically adjust the control parameters of the authorization control units based on actual performance indicators to update the flow carrying capacity for subsequent inference requests, thereby achieving dynamic adaptation between flow control and the inference performance of the LLM processing engine. This performance feedback mechanism breaks through the limitations of traditional static rate limiting methods. It not only tightens traffic in a timely manner to prevent service overload or crash when the LLM processing engine is under high load, but also automatically relaxes restrictions to improve resource utilization when the load is low. Thus, while ensuring high system availability, it also considers the response speed and service quality of inference requests, improving the user experience.

[0132] This specification, in another aspect, provides a computer-readable non-transitory storage medium storing at least one instruction set for flow control. When the at least one instruction set is executed by a processor, it instructs the processor to implement the steps of the flow control method P300 described herein. In some possible embodiments, various aspects of this specification may also be implemented as a program product comprising program code. When the program product is run on a computing system 200, the program code causes the computing system 200 to perform the steps of the flow control method P300 described herein. The program product for implementing the above method may employ a portable compact disc read-only memory (CD-ROM) containing program code and may run on the computing system 200. However, the program product of this specification is not limited thereto. In this specification, the readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system. The program product may employ any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media include: portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. The computer-readable storage medium may include a data signal propagated as part of a carrier wave in baseband, carrying readable program code. This propagated data signal may take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable storage medium may also be any readable medium other than a readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical fiber, RF, etc., or any suitable combination thereof. Program code for performing operations of this specification may be written in any combination of one or more programming languages, including object-oriented programming languages—such as Java, C++, etc.—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on computing system 200, partially on computing system 200, as a standalone software package, partially on computing system 200 and partially on a remote computing system, or entirely on a remote computing system.

[0133] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0134] In summary, after reading this detailed disclosure, those skilled in the art will understand that the foregoing detailed disclosure is presented by way of example only and is not restrictive. Although not explicitly stated herein, those skilled in the art will understand that this specification requires various reasonable changes, improvements, and modifications to the embodiments. These changes, improvements, and modifications are intended to be made by this specification and are within the spirit and scope of the exemplary embodiments described herein.

[0135] Furthermore, certain terms in this specification have been used to describe embodiments of this specification. For example, "an embodiment," "an embodiment," and / or "some embodiments" mean that a particular feature, structure, or characteristic described in connection with that embodiment may be included in at least one embodiment of this specification. Therefore, it is to be emphasized and understood that two or more references to "an embodiment" or "an embodiment" or "alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. Moreover, specific features, structures, or characteristics may be suitably combined in one or more embodiments of this specification.

[0136] It should be understood that in the foregoing description of the embodiments in this specification, various features are combined in a single embodiment, drawing, or description for the purpose of simplifying the description and to aid in understanding a feature. However, this does not mean that the combination of these features is necessary, and those skilled in the art, upon reading this specification, may readily identify some of the devices as separate embodiments. That is, the embodiments in this specification can also be understood as an integration of multiple secondary embodiments. And the content of each secondary embodiment is valid even if it contains fewer than all the features of a single foregoing disclosed embodiment.

[0137] Every patent, patent application, publication of a patent application, and other material such as articles, books, specifications, publications, documents, articles, etc., cited herein, except for those inconsistent with or conflicting with this document, or those having a restrictive effect on the widest scope of the claims, may be incorporated herein by reference for all purposes now or hereafter associated with this document. Furthermore, in the event of any inconsistency or conflict between the description, definition, and / or use of relevant terms in any material and the description, definition, and / or use of relevant terms in this document, the terms in this document shall prevail.

[0138] Finally, it should be understood that the embodiments disclosed herein are illustrative of the principles of the embodiments described in this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are merely examples and not limitations. Those skilled in the art can implement the applications described in this specification using alternative configurations based on the embodiments in this specification. Therefore, the embodiments in this specification are not limited to the embodiments precisely described in the applications.

Claims

1. A flow control method applied to a flow control system communicatively connected to a Large Language Model (LLM) processing engine, wherein the flow control system deploys K authorized control units, each authorized control unit maintaining a token bucket based on dynamically adjustable control parameters, wherein K is an integer greater than or equal to 1, the method comprising: In response to receiving a target inference request, the target inference request is authorized by consuming tokens from the token buckets of the K authorized control units; If authorization is successful, the target inference request is sent to the LLM processing engine for inference processing, and the actual performance index corresponding to the target inference request is received from the LLM processing engine. The actual performance index is used to characterize the response latency of the LLM processing engine to the target inference request. as well as Based on the actual performance indicators, adjust the control parameters of at least some of the K authorized control units to update the traffic carrying capacity of the at least some authorized control units for subsequent inference requests.

2. The method according to claim 1, wherein, K is an integer greater than 1. Different authorized control units are associated with different traffic priorities and correspond to different control parameters. The step of authorizing the target inference request by consuming tokens from the token buckets of the K authorization control units includes: Obtain the target traffic priority corresponding to the target inference request, determine the authorization control unit associated with the target traffic priority among the K authorization control units as the target authorization control unit, and authorize the target inference request by consuming tokens from the token bucket of the target authorization control unit; and The step of adjusting the control parameters of at least some of the K authorized control units according to the actual performance indicators includes: Based on the actual performance indicators, the control parameters of the target authorized control unit are adjusted.

3. The method according to claim 2, wherein, The control parameters for each authorized control unit include at least one of the following: Token replenishment rate, used to control the rate at which tokens are replenished in the token bucket; Token bucket capacity controls the maximum number of tokens that can be held in the token bucket; or Queue length controls the upper limit of the number of inference requests waiting in line outside the token bucket.

4. The method according to claim 3, wherein, The step of adjusting the control parameters of the target authorized control unit based on the actual performance indicators includes: Obtain the reference performance index corresponding to the target authorized control unit and the parameter adjustment strategy corresponding to the target authorized control unit, wherein the parameter adjustment strategy is used to indicate the magnitude of parameter adjustment; Based on the difference between the actual performance index and the reference performance index, and the parameter adjustment strategy, the control parameters of the target authorized control unit are adjusted.

5. The method according to claim 4, wherein, Different authorized control units among the K authorized control units correspond to different parameter adjustment strategies. For any first authorized control unit and second authorized control unit among the K authorized control units: If the traffic priority associated with the first authorized control unit is lower than the traffic priority associated with the second authorized control unit, then the parameter adjustment range indicated by the parameter adjustment strategy corresponding to the first authorized control unit is greater than the parameter adjustment range indicated by the parameter adjustment strategy corresponding to the second authorized control unit.

6. The method according to claim 4, wherein, If the response delay represented by the actual performance index is greater than the response delay represented by the reference performance index, the adjusted control parameters of the target authorization control unit shall satisfy at least one of the following: the token replenishment rate is reduced, the token bucket capacity is reduced, or the queue length is reduced. If the response delay represented by the actual performance indicator is less than the response delay represented by the reference performance indicator, the adjusted control parameters of the target authorization control unit shall satisfy at least one of the following: the token replenishment rate is increased, the token bucket capacity is increased, or the queue length is increased.

7. The method according to claim 2, wherein, Each authorization control unit has a corresponding waiting queue, and the authorization of the target inference request by consuming tokens from the token bucket of the target authorization control unit includes: An attempt is made to consume a token from the token bucket of the target authorization control unit. If the consumption is successful, the target inference request is determined to be authorized successfully. If the consumption fails, the target inference request is added to the waiting queue corresponding to the target authorization control unit.

8. The method according to claim 7, wherein, After adding the target inference request to the waiting queue corresponding to the target authorization control unit, the method further includes: If the waiting time for the target inference request in the waiting queue exceeds a preset time, then the authorization of the target inference request is determined to have failed; and In response to the authorization failure, the target inference request is removed from the waiting queue, and a failure message is returned to the initiator of the target inference request.

9. The method according to claim 8, wherein, The method further includes: The number of requests deemed as authorization failures by the target authorization control unit within the first time window is counted; and If the number of requests exceeds a preset threshold, the target authorization control unit is controlled to enter a circuit breaker state. In the circuit breaker state, the waiting queue corresponding to the target authorization control unit does not accept new inference requests within a second time window.

10. The method according to claim 9, wherein, After controlling the target authorized control unit to enter the fuse-off state, the method further includes: After the second time window, in response to receiving a new inference request corresponding to the traffic priority associated with the target authorization control unit, the new inference request is authorized by consuming a token from the token bucket of the target authorization control unit. If the authorization is successful, the target authorization control unit is controlled to exit the circuit breaker state.

11. The method according to claim 2, wherein, The target inference request is a request initiated by the target user, and obtaining the target traffic priority corresponding to the target inference request includes: The target traffic priority corresponding to the target inference request is determined based on at least one of the attribute information of the target user or the attribute information of the target inference request.

12. The method according to claim 11, wherein, The attribute information of the target user includes at least one of the following: the type of the target user, or the service level bound to the target user.

13. The method according to claim 11, wherein, The attribute information of the target inference request includes at least one of the following: inference request type or inference request content.

14. The method according to claim 1, wherein, The target inference request is initiated by the target user, and the actual performance metric includes at least one of the following: The first-character delay index (TTFT) represents the time interval between when the target user initiates the target inference request and when the target user receives the first character of the inference response. or The inter-character latency index TPOT represents the time interval between adjacent characters received by the target user in the inference response.

15. A flow control system, comprising: At least one storage medium storing at least one instruction set for flow control; as well as At least one processor is communicatively connected to the at least one storage medium, wherein, when the flow control system is running, the at least one processor reads the at least one instruction set and executes the flow control method as described in any one of claims 1-14 according to the instructions of the at least one instruction set.

16. A computer-readable non-transitory storage medium, wherein, The computer-readable non-transitory storage medium stores at least one set of instructions, which, when executed by at least one processor, implement the method as described in any one of claims 1-14.

Images