Adaptive routing and traffic control system for large model inference

Abstract

The present application relates to the technical field of large model adaptive routing and traffic control, and particularly relates to an adaptive routing and traffic control system for large model inference, which comprises a collection module, a screening module, a display memory processing module, a calculation processing module, a throttling calibration module and a prediction calibration module.The present application divides the to-be-screened requests into display memory requests or calculation requests by monitoring the length of each to-be-screened request and combining the dynamic classification threshold of all routing nodes, so as to control the contention range of display memory and calculation resources, and for the display memory requests, screening out release requests within the query period and combining the adaptive nodes of each release request for inference, realizing display memory overload protection and smooth peak clipping of burst traffic, forming a routing and traffic control logic from real-time perception and classification scheduling, and effectively alleviating the problem of sharp deterioration of short request tail delay caused by competition for node resources between long requests and short requests under load during peak periods.

Description

Technical Field

[0001] This invention relates to the field of large model adaptive routing and flow control technology, specifically to an adaptive routing and flow control system for large model inference. Background Technology

[0002] In large-scale model inference services, adaptive routing and traffic control systems are key technologies for ensuring model service stability and maintaining low system latency. Existing large-scale model service systems allocate requests to the routing nodes with the best overall scores by collecting real-time indicators such as request queuing depth and key-value cache utilization. This can alleviate the inadequacy of simple round-robin strategies under heterogeneous loads to some extent. However, in high-load mixed request scenarios, the above-mentioned traditional routing strategies still cannot fundamentally solve the problem of the coupling between request memory contention and queue blocking, resulting in poor service quality of the model during peak hours. Taking the peak hours of a weekday afternoon as an example, code generation requests and document summary requests often emerge concurrently. There are natural differences in memory usage and processing time between the two types of requests. Even if the routing module can sense the queuing depth and memory utilization of nodes, there is still a high probability that nodes with high memory but low queues will be selected first, or nodes with low memory but high queues will be ignored during peak hours. This exacerbates the risk of memory overflow and queue backlog on different nodes. Therefore, it is urgent to develop an adaptive routing and traffic control system that can collaboratively sense the resource status and traffic intensity of nodes.

[0003] Chinese Patent Application Publication No. CN121501479A discloses a performance-aware large model cluster traffic splitting system and method. The system includes: a routing decision module for receiving inference requests sent by upstream business systems to Unicorn via FastAPI and initiating a scheduling process; a resource pooling module for matching model instance groups from the resource pool based on model names and performing logical isolation and dynamic resource allocation based on concurrency and token length attributes; a KV cache awareness module for performing hash calculations on request inputs, querying the global KV index table, and returning the hit instance ID and hit rate; and a performance awareness module for periodically collecting instance performance indicators and calculating real-time load to maintain instance health status. The routing decision module calculates a scheduling score based on the hit rate and instance load, and selects the optimal instance for request forwarding.

[0004] Therefore, the aforementioned existing technologies have at least the following problems: they are difficult to respond to dynamic changes in mixed loads in real time, resulting in long requests still having a high probability of blocking short requests, and the tail latency of short requests remaining high for a long time; they lack a traffic control mechanism linked to routing decisions, and the statically configured rate limiting thresholds cannot smoothly transition when the load is in the middle range, while the nodes selected by the route are often unable to receive requests due to their own rate limitations; their classification and scoring parameters remain unchanged for a long time once set, while the request characteristics may gradually evolve in actual business scenarios, and the static parameters cannot proactively adapt to the needs of actual load scenarios, making it difficult to adjust the request processing strategy in a timely manner, thus generating unnecessary resource waste or overflow risks. Summary of the Invention

[0005] To address this, the present invention provides an adaptive routing and flow control system for large model inference, which overcomes the problem of rapidly deteriorating tail latency of short requests caused by competition for node resources between long and short requests under load in the prior art by classifying requests according to resource bottlenecks and routing them separately.

[0006] To achieve the above objectives, the present invention provides an adaptive routing and flow control system for large model inference, comprising: The data acquisition module is used to obtain in real time the length of requests to be screened at the gateway during peak hours of the language inference model, the total concurrency of each routing node in the language inference model, the memory usage, and the request rate between the gateway and the corresponding routing node. The screening module is used to determine whether a request to be screened is a memory request or a computing request based on the length of each request to be screened, the memory usage of all routing nodes, and the classification threshold, and to determine the backlog index of the corresponding routing node based on the total concurrency and request rate of each routing node. The video memory processing module is used to determine the appropriate node for each pass request for inference based on the total concurrency of each routing node, video memory utilization, and backlog index within the query period. The pass request is determined based on the request rate of each routing node, the preset throttling rate, video memory utilization, backlog index, and the length of each video memory request within the query period. The calculation processing module is used to determine the appropriate node for each calculation request for inference based on the total concurrency of each routing node, the request rate, and the backlog index within the query period. The throttling calibration module is used to calibrate the preset throttling rate of each adapter node based on the number of memory requests, pass requests and screening requests within the calibration period, the memory usage rate of each routing node and the backlog index. The predictive calibration module is used to calibrate the classification threshold based on the number of memory requests and the length of each memory request within a consecutive preset number of calibration cycles.

[0007] Furthermore, the screening module includes: The estimated demand determination unit is used to determine the estimated memory demand based on the length of each request to be screened, the preset minimum window, and the preset memory coefficient. The screening and determination unit is used to determine whether the corresponding request to be screened is a memory request or a computing request based on the estimated memory requirements and dynamic thresholds. The dynamic thresholds are determined based on the average memory usage of all routing nodes and the classification threshold. The index determination unit is used to determine the backlog index of the corresponding routing node based on the total concurrency and request rate of each routing node.

[0008] Furthermore, the video memory processing module includes: The system load calculation unit is used to determine the system load rate based on the memory load rate, traffic load rate and backlog load rate of each routing node within the query period. The memory load rate is determined based on the memory utilization rate of each routing node, the traffic load rate is determined based on the request rate and preset throttling rate of each routing node, and the backlog load rate is determined based on the backlog index of each routing node. The request filtering unit is used to filter and allow requests from all video memory requests based on the allowance ratio and the estimated video memory demand of each video memory request within the query period. The allowance ratio is determined based on the system load rate within the query period. The node filtering unit is used to filter candidate nodes for video memory from all routing nodes based on the video memory pressure value, total concurrency, preset video memory pressure threshold and preset concurrency threshold of each routing node within the query period. The video memory pressure value is determined based on the video memory utilization rate of each routing node within the query period. The memory matching unit is used to determine the appropriate node for each request to be allowed in the query period based on the backlog index, total concurrency and preset concurrency threshold of each candidate memory node in the query period for inference.

[0009] Furthermore, the system load calculation unit includes: The memory load determination subunit is used to determine the memory load rate of the corresponding routing node based on the memory usage rate of each routing node and the preset memory occupancy threshold. The traffic load determination subunit is used to determine the traffic load rate of the corresponding routing node based on the request rate of each routing node and the preset throttling rate. The backlog load determination subunit is used to determine the backlog load rate of the corresponding routing node based on the backlog index of each routing node and a preset backlog threshold. The system load determination subunit is used to determine the system load rate based on the average of the memory load rate, the average of the traffic load rate, and the average of the backlog load rate of all routing nodes during the query period.

[0010] Furthermore, the node filtering unit includes: The memory pressure determination subunit is used to determine the memory pressure value based on the average memory usage of each routing node within the query cycle and the change in memory usage compared to the previous query cycle. The node filtering subunit is used to determine the corresponding routing node as a candidate node for video memory when the video memory pressure value of the routing node is less than the preset video memory pressure threshold and the total concurrency is less than the preset concurrency threshold.

[0011] Furthermore, the computational processing module includes: The node determination unit is used to filter and calculate candidate nodes from all routing nodes based on the request rate, preset request threshold, backlog index and preset backlog threshold of each routing node within the query period. The calculation matching unit is used to determine the appropriate node for each calculation request in the query period for inference based on the backlog index, total concurrency and preset concurrency threshold of each candidate calculation node in the query period.

[0012] Furthermore, the throttling calibration module includes: The ratio determination module is used to determine the release request ratio based on the number of memory requests and the number of release requests within the calibration cycle, and to determine the rollback request ratio based on the number of memory requests, release requests, and the number of requests to be screened within the calibration cycle. The periodic load calculation unit is used to determine the periodic load index based on the average memory usage of all routing nodes within the calibration period, the preset memory occupancy threshold, the average backlog index, and the preset backlog threshold. The throttling calibration unit is used to calibrate the preset throttling rate of each adapter node based on the periodic load index, the release request ratio, and the backoff request ratio.

[0013] Furthermore, the throttling calibration unit includes: The throttling direction determination subunit is used to determine the rate calibration direction based on the periodic load index, preset high load threshold, preset low load threshold, release request ratio, preset release threshold, backoff request ratio, and preset backoff threshold. The final throttling calibration subunit is used to calibrate the preset throttling rate of each adapter node based on the rate calibration direction and preset calibration step size.

[0014] Furthermore, the predictive calibration module includes: The ratio calculation unit is used to determine the ratio of video memory requests for the corresponding calibration period based on the number of video memory requests and the number of requests to be screened within the calibration period. The trend calibration unit is used to calibrate the classification threshold based on the changing trend of the proportion of video memory requests within a consecutive preset number of calibration cycles and the changing trend of the length of all video memory requests within each calibration cycle.

[0015] Furthermore, the trend calibration unit includes: The trend calculation subunit is used to determine the memory change rate of the corresponding calibration cycle based on the memory request ratio of each calibration cycle and the memory request ratio of the previous calibration cycle, and to determine the length change rate of the corresponding calibration cycle based on the average length of all memory requests in each calibration cycle and the average length of all memory requests in the previous calibration cycle. The trend direction determination subunit is used to determine the threshold calibration direction based on the average value of the memory change rate within a consecutive preset number of calibration cycles, the average value of the length change rate within all calibration cycles, the preset high memory threshold, the preset low memory threshold, and the preset length threshold. The final calibration subunit is used to calibrate the classification threshold based on the threshold calibration direction and the average value of the length change rate within each calibration cycle.

[0016] Compared with the prior art, the beneficial effects of this invention are as follows: This embodiment obtains the length of each request to be screened in real time to reflect the physical differences in its memory usage and computing requirements, and quantifies the real-time load and congestion level by monitoring the total concurrency of each routing node, memory usage, and request rate between the gateway and the node. The requests to be screened are then dynamically classified into memory requests or computing requests by adjusting the classification threshold. Based on the backlog index of the corresponding node, the cumulative effect of queuing pressure is reflected, thus initially separating two types of requests with vastly different resource requirements to control the contention for memory and computing resources. Specifically, for memory requests, within the query cycle, the request rate and memory usage of each routing node are considered... The system uses a combination of memory utilization rate and backlog index to filter out requests for passage. It then selects the appropriate node with sufficient memory and the lowest backlog for inference based on the length of each passed request. This achieves memory overload protection and smoothing of burst traffic. For computation requests, the appropriate node is determined based on the total concurrency of each routing node, request rate, and backlog index to ensure low-latency response for short requests. In addition, the system uses statistical analysis of the changes in the number of memory requests, passed requests, and pending requests during the calibration period. Combined with the long-term trends of memory utilization rate and backlog index of each routing node, the preset throttling rate of each appropriate node is calibrated in reverse. This allows the system to adaptively improve throughput while avoiding memory overflow. Attached Figure Description

[0017] Figure 1 This is a system diagram of the adaptive routing and flow control system used for large model inference in this embodiment; Figure 2 This is a logic diagram for determining the corresponding request to be screened in this embodiment; Figure 3 This is the logic diagram for determining the rate calibration direction in this embodiment; Figure 4 This is a logic diagram for determining the threshold calibration direction in this embodiment. Detailed Implementation

[0018] To make the objectives and advantages of the present invention clearer, the present invention will be further described below with reference to embodiments; it should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0019] Preferred embodiments of the present invention will now be described with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0020] Please see Figure 1 As shown, this is a system diagram of the adaptive routing and flow control system used for large model inference in this embodiment, including: The data acquisition module is used to obtain in real time the length of requests to be screened at the gateway during peak hours of the language inference model, the total concurrency of each routing node in the language inference model, the memory usage, and the request rate between the gateway and the corresponding routing node. The screening module, which is electrically connected to the acquisition module, is used to determine whether a request to be screened is a memory request or a computing request based on the length of each request to be screened, the memory usage rate of all routing nodes, and the classification threshold, and to determine the backlog index of the corresponding routing node based on the total concurrency and request rate of each routing node. The video memory processing module is electrically connected to the acquisition module and the screening module. It is used to determine the appropriate node for each release request for inference based on the total concurrency of each routing node, video memory utilization rate and backlog index within the query period. The release request is determined based on the request rate of each routing node, the preset throttling rate, video memory utilization rate, backlog index and the length of each video memory request within the query period. The calculation and processing module, which is electrically connected to the acquisition module and the screening module, is used to determine the appropriate node for each calculation request for inference based on the total concurrency of each routing node, the request rate and the backlog index within the query period. The throttling calibration module is electrically connected to the acquisition module, screening module and video memory processing module. It is used to calibrate the preset throttling rate of each adapter node based on the number of video memory requests, release requests and screening requests within the calibration period, the video memory usage rate of each routing node and the backlog index. The predictive calibration module is electrically connected to the acquisition module, screening module, calculation and processing module and throttling calibration module. It is used to calibrate the classification threshold based on the number of memory requests and screening requests and the length of each memory request within a consecutive preset number of calibration cycles.

[0021] This embodiment is deployed on a large language inference model cluster for enterprise-level services, specifically a general-purpose large language model based on the Transformer architecture, such as LLaMA370B. This embodiment selects LLaMA370B as the basic architecture, with the number of parameters ranging from billions to tens of billions. It mainly handles two types of typical requests: one is code generation and interpretation requests, which have short inputs, usually tens to hundreds of tokens, and generate outputs of the same number of tokens. These requests are frequent and sensitive to latency. The other type is long document summarization and analysis requests, which are characterized by input lengths of hundreds to thousands of tokens. Their generated outputs are usually also long, with large memory usage and processing time much longer than short requests.

[0022] In this embodiment, the peak period refers to the period of intensive data transmission for the enterprise, which is dynamically determined based on the actual number of requests in real time.

[0023] In this embodiment, the data acquisition module includes: The length acquisition unit is used to obtain the length of the token sequence contained in the user input prompt in each request entering the gateway, so as to obtain the length of the request to be screened.

[0024] The concurrent acquisition unit is used to obtain the total number of requests that the routing node is currently processing and waiting in the queue, so as to obtain the total concurrency of each routing node.

[0025] The memory usage acquisition unit is used to obtain the percentage of GPU memory used by each routing node relative to the total memory, in order to obtain the memory usage rate of each routing node.

[0026] The request rate acquisition unit is used to obtain the request arrival rate from the gateway to each routing node, so as to obtain the request rate of each routing node.

[0027] In this embodiment, the classification threshold is the criterion for determining whether a request should be classified as a memory request or a computing request. It is based on a simulation experiment using manually labeled historical data during peak periods and candidate thresholds. The candidate threshold with the highest classification consistency and a long document summary consistency rate of no less than 70% is selected as the classification threshold.

[0028] The preset throttling rate is used as the benchmark for determining whether the current system traffic is overloaded. It is based on selecting a regular routing node, gradually increasing the sending rate and continuously monitoring the node's memory usage, average processing latency, and instantaneous concurrency. When any indicator reaches its preset overload limit, the rate is recorded as the node's maximum stable request rate to determine the preset throttling rate. This is to reserve buffer space for the system under sudden traffic surges and prevent direct triggering of the overload threshold.

[0029] In this embodiment, the query period is the time interval for obtaining the latest routing node status and performing route allocation. It is set and kept synchronized with the update frequency actively pushed by each node to ensure that each routing decision is based on the latest available status, while avoiding missing status changes due to an excessively long period or causing system waste due to an excessively short period.

[0030] In this embodiment, the preset calibration period is the time window for determining the throttling calibration module to update the preset throttling rate. It is determined based on the statistical granularity of load changes in historical data or simulation experiments and the stability requirements of statistical samples, so as to achieve a balance between timely response to load fluctuations and avoiding instantaneous disturbances of samples.

[0031] In this embodiment, the preset number is the number of calibration cycles included in a single long-cycle analysis by the prediction calibration module. It is driven by the hourly time feature of the peak period request feature evolution to smooth minute-level random fluctuations and capture hourly trend changes, preventing the classification threshold from being frequently misadjusted due to short-term disturbances or lagging behind the pattern evolution.

[0032] Specifically, this embodiment obtains the length of each request to be screened in real time to reflect the physical difference between its memory usage and computing requirements. It also quantifies real-time load and congestion by monitoring the total concurrency of each routing node, memory utilization, and the request rate between the gateway and nodes. By dynamically adjusting the classification threshold, requests to be screened are divided into memory requests or computing requests. Based on the backlog index of the corresponding nodes, the cumulative effect of queuing pressure is reflected, initially separating the two types of requests with significantly different resource requirements to control the contention for memory and computing resources. Specifically, for memory requests, within the query period, the request rate, memory utilization, and backlog index of each routing node are considered... The system filters out requests and selects the appropriate node with sufficient video memory and the lowest backlog based on the length of each request. This achieves video memory overload protection and smoothing of burst traffic. For computation requests, the appropriate node is determined based on the total concurrency of each routing node, the request rate, and the backlog index to ensure low-latency response for short requests. In addition, the system reverse-calibrates the preset throttling rate of each adapter node by statistically analyzing the changes in the number of video memory requests, approved requests, and pending requests during the calibration period, combined with the long-term trends of video memory utilization and backlog index of each routing node. This allows the system to adaptively improve throughput while avoiding video memory overflow.

[0033] In this embodiment, the screening module includes: The estimated memory requirement determination unit determines the estimated memory requirement based on the product of a preset memory coefficient and the initial estimated memory requirement of each request to be screened. It also estimates the corresponding memory consumption based on the sum of the estimated memory consumption caused by each request to be screened. The initial estimated memory requirement is determined based on the sum of the length of each request to be screened and the corresponding estimated output length, in order to estimate the total length of the input and output of each request to be screened; The estimated output length is determined based on the maximum value between the length of each screening request and the preset minimum window, and the output length of each screening request is estimated based on the length of the input of each screening request.

[0034] In this embodiment, the preset minimum window is used as the initial criterion for distinguishing the input length when calculating the estimated output length, so as to avoid underestimating the video memory demand due to the input being too short. It is determined based on the statistical quantile of the historical requested output length and can be adjusted according to the business scenario to adapt to the video memory estimation of different load types.

[0035] In this embodiment, the preset video memory coefficient is a conversion benchmark for converting the total number of requested tokens into the estimated video memory requirement. It is determined by taking the median of the ratio of the requested video memory increment to the total number of tokens under different load types measured by offline benchmark tests, so as to reflect the average video memory usage ratio of each token.

[0036] Specifically, this embodiment conservatively estimates the output length by adding the input length of each request to be screened to the larger value in the preset minimum window, thereby avoiding underestimating the actual video memory usage due to the input being too short. In addition, it combines a preset video memory coefficient to linearly map the total number of tokens to the estimated video memory requirement, so that the requirement can be compared with the subsequent dynamic threshold.

[0037] Please see Figure 2 As shown, this is a logic diagram for determining the corresponding screening request in this embodiment. The screening module also includes: The screening and determination unit is used to determine whether the corresponding request to be screened is a memory request or a computation request based on the estimated memory requirement and a dynamic threshold. The dynamic threshold is determined based on the product of the remaining video memory rate and the classification threshold, so as to dynamically adjust the strictness of the classification threshold according to the remaining video memory of all routing nodes. The remaining video memory rate is determined based on the difference between one and the average video memory usage of all routing nodes, in order to determine the amount of video memory remaining for all routing nodes. Specifically, when the estimated memory requirement of a request to be screened is greater than the dynamic threshold, the corresponding request to be screened is a memory request; otherwise, it is a computation request. The index determination unit is used to determine the backlog index of the corresponding routing node based on the ratio of the total concurrency of each routing node to the request rate, so as to measure the actual memory load pressure of each routing node and provide a theoretical basis for subsequent selection of routing nodes with sufficient resources.

[0038] Specifically, this embodiment dynamically adjusts the classification threshold based on the average memory usage of all routing nodes, decreasing it as the overall memory usage increases. This lowers the threshold for becoming a memory request when memory pressure rises, proactively bringing more requests that might consume memory resources into the control scope of the memory processing module. This tightens the defenses at the source, ensuring the low-load and low-latency response capabilities of the computing module. Conversely, when memory is plentiful, the threshold increases, allowing only requests with truly large estimated memory demands to enter the memory processing module, enabling most requests to be efficiently processed by the computing module. This achieves adaptive classification based on system load awareness, achieving a dynamic balance between memory pressure and processing efficiency.

[0039] In this embodiment, the video memory processing module includes: The system load calculation unit determines the system load rate based on the memory load rate, traffic load rate, and backlog load rate of each routing node within the query period. The memory load rate is determined based on the memory utilization rate of each routing node, the traffic load rate is determined based on the request rate and preset throttling rate of each routing node, and the backlog load rate is determined based on the backlog index of each routing node. It characterizes the overall busyness of the system from three complementary dimensions: space capacity, flow capacity, and queuing time, thereby determining the system load rate and comprehensively reflecting the current resource tension status to avoid misjudgment based on a single indicator.

[0040] The request filtering unit is used to filter and allow requests from all video memory requests based on the release ratio and the estimated video memory demand of each video memory request within the query cycle. The release ratio is determined based on the system idle ratio within the query cycle so that the release ratio decreases linearly as the system processing capacity decreases, thus providing smooth overload protection for the system. The system idle ratio is determined based on the difference between a and the system load rate, with its minimum value truncated to zero, in order to estimate the current processing capacity of the system in combination with the overall busy level of the system. The screening process is as follows: within the query period, based on the estimated memory requirements of each memory request, memory requests with the highest approval ratio are selected as approved requests. In this embodiment, for video memory requests that are not identified as allowed requests, the system directly rejects them and returns an HTTP 503 status code to the upstream client. It also generates a response message to the client stating that the current traffic load is high and requests should be retried later. By sorting all video memory requests within the query period according to their estimated video memory requirements from smallest to largest, requests with smaller requirements are allowed first. This allows lightweight requests to be processed first when resources are scarce, thereby shortening the average waiting time and quickly releasing video memory resources.

[0041] The node filtering unit is used to filter candidate nodes for video memory from all routing nodes based on the video memory pressure value, total concurrency, preset video memory pressure threshold and preset concurrency threshold of each routing node within the query period. The video memory pressure value is determined based on the video memory utilization rate of each routing node within the query period. The memory matching unit is used to determine the appropriate node for each request to be allowed in the query period based on the backlog index, total concurrency and preset concurrency threshold of each candidate memory node in the query period for inference.

[0042] In this embodiment, for requests that have passed flow control, a round-by-round filling scheduling strategy is used to allocate them to the memory candidate nodes. The specific process is as follows: First, all memory candidate nodes are sorted in ascending order according to their current backlog index to prioritize the node with the shortest expected waiting time. Then, multiple rounds of allocation are performed: In each round, a request is allocated to each memory candidate node in the sorted order. This memory candidate node is the matching node for the corresponding request. Each time a request is allocated, the total concurrency of the memory candidate nodes increases by one. After one round of allocation, it is checked whether the total concurrency of each node has reached the preset concurrency threshold. If it has not reached the threshold... If the target memory node is reached, it will be retained for the next round of allocation. If the target memory node has already been reached, it will be temporarily removed from subsequent rounds. The above allocation process is repeated round by round until all requests are allocated or the total concurrency of all remaining memory nodes has reached the preset concurrency threshold. If there are still requests that have not been allocated, these requests will be sent back to the upstream with a backpressure signal. The client will generate a response message that the current traffic load is high and you should try again later. In the above process, the total concurrency is corrected to be the instantaneous total concurrency, which is synchronized with the internal counter of the actual routing node, that is, it is included in the total concurrency updated every 100ms.

[0043] In this embodiment, the preset memory pressure threshold is used as the criterion for eliminating routing nodes with excessive memory pressure. It is determined by gradually increasing the load on the node and monitoring the memory usage and its upward trend. The threshold is set when the node approaches the overflow threshold to avoid the node from failing due to memory overflow.

[0044] In this embodiment, the preset concurrency threshold is used as the criterion for eliminating routing nodes with excessively high instantaneous total concurrency. It is based on gradually increasing the concurrent load on the nodes and monitoring the average processing latency and memory usage. When the latency increases significantly or the memory approaches its limit, the maximum stable concurrency is determined to ensure that the nodes remain stable during concurrent processing.

[0045] Specifically, this embodiment compares the memory usage and total concurrency of each routing node with preset memory pressure thresholds and preset concurrency thresholds, respectively. Only nodes that simultaneously meet the requirements of sufficient memory and no concurrency exceeding the limit are retained as candidate nodes for memory, while nodes that are already on the verge of overload are excluded, thereby reducing the allocation of new requests to routing nodes that have a high probability of causing high latency. Finally, the node with the smallest backlog index and the lowest total concurrency is selected from the candidate nodes for memory as the matching node. From global load awareness, short job priority screening, node health filtering to backlog awareness matching, a progressive approach is taken to achieve fine-grained load balancing.

[0046] In this embodiment, the system load calculation unit includes: The memory load determination subunit is used to determine the memory load rate of the corresponding routing node based on the ratio of the memory pressure of each routing node to a preset memory usage threshold. The memory pressure is determined based on the difference between the memory load rate and a preset memory usage threshold, with the minimum value truncated to zero, to identify the memory load of each routing node that exceeds the preset memory usage threshold. In this embodiment, the preset memory usage threshold is used as the benchmark for determining whether the memory usage rate of a node exceeds the normal range. It is determined based on the median of the memory usage rate distribution of each node during peak hours, so as to sensitively capture the upward trend from the normal range to the overload range.

[0047] Specifically, this embodiment avoids excessively affecting the subsequent steps of determining the system load rate by keeping the video memory load rate at zero when the video memory utilization rate is within a safe range. After exceeding the threshold, the video memory load rate increases linearly with the increase of video memory utilization rate, reflecting the actual pressure on the video memory and quantifying the tension of the remaining video memory capacity of each routing node.

[0048] The traffic load determination subunit is used to determine the traffic load rate of the corresponding routing node based on the ratio of the request rate of each routing node to the preset throttling rate, and to identify whether the pressure of incoming requests to each routing node exceeds the node's processing capacity, so as to identify the overload state at the flow rate level. The backlog load determination subunit is used to determine the backlog load rate of the corresponding routing node based on the ratio of the backlog index of each routing node to the preset backlog threshold. The queuing delay is linked to the acceptable waiting limit of the service to quantify the current congestion level of the node. The system load determination subunit is used to determine the system load rate by taking the maximum value among the average values ​​of the memory load rate, traffic load rate, and backlog load rate of all routing nodes during the query period, in order to follow the bottleneck effect and make the system load rate determined by the most strained resource dimension.

[0049] In this embodiment, the preset backlog threshold is used as a benchmark to determine whether the node backlog index exceeds the normal range. It is determined based on the specific requirements of the service level agreement for latency, so as to provide early warning of queuing backlog and reduce the probability of normal load being misjudged as overload.

[0050] Specifically, this embodiment identifies the urgency of the remaining video memory capacity of each routing node, the overload status at the flow rate level, and the current congestion level of the routing node. It then calculates the average of the three load rates of all nodes to obtain the overall pressure level of the system in each dimension. Finally, it takes the maximum value of these three averages as the system load rate. Following the bottleneck effect, the system load rate is determined by the most strained resource dimension, thus comprehensively and accurately reflecting the overall bottleneck of the system.

[0051] In this embodiment, the node filtering unit includes: The memory pressure determination subunit determines the memory pressure value based on the product of the average memory usage of each routing node during the query cycle and the change in memory usage. This is used to identify routing nodes that are about to enter a memory overload state in advance. The video memory change rate is determined based on the sum of the changes in video memory usage and the previous query cycle, in order to identify the trend of video memory change; This embodiment uses the current memory usage level and its trend to determine the memory pressure value. Thus, when the memory usage rate rises rapidly, even if the current usage level is still acceptable, it will be amplified and an early warning will be issued to identify routing nodes that are about to enter a memory overload state.

[0052] The node filtering subunit is used to determine the corresponding routing node as a candidate node for video memory when the video memory pressure value of the routing node is less than the preset video memory pressure threshold and the total concurrency is less than the preset concurrency threshold.

[0053] Specifically, this embodiment identifies routing nodes that are about to enter a state of video memory overload in advance, requiring that a routing node be identified as a candidate node for video memory only when its video memory pressure value is less than a preset video memory pressure threshold and its total concurrency is less than a preset concurrency threshold. In this way, health checks are performed simultaneously from two independent dimensions: video memory capacity and task queue length, reducing the probability of allocating new requests to routing nodes that are about to run out of video memory or have a high probability of congestion.

[0054] In this embodiment, the calculation processing module includes: The node determination unit is used to filter and calculate candidate nodes from all routing nodes based on the request rate, preset request threshold, backlog index and preset backlog threshold of each routing node within the query period. Specifically, when the request rate of a routing node is less than a preset request threshold and the backlog index is less than a preset backlog threshold, the corresponding routing node will be determined as a candidate node for calculation. The calculation matching unit is used to determine the appropriate node for each calculation request in the query period for inference based on the backlog index, total concurrency and preset concurrency threshold of each candidate calculation node in the query period.

[0055] In this embodiment, for computation requests, suitable nodes are determined and inference-based allocation is performed based on the backlog index, total concurrency, and preset concurrency threshold of each computation candidate node within the query period. The specific operations are as follows: First, computation candidate nodes with a total concurrency greater than or equal to the preset concurrency threshold are removed. Then, all computation candidate nodes are sorted in ascending order of backlog index to prioritize the computation candidate node with the shortest expected waiting time. A round-by-round allocation strategy is then adopted: In each round, a computation request is allocated to each computation candidate node in the sorted order. For each allocated request, the total concurrency of that computation candidate node increases by one. After one round of allocation is completed, each node is checked. If the total concurrency has not reached the preset concurrency threshold, the candidate node for computation will be retained to participate in the next round of allocation. If it has reached the threshold, the candidate node for computation will be temporarily removed from subsequent rounds. The above process is repeated until all computation requests are allocated, or the total concurrency of all remaining candidate nodes for computation has reached the preset concurrency threshold. If there are still computation requests that have not been arranged, these requests will be sent back to the upstream with a backpressure signal. The client will generate a message indicating that the current traffic load is high and requests to retry later. In the above process, the corrected total concurrency is the instantaneous total concurrency, which is synchronized with the internal counter of the actual routing node, i.e., it is included in the total concurrency updated every 100ms.

[0056] In this embodiment, the video memory processing module and the computing processing module run in parallel and independently. Each module selects and allocates requests to suitable nodes in real time based on the status of the selected routing nodes, without waiting for each other. If the two modules select the same node in the same round of allocation during parallel operation, and each allocates a request to it, the total concurrency of the routing node may temporarily exceed the preset concurrency threshold. However, since the magnitude of the temporary exceedance of the preset concurrency threshold is only one, it will not cause the routing node to crash or the latency to deteriorate sharply. It will only affect its temporary removal in the subsequent arrangement process, so it is an acceptable system error.

[0057] In this embodiment, the preset request threshold is used as the criterion for determining whether the request rate of a node is too high. It is determined by gradually increasing the request rate during stress testing until the memory usage or average processing latency reaches its respective upper limit, in order to obtain the maximum stable request rate and eliminate routing nodes with excessively high request rates.

[0058] Specifically, this embodiment filters candidate nodes for computation based on two independent conditions: whether the request rate of each routing node is less than a preset request threshold and whether the backlog index is less than a preset backlog threshold. The request rate reflects the inflow pressure per unit time, while the backlog index reflects the expected clearing time of the current queue. Both conditions determine whether a node is in a healthy state from the perspectives of flow rate and inventory, thereby excluding nodes that are overloaded or severely congested to ensure that computation requests are only allocated to targets that still have processing capacity. Subsequently, among the candidate nodes for computation, based on the comparison of the backlog index, total concurrency, and preset concurrency threshold, the node with the smallest backlog index and the concurrency not exceeding the limit is preferentially selected as the adaptation node. The backlog index, as a physical measure of expected waiting time, minimizes the queuing delay of new requests by selecting the minimum value, while the concurrency limit avoids intensified competition for computing resources due to processing too many computation requests at the same time. This effectively balances the current load pressure of the routing node, the queue length, and the concurrent processing capacity when allocating nodes for computation requests, thus balancing the low-latency response requirements of computation requests with the need for balanced utilization of computing resources.

[0059] In this embodiment, the throttling calibration module includes: The ratio determination module determines the release request ratio based on the ratio of the number of memory requests to the number of release requests within the calibration cycle, in order to identify the proportion of memory requests that are actually accepted into the inference stage. The rollback request ratio is determined based on the ratio of the number of memory requests rejected and rolled back by the requested filtering unit to the number of requests to be filtered within the calibration cycle. This reflects the proportion of memory requests rejected or cached by the system due to overload among all input requests. The number of memory requests rejected and rolled back by the requested filtering unit is determined based on the difference between the number of memory requests and the number of allowed requests during the calibration cycle. The periodic load calculation unit is used to determine the periodic load index by the maximum value between the average value of the memory pressure of all routing nodes and the average value of the backlog load rate during the calibration period. Following the weakest link effect, the periodic load index is determined by the most strained resource dimension to comprehensively evaluate the overall overload of the system during the period. The throttling calibration unit is used to calibrate the preset throttling rate of each adapter node based on the periodic load index, the release request ratio, and the backoff request ratio.

[0060] Specifically, this embodiment characterizes the system's traffic control effect from two complementary perspectives—acceptance efficiency and rejection pressure—by using the allow request ratio and the fallback request ratio. Subsequently, the overall overload of the system within a given period is comprehensively evaluated using the periodic load index. The preset throttling rate of each adapter node is calibrated by using the periodic load index, the allow request ratio, and the fallback request ratio together. A high allow request ratio and a high periodic load index indicate that the current throttling rate is too high, resulting in continuous memory shortage and thus requiring a rate reduction. A high fallback request ratio and a low periodic load index indicate that the throttling is too conservative, resulting in idle resources and thus requiring a rate increase. This forms the basic adjustment logic of closed-loop feedback regulation.

[0061] Please see Figure 3 As shown, this is the logic diagram for determining the rate calibration direction in this embodiment. The throttling calibration unit includes: The throttling direction determination subunit is used to determine the rate calibration direction based on the periodic load index, preset high load threshold, preset low load threshold, release request ratio, preset release threshold, backoff request ratio, and preset backoff threshold. Among them, when the periodic load index is less than the preset low load threshold, the proportion of release requests is greater than the preset release threshold, and the proportion of backoff requests is less than the preset backoff threshold, the rate calibration direction is determined to increase the preset throttling rate. When the periodic load index is greater than the preset high load threshold, the proportion of release requests is less than the preset release threshold, and the proportion of backoff requests is greater than the preset backoff threshold, the rate calibration direction is determined to reduce the preset throttling rate. When the rate calibration direction is not to increase or decrease the preset throttling rate, the preset throttling rate is not calibrated; The final throttling calibration subunit is used to determine the new preset throttling rate for each adapter node based on the product of a preset dynamic step size and the current preset throttling rate of each adapter node. Preset dynamic step size based on rate calibration direction calibration: When the rate calibration direction is determined to increase the preset throttling rate, the preset dynamic step size is the sum of one and the preset calibration step size; When the rate calibration direction is determined to reduce the preset throttling rate, the preset dynamic step size is the difference between one and the preset calibration step size.

[0062] In this embodiment, the above calibration method cannot be regarded as the only calibration method for the preset throttling rate. The staff can reset the preset throttling rate or manually calibrate it separately during the maintenance cycle.

[0063] In this embodiment, a preset low load threshold is used as the criterion for determining whether the system is in an idle state. It is determined based on the low quantile of the load index distribution during offline analysis of peak periods, so as to relax the flow limit when the system has a margin.

[0064] In this embodiment, a preset high load threshold is used as the criterion for determining whether the system is in an overload state. It is determined by gradually increasing the load during stress testing until the node memory utilization or average processing latency reaches its respective upper limit, so as to tighten the current limiting and achieve early warning when the system is close to overload.

[0065] In this embodiment, the preset release threshold is used as the criterion for determining whether the proportion of memory request releases is too high. It is determined by taking the average value and standard deviation of the proportion of release requests during multiple consecutive calibration cycles when the system is running stably, so as to identify the risk of the rate limiting policy being too lenient before the load increases.

[0066] In this embodiment, the preset fallback threshold is used as the criterion for determining whether the proportion of rejected memory requests to the total number of requests is too high. It is determined based on the maximum request rejection rate allowed in the service level agreement, so as to trigger adjustment in advance before the rejection rate approaches the upper limit.

[0067] In this embodiment, the preset calibration step size is used as the benchmark for determining the adjustment range of the throttling rate each time. It is determined based on the influence of different step sizes on response time and stability tested by system simulation or field test, so as to ensure that the system can converge to a new equilibrium within a finite period after the load changes without generating oscillations.

[0068] Specifically, this embodiment determines whether to increase or decrease the preset throttling rate based on three conditions: the periodic load index, the proportion of release requests, and the proportion of rollback requests. It uses multi-dimensional and logical precision to distinguish between the system's true idle state, true overload state, and normal fluctuations, avoiding rate oscillations caused by false triggering of a single indicator. Then, the preset throttling rate is multiplicatively adjusted according to the direction of the determination, so that the rate change is in a fixed proportion to the current value to maintain a constant relative rate of change. The preset throttling rate value range is combined with the amplitude limit to prevent excessive reduction from causing insufficient throughput or excessive increase from causing memory overload. This allows the preset throttling rate of each adapter node to adaptively follow the load trend, finding a dynamic balance between not wasting resources and not overloading the system.

[0069] In this embodiment, the prediction calibration module includes: The ratio calculation unit is used to determine the ratio of video memory requests in the corresponding calibration period based on the ratio of the number of video memory requests to the number of requests to be screened in the calibration period, so as to reflect the macro proportion of video memory sensitive requests under the current classification threshold. The trend calibration unit is used to calibrate the classification threshold based on the changing trend of the proportion of video memory requests within a consecutive preset number of calibration cycles and the changing trend of the length of all video memory requests within each calibration cycle.

[0070] Specifically, this embodiment identifies the macro-level proportion of memory-sensitive requests under the current classification threshold. Then, based on the changing trend of the proportion of memory requests and the changing trend of the length of all memory requests in each cycle within a preset number of calibration periods, it determines the drift direction and magnitude of the actual business mode. When the proportion of memory requests and the length of requests both increase in the same direction, it means that the system is facing dual pressure and needs to increase the classification threshold to transfer more requests that are hovering between the two types of requests to the computing processing module. The existing request filtering mode is used to focus on protecting the memory resources of the routing nodes. When both decrease in the same direction, the classification threshold is lowered to make full use of the idle memory resources. When the trends of the two diverge, no adjustment is made temporarily to avoid misjudgment.

[0071] Please see Figure 4 As shown, this is the logic diagram for determining the threshold calibration direction in this embodiment. The trend calibration unit includes: The trend calculation subunit is used to determine the memory change rate of the corresponding calibration cycle based on the memory request ratio of each calibration cycle and the memory request ratio of the previous calibration cycle, and to determine the length change rate of the corresponding calibration cycle based on the average length of all memory requests in each calibration cycle and the average length of all memory requests in the previous calibration cycle. The trend direction determination subunit is used to determine the threshold calibration direction based on the average value of the memory change rate within a consecutive preset number of calibration cycles, the average value of the length change rate within all calibration cycles, the preset high memory threshold, the preset low memory threshold, and the preset length threshold. Among them, when the average value of the memory change rate within a consecutive preset number of calibration cycles is greater than the preset high memory threshold and the average value of the length change rate within all calibration cycles is greater than the preset length threshold, the threshold calibration direction is to reduce the classification threshold. When the average value of the memory change rate is less than the preset low memory threshold within a preset number of consecutive calibration cycles, the threshold calibration direction is to increase the classification threshold. When the threshold calibration direction is not to decrease or increase the calibration classification threshold, the classification threshold is not calibrated; In this embodiment, a preset high video memory threshold is used as the criterion for determining whether the upward trend of the video memory request ratio is significant. It is determined based on the statistical distribution of the rate of change of the video memory request ratio in adjacent cycles during a stable running period, in order to determine when the classification threshold needs to be lowered to increase the strictness of screening.

[0072] In this embodiment, a preset low video memory threshold is used as the criterion for determining whether the downward trend of the video memory request ratio is significant. It is determined based on the statistical distribution of the rate of change of the video memory request ratio in adjacent cycles during a stable operating period, in order to determine when the system is shifting to a compute-intensive load, i.e., it is necessary to raise the classification threshold to release computing resources.

[0073] In this embodiment, the preset length threshold is used as the criterion for determining whether the change in the average input length of the video memory request is significant. It is determined based on the statistical distribution of the length change rate of adjacent calibration cycles during the stable operation period, in order to determine whether it is necessary to reduce the classification threshold.

[0074] Specifically, this embodiment determines the memory change rate and length change rate for each calibration cycle, captures the short-term fluctuation direction and magnitude of the memory request ratio and request length, and then makes a joint judgment. When the average memory change rate is higher than the preset high memory threshold and the average length change rate is higher than the preset length threshold, the threshold calibration direction is determined to lower the classification threshold, so that more requests are assigned to the memory processing module to cope with the dual upward pressure. When the average memory change rate is lower than the preset low memory threshold, the classification threshold is determined to be increased to reduce the proportion of memory requests and avoid resource idleness. No adjustments are made in other cases to avoid misadjustment caused by single-point fluctuations.

[0075] In this embodiment, the trend calibration unit further includes: The final calibration subunit is used to calibrate the classification threshold based on the threshold calibration direction and the dynamic threshold length within each calibration cycle. The dynamic threshold length is determined based on the ratio of the average length change rate within each calibration period to the maximum context length of the model. The dynamic threshold length is truncated between a preset minimum length and a preset maximum length to constrain the adjustment step size of the classification threshold within a reasonable range.

[0076] When the threshold calibration direction is to reduce the classification threshold, a new classification threshold is determined based on the difference between the current classification threshold and the dynamic threshold length; When the threshold calibration direction is to increase the classification threshold, a new classification threshold is determined based on the sum of the current classification threshold and the dynamic threshold length.

[0077] In this embodiment, the above calibration method cannot be regarded as the only calibration method for the preset throttling rate. The staff can reset the preset throttling rate or manually calibrate it separately during the maintenance cycle.

[0078] In this embodiment, the maximum context length of the model represents the maximum total number of input and output tokens that the large language inference model can support, i.e., the context window size, which is determined by the model architecture itself, in order to normalize the variation of the input length of the memory request to a uniform range in order to calculate the classification threshold adjustment step size.

[0079] In this embodiment, the preset minimum length is the minimum adjustment step size for each calibration of the classification threshold. It is determined based on the ratio of the statistical distribution of the rate of change of the input length during the steady-state operation to the maximum context length of the model, so as to filter random noise and ensure the slow accumulation of long-term trends.

[0080] In this embodiment, the preset maximum length is the maximum adjustment step size for each calibration of the classification threshold. It is determined based on the ratio of the typical rate of change of the average input length of memory requests during peak periods in historical data to the maximum context length of the model, so as to prevent the threshold from oscillating violently due to excessive adjustment amplitude in a single adjustment.

[0081] Specifically, in this embodiment, when adjusting the step size, the ratio of the absolute value of the length change rate to the maximum context length of the model is used, along with the preset minimum length and preset maximum length, to set upper and lower limits. This ensures that the adjustment step size is positively correlated with the degree of length change and is always controlled within a safe range, thereby reducing the probability of system oscillation caused by threshold abrupt changes while ensuring the smoothness of the adjustment.

[0082] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An adaptive routing and flow control system for large model inference, characterized in that, include: The data acquisition module is used to obtain in real time the length of requests to be screened at the gateway during peak hours of the language inference model, the total concurrency of each routing node in the language inference model, the memory usage, and the request rate between the gateway and the corresponding routing node. The screening module is used to determine whether a request to be screened is a memory request or a computing request based on the length of each request to be screened, the memory usage of all routing nodes, and the classification threshold, and to determine the backlog index of the corresponding routing node based on the total concurrency and request rate of each routing node. The video memory processing module is used to determine the appropriate node for each pass request for inference based on the total concurrency of each routing node, video memory utilization, and backlog index within the query period. The pass request is determined based on the request rate of each routing node, the preset throttling rate, video memory utilization, backlog index, and the length of each video memory request within the query period. The calculation processing module is used to determine the appropriate node for each calculation request for inference based on the total concurrency of each routing node, the request rate, and the backlog index within the query period. The throttling calibration module is used to calibrate the preset throttling rate of each adapter node based on the number of memory requests, pass requests and screening requests within the calibration period, the memory usage rate of each routing node and the backlog index. The predictive calibration module is used to calibrate the classification threshold based on the number of memory requests and the length of each memory request within a consecutive preset number of calibration cycles.

2. The adaptive routing and flow control system for large model inference according to claim 1, characterized in that, The screening module includes: The estimated demand determination unit is used to determine the estimated memory demand based on the length of each request to be screened, the preset minimum window, and the preset memory coefficient. The screening and determination unit is used to determine whether the corresponding request to be screened is a memory request or a computing request based on the estimated memory requirements and dynamic thresholds. The dynamic thresholds are determined based on the average memory usage of all routing nodes and the classification threshold. The index determination unit is used to determine the backlog index of the corresponding routing node based on the total concurrency and request rate of each routing node.

3. The adaptive routing and flow control system for large model inference according to claim 2, characterized in that, The video memory processing module includes: The system load calculation unit determines the system load rate based on the memory load rate, traffic load rate, and backlog load rate of each routing node during the query period. The video memory load rate is determined based on the video memory usage rate of each routing node. Traffic load rate is determined based on the request rate of each routing node and the preset throttling rate. The backlog load rate is determined based on the backlog index of each routing node; The request filtering unit is used to filter and allow requests from all video memory requests based on the allowance ratio and the estimated video memory demand of each video memory request within the query period. The allowance ratio is determined based on the system load rate within the query period. The node filtering unit is used to filter candidate nodes for video memory from all routing nodes based on the video memory pressure value, total concurrency, preset video memory pressure threshold and preset concurrency threshold of each routing node within the query period. The video memory pressure value is determined based on the video memory utilization rate of each routing node within the query period. The memory matching unit is used to determine the appropriate node for each request to be allowed in the query period based on the backlog index, total concurrency and preset concurrency threshold of each candidate memory node in the query period for inference.

4. The adaptive routing and flow control system for large model inference according to claim 3, characterized in that, The system load calculation unit includes: The memory load determination subunit is used to determine the memory load rate of the corresponding routing node based on the memory usage rate of each routing node and the preset memory occupancy threshold. The traffic load determination subunit is used to determine the traffic load rate of the corresponding routing node based on the request rate of each routing node and the preset throttling rate. The backlog load determination subunit is used to determine the backlog load rate of the corresponding routing node based on the backlog index of each routing node and a preset backlog threshold. The system load determination subunit is used to determine the system load rate based on the average of the memory load rate, the average of the traffic load rate, and the average of the backlog load rate of all routing nodes during the query period.

5. The adaptive routing and flow control system for large model inference according to claim 4, characterized in that, The node filtering unit includes: The memory pressure determination subunit is used to determine the memory pressure value based on the average memory usage of each routing node within the query cycle and the change in memory usage compared to the previous query cycle. The node filtering subunit is used to determine the corresponding routing node as a candidate node for video memory when the video memory pressure value of the routing node is less than the preset video memory pressure threshold and the total concurrency is less than the preset concurrency threshold.

6. The adaptive routing and flow control system for large model inference according to claim 5, characterized in that, The computational processing module includes: The node determination unit is used to filter and calculate candidate nodes from all routing nodes based on the request rate, preset request threshold, backlog index and preset backlog threshold of each routing node within the query period. The calculation matching unit is used to determine the appropriate node for each calculation request in the query period for inference based on the backlog index, total concurrency and preset concurrency threshold of each candidate calculation node in the query period.

7. The adaptive routing and flow control system for large model inference according to claim 6, characterized in that, The throttling calibration module includes: The ratio determination module is used to determine the release request ratio based on the number of memory requests and the number of release requests within the calibration cycle, and to determine the rollback request ratio based on the number of memory requests, release requests, and the number of requests to be screened within the calibration cycle. The periodic load calculation unit is used to determine the periodic load index based on the average memory usage of all routing nodes within the calibration period, the preset memory occupancy threshold, the average backlog index, and the preset backlog threshold. The throttling calibration unit is used to calibrate the preset throttling rate of each adapter node based on the periodic load index, the release request ratio, and the backoff request ratio.

8. The adaptive routing and flow control system for large model inference according to claim 7, characterized in that, The throttling calibration unit includes: The throttling direction determination subunit is used to determine the rate calibration direction based on the periodic load index, preset high load threshold, preset low load threshold, release request ratio, preset release threshold, backoff request ratio, and preset backoff threshold. The final throttling calibration subunit is used to calibrate the preset throttling rate of each adapter node based on the rate calibration direction and preset calibration step size.

9. The adaptive routing and flow control system for large model inference according to claim 8, characterized in that, The predictive calibration module includes: The ratio calculation unit is used to determine the ratio of video memory requests for the corresponding calibration period based on the number of video memory requests and the number of requests to be screened within the calibration period. The trend calibration unit is used to calibrate the classification threshold based on the changing trend of the proportion of video memory requests within a consecutive preset number of calibration cycles and the changing trend of the length of all video memory requests within each calibration cycle.

10. The adaptive routing and flow control system for large model inference according to claim 9, characterized in that, The trend calibration unit includes: The trend calculation subunit is used to determine the memory change rate of the corresponding calibration cycle based on the memory request ratio of each calibration cycle and the memory request ratio of the previous calibration cycle, and to determine the length change rate of the corresponding calibration cycle based on the average length of all memory requests in each calibration cycle and the average length of all memory requests in the previous calibration cycle. The trend direction determination subunit is used to determine the threshold calibration direction based on the average value of the memory change rate within a consecutive preset number of calibration cycles, the average value of the length change rate within all calibration cycles, the preset high memory threshold, the preset low memory threshold, and the preset length threshold. The final calibration subunit is used to calibrate the classification threshold based on the threshold calibration direction and the average value of the length change rate within each calibration cycle.

Images