High-concurrency lightweight data channel adaptive load balancing method based on large model

Abstract

The invention relates to the technical field of distributed computing, in particular to a high-concurrency lightweight data channel adaptive load balancing method based on a large model, and the method comprises the steps: when a current LLM reasoning request reaches a load balancing device, according to the hardness similarity between each historical request in a historical request record set of any server and the current LLM reasoning request, calculating the load balancing device according to the hardness similarity between each historical request in the historical request record set of any server and the current LLM reasoning request; performing initial optimization on the utilization item to obtain an initial optimization utilization item for processing the current LLM reasoning request by any server; an expected residual compensation factor is constructed, the initial optimization utilization item is optimized again by using the expected residual compensation factor, and an optimal utilization item is obtained; according to the method, the decision score of any server is obtained according to the optimal utilization item of any server, and adaptive load balancing is performed on the current LLM reasoning request according to the decision score of each server, so that efficient load balancing capability is kept in long-term operation.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, and in particular to a high-concurrency lightweight data channel adaptive load balancing method based on a large model. Background Technology

[0002] In distributed computing systems, load balancing is a key technology for distributing network requests or computational tasks across multiple backend servers. Its purpose is to improve overall system processing capacity, reduce response latency, and ensure stable service availability. With the rise of Large Language Model (LLM) inference services, efficient load balancing for such services has become a new technical challenge. The request flow of LLM inference services exhibits heterogeneity, with requests of varying complexity consuming vastly different amounts of computing resources. This places extremely high demands on the accuracy and adaptability of load balancing algorithms. Currently, there are several technical approaches for load balancing of LLM inference services or similar scenarios:

[0003] The first method is resource pool isolation based on request feature bucketing. This method analyzes quantifiable features of requests (input text length) to divide requests into different buckets and allocates a dedicated physical server resource pool to each bucket. While this method achieves differentiated processing of heterogeneous requests to some extent, the physical resource isolation prevents resources from being shared between different resource pools. In scenarios with dynamically changing business traffic, this can easily lead to some resource pools being idle while others are overloaded, resulting in low overall resource utilization.

[0004] The second approach is based on centralized scheduling and batch processing. This method uses a central scheduler to combine multiple incoming requests into a batch, and then sends the entire batch as a unit to the server for processing, maximizing the parallel computing efficiency of the hardware. The core goal of this approach is to increase the overall system throughput, but at the cost of increased queuing time for individual requests and sacrificed instant response latency, making it unsuitable for real-time interactive applications with latency requirements.

[0005] To achieve intelligent allocation of heterogeneous requests while maintaining low-latency per-request decision-making, a third technical approach is to employ a multi-armed slot machine algorithm. This method uses a decision scoring formula to balance the utilization and exploration of server performance, dynamically learning and selecting the optimal server in an uncertain environment. The utilization term in the decision scoring formula is key to achieving adaptive decision-making; it is calculated based on the server's historical performance data to evaluate its expected performance in handling future tasks.

[0006] In the standard multi-armed slot machine algorithm, the utilization term in its decision scoring formula is calculated by arithmetically averaging the performance gains of all historical requests processed by the server. When this algorithm is directly applied to LLM inference service scenarios with extremely heterogeneous characteristics, this indiscriminate arithmetic averaging method ignores the quantifiable differences in computational complexity between requests and fails to account for the dependence of the semantic associations of request content on the server's internal state. This results in the calculated utilization term value failing to accurately reflect the expected performance of the server in processing the next specific request, thus causing systemic decision bias and reducing the accuracy of load balancing decisions. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a high-concurrency lightweight data channel adaptive load balancing method based on a large model to solve the problem of how to improve the accuracy of load balancing decisions.

[0008] This invention provides a high-concurrency lightweight data channel adaptive load balancing method based on a large model, which includes the following steps:

[0009] When the current LLM inference request arrives at the load balancer, obtain the hardness factor, semantic vector, and request time of the current LLM inference request, as well as the historical request record set and recent performance gain set for each server. The historical request record set includes the hardness factor, semantic vector, original performance gain, and request time of each historical request.

[0010] For any server, based on the hardness similarity between each historical request in the historical request record set of the server and the current LLM inference request, the utilization items of the server are initially optimized to obtain the initial optimized utilization items for the server to process the current LLM inference request.

[0011] Based on the recent performance gain set, the historical request record set, and the semantic vector and request time of the current LLM inference request, an expected residual compensation factor is constructed. The expected residual compensation factor is then used to further optimize the initial optimization utilization item to obtain the best utilization item for any server to handle the current LLM inference request.

[0012] Based on the best utilization of each server in handling the current LLM inference request, a decision score for each server is obtained. Based on the decision score of each server, adaptive load balancing is performed on the current LLM inference request.

[0013] Preferably, the initial optimization of the utilization items of any server based on the hardness similarity between each historical request in the historical request record set of any server and the current LLM inference request, to obtain the initial optimized utilization items for any server to process the current LLM inference request, includes:

[0014] For any historical request in the historical request record set of any server, obtain the weight factor of any historical request based on the difference between the hardness factor of any historical request and the hardness factor of the current LLM inference request.

[0015] Obtain the weight factor of each historical request in the historical request record set. Based on the weight factor of each historical request in the historical request record set, perform a weighted average of the original performance gains of each historical request in the historical request record set to obtain the initial optimization utilization item for any server to process the current LLM inference request.

[0016] Preferably, obtaining the weight factor of any historical request based on the difference between the hardness factor of any historical request and the hardness factor of the current LLM inference request includes:

[0017] Calculate the squared difference between the hardness factor of any historical request and the hardness factor of the current LLM inference request. Calculate the variance of the hardness factor of all historical requests in the historical request record set. Use a preset multiple of the hardness factor variance as the denominator and the squared difference as the numerator to obtain the corresponding ratio. Use the negative of the ratio as the independent variable of an exponential function with the natural constant as the base to obtain the weight factor of any historical request.

[0018] Preferably, the recent performance gain set consists of all original performance gains within a fixed-size time window. Then, based on the recent performance gain set, the historical request record set, and the semantic vector and request time of the current LLM inference request, an expected residual compensation factor is constructed, including:

[0019] For any historical request in the historical request record set, obtain the initial optimization utilization item of any server in processing any historical request, denoted as the baseline performance benefit prediction value, and calculate the difference between the original performance benefit of any historical request and the baseline performance benefit prediction value, denoted as the performance prediction residual.

[0020] Based on all the original performance gains in the recent performance gain set, obtain the adaptive state decay rate of any server. Based on the semantic vector similarity and request time difference between any historical request and the current LLM inference request, and combined with the adaptive state decay rate of any server, obtain the residual weight factor of any historical request.

[0021] Obtain the performance prediction residual and residual weight factor for each historical request in the historical request record set. Based on the residual weight factor, perform a weighted average on the performance prediction residuals of all historical requests in the historical request record set to obtain the expected residual compensation factor for any server to process the current LLM inference request.

[0022] Preferably, obtaining the adaptive state decay rate of any server based on all the original performance gains in the recent performance gain set includes:

[0023] Calculate the mean and variance of performance gains for all original performance gains in the recent performance gain set, and obtain the adaptive state decay rate for any server based on the ratio between the variance of performance gains and the square of the mean of performance gains.

[0024] Preferably, obtaining the residual weight factor for any historical request based on the semantic vector similarity and request time difference between any historical request and the current LLM inference request, and in conjunction with the adaptive state decay rate of any server, includes:

[0025] Calculate the cosine similarity between the semantic vector of any historical request and the semantic vector of the current LLM inference request. Calculate the time difference between the request time of any historical request and the request time of the current LLM inference request. Substitute the negative of the product of the time difference and the adaptive state decay rate into an exponential function with the natural constant as the base to obtain an adaptive timeliness factor. Use the product of the cosine similarity and the adaptive timeliness factor as the residual weight factor for any historical request.

[0026] Preferably, the step of further optimizing the initial optimization utilization item using the expected residual compensation factor to obtain the optimal utilization item for any server to handle the current LLM inference request includes:

[0027] The sum of the expected residual compensation factor and the initial optimization utilization term is taken as the optimal utilization term for any server to process the current LLM inference request.

[0028] Preferably, obtaining the decision score of any server based on the best utilization item for processing the current LLM inference request by any server includes:

[0029] Obtain the exploration items for any of the servers, and obtain the decision score for any of the servers based on the sum of the best utilization items and the exploration items.

[0030] Preferably, the step of adaptively load balancing the current LLM inference request based on the decision score of each of the servers includes:

[0031] Based on the decision score of each server, the server with the highest decision score is selected as the best server for the current LLM inference request, and the best server is used to process the current LLM inference request.

[0032] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0033] This invention introduces a weighted average method based on request workload similarity to initially optimize traditional exploit items, resulting in an initially optimized exploit item. This addresses the performance evaluation bias caused by request heterogeneity in the standard multi-armed slot machine algorithm, providing a more reliable basis for technical performance prediction in subsequent load balancing decisions. Furthermore, based on the initial optimized item, an expected residual compensation factor is introduced to further optimize it, resulting in a more accurate exploit item. This ensures that the actual performance of the server depends not only on the computational load of the requests but also on the correlation between the request content and the current internal state of the server. It can continuously learn from the changes in request flow patterns and the latest performance status of the server cluster, thereby maintaining efficient load balancing capabilities in long-term operation. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart of a high-concurrency lightweight data channel adaptive load balancing method based on a large model, provided in Embodiment 1 of the present invention. Detailed Implementation

[0036] Embodiments of this disclosure are described in detail below, with examples of these embodiments illustrated in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this disclosure, and should not be construed as limiting it.

[0037] It should be noted that the terms "first," "second," etc., used in this disclosure and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this disclosure.

[0038] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0039] See Figure 1 This is a flowchart of a high-concurrency lightweight data channel adaptive load balancing method based on a large model, provided in Embodiment 1 of the present invention. Figure 1 As shown, the method may include:

[0040] Step S101: When the current LLM inference request arrives at the load balancer, obtain the hardness factor, semantic vector, and request time of the current LLM inference request, as well as the historical request record set and recent performance gain set for each server.

[0041] When making load balancing decisions, a series of data needs to be collected and maintained in real time. This data collection and maintenance is based on existing technology, and will be briefly described here: First, when an LLM inference request arrives at the load balancer, the system needs to extract its raw information from the request. Specifically, the system parses the request's metadata and payload to obtain the length of the request's input text. and estimated output text length The hardness factor H is used to quantify the computational complexity of an LLM inference request. In the existing field of large language model technology, the computational workload of a request is mainly determined by the total number of tokens it needs to process. This has become a recognized core metric in the industry. Therefore, this embodiment directly adopts this standard metric to define the hardness factor H, and its calculation method is as follows: Simultaneously, the system uses a pre-trained sentence encoding model to convert the input text of the request into a d-dimensional semantic vector. .

[0042] Secondly, the request is assigned to a certain server. After processing is complete, the system needs to record the performance results of this service, that is, measure and record the actual processing latency of this request. And based on this value, the original performance gain of this service is calculated. In this embodiment At the same time, the system records the moment when the request is completed. .

[0043] Additionally, each server maintains a set of historical request records and a set of recent performance gains. The set of historical request records stores server... Detailed records of the 1000 most recently processed historical requests, for any given historical request. This set contains the hardness factor requested. semantic vectors Original performance benefits and request time For the set of recent performance gains, this set of storage servers... In the most recent time window All original performance gains It is implemented using a time window queue of fixed size. When a new raw performance gain is generated, the new raw performance gain is added to the set, while the oldest raw performance gain is removed.

[0044] Based on the above, when the current LLM inference request reaches the load balancer, the hardness factor of the current LLM inference request can be obtained. semantic vectors and request time The system includes a historical request record set S1 and a recent performance gain set S2 for each server, which are used to evaluate the expected performance of each server in handling the current LLM inference request based on the performance gains of the historical requests processed by each server, so as to determine the optimal server for processing the current LLM inference request.

[0045] Step S102: For any server, based on the hardness similarity between each historical request in the historical request record set of any server and the current LLM inference request, perform initial optimization on the exploit items of any server to obtain the initial optimized exploit items for any server to process the current LLM inference request.

[0046] In the prior art, the hardness factor of the current LLM inference request is obtained. semantic vectors and request time After obtaining the historical request record set S1 and the recent performance gain set S2 for each server, a multi-armed slot machine algorithm is typically used to achieve intelligent allocation of heterogeneous requests while maintaining low-latency per-request decision-making. This algorithm balances the utilization and exploration of server performance based on a decision scoring formula, dynamically learning and selecting the optimal server in an uncertain environment. The utilization term in the decision scoring formula, based on the arithmetic average of the original performance gains of all historical requests processed by the server, evaluates the expected performance of the server in handling future requests, which is crucial for achieving adaptive decision-making.

[0047] However, this indiscriminate arithmetic averaging method ignores the quantifiable differences in computational complexity between requests when dealing with scenarios like LLM inference services, where the computational workload varies greatly. In other words, the computational workload of an LLM inference request is strongly correlated with the total number of tokens it needs to process (the sum of the number of input tokens and the number of output tokens). In actual service scenarios, the server's historical request record set will inevitably contain a large number of low-load requests with a small total number of tokens and a small number of high-load requests with an extremely high total number of tokens. When using the arithmetic averaging method to calculate the exploit, the average will be significantly affected by the requests with the larger sample size. This effect causes the exploit to fail to accurately reflect the server's expected performance when processing a request with a specific workload.

[0048] For example, a server's historical request record set will inevitably contain both requests with short processing times and low computational complexity, as well as requests with long processing times and high computational complexity. Due to the asymmetry of data distribution, the arithmetic mean of the exploits becomes a statistically ineffective expected value that cannot represent any specific request type. When the next request to be assigned is a high-complexity request, the exploit, influenced by the performance data of a large number of low-complexity requests, will be a value much smaller than the actual expected processing time, thus underestimating the cost of processing the complex request and making the wrong decision to assign the high-cost task to an unsuitable server. Conversely, when the next request to be assigned is a low-complexity request, the exploit, influenced by the performance data of a few high-complexity requests, will be a value larger than the actual expected processing time, thus overestimating the cost of processing the simple request and missing the optimal server selection.

[0049] Therefore, in this embodiment, to address the performance evaluation bias caused by request heterogeneity in the standard multi-armed slot machine algorithm and to provide a more reliable basis for subsequent load balancing decisions, a weighted average method based on request workload similarity is introduced. This method is used to evaluate the current LLM inference request server. When considering expected performance, servers should not be viewed as equals. Instead of relying solely on all historical processing records, higher reference weights should be given to historical requests that are more similar to the current LLM inference request in terms of total token processing volume. The greater the difference in workload between a historical request and the current LLM inference request, the lower its reference value for this prediction, and its weight should be reduced accordingly. In this way, the calculated utilization will be a value that is "tailor-made" for the specific workload of the current LLM inference request and better reflects its expected performance, thereby eliminating evaluation bias caused by request heterogeneity.

[0050] Taking any server as an example, let's call it server. For any historical request in the historical request record set of any server, calculate the squared difference between the hardness factor of the historical request and the hardness factor of the current LLM inference request, calculate the variance of the hardness factor of all historical requests in the historical request record set, use the hardness factor variance of a preset multiple as the denominator and the squared difference as the numerator to obtain the corresponding ratio, and use the negative of the ratio as the independent variable of an exponential function with the natural constant as the base to obtain the weight factor of the historical request.

[0051] The formula for calculating the weight factor of any historical request is as follows:

[0052]

[0053] in, This represents the weighting factor for any historical request. This represents an exponential function with the natural constant as its base. This represents the hardness factor of the current LLM inference request. This represents the hardness factor of any historical request. This represents the variance of the hardness factor of all historical requests in the historical request record set of any server, where 2 represents the preset multiple.

[0054] It should be noted that, The squared difference between the current LLM inference request and historical requests in terms of total token processing is used to calculate this difference. This squared difference is placed in a negative exponential Gaussian function to achieve smooth, non-linear weight decay: when the total token processing of any historical request (hardness factor) is... ) and the total number of tokens processed for the current LLM inference request (hardness factor) When they are very close, The value will approach This means the original performance gain of any historical request. It will be included in the final weighted average with its full weight, as it constitutes a highly relevant reference sample for evaluating the performance gains of either server in processing the current LLM inference request. Conversely, as the difference between the two requests in terms of total token processing volume increases, The value will decay exponentially and rapidly approach zero, meaning that historical requests with computational loads drastically different from the current LLM inference request will have their weights reduced and be filtered out.

[0055] Similarly, the weight factor of each historical request in the historical request record set is obtained, and the original performance gains of each historical request in the historical request record set are weighted and averaged according to the weight factor of each historical request in the historical request record set to obtain the initial optimization utilization item for any server to process the current LLM inference request.

[0056] The formula for calculating the initial optimization utilization term for any server processing the current LLM inference request is as follows:

[0057]

[0058] in, This represents the initial optimizations used by any server to process the current LLM inference request. This represents the original performance gain of any historical request in the historical request record set. S1 represents the weight factor of any historical request and S1 represents the set of historical request records.

[0059] At this point, the initial optimization utilization terms obtained by any server in processing the current LLM inference request are... No longer a biased average that is indiscriminately affected by all historical records, when dealing with a high-load request that needs to process a large number of tokens, the initial optimization utilization formula will automatically refer to the performance data of other high-load requests processed by the server in the past, so as to derive a performance gain prediction that will not be diluted by a large number of low-load requests, providing a benchmark prediction for server performance evaluation.

[0060] Step S103: Based on the recent performance gain set, the historical request record set, and the semantic vector and request time of the current LLM inference request, construct the expected residual compensation factor, and use the expected residual compensation factor to further optimize the initial optimization utilization item to obtain the best utilization item for any server to handle the current LLM inference request.

[0061] This embodiment further reveals that relying solely on the static computational complexity (hardness factor) of a request for prediction still has inherent limitations. When calculating exploits using the arithmetic mean method, the dependency of the request content's semantic association on the server's internal state cannot be taken into account. The actual performance of the server depends not only on the computational load of the request but also on the correlation between the request content and the server's current internal state. In modern large language model inference engines, key-value caching technology is commonly used to accelerate the generation of subsequent tokens. This means that for two requests with the same computational complexity (hardness factor), if the content of one request is highly relevant to the context in the server's cache, its subsequent actual computational load will be significantly reduced. Standard exploit calculation methods indiscriminately include this conditional high performance resulting from cache hits in the same average calculation as performance generated by regular processing. This approach ignores the prerequisites for performance data generation, i.e., it mixes performance data generated under different internal states in the calculation, which further affects the reliability of exploits as a stable performance prediction indicator.

[0062] For example, when the content of a request is highly relevant to the context already cached in the server's memory, subsequent questions in a multi-turn dialogue can reuse a large number of calculated results, thus greatly reducing processing time. Conversely, a request with entirely new content requires executing a complete calculation process. The baseline performance gain prediction calculated by the initial optimization factor is unaware of this cache hit effect caused by content semantic relevance. Therefore, there is a performance prediction residual between this baseline performance gain prediction and the original performance gain. The magnitude and sign of this residual contain implicit contextual information that the hardness factor fails to capture.

[0063] Therefore, to address the aforementioned issues, this implementation further optimizes the initial utilization term by introducing an expected residual compensation factor to obtain a more precise utilization term, which is the optimal utilization term. Specifically, taking any server as an example, that is, server... First, based on the recent performance gain set S2 of any server, the historical request record set S1, and the semantic vector of the current LLM inference request... and request time Construct the expected residual compensation factor:

[0064] (1) For any historical request in the set of historical request records, according to the calculation formula of the initial optimization utilization item above, obtain the initial optimization utilization item of any server for processing any historical request, and record it as the baseline performance benefit prediction value. Calculate the difference between the original performance gain of any historical request and the baseline performance gain prediction value, and denote it as the performance prediction residual. .

[0065] (2) Obtain the adaptive state decay rate of any server based on all the original performance gains in the recent performance gain set.

[0066] Specifically, the mean and variance of performance gains of all original performance gains in the recent performance gain set are calculated, and the adaptive state decay rate of any server is obtained based on the ratio between the variance of performance gains and the square of the mean of performance gains.

[0067] The formula for calculating the adaptive state decay rate of any server is as follows:

[0068]

[0069] in, This represents the adaptive state decay rate of any server. The variance of the performance gains represents the total variance of all raw performance gains in the set of recent performance gains for any server. This represents the average performance gain across all raw performance gains in the recent performance gain set for any given server. This represents a preset constant used to prevent the denominator from being 0. In this embodiment of the invention, it is set... There are no restrictions here; implementers can set them according to the specific scenario.

[0070] (3) Based on the semantic vector similarity and request time difference between any historical request and the current LLM inference request, and combined with the adaptive state decay rate of any server, obtain the residual weight factor of any historical request.

[0071] Specifically, the cosine similarity between the semantic vector of any historical request and the semantic vector of the current LLM inference request is calculated. The time difference between the request time of any historical request and the request time of the current LLM inference request is calculated. The negative of the product of the time difference and the adaptive state decay rate is substituted into an exponential function with the natural constant as the base to obtain an adaptive timeliness factor. The product of the cosine similarity and the adaptive timeliness factor is used as the residual weight factor of any historical request.

[0072] The formula for calculating the residual weighting factor for any historical request is as follows:

[0073]

[0074] in, This represents the residual weighting factor for any historical request pair. This represents the semantic vector of the current LLM inference request. A semantic vector representing any historical request. This represents the adaptive state decay rate of any server. This represents the time difference between the request time of any historical request and the request time of the current LLM inference request. This represents an exponential function with the natural constant as its base.

[0075] It should be noted that, Used to characterize the semantic similarity between two requests, the residual weight factor approaches 1 as the semantic similarity of the two requests in terms of content increases, and approaches 0 as the semantic similarity decreases. When the system attempts to estimate the residual of the current LLM inference request, it will prioritize referring to the historical residuals that are most relevant to the content of the current LLM inference request. For example, if the current LLM inference request is a follow-up question about Python programming, the residual weight factor will give high weight to the residuals generated by historical questions and answers about Python programming. The adaptive timeliness factor used to characterize the current LLM inference request decays exponentially as the time difference increases, ensuring that the estimation process relies more on recent data that reflects the latest state of the server. It is directly proportional to the variance of the server's recent performance (performance gain variance) and inversely proportional to the square of the average performance (performance gain mean). This means that when the server's recent performance is very stable, The value is very small, and the decay of the adaptive timeliness factor is very slow. Even slightly earlier historical data still has high reference value. Conversely, when server performance fluctuates drastically, The value of can become very large, causing the adaptive timeliness factor to decay rapidly to zero over time. This indicates that in this unstable state, the system will quickly discard old data and only trust the latest performance feedback.

[0076] (4) Obtain the performance prediction residual and residual weight factor of each historical request in the historical request record set. Based on the residual weight factor, perform weighted average processing on the performance prediction residual of all historical requests in the historical request record set to obtain the expected residual compensation factor for any server to process the current LLM inference request.

[0077] The formula for calculating the expected residual compensation factor for any server processing the current LLM inference request is as follows:

[0078]

[0079] in, S1 represents the expected residual compensation factor for any server processing the current LLM inference request, and S1 represents the set of historical request records. This represents the residual weighting factor for any historical request pair. This represents the original performance gain of any historical request. This represents the initial optimization utilization (i.e., the baseline performance gain prediction) for any server processing any of the historical requests.

[0080] It should be noted that the expected residual compensation factor is a weighted average process, in which... It quantifies the residual performance gain of any server in processing each historical request throughout history. If the value is positive, the residual is positive, indicating a key-value cache hit; otherwise, the residual is negative. It is a residual weighting factor based on semantic similarity and timeliness. For the residual of any historical request in the historical request record set, it can only be applied when both the content semantic relevance and the freshness and validity of the state information are met simultaneously. This has a significant impact on the calculation.

[0081] After obtaining the expected residual compensation factor for any server, the sum of the expected residual compensation factor and the initial optimization utilization term is taken as the optimal utilization term for that server to handle the current LLM inference request. This is to further optimize the initial utilization. The formula for calculating the optimal utilization is: .

[0082] This gives us the optimal exploit for any server to handle the current LLM inference request.

[0083] Step S104: Based on the best utilization item for handling the current LLM inference request by any server, obtain the decision score of any server, obtain the decision score of each server, and perform adaptive load balancing on the current LLM inference request based on the decision score of each server.

[0084] The optimal exploit for any server to handle the current LLM inference request is obtained through steps S102-S103. Subsequently, this embodiment follows the decision-making framework of the multi-armed slot machine algorithm, combining the best utilization term with an exploration term to calculate the decision score for any server. The decision score is calculated using the existing algorithm formula as follows:

[0085]

[0086] in, Best use case for handling current LLM inference requests on any server , As a standard exploration item, in this known technology, This indicates the total number of requests processed so far. Indicates server The number of times selected, It is a constant used to balance utilization and exploration, in this embodiment The value is set to This is a typical value that has proven to have good performance in practice.

[0087] The purpose of the exploration term is to ensure that the algorithm can explore servers that have not performed optimally in the past or have been selected less frequently, thus preventing the algorithm from prematurely converging to a local optimum. Similarly, the decision score for each server in handling the current LLM inference request is obtained. Based on the decision score of each server, the server with the highest decision score is selected as the best server for the current LLM inference request. The load balancer performs the request forwarding operation, sending the current LLM inference request to the best server for processing, that is, using the best server to handle the current LLM inference request.

[0088] It should be noted that after the current LLM inference request is processed and returns a result, the system will collect the performance data of this processing and use this newly generated data to asynchronously and in real time update the historical request record set S1 and the recent performance gain set S2 corresponding to each server. The system will also update the relevant records of the current LLM inference request (including hardness factor) semantic vectors and request time The data is updated to the historical request record set S1 of the corresponding server, and the performance gains of the current LLM inference request are updated to the recent performance gain set S2. Through this continuous data feedback and parameter update, the load balancing method proposed in this embodiment can continuously learn the changes in request flow patterns and the latest performance status of the server cluster, thereby maintaining efficient load balancing capabilities in long-term operation.

[0089] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A large model-based high-concurrency lightweight data channel adaptive load balancing method, characterized in that, The method comprises: When a current LLM inference request arrives at the balanced load ware, a hardness factor, a semantic vector and a request time of the current LLM inference request are obtained, and a historical request record set and a recent performance benefit set of each server are obtained, the historical request record set comprising a hardness factor, a semantic vector, an original performance benefit and a request time of each historical request; For any server, an initial optimization utilization item of the any server is optimized according to the hardness similarity of each historical request in the historical request record set of the any server and the current LLM inference request, to obtain an initial optimization utilization item of the any server for processing the current LLM inference request; An expected residual compensation factor is constructed according to the recent performance benefit set, the historical request record set and the semantic vector and the request time of the current LLM inference request, and the initial optimization utilization item is re-optimized by using the expected residual compensation factor to obtain an optimal utilization item of the any server for processing the current LLM inference request; According to the optimal utilization item of the any server for processing the current LLM inference request, a decision score of the any server is obtained, a decision score of each server is obtained, and the current LLM inference request is adaptively balanced according to the decision score of each server.

2. The large model-based high-concurrency lightweight data channel adaptive load balancing method according to claim 1, characterized in that, The initial optimization utilization item of the any server for processing the current LLM inference request is obtained by optimizing the utilization item of the any server according to the hardness similarity of each historical request in the historical request record set of the any server and the current LLM inference request, comprising: For any historical request in the historical request record set of the any server, a weight factor of the any historical request is obtained according to the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request; The weight factor of each historical request in the historical request record set is obtained, and the original performance benefit of each historical request in the historical request record set is weighted and averaged according to the weight factor of each historical request in the historical request record set, to obtain the initial optimization utilization item of the any server for processing the current LLM inference request.

3. The large model-based high-concurrency lightweight data channel adaptive load balancing method according to claim 2, characterized in that, The weight factor of the any historical request is obtained according to the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request, comprising: The square of the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request is calculated, the hardness factor variance of all historical requests in the historical request record set is calculated, the hardness factor variance of the preset multiple is taken as the denominator, the square of the difference is taken as the numerator to obtain the corresponding ratio, the inverse of the ratio is taken as the independent variable of the exponential function with the natural constant as the base, and the weight factor of the any historical request is obtained.

4. The large model-based high-concurrency lightweight data channel adaptive load balancing method according to claim 1, characterized in that, The recent performance benefit set is composed of all original performance benefits in a fixed-size time window, and an expected residual compensation factor is constructed according to the recent performance benefit set, the historical request record set, and a semantic vector and a request time of a current LLM inference request, including: For any historical request in the historical request record set, an initial optimized utilization item of the any server processing the any historical request is obtained, denoted as a benchmark performance benefit prediction value, a difference between an original performance benefit of the any historical request and the benchmark performance benefit prediction value is calculated, denoted as a performance prediction residual; According to all original performance benefits in the recent performance benefit set, an adaptive state decay rate of the any server is obtained, a residual weight factor of the any historical request is obtained according to a semantic vector similarity and a request time difference between the any historical request and the current LLM inference request, and in combination with the adaptive state decay rate of the any server; The performance prediction residual and the residual weight factor of each historical request in the historical request record set are obtained, and the performance prediction residuals of all historical requests in the historical request record set are weighted and averaged based on the residual weight factor, to obtain an expected residual compensation factor of the any server processing the current LLM inference request.

5. The large model-based high-concurrency lightweight data channel adaptive load balancing method according to claim 4, characterized in that, The adaptive state decay rate of the any server is obtained according to all original performance benefits in the recent performance benefit set, including: The performance benefit mean and the performance benefit variance of all original performance benefits in the recent performance benefit set are calculated, and the adaptive state decay rate of the any server is obtained according to a ratio between the performance benefit variance and the square of the performance benefit mean.

6. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 4, characterized in that, The residual weight factor of the any historical request is obtained according to a semantic vector similarity and a request time difference between the any historical request and the current LLM inference request, and in combination with the adaptive state decay rate of the any server, including: The cosine similarity between the semantic vector of the any historical request and the semantic vector of the current LLM inference request is calculated, the time difference between the request time of the any historical request and the request time of the current LLM inference request is calculated, the inverse of the product of the time difference and the adaptive state decay rate is substituted into an exponential function with a natural constant as the base, to obtain an adaptive timeliness factor, and the product between the cosine similarity and the adaptive timeliness factor is taken as the residual weight factor of the any historical request.

7. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 1, characterized in that, The initial optimized utilization item is re-optimized by using the expected residual compensation factor, to obtain an optimal utilization item of the any server processing the current LLM inference request, including: The sum between the expected residual compensation factor and the initial optimized utilization item is taken as the optimal utilization item of the any server processing the current LLM inference request.

8. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 1, characterized in that, The decision score of the any server is obtained according to the optimal utilization item of the any server processing the current LLM inference request, including: An exploration item of the any server is acquired, and a decision score of the any server is obtained according to a sum between the best utilization item and the exploration item.

9. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 1, characterized in that, The adaptive load balancing of the current LLM inference request according to the decision score of each server comprises: The server corresponding to the maximum decision score is taken as a best server of the current LLM inference request according to the decision score of each server, and the current LLM inference request is processed by using the best server.

Images