Big model based high concurrency lightweight data channel adaptive load balancing method

Abstract

The present application relates to the technical field of distributed computing, and particularly relates to a high-concurrency lightweight data channel adaptive load balancing method based on a large model, which, when a current LLM inference request reaches a balanced load ware, carries out initial optimization on a utilization item according to the hardness similarity of each historical request in a historical request record set of any server and the current LLM inference request, to obtain an initial optimized utilization item of the current LLM inference request processed by any server; an expected residual compensation factor is constructed, the initial optimized utilization item is optimized again by using the expected residual compensation factor, to obtain an optimal utilization item; a decision score of any server is obtained according to the optimal utilization item of any server, and the current LLM inference request is adaptively load balanced according to the decision score of each server, so that the efficient load balancing capability is maintained in long-term operation.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing, and in particular to a high-concurrency lightweight data channel adaptive load balancing method based on large models. BACKGROUND

[0002] In a distributed computing system, load balancing is a key technology for distributing network requests or computing tasks to multiple backend servers, aiming to improve the overall processing capacity of the system, reduce response delay, and ensure the stability and availability of services. With the rise of large language model (LLM) inference services, how to efficiently load balance such services has become a new technical challenge. The request stream of LLM inference services presents heterogeneity, and the consumption of computing resources by requests of different complexity varies greatly, which puts high demands on the accuracy and adaptability of load balancing algorithms. In existing technologies, there are multiple technical paths for load balancing of LLM inference services or similar scenarios:

[0003] The first is a resource pool isolation method based on request feature bucketing. This method analyzes the quantifiable features of requests (input text length) to divide requests into different buckets and allocate dedicated physical server resource pools for each bucket. Although this method achieves some differentiation in handling heterogeneous requests, physical resource isolation prevents resource sharing between different resource pools, and in dynamic traffic scenarios, it is easy to cause some resource pools to be idle while others are overloaded, resulting in low overall resource utilization.

[0004] The second is a centralized scheduling and batch processing method. This method uses a central scheduler to combine multiple incoming requests into a batch, and then sends the entire batch to a server for processing to maximize hardware parallel computing efficiency. The core goal of this method is to improve the total throughput of the system, but at the cost of increasing the queuing time of individual requests, sacrificing the immediate response delay of requests, and is not suitable for real-time interactive applications that require low latency.

[0005] To achieve intelligent allocation of heterogeneous requests while maintaining low-latency per-request decision-making, the third technical path is to use a multi-armed bandit algorithm. This method uses a decision score formula to balance the utilization and exploration of server performance, dynamically learning and selecting the optimal server in an uncertain environment, where the utilization term in the decision score formula is the key to adaptive decision-making, calculated based on historical performance data of the server to evaluate its expected performance in handling future tasks.

[0006] In the standard multi-arm bandit algorithm, the utilization item in the decision score formula is calculated by taking an arithmetic mean of the performance benefits of all historical requests processed by the server. When this algorithm is directly applied to the LLM inference service scenario with extreme heterogeneity characteristics, this indiscriminate arithmetic mean method ignores the differences between requests in quantifiable computational complexity and cannot take into account the dependence of the server's internal state on the semantic association of request content when calculating, so that the calculated utilization item value cannot accurately reflect the expected performance of the server processing the next specific request, thereby causing systematic decision bias and reducing the accuracy of load balancing decisions. SUMMARY

[0007] Therefore, the embodiment of the present application provides a large model-based high-concurrency lightweight data channel adaptive load balancing method to solve the problem of how to improve the accuracy of load balancing decisions.

[0008] The embodiment of the present application provides a large model-based high-concurrency lightweight data channel adaptive load balancing method, which comprises the following steps:

[0009] When the current LLM inference request arrives at the balanced load, the hardness factor, semantic vector and request time of the current LLM inference request, and the historical request record set and recent performance benefit set of each server are obtained, the historical request record set includes the hardness factor, semantic vector, original performance benefit and request time of each historical request;

[0010] For any server, the utilization item of the any server is initially optimized according to the hardness similarity of each historical request in the historical request record set of the any server and the current LLM inference request, to obtain the initial optimization utilization item of the any server processing the current LLM inference request;

[0011] According to the recent performance benefit set, the historical request record set and the semantic vector and request time of the current LLM inference request, an expected residual compensation factor is constructed, and the initial optimization utilization item is optimized again by using the expected residual compensation factor, to obtain the best utilization item of the any server processing the current LLM inference request;

[0012] According to the best utilization item of the any server processing the current LLM inference request, the decision score of the any server is obtained, the decision score of each server is obtained, and the current LLM inference request is adaptively load balanced according to the decision score of each server.

[0013] Preferably, the utilization of the any server is initially optimized according to the hardness similarity of each historical request in the historical request record set of the any server and the current LLM inference request, to obtain an initial optimization utilization of the any server for processing the current LLM inference request, comprising:

[0014] For any historical request in the historical request record set of the any server, a weight factor of the any historical request is obtained according to the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request;

[0015] The weight factor of each historical request in the historical request record set is obtained, and the original performance benefit of each historical request in the historical request record set is weighted and averaged according to the weight factor of each historical request in the historical request record set, to obtain the initial optimization utilization of the any server for processing the current LLM inference request.

[0016] Preferably, the weight factor of the any historical request is obtained according to the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request, comprising:

[0017] The difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request is squared, the hardness factor variance of all historical requests in the historical request record set is calculated, and the hardness factor variance of the preset multiple is taken as the denominator, the square of the difference is taken as the numerator to obtain the corresponding ratio, the inverse of the ratio is taken as the independent variable of the exponential function with the natural constant as the base, and the weight factor of the any historical request is obtained.

[0018] Preferably, the recent performance benefit set is composed of all original performance benefits in a fixed size time window, and then the expected residual compensation factor is constructed according to the recent performance benefit set, the historical request record set, and the semantic vector and request time of the current LLM inference request, comprising:

[0019] For any historical request in the historical request record set, the initial optimization utilization of the any server for processing the any historical request is obtained, denoted as a benchmark performance benefit prediction value, and the difference between the original performance benefit of the any historical request and the benchmark performance benefit prediction value is calculated, denoted as a performance prediction residual;

[0020] According to all original performance benefits in the recent performance benefit set, an adaptive state decay rate of the any server is obtained, and according to the semantic vector similarity and request time difference between the any historical request and the current LLM inference request, and in combination with the adaptive state decay rate of the any server, a residual weight factor of the any historical request is obtained.

[0021] obtaining a performance prediction residual and a residual weight factor of each historical request in the historical request record set, performing weighted average processing on the performance prediction residuals of all historical requests in the historical request record set based on the residual weight factors to obtain an expected residual compensation factor of the any server processing the current LLM inference request.

[0022] Preferably, the adaptive state decay rate of the any server is obtained according to all original performance benefits in the recent performance benefit set, comprising:

[0023] calculating a performance benefit mean and a performance benefit variance of all original performance benefits in the recent performance benefit set, and obtaining the adaptive state decay rate of the any server according to a ratio between the performance benefit variance and a square of the performance benefit mean.

[0024] Preferably, the residual weight factor of the any historical request is obtained according to a semantic vector similarity and a request time difference between the any historical request and the current LLM inference request, and in combination with the adaptive state decay rate of the any server, comprising:

[0025] calculating a cosine similarity between the semantic vector of the any historical request and the semantic vector of the current LLM inference request, calculating a time difference between the request time of the any historical request and the request time of the current LLM inference request, obtaining an adaptive timeliness factor by substituting an inverse of a product of the time difference and the adaptive state decay rate into an exponential function with a natural constant as a base, and taking a product between the cosine similarity and the adaptive timeliness factor as the residual weight factor of the any historical request.

[0026] Preferably, the initial optimization utilization item is re-optimized by using the expected residual compensation factor to obtain the optimal utilization item of the any server processing the current LLM inference request, comprising:

[0027] taking a sum between the expected residual compensation factor and the initial optimization utilization item as the optimal utilization item of the any server processing the current LLM inference request.

[0028] Preferably, the decision score of the any server is obtained according to the optimal utilization item of the any server processing the current LLM inference request, comprising:

[0029] obtaining an exploration item of the any server, and obtaining the decision score of the any server according to a sum between the optimal utilization item and the exploration item.

[0030] Preferably, the adaptive load balancing of the current LLM inference request according to the decision score of each server comprises:

[0031] According to the decision score of each server, the server corresponding to the maximum decision score is regarded as the optimal server of the current LLM inference request, and the current LLM inference request is processed by using the optimal server.

[0032] Compared with the prior art, the embodiment of the present application has the beneficial effects that:

[0033] The present application introduces a weighted average method based on request workload similarity to initially optimize the traditional utilization item to obtain an initial optimization utilization item, so as to solve the performance evaluation deviation problem of the utilization item of the standard multi-arm bandit algorithm due to request heterogeneity, and provide a more reliable technical performance prediction basis for subsequent load balancing decisions, and then introduce an expected residual compensation factor again on the basis of the initial optimization item to optimize the initial optimization item again to obtain a more accurate utilization item, so that the actual performance of the server depends not only on the calculation load of the request, but also on the relevance between the request content and the current internal state of the server, can constantly learn the mode change of the request flow and the latest performance status of the server cluster, and thus maintain high efficient load balancing ability in long-term operation. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0035] Figure 1 It is a method flowchart of a high-concurrency lightweight data channel adaptive load balancing method based on a large model provided by the first embodiment of the present application. DETAILED DESCRIPTION

[0036] The embodiments of the present disclosure will be described in detail below, and examples of the embodiments are shown in the drawings. The embodiments described below by referring to the drawings are exemplary and are intended to explain the present disclosure, and cannot be understood as a limitation of the present disclosure.

[0037] It should be noted that the terms "first", "second", etc. in the specification of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation described in the following exemplary embodiments does not represent all implementations consistent with the present disclosure. Rather, they are merely examples of devices and methods consistent with some aspects of the present disclosure.

[0038] In order to illustrate the technical solutions of the present application, the following will be described by specific embodiments.

[0039] Referring to Figure 1 , a method flowchart of a high-concurrency lightweight data channel adaptive load balancing method based on a large model is provided in Embodiment One of the present application, as Figure 1 shown, the method can include:

[0040] Step S101, when a current LLM inference request arrives at the load balancer, the hardness factor, semantic vector and request time of the current LLM inference request are obtained, as well as the historical request record set and recent performance benefit set of each server.

[0041] When performing load balancing decision, a series of data needs to be collected and maintained in real time, wherein the data collection and maintenance belongs to the prior art, which is described as follows: first, when an LLM inference request arrives at the load balancer, the system needs to extract the original information from the request, specifically, the system parses the metadata and payload of the request to obtain the input text length and the estimated output text length , which are used to quantify the computational complexity of the LLM inference request, i.e. the hardness factor H. In the existing large language model technology field, the computational workload of a request is mainly determined by the total number of tokens it needs to process, which has become the core measurement standard recognized by the industry, therefore, this embodiment directly uses this standard measurement to define the hardness factor H, and the calculation method is as follows: At the same time, the system converts the input text of the request into a d-dimensional semantic vector by using a pre-trained sentence encoding model.

[0042] Secondly, after the request is assigned to a certain server and processed, the system needs to record the performance results of this service, that is, to measure and record the actual processing delay time of this request, and based on this value, the original performance benefit of this service is calculated, in this embodiment Meanwhile, the system records the request time when the current request is processed .

[0043] Meanwhile, a historical request record set and a recent performance benefit set are maintained on each server, where, for the historical request record set, the set stores detailed records of 1000 historical requests recently processed by the server , for any historical request , the set contains the hardness factor , semantic vector , original performance benefit and request time . For the recent performance benefit set, the set stores all original performance benefits of the server in a recent time window , and is implemented by a fixed-size time window queue, when a new original performance benefit is generated, the new original performance benefit is added to the set, and the oldest original performance benefit is removed.

[0044] Based on the above, when the current LLM inference request arrives at the balanced load ware, the hardness factor , semantic vector and request time of the current LLM inference request can be obtained, as well as the historical request record set S1 and the recent performance benefit set S2 of each server, which are used to evaluate the expected performance of each server processing the current LLM inference request according to the performance benefits of the historical requests processed by each server, so as to decide the optimal server to process the current LLM inference request.

[0045] Step S102, for any server, according to the hardness similarity of each historical request in the historical request record set of any server to the current LLM inference request, the utilization item of any server is initially optimized to obtain the initial optimization utilization item of any server processing the current LLM inference request.

[0046] In the prior art, the hardness factor , semantic vector and request time of the current LLM inference request are obtainedAfter obtaining the historical request record set S1 and the recent performance benefit set S2 of each server, in order to achieve intelligent allocation of heterogeneous requests while maintaining low-latency per-request decision-making, a multi-armed bandit algorithm is usually used to balance the utilization of server performance and exploration according to a decision score formula to dynamically learn and select the optimal server in an uncertain environment. The utilization term in the decision score formula evaluates the expected performance of the server in processing future requests by arithmetic mean according to the original performance benefits of all historical requests processed by the server, which is the key to realizing adaptive decision-making.

[0047] However, this indiscriminate arithmetic mean method ignores the differences between requests in quantifiable computational complexity when facing the LLM inference service scenario with a large difference in request computational workloads, that is, the computational workload of an LLM inference request is strongly correlated with the total number of tokens (the sum of the number of input tokens and the number of output tokens) it needs to process. In actual service scenarios, the historical request record set of the server will inevitably contain a large number of low-load requests with a small total number of tokens and a small number of high-load requests with a large total number of tokens. When the arithmetic mean method is used to calculate the utilization term, the average value will be significantly affected by the sample number-dominant requests. This effect causes the utilization term to fail to accurately reflect the expected performance of the server in processing a request with a specific workload.

[0048] For example, in a server's historical request record set, there will be both short-time-consuming, low-complexity requests and long-time-consuming, high-complexity requests. Due to the asymmetry of data distribution, the arithmetic mean of the utilization term becomes an effective expected value that cannot represent any specific request type. When the next request to be allocated is a high-complexity request, the utilization term affected by the performance data of a large number of low-complexity requests is a value much smaller than the actual expected time consumption, thereby underestimating the cost of processing the complex request and making an incorrect decision to allocate a high-cost task to an inappropriate server. Conversely, when the next request to be allocated is a low-complexity request, the utilization term affected by the performance data of a small number of high-complexity requests is a value greater than the actual expected time consumption, thereby overestimating the cost of processing the simple request and missing the optimal server selection.

[0049] Therefore, in this embodiment, in order to solve the performance evaluation deviation problem of the utilization term of the standard multi-armed bandit algorithm due to request heterogeneity and provide a more reliable technical performance prediction basis for subsequent load balancing decisions, a weighted average method based on request workload similarity is introduced to evaluate the expected performance of the server for the current LLM inference request without equal consideration of the Instead of relying solely on all historical processing records, higher reference weights should be given to historical requests that are more similar to the current LLM inference request in terms of total token processing volume. The greater the difference in workload between a historical request and the current LLM inference request, the lower its reference value for this prediction, and its weight should be reduced accordingly. In this way, the calculated utilization will be a value that is "tailor-made" for the specific workload of the current LLM inference request and better reflects its expected performance, thereby eliminating evaluation bias caused by request heterogeneity.

[0050] Taking any server as an example, let's call it server. For any historical request in the historical request record set of any server, calculate the squared difference between the hardness factor of the historical request and the hardness factor of the current LLM inference request, calculate the variance of the hardness factor of all historical requests in the historical request record set, use the hardness factor variance of a preset multiple as the denominator and the squared difference as the numerator to obtain the corresponding ratio, and use the negative of the ratio as the independent variable of an exponential function with the natural constant as the base to obtain the weight factor of the historical request.

[0051] The formula for calculating the weight factor of any historical request is as follows:

[0052]

[0053] in, This represents the weighting factor for any historical request. This represents an exponential function with the natural constant as its base. This represents the hardness factor of the current LLM inference request. This represents the hardness factor of any historical request. This represents the variance of the hardness factor of all historical requests in the historical request record set of any server, where 2 represents the preset multiple.

[0054] It should be noted that, The squared difference between the current LLM inference request and historical requests in terms of total token processing is used to calculate this difference. This squared difference is placed in a negative exponential Gaussian function to achieve smooth, non-linear weight decay: when the total token processing of any historical request (hardness factor) is... ) and the total number of tokens processed for the current LLM inference request (hardness factor) When they are very close, The value will approach This means the original performance gain of any historical request. will be counted into the final weighted average with its full weight, as it constitutes a highly relevant reference sample for the performance benefit evaluation of any server when processing the current LLM inference request. Conversely, as the difference between these two requests in total token processing volume increases, the value of will decay exponentially and approach zero rapidly, which means that those historical requests whose computational loads are quite different from the current LLM inference request will have their weights reduced and be filtered out.

[0055] Similarly, obtain the weight factor of each historical request in the historical request record set, and perform weighted average processing on the original performance benefits of each historical request in the historical request record set according to the weight factor of each historical request in the historical request record set, to obtain the initial optimization utilization item of any server processing the current LLM inference request.

[0056] wherein the calculation formula of the initial optimization utilization item of any server processing the current LLM inference request is:

[0057]

[0058] wherein, represents the initial optimization utilization item of any server processing the current LLM inference request, represents the original performance benefit of any historical request in the historical request record set, represents the weight factor of any historical request, and S1 represents the historical request record set.

[0059] At this time, the obtained initial optimization utilization item of any server processing the current LLM inference request is no longer a biased average value affected by all historical records indiscriminately. When processing a high-load request that needs to process a large number of tokens, the initial optimization utilization item formula will automatically refer to the performance data when the server processes other high-load requests in the past, so as to obtain a performance benefit prediction value that is not diluted by a large number of low-load requests, thereby providing a benchmark prediction for the performance evaluation of the server.

[0060] Step S103, constructing an expected residual compensation factor according to the recent performance benefit set, the historical request record set, and the semantic vector and request time of the current LLM inference request, and using the expected residual compensation factor to further optimize the initial optimization utilization item to obtain the best utilization item of any server processing the current LLM inference request.

[0061] The embodiment further finds that there are inherent limitations in relying solely on the static computational complexity (hardness factor) of the request for prediction. When using the arithmetic average method to calculate the utilization item, the dependence of the server internal state on the semantic association of the request content cannot be taken into account. The actual performance of the server depends not only on the computational load of the request, but also on the association between the request content and the current internal state of the server. In modern large language model inference engines, the key-value cache technology is widely used to speed up the generation of subsequent tokens. This means that if the content of one of the two requests with the same computational complexity (hardness factor) is highly related to the context in the server cache, the actual amount of subsequent calculation will be significantly reduced. The standard utilization item calculation method does not distinguish between the conditional high performance generated by cache hits and the performance generated by regular processing, and includes them in the same average calculation. This processing method ignores the prerequisites for generating performance data, i.e., mixing performance data generated under different internal states for calculation, which further affects the reliability of the utilization item as a stable performance prediction indicator.

[0062] For example, when the content of a request is highly related to the cached context in the server memory, the inference process of subsequent questions in a multi-round conversation can reuse a large number of calculated results, greatly shortening the processing time. Conversely, a completely new request requires a complete calculation process. The baseline performance benefit prediction value calculated by the initial optimization utilization item is not aware of the effect of cache hits caused by content semantic association. Therefore, there is a performance prediction residual between the baseline performance benefit prediction value and the original performance benefit. The size and sign of this residual contain implicit context information that the hardness factor fails to capture.

[0063] Therefore, to solve the above problems, the embodiment further optimizes the initial optimization utilization item by introducing an expected residual compensation factor to obtain a more accurate utilization item, i.e., the best utilization item. Specifically, still taking any server as an example, i.e., server , first, according to the recent performance benefit set S2, the historical request record set S1, and the semantic vector and the request time of the current LLM inference request, an expected residual compensation factor is constructed:

[0064] (1) For any historical request in the historical request record set, the initial optimization utilization item of the any server processing the any historical request is obtained according to the calculation formula of the above initial optimization utilization item, denoted as the baseline performance benefit prediction value , the difference between the original performance benefit of the any historical request and the baseline performance benefit prediction value is calculated, denoted as the performance prediction residual .

[0065] (2) Obtain an adaptive state decay rate of any server according to all original performance benefits in the set of recent performance benefits.

[0066] Specifically, a performance benefit mean and a performance benefit variance of all original performance benefits in the set of recent performance benefits are calculated, and the adaptive state decay rate of any server is obtained according to a ratio between the performance benefit variance and a square of the performance benefit mean.

[0067] wherein a calculation formula of the adaptive state decay rate of any server is:

[0068]

[0069] wherein, denotes the adaptive state decay rate of any server, denotes a performance benefit variance of all original performance benefits in the set of recent performance benefits of any server, denotes a performance benefit mean of all original performance benefits in the set of recent performance benefits of any server, denotes a preset constant for preventing a denominator from being 0, and in the embodiment of the present application, is set to Here, no limitation is made, and implementers can set it according to specific scenarios.

[0070] (3) Obtain a residual weight factor of any historical request according to semantic vector similarity and request time difference between the any historical request and the current LLM inference request, and in combination with the adaptive state decay rate of any server.

[0071] Specifically, a cosine similarity between a semantic vector of the any historical request and a semantic vector of the current LLM inference request is calculated, a time difference between a request time of the any historical request and a request time of the current LLM inference request is calculated, a reciprocal of a product of the time difference and the adaptive state decay rate is substituted into an exponential function with a natural constant as a base number to obtain an adaptive timeliness factor, and a product between the cosine similarity and the adaptive timeliness factor is taken as the residual weight factor of the any historical request.

[0072] wherein a calculation formula of the residual weight factor of any historical request is:

[0073]

[0074] wherein, denotes the residual weight factor of any historical request pair, denotes a semantic vector of the current LLM inference request, a semantic vector representing any historical request, an adaptive state decay rate representing any server, a time difference between the request time of any historical request and the request time of the current LLM inference request, an exponential function with a natural constant as the base number.

[0075] It should be noted that, is used to represent the semantic similarity between two requests. As the semantic similarity between the two requests increases, the residual weight factor tends to 1, and vice versa, when the system tries to estimate the residual of the current LLM inference request, it will prefer to refer to the historical residual that is most relevant to the content of the current LLM inference request, for example, if the current LLM inference request is a follow-up question about Python programming, the residual weight factor will make the residual generated by the historical question and answer about Python programming be given a high weight; is used to represent the adaptive timeliness factor of the current LLM inference request, which decays exponentially with the increase of the time difference, ensuring that the estimation process relies more on recent data that reflects the latest state of the server, is proportional to the variance of the server's recent performance (performance benefit variance) and inversely proportional to the square of the average performance (performance benefit mean). This means that when the server's recent performance is very stable, the value is small, and the decay of the adaptive timeliness factor will be very slow, even if the historical data is slightly earlier, it still has high reference value, on the contrary, when the server performance fluctuates sharply, the value will become very large, causing the adaptive timeliness factor to decay rapidly to zero with time, indicating that the system will quickly discard old data and only trust the latest performance feedback in this unstable state.

[0076] (4) Obtain the performance prediction residual and residual weight factor of each historical request in the historical request record set, and perform weighted average processing on the performance prediction residuals of all historical requests in the historical request record set based on the residual weight factor, to obtain the expected residual compensation factor of the any server processing the current LLM inference request.

[0077] wherein the calculation formula of the expected residual compensation factor of the any server processing the current LLM inference request is:

[0078]

[0079] wherein, represents the expected residual compensation factor of the any server processing the current LLM inference request, S1 represents the historical request record set, represents the residual weight factor of any historical request pair, represents the original performance benefit of any historical request, represents the initial optimized utilization item of any server processing the any historical request (i.e. benchmark performance benefit prediction value).

[0080] It should be noted that the expected residual compensation factor is a weighted average process, wherein, quantifies the residual of the performance benefit of any server in history processing each historical request, if , the residual is positive, corresponding to the occurrence of key-value cache hit, otherwise, the residual is negative; is a residual weight factor based on semantic similarity and timeliness, for the residual of any historical request in the historical request record set, only when both the content semantic correlation and the state information freshness are satisfied, can it have a significant impact on the calculation of .

[0081] After obtaining the expected residual compensation factor of any server, the sum between the expected residual compensation factor and the initial optimized utilization item is taken as the optimal utilization item of any server processing the current LLM inference request , so as to realize the re-optimization of the initial utilization item. Wherein, the calculation formula of the optimal utilization item is: .

[0082] At this point, the optimal utilization item of any server processing the current LLM inference request is obtained.

[0083] Step S104, according to the optimal utilization item of any server processing the current LLM inference request, the decision score of any server is obtained, the decision score of each server is obtained, and according to the decision score of each server, the current LLM inference request is adaptively load balanced.

[0084] After obtaining the optimal utilization item of any server processing the current LLM inference request by using steps S102-S103, the embodiment follows the decision framework of the multi-armed bandit algorithm, combines the optimal utilization item with an exploration item, and calculates the decision score of any server . The calculation of the decision score adopts the existing algorithm formula, which is as follows:

[0085]

[0086] Wherein, is the optimal utilization item of any server processing the current LLM inference request , is a standard exploration item, in this known technology, represents the total number of requests processed so far, representative server the number of times of being selected, is a constant for balancing exploitation and exploration, in the embodiment is set to , which is a typical value that has been proven to have good performance in practice.

[0087] The role of the exploration term is to ensure that the algorithm can explore servers whose historical performance is not optimal or the number of times of being selected is less to some extent, so as to prevent the algorithm from converging to a local optimal solution too early. Similarly, the decision score of each server for processing the current LLM inference request is obtained, and according to the decision score of each server, the server corresponding to the maximum decision score is taken as the optimal server for the current LLM inference request, and the load balancer performs a forwarding operation of the request and sends the current LLM inference request to the optimal server for processing, that is, the optimal server is used to process the current LLM inference request.

[0088] It should be noted that after the current LLM inference request is processed and the result is returned, the system will collect the performance data of this processing, and use these newly generated data to update the historical request record set S1 and the recent performance benefit set S2 corresponding to each server asynchronously and in real time. The system will update the related records (including the hardness factor , semantic vector and request time , etc.) of the current LLM inference request to the historical request record set S1 of the corresponding server, and update the performance benefit of the current LLM inference request to the recent performance benefit set S2. Through this continuous data feedback and parameter update, the load balancing method proposed in the embodiment can continuously learn the pattern changes of the request stream and the latest performance status of the server cluster, so as to maintain high-efficiency load balancing ability in long-term operation.

[0089] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A large model-based high-concurrency lightweight data channel adaptive load balancing method, characterized in that, The method comprises: When a current LLM inference request arrives at the balanced load ware, a hardness factor, a semantic vector and a request time of the current LLM inference request are obtained, and a historical request record set and a recent performance benefit set of each server are obtained, the historical request record set comprising a hardness factor, a semantic vector, an original performance benefit and a request time of each historical request; For any server, an initial optimization utilization item of the any server is optimized according to the hardness similarity of each historical request in the historical request record set of the any server and the current LLM inference request, to obtain an initial optimization utilization item of the any server for processing the current LLM inference request; An expected residual compensation factor is constructed according to the recent performance benefit set, the historical request record set and the semantic vector and the request time of the current LLM inference request, and the initial optimization utilization item is re-optimized by using the expected residual compensation factor to obtain an optimal utilization item of the any server for processing the current LLM inference request; According to the optimal utilization item of the any server for processing the current LLM inference request, a decision score of the any server is obtained, a decision score of each server is obtained, and the current LLM inference request is adaptively balanced according to the decision score of each server.

2. The large model-based high-concurrency lightweight data channel adaptive load balancing method according to claim 1, characterized in that, The initial optimization utilization item of the any server for processing the current LLM inference request is obtained by optimizing the utilization item of the any server according to the hardness similarity of each historical request in the historical request record set of the any server and the current LLM inference request, comprising: For any historical request in the historical request record set of the any server, a weight factor of the any historical request is obtained according to the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request; The weight factor of each historical request in the historical request record set is obtained, and the original performance benefit of each historical request in the historical request record set is weighted and averaged according to the weight factor of each historical request in the historical request record set, to obtain the initial optimization utilization item of the any server for processing the current LLM inference request.

3. The large model-based high-concurrency lightweight data channel adaptive load balancing method according to claim 2, characterized in that, The weight factor of the any historical request is obtained according to the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request, comprising: The square of the difference between the hardness factor of the any historical request and the hardness factor of the current LLM inference request is calculated, the hardness factor variance of all historical requests in the historical request record set is calculated, the hardness factor variance of the preset multiple is taken as the denominator, the square of the difference is taken as the numerator to obtain the corresponding ratio, the inverse of the ratio is taken as the independent variable of the exponential function with the natural constant as the base, and the weight factor of the any historical request is obtained.

4. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 1, characterized in that, The recent performance benefit set is composed of all original performance benefits in a fixed-size time window, and an expected residual compensation factor is constructed according to the recent performance benefit set, the historical request record set, and a semantic vector and a request time of a current LLM inference request, including: For any historical request in the historical request record set, an initial optimized utilization item of the any server processing the any historical request is obtained, denoted as a benchmark performance benefit prediction value, a difference between an original performance benefit of the any historical request and the benchmark performance benefit prediction value is calculated, denoted as a performance prediction residual; According to all original performance benefits in the recent performance benefit set, an adaptive state decay rate of the any server is obtained, a residual weight factor of the any historical request is obtained according to a semantic vector similarity and a request time difference between the any historical request and the current LLM inference request, and in combination with the adaptive state decay rate of the any server; The performance prediction residual and the residual weight factor of each historical request in the historical request record set are obtained, and the performance prediction residuals of all historical requests in the historical request record set are weighted and averaged based on the residual weight factor, to obtain an expected residual compensation factor of the any server processing the current LLM inference request.

5. The large model-based high-concurrency lightweight data channel adaptive load balancing method according to claim 4, characterized in that, The adaptive state decay rate of the any server is obtained according to all original performance benefits in the recent performance benefit set, including: The performance benefit mean and the performance benefit variance of all original performance benefits in the recent performance benefit set are calculated, and the adaptive state decay rate of the any server is obtained according to a ratio between the performance benefit variance and the square of the performance benefit mean.

6. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 4, characterized in that, The residual weight factor of the any historical request is obtained according to a semantic vector similarity and a request time difference between the any historical request and the current LLM inference request, and in combination with the adaptive state decay rate of the any server, including: The cosine similarity between the semantic vector of the any historical request and the semantic vector of the current LLM inference request is calculated, the time difference between the request time of the any historical request and the request time of the current LLM inference request is calculated, the inverse of the product of the time difference and the adaptive state decay rate is substituted into an exponential function with a natural constant as the base, to obtain an adaptive timeliness factor, and the product between the cosine similarity and the adaptive timeliness factor is taken as the residual weight factor of the any historical request.

7. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 1, characterized in that, The initial optimized utilization item is re-optimized by using the expected residual compensation factor, to obtain an optimal utilization item of the any server processing the current LLM inference request, including: The sum between the expected residual compensation factor and the initial optimized utilization item is taken as the optimal utilization item of the any server processing the current LLM inference request.

8. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 1, characterized in that, The decision score of the any server is obtained according to the optimal utilization item of the any server processing the current LLM inference request, including: An exploration item of the any server is acquired, and a decision score of the any server is obtained according to a sum between the best utilization item and the exploration item.

9. The large model based high concurrency light weight data channel adaptive load balancing method according to claim 1, characterized in that, The adaptive load balancing of the current LLM inference request according to the decision score of each server comprises: The server corresponding to the maximum decision score is taken as a best server of the current LLM inference request according to the decision score of each server, and the current LLM inference request is processed by using the best server.

Images