Load balancing routing system and method for large models

Abstract

The present disclosure relates to a load balancing routing system and method for large models. The system comprises: a request parsing component that receives a data processing request of a user to obtain token information of the data processing request; a subset segmentation component that divides a token subset from a head of the token information of the received data processing request each time based on a predetermined token number, and generates a hash value of each divided token subset to form one or more hash value sets; a load information collection component that collects, for each destination processing instance, the hash value set and a load amount of data processing requests being processed; a scoring component that scores each destination processing instance based on a matching degree between the hash value set of the current data processing request and each destination processing instance and a load amount of each destination processing instance; and a forwarding component that routes and forwards the current data processing request to a destination processing instance with the highest score or one of destination processing instances with the highest scores.

Description

Technical Field

[0001] This disclosure relates to the field of computer information processing, and more specifically, to a large-scale load balancing routing system and method. Background Technology

[0002] Large language model inference engines often receive a large number of user requests, which typically require significant computing resources. However, hardware resources are limited and cannot handle all requests simultaneously. In practical applications, to efficiently and stably handle a large number of concurrent requests while optimizing resource utilization and user experience, traditional large language model inference engines introduce a request scheduling layer to queue requests, wait for the inference engine to have available computing resources, dynamically adjust the processing order based on priority, and merge multiple requests to reuse hardware parallel computing capabilities and improve throughput. Therefore, traditional deep learning models typically employ data parallel processing (DP) during inference to fully utilize computing power and accelerate the inference process. In DP parallel processing, requests are sent to different DP groups (or processing instances). With a large number of parallel processing instances, if forwarding or task allocation is not managed, load imbalance can easily occur, causing some cards to operate at high load for significantly longer periods, or experience significantly longer processing wait or queuing times than others, resulting in waiting times for other cards and wasted computing power. Therefore, load balancing must be considered in the design of routing components or data distribution components.

[0003] This is a traditional technique known as continuous batching. It combines multiple batches to flexibly handle input sequences of varying lengths and updates the batches in real-time during inference, reducing resource idleness and inference latency. Traditional batching requires waiting for all requests to complete, which can lead to low hardware utilization and increased response time. Continuous batching, on the other hand, allows the model to dynamically merge requests during inference, supports different sequence lengths, and immediately replaces a new request after one is completed, thus significantly improving throughput and efficiency. This technique is particularly suitable for high-concurrency scenarios, better balancing computational resource utilization and response speed.

[0004] Furthermore, in the data inference process of traditional large-scale models, request scheduling strategies are typically prefill-first, decode-first, and Chunked Prefill. The prefill-first strategy prioritizes prefill requests when both prefill and decode requests exist. The decode-first strategy prioritizes decode requests when both exist. The Chunked Prefill strategy, when both exist, splits a prefill request into multiple parts and executes them in multiple rounds, combining them with the decode request in each round to form a complete batch before sending it to the engine.

[0005] Clearly, considering the efficiency of large model inference and the stability of the inference service, using a prefill-first or decode-first strategy will increase the inference service latency. The prefill-first strategy prioritizes prefill requests into the engine, delaying decode requests, which significantly impacts decoding requests. The decode-first strategy prioritizes decode requests into the engine, which can lead to a small number of input tokens, reduced GPU utilization, and a significant increase in the latency of the first token, affecting user experience. Similarly, the Chunked Prefill strategy, by combining prefill and decode requests into a complete batch, increases the number of tokens processed by the engine in each round, improving inference throughput. Since the number of tokens processed by the inference engine in each round is relatively stable, the latency between tokens is relatively stable. However, its total latency is still affected by the insertion of prefill requests, slowing down the processing speed of decode requests, and because prefill requests still occupy GPU memory for a long time, the concurrency of decode requests is limited.

[0006] Furthermore, due to the existence of key-value (kV) cache, if multiple rounds of communication enter the same DP group, the identical preamble can directly reuse the previous kV cache without recalculation. Therefore, the router must also consider the kV cache situation. A more complex and potentially dangerous scenario arises: if an enterprise user adds the same long prompt to all requests, then a large number of identical requests from that enterprise user will have the same prefix (kV cache). Placing these requests in the same DP group while leaving other groups empty is unreasonable.

[0007] Therefore, there is a need for a new data request routing system and method that does not require prioritizing pre-filling and decoding while fully utilizing repetitive computation and taking into account the load of specific parallel processing instances to optimize data request routing and achieve load balancing.

[0008] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] In view of this, this disclosure provides a load balancing routing system for large models, especially large-scale hardware clusters, for efficient large language models. By real-time caching to perceive the number of already computed results and idle computations that each parallel data processing instance can reuse, in the overall environment of parallel processing, the system selects the processing instance with the shortest possible processing time and the smallest possible increase in its own load as the forwarding destination. This can optimize load balancing and parallelism, improve the parallel efficiency of the entire model, and avoid unbalanced load situations.

[0010] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0011] According to one aspect of this disclosure, a load balancing routing system for large models is proposed, comprising: a request parsing component, which receives a user's data processing request and obtains token information of the data processing request; a subset partitioning component, which, based on a predetermined number of tokens, partitions a subset of tokens from the header of the received data processing request's token information each time, until the end of the token information is reached, and generates a hash value for each partitioned token subset using a hash function, thereby forming one or more hash value sets of the data processing request; a load information collection component, which, for each destination processing instance, collects and caches the hash value set of each data processing request forwarded to it by the forwarding component and the load of the data processing requests being processed by each destination processing instance; a scoring component, which scores each destination processing instance based on the matching degree between the hash value set corresponding to the current data processing request and the hash value set corresponding to the data processing requests received by each destination processing instance collected by the load information collection component for each destination processing instance, and the load of each destination processing instance; and a forwarding component, which routes and forwards the current data processing request to the destination processing instance with the highest score or one of the highest-scoring destination processing instances.

[0012] According to the load balancing routing system of the large model of this disclosure, when the subset partitioning component divides the token subsets, in two adjacent partitions, the number of token subsets divided in the later partition is twice the number of token subsets divided in the previous partition, or the number of token subsets divided in the later partition is increased by a base number relative to the number of token subsets divided in the previous partition.

[0013] According to the large-scale load balancing routing system disclosed herein, the load information collection component allocates a predetermined length of memory for each destination processing instance, and when the memory is full, it replaces the cached hash values ​​in the memory using a first-in-first-out (FIFO) approach.

[0014] According to the load balancing routing system of the large model disclosed herein, the scoring component scores both the matching degree and the load. The matching degree is scored by calculating, for any destination processing instance, the ratio of the number of matches between the hash value of the current data processing request and the number of hash values ​​cached by the load information collection component for the destination processing instance to the total number of hash values ​​in the current data processing request's hash set, as the matching hit rate. A corresponding hit rate score is assigned based on a multiple of the base number contained in the partition page corresponding to the matched hash value, thereby accumulating the total number of matches between the current data processing request and the destination processing instance. The hit rate score is obtained by summing the hit rate scores as the total hit rate score for the current data processing request for each destination processing instance; and the load score is obtained by calculating the negative of each data processing request being processed by the destination data processing instance for any given destination processing instance as a multiple of a preset length as the load score for each data processing request, thereby accumulating the load scores of all data processing requests being processed by the destination data processing instance as the total load score for that destination processing instance, and summing the total hit rate score and the total load score for each destination processing instance as the total score for the current data request for each destination processing instance.

[0015] According to the load balancing routing system of the large model disclosed herein, when the destination processing instance has a pre-filled component and a decoding service component, the forwarding component directly selects the destination processing instance with the highest total score in the pre-filled component for forwarding when the total hit rate score for any destination processing instance is lower than a predetermined threshold for the total hit rate score.

[0016] According to the load balancing routing system of the large model disclosed herein, when the destination processing instance has a pre-filled component and a decoding service component, the forwarding component directly selects the destination processing instance with the highest total score in the decoding service component for forwarding when the total hit rate score for a certain destination processing instance is higher than a predetermined threshold.

[0017] According to the large-scale load balancing routing system of this disclosure, the scoring component directly assigns a score of -1 when the length of the data processing request being processed by the destination data processing instance is less than a preset length.

[0018] According to another aspect of this disclosure, a load balancing routing method for large models is provided, comprising: receiving a user's data processing request through a request parsing component and parsing to obtain token information of the data processing request; using a subset partitioning component, with a predetermined number of tokens as the base, partitioning a subset of tokens each time starting from the header of the received data processing request's token information until the end of the token information is reached, and generating a hash value for each partitioned token subset using a hash function, thereby forming one or more hash value sets of the data processing request; using a load information collection component, for each destination processing instance, collecting the hash value set of each data processing request forwarded to it by the forwarding component and the load of the data processing requests being processed by each destination processing instance; using a scoring component, scoring each destination processing instance based on the matching degree between the hash value set corresponding to the current data processing request and the hash value set corresponding to the data processing requests received by each destination processing instance collected by the load information collection component for each destination processing instance, and the load of each destination processing instance; and using a forwarding component, routing and forwarding the current data processing request to the destination processing instance with the highest score or multiple highest-scoring destination processing instances.

[0019] According to the load balancing routing method of the large model disclosed herein, when the subset partitioning component divides the token subsets, in two adjacent partitions, the number of token subsets divided in the latter partition is twice the number of token subsets divided in the former partition, or the number of token subsets divided in the latter partition is increased by a base number relative to the number of token subsets divided in the former partition.

[0020] According to the load balancing routing method of the large model disclosed herein, the load information collection component allocates a predetermined length of memory for each destination processing instance, and when the memory is full, it replaces the cached hash value in memory in a first-in-first-out manner.

[0021] According to the load balancing routing method of the large model disclosed herein, the scoring component, when performing scoring, includes: calculating the ratio of the number of matching hash values ​​of the current data processing request to the number of hash values ​​cached by the load information collection component for the destination processing instance to the number of hash values ​​of the current data processing request as the matching hit rate for any destination processing instance; assigning a corresponding hit rate score according to a multiple of the base number contained in the partition page corresponding to the matched hash value; thereby accumulating all hit rate scores of the current data processing request relative to the destination processing instance, and obtaining the sum of the hit rate scores as the total hit rate score of the current data processing request for each destination processing instance; calculating the negative of each data processing request being processed by the destination data processing instance relative to a multiple of a preset length as the load score of each data processing request; thereby accumulating the load scores of all data processing requests being processed by the destination data processing instance as the total load score of the destination processing instance; and summing the total hit rate score and the total load score for each destination processing instance as the total score of the current data request for each destination processing instance.

[0022] According to the load balancing routing system of the large model disclosed herein, forwarding through the forwarding component further includes: when the destination processing instance has a pre-filling component and a decoding service component separately, if the total hit rate score for any destination processing instance is lower than a predetermined threshold for the total hit rate score, the destination processing instance with the highest total score in the pre-filling component is directly selected for forwarding.

[0023] According to the load balancing routing system of the large model disclosed herein, forwarding through the forwarding component further includes: when the destination processing instance has a pre-filling component and a decoding service component separately, when the total hit rate score for a certain destination processing instance is higher than a predetermined threshold for the total hit rate score, the destination processing instance with the highest total score in the decoding service component is directly selected for forwarding.

[0024] According to the large-scale load balancing routing system of this disclosure, the scoring component directly assigns a score of -1 when the length of the data processing request being processed by the destination data processing instance is less than a preset length.

[0025] Based on the load balancing routing and method of the large model disclosed herein, specific parallel processing instances are selected for resource optimization, parallel processing, and load balancing by comprehensively considering the repeatable computational load of each parallel processing instance and the data request load being processed. This significantly improves inference speed and reduces resource consumption to meet the needs of different models and tasks. In other words, this disclosure introduces a cache-aware routing mechanism that polls each node (or processing instance) to record its current load. The routing system automatically distributes requests based on the load of each node, striving to ensure relatively balanced load and stable operation across all nodes. This fully utilizes the resources of all nodes, enabling load-balanced parallel processing of the respective processing instances for filling and decoding computations.

[0026] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0027] The above and other objects, features, and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this disclosure, and those skilled in the art will be able to obtain other drawings based on these drawings without any inventive effort.

[0028] Figure 1 This is a block diagram illustrating a first embodiment of a large-scale load balancing routing system according to an exemplary embodiment.

[0029] Figure 2 This is a block diagram illustrating a second embodiment of a large-scale load balancing routing system according to an exemplary embodiment.

[0030] Figure 3 This is a flowchart of a first embodiment of a load balancing routing method for a large model, according to an exemplary embodiment.

[0031] Figure 4 This is a flowchart of a second embodiment of a load balancing routing method for a large model, as illustrated in an exemplary embodiment. Detailed Implementation

[0032] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0033] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0034] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0035] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0036] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first computing device discussed below may be referred to as the second computing device without departing from the teachings of this disclosure. As used herein, the term "and / or" includes all combinations of any and more of the associated listed items.

[0037] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing this disclosure, and therefore cannot be used to limit the scope of protection of this disclosure.

[0038] Figure 1 This is a block diagram illustrating a first embodiment of a large-scale load balancing routing system according to an exemplary embodiment. Figure 1 As shown, the large-scale load balancing routing system 100 includes: a request parsing component 110, a subset segmentation component 120, a load information collection component 130, a scoring component 140, and a forwarding component 150.

[0039] like Figure 1As shown, the request parsing component 110 receives a user's data processing request to obtain the token information for the data processing request. In a Large Language Model (LLM), a token is the basic unit of text, which can be a word, subword, or character. The model breaks down the input text into tokens for processing and understanding. The number of tokens determines the input length and computational complexity of the model. The text data input by the user is first converted into tokens and input into the model. The request parsing component 110 parses the current data processing request into one or more tokens.

[0040] Next, the subset segmentation component 120 divides one or more tokens of each data processing request into a subset of tokens, starting from the header of the received data processing request token information, based on a predetermined number of tokens, until the end of the token information is reached. Then, it uses a hash function to generate a hash value for each subset of tokens, thereby forming one or more hash value sets of the data processing request.

[0041] For example, in one segmentation method, the subset segmentation component 120 divides the tokens converted from the current data processing request according to the number of tokens per page of page attention, and sets the dividing line according to the rule of doubling each time. Assuming that the number of tokens per page in the system is 32 (cardinality), and the number of tokens in a request is 555, then the dividing line is [32, 64, 128, 256, 512]. Usually, the small number of tokens that are more than 512 in 555 are ignored because they will not affect the accuracy of the final pre-padding or decoding. Thus, the first token subset has 32 tokens, the second token subset has 64 tokens, which contains the 32 tokens in the first token subset, and so on. Each time, the resulting token subsets, starting from the 0th token of the request, are tokens[0:32], tokens[0:64], tokens[0:128], tokens[0:256], and tokens[0:512], where tokens[0:i] represents the values ​​from the 0th to the i-th token. These token subsets (or token pages) can be converted into 5 hash values ​​using a fixed hash function. The advantage of this partitioning is that it maintains an overall linear complexity; for example, the complexity above is only 32 + 64 + 128 + 256 + 512 = 992 <= 2 * 555.

[0042] Alternatively, each page can be divided once, meaning a subset of tokens is partitioned from the header of the current data processing request each time, based on a predetermined number of tokens. In two consecutive partitions, the number of tokens in the later partition increases by the base number relative to the previous partition. For example, 32, 64, 96, 128, 160, ..., 512, 544. However, the complexity would be n^2: 32 + 64 + 96 + 128 + 160 + ... + 512 + 544 = 32 * (1 + 17) * 17 / 2. Here, 17 or n increases linearly with the total number of tokens, so the complexity is quadratic.

[0043] For each destination processing instance, the load information collection component 130 collects and caches the set of hash values ​​of each data processing request forwarded by the forwarding component 150 and the load of data processing requests being processed by each destination processing instance. In other words, when the forwarding component 150 forwards each data processing request, the load information collection component 130 obtains the ID of the processing instance as the forwarding destination and caches the set of hash values ​​of the forwarded data processing requests, thus also knowing how many data processing requests the forwarding component 150 forwarded to each data processing request. Specifically, for each destination processing instance, a data structure is established to cache information related to the data processing requests it receives. This data structure only needs to be able to record the hash values ​​of the segmented token subsets of the received data requests in the order of input. For example, the load information collection component 130 uses a dict and a FIFO list to store the hash values ​​of existing requests from 0 to the dividing line for each processing instance used for data parallel inference. Therefore, the load information collection component 130 allocates a predetermined length of memory for each destination processing instance, and when the memory is full, it replaces the cached hash value in the memory using a first-in-first-out method.

[0044] The hash values ​​in the dictionary here are somewhat like indices of the subset of tokens corresponding to those hash values. In other words, a dictionary is a mapping from a hash value to the number of times that hash value appears in the current processing instance. The FIFO list (First In First Out) means that the oldest hash is popped first. Because the number of pages is limited, after the cache runs out, new requests will push out the caches of older requests. Therefore, the FIFO list also has a size. Before the number of hash values ​​stored reaches the maximum size of the FIFO list, it stores the hash values ​​of all incoming requests according to the dividing lines. Each request may store multiple hash values, and the number of dividing lines, i.e., the number of hash values, is determined by the token length. Once the number of hash values ​​stored reaches the maximum size of the FIFO list, the FIFO list will pop the oldest hash value each time, and then put a new hash value in, repeating this process multiple times. Multiple hash values ​​from a single request are input in ascending order of page number. Therefore, multiple hash values ​​from the same request also have a specific order.

[0045] Examples of such FIFO lists are as follows:

[0046] For the first processing instance, the load information collection component 130 recorded 3 requests.

[0047] I ate an apple and a banana.

[0048] I ate an apple and a pineapple.

[0049] "I ate apples and bananas, they were so delicious I couldn't stop eating them."

[0050] For ease of explanation, let's assume there's only 1 token per page, although this is practically impossible; it's usually 32 or 128. Here, we simply assume each character and each punctuation mark represents one token. Also, for simplicity, let's assume the corresponding hash value is 4 digits (hash values ​​are very long; 4 digits are used here for clarity). The hash value of the token subset after splitting the aforementioned request tokens (for simplicity, let's assume it conforms to a predetermined partitioning rule; this rule can be manually set and isn't necessarily the one exemplified in this disclosure) is:

[0051] "Me" → 1234

[0052] “I eat” → 2222

[0053] "I ate an apple" → 3214

[0054] "I ate an apple and a banana" → 4444

[0055] "I ate an apple and a pineapple" → 8888

[0056] "I ate apples and bananas, they were so delicious I couldn't stop eating!" → 5555

[0057] Assuming the FIFO list has a length of 7 (for example, it can only store 7 hash values. In real-world scenarios, the length can be set to any finite value depending on the data processing volume), then both the FIFO list and the dictionary are empty before the first request arrives.

[0058] FIFO_list = [], dict = {}

[0059] After the first request "I ate an apple and a banana" comes in, its data structure is as follows:

[0060] FIFO_list=[1234, 2222, 3214, 4444]

[0061] Dict = {1234:1, 2222:1, 3214:1, 4444:1}, where 1 indicates that the hash value exists only once in FIFI_list.

[0062] If the second request "I ate apples and pineapples" comes in, the data structure becomes:

[0063] FIFO_list = [2222, 3214, 4444, 1234, 2222, 3214, 8888]

[0064] Dict = {1234: 1, 2222: 2, 3214: 2, 4444: 1, 8888: 1}. Here, 1234 appears only once; 2222 appears twice, in positions 1 and 5; 3214 appears twice, in positions 2 and 6; 4444 appears only once; and 8888 appears only once. It can be seen that in the FIFO_list, the oldest hash 1234 was pushed out before the last hash 8888 even entered.

[0065] If the third request, "I ate apples and bananas, and they were so delicious I couldn't stop eating them," comes in, the data structure becomes:

[0066] FIFO_list = [3214, 8888, 1234, 2222, 3214, 4444, 5555]

[0067] Dict = {8888: 1, 1234: 1, 2222: 1, 3214: 2, 4444: 1, 5555: 1}

[0068] Because the FIFO list was already full before the third request came in, for every new hash value that was added, an old hash value was pushed out.

[0069] Using these two data structures, in the first partitioning, the maintenance of the mapping only requires O(1) complexity. Not only can it immediately confirm whether a hash value exists in the corresponding processing instance with O(1) complexity, but it can also squeeze out the oldest hash value in chronological order after the FIFO_list is full, perfectly simulating the key-value cache process. This allows the load information collection component 130 to restore the true load situation of each processing instance as accurately as possible. If the simulation is poor, for example, it might assume that the first processing instance contains two requests, a and b, but in reality, the first processing instance contains two requests, c and d, or that the first processing instance has 100 requests while the second processing instance is empty. When encountering subsequent problems with request a, because the simulation was poor and the true load situation was not understood, placing it in the first processing instance will increase the load on the first processing instance.

[0070] Furthermore, the load information collection component 130 collects the load of data processing requests being processed by each destination processing instance by probing the current running status of each processing instance. Specifically, after each processing instance completes a data processing request or a subset of tokens, it can report the task completion information to the load information collection component 130. The load information collection component 130, based on the previously known number of data processing requests or the number and order of token subsets forwarded by the forwarding component 150 to each data processing request, subtracts the number of completed requests to determine the number of data requests currently being processed by each processing instance and the number of tokens for each data request.

[0071] Subsequently, the scoring component 140 scores each destination processing instance based on the matching degree between the hash value set corresponding to the current data processing request and the hash value set corresponding to the data processing requests received by each destination processing instance collected by the load information collection component, as well as the load of each destination processing instance. Specifically, the matching degree scoring is performed by calculating, for any destination processing instance, the ratio of the number of matching hash values ​​of the current data processing request with the number of hash values ​​cached by the load information collection component for the destination processing instance to the number of hash values ​​in the current data processing request's hash value set as the matching hit rate. A corresponding hit rate score is assigned according to a multiple of the base number contained in the partition page corresponding to the matched hash value, thereby accumulating all hit rate scores of the current data processing request relative to the destination processing instance, and obtaining the sum of the hit rate scores as the total hit rate score of the current data processing request for each destination processing instance.

[0072] For example, for a processing instance, after the first processing instance has calculated the request "I ate apples and bananas", the DIct for that first processing instance is:

[0073] Dict = {1234: 1, 2222: 1, 3214: 1, 4444: 1},

[0074] After the second processing instance has calculated a request: "I ate an apple and a pineapple", the Dict for that second processing instance is:

[0075] Dict = {1234: 1, 2222: 1, 3214: 1, 8888: 1}

[0076] Then, there's a current data processing request: "I ate apples and bananas, and they were so delicious I couldn't stop eating them." The hash value of each split page is: Hash: 1234, 2222, 3214, 4444, 5555

[0077] As mentioned above, multiple hash values ​​are calculated for a single request based on the dividing line. Since different hash values ​​are assigned scores, the score is based on the number of pages. That is, one point is awarded for each page hit. The score for hitting different hash values ​​will vary. The score is assigned as a multiple of a predetermined number of tokens (e.g., 32 tokens). Therefore, the scores for hitting the 1st to 5th hash values ​​are: 1, 2, 4, 8, 16 respectively. As previously mentioned, assuming the dividing line is obtained by doubling the number of tokens per page, hitting the first hash value (one page of tokens) means that in actual data inference, one page of tokens can be counted less; hitting the second hash value means two pages less; hitting the third hash value means four pages less; hitting the fourth hash value means eight pages less, and so on. Therefore, the hash value set for the current data processing request, "I ate apples and bananas, and they were so delicious I couldn't stop eating them," is: Hash: 1234, 2222, 3214, 4444, 5555. These hash values ​​are matched against the hash values ​​in the Dict cached by the first and second processing instances, respectively. This assigns a score to the hash value hit rate for the current data processing request. For the first processing instance, hitting the first four hashes results in a total hit rate of 15, while for the second processing instance, hitting the first three hashes results in a total hit rate of 7. Therefore, the forwarding component 150 will route the current data processing request to the data processing instance with the highest matching hit rate score in the pre-filling or decoding service component 10, i.e., the first processing instance, thereby reducing the computation and / or access volume of the first processing instance performing the data filling or decoding service.

[0078] Furthermore, load is scored by calculating the negative of each data processing request being processed by the destination data processing instance for any given destination data processing instance, multiplied by a multiple of a preset length. This is used as the load score for each data processing request. The total load score for that destination data processing instance is calculated by accumulating the load scores of all data processing requests being processed by that instance. Finally, the total hit rate score and the total load score for each destination processing instance are summed to obtain the total score for the current data request for that destination processing instance. Load balancing requires consideration of both computation time and memory load. Simply put, the memory usage of a request with 100,000 tokens is definitely different from that of a request with 20 tokens. However, at the router, only the prefill part is used, not the decode part. A request with 100,000 tokens might only generate one token and then terminate. A request for 20 tokens might generate 8000 tokens (a simple example is a request like "Write me an 8000-word essay"). The length of the generated tokens is unpredictable, but fortunately, based on statistics, we can know the average length of a typical request. The same applies to computation time. Request A's prefill length might be 5000 times that of request B, but the actual computation time of A might only be 50 times that of B. In the LLM model, the attention component increases linearly with the number of tokens for each decoded token, but the MLP's time remains constant. It should be noted that generating one token for a request with 100,000 tokens will definitely take longer than generating one token for a request with 20 tokens, but it's certainly not 5000 times longer.

[0079] For example, for a specific model, an experience token value for a request can be set based on experience to evaluate the load score. To show that the more load, the lower the score, a negative score is applied to the load added to a specific processing instance; that is, -1 point is assigned for each additional basic unit (experience token value) of load request, and -1 point is assigned to any subset of tokens, even if its length is less than one basic unit. Therefore, the score can be calculated using the formula: Score = -(token_len / / 2048 + 1), where / / represents the quotient (without remainder, no decimal part), token_len is the length of the token set, and 2048 is the experience length. Of course, for simplicity, a score can also be directly assigned based on the number of load requests, without considering the length of the load requests, that is, without using the experience value, because from the perspective of normal distribution, when there are a large number of requests, the overall average length does not change much. Therefore, directly using a counting method, assigning -1 point for each additional load request for each processing instance, can also achieve the purpose of this disclosure. Of course, any method that achieves the result of a linear decrease in the assigned score as the load increases can realize the technical effect of this disclosure. Alternatively, for a specific purpose, a non-linear decrease or variation in the assigned score can also be used. This means that for each request accepted by a processing instance, the increased score is added to the total load score of that processing instance. Note that this total load score is negative, meaning that the more requests a processing instance accepts, the lower its total load score becomes, and new requests will tend to go to other processing instances.

[0080] Furthermore, it should be noted that the total load score of the data processing requests being processed by the data processing instance gradually increases as the tasks within it are completed. That is, each time a request finishes processing in an instance, the corresponding score is deducted from the total load score of that instance. (Because the score is negative, the instance's score increases when the request ends). Also, due to certain special requests, a request may not finish normally in a processing instance (e.g., it is canceled), so the processing instance cleans up requests that exceed a certain duration (temporarily set at 80 seconds. This is sufficient for short requests. For scenarios requiring long requests, or where the model runs slowly, such as Deepseek-R1, it can be set to 10 minutes or 30 minutes). In other words, the load information collection component 130 obtains the completion or cancellation status of data processing requests reported by each processing instance, or in other words, it obtains the number and length of data processing requests being processed by each processing instance. The load information collection component 130 also obtains the liveness status of each processing instance.

[0081] Clearly, this disclosure can consider only the hit rate to obtain a total hit rate score as the final score for subsequent forwarding processing, or it can consider only the load balance to obtain a total load score for forwarding processing, or it can consider both the hit rate and load balance to obtain a comprehensive score and obtain the sum of the total hit rate score and the total load score for forwarding processing. That is, the sum of the total hit rate score and the total load score for each destination processing instance is used as the total score of the current data request for each destination processing instance.

[0082] Finally, the forwarding component 150 routes the current data processing request to the destination processing instance with the highest score or one of the highest-scoring destination processing instances. The specific forwarding process is not detailed here; conventional forwarding techniques can be used. The destination processing instance is the processing instance used for pre-filling or decoding processing, for example... Figure 1 Processing instances 1-n in the pre-filled or decoding component 10 of the seed.

[0083] Figure 2 This is a block diagram illustrating a second embodiment of a large-scale load balancing routing system 200 according to an exemplary embodiment. Figure 1 Compared to the large-scale load balancing routing system 100 shown, the pre-filling or decoding component 10 is divided into two independent pre-filling components 20 or decoding service components 30, with reference markers for other identical parts being the same. Figure 1 The similarities are only in that they begin with the number "2". Therefore, the descriptions of the same parts adopt a targeted approach. Figure 1 The descriptions are not detailed here.

[0084] Since pre-padding computation is computationally intensive, it can be executed on machines with higher computing power (metric: tflops). Decoding, on the other hand, is memory-bandwidth limited and can be performed on machines with greater video memory bandwidth. Therefore, to optimize the latency of the first token and the latency between tokens by splitting the pre-padding and decoding computations onto different machines without interference, as follows... Figure 2As shown, the prefilling component 20 and the decoding service component 30 are deployed separately. For example, the prefilling component 20 is deployed on the first computing device, and the decoding service component 30 is deployed on the second computing device. Alternatively, they can be deployed on different physical parts of the same computing device. Therefore, from the perspective of the large model, the traditionally monolithic large model is divided into a sub-model for performing prefilling inference and a sub-model for performing decoding inference. This separate deployment eliminates the situation where prioritizing prefilling inference and decoding inference leads to one being neglected for the other. That is, it eliminates the defect that a prefilling priority strategy has a greater impact on decoding requests, while a decoding priority strategy results in too few input tokens, reduced GPU utilization efficiency, and a significant increase in the latency of the first token, affecting the user experience. Therefore, it also eliminates the defect that a prefilling priority strategy or a decoding priority strategy will increase the latency of the inference service. Similarly, it also eliminates the defect that the total latency of the Chunked Prefill strategy is still affected by the insertion of prefilling requests, which slows down the processing speed of decoding requests, and that the concurrency of decoding requests is limited because prefilling requests still occupy video memory for a long time.

[0085] Therefore, even when the pre-filled component 20 and the decoding service component 30 are deployed separately, it is still possible to follow the... Figure 1 In addition to routing, the forwarding component 250 can also determine the length of a request based on the total hit rate score, total load score, and the sum of the two scores given by the scoring component 240. Specifically, if the total hit rate score for any destination processing instance is lower than a predetermined threshold, the forwarding component 250 directly selects the destination processing instance with the highest total score from the pre-filling component 20 for forwarding. Conversely, if the total hit rate score for any destination processing instance is higher than the predetermined threshold, the forwarding component 250 directly selects the destination processing instance with the highest total score from the decoding service component for forwarding.

[0086] Requests whose total hit rate score is below a predetermined threshold are called long requests. These are typically relatively new requests, and the cached context representations related to long requests are relatively few throughout the model. Since the decoding phase involves the model generating output tokens one by one, it is serialized, requiring frequent memory accesses each time a token is generated (e.g., loading model parameters and KV cache). Due to the relatively small computational load and the underutilization of GPU parallel computing capabilities, the performance bottleneck is mainly memory bandwidth; therefore, the decoding phase is a typical "memory-constrained" phase. Therefore, routing long requests to the pre-filling component 20 for pre-filling before decoding is a better approach, while routing short requests directly to the decoding service component 30 is also a good solution. Thus, to determine the length of a request, in addition to the length of the request text itself, it is also necessary to consider whether the context representation cached by the trained model exists or is mostly present. If most of it already exists, it means that even a long request is relatively short relative to the larger model, because only the remaining tokens need to be decoded. Therefore, the relative length of the hit rate can be determined based on the model's cached tokens within the existing data of the large model. The predetermined threshold for the total hit rate score here can be 50%, 60%, or 70%. Setting it to 60% typically achieves a compromise in efficiency. If a higher predetermined threshold for the total hit rate score is required, the second method of segmenting the request's token information can be used. This way, when calculating the score, the matching degree for more than half of the requests will exceed 50%.

[0087] like Figure 2As shown, the pre-filling component 20, through its processing instance, pre-fills long requests forwarded from the forwarding component 250, generating an initial context representation (KV cache). Its processing instance then updates the load information collection component 230 in real-time with information including its own cache, load, and liveness status. Simultaneously, it transmits the original long request and the generated initial context to the decoding service component 30. The decoding service component 30, through its processing instance, performs decoding based on short requests forwarded from the forwarding component 250 and the cached context associated with those short requests, generating an output token corresponding to the short request. It also performs decoding based on the original long request transmitted from the pre-filling component 20 via a communication component (not shown) and the initial context generated for the long request, generating an output token corresponding to the long request. Its processing instance then updates the load information collection component 230 with information including its own cache, load, and liveness status. The communication component between the decoding service component 30 and the pre-filling component 20 is an RDMA communication component. KVCache is a caching mechanism used to store intermediate states generated during the inference process of the Transformer model. In the Transformer model, each inference step (whether in the pre-filling or decoding phase) requires generating a corresponding Key and Value based on the current input. These intermediate states are reused in subsequent inference steps. The role of KVCache is to cache these intermediate states, avoiding repeated calculations during each inference, thereby significantly improving inference efficiency. In the pre-filling and decoding separation scenario disclosed in this paper, after the pre-filling instance completes the pre-filling phase, it needs to transfer the KVCache to the decoding instance. The core objective of the KVCache transfer module is to achieve high-speed, low-overhead data transmission to ensure that the performance advantages of the inference system in the pre-filling and decoding separation mode can be fully realized; therefore, RDMA is adopted. RDMA is a high-performance network transmission technology that allows network devices to directly access the memory of remote hosts without CPU involvement in data copying, significantly reducing transmission latency and CPU overhead. This RDMA communication can be implemented using the network hardware of the computing devices deployed by the pre-filling component and the decoding service component using conventional RDMA settings. Alternatively, zero-copy technology can be used to directly transfer the KVCache from the memory of the Prefill node to the memory of the Decode node by avoiding multiple copies of data between the kernel space and user space, thereby further reducing transmission latency. Alternatively, an efficient network protocol can be used, such as RoCEv2 or InfiniBand, to ensure low-latency, high-bandwidth data transmission in a high-speed network environment.The communication components can also be implemented using the RDMA communication ports of the computing devices deployed by the decoding service component 30 and the pre-filled component 20, respectively.

[0088] When the total hit rate score of a certain destination processing instance is higher than a predetermined threshold, the forwarding component 250 directly selects the destination processing instance with the highest total score in the decoding service component for forwarding. This means that regardless of whether the new processing request is long or short, most of it has already been processed by a certain destination processing instance. Therefore, further processing by that destination processing instance will save a lot of computing and memory, which will significantly reduce the load on the entire data processing system and speed up the processing of the new request.

[0089] Although the request token can be directly used to evaluate the hit rate, using the hash values ​​of the segmented token subset obtained from the above-mentioned request segmentation to evaluate the length of the request makes the hit rate evaluation faster and more efficient. It also allows for load balancing within the pre-filling component 20 or the decoding service component 30. Specifically, after forwarding each received data processing request to the pre-filling component 20 or the decoding service component 30, the load information collection component 230 collects and caches the hash values ​​of the data processing requests routed from the forwarding component to the destination data processing instance. It then calculates the hit rate score between the hash value of the current data processing request and the hash values ​​of the data processing requests processed by each data processing instance. Based on the calculated hit rate score, the forwarding component 250 routes the received current data processing request to the data processing instance with the highest hit rate score in the pre-filling component 20 or the decoding service component 30. In layman's terms, this means allocating new requests to processing instances that have previously processed highly similar requests, thereby making full use of the computational results already performed during the processing of new requests and reducing the amount of computation.

[0090] Alternatively, when the pre-filling component 20 and the decoding service component 30 are separate, request delivery can be performed only after considering load balancing and scoring. After forwarding each received data processing request to the pre-filling component or the decoding service component, the load information collection component 230 collects and caches the hash values ​​of the data processing requests routed from the forwarding component to the destination data processing instance, in order to calculate the load score of the data processing requests routed from the routing component that the destination data processing instance is currently processing. Based on the calculated load score, the forwarding component 250 load-balances the current data processing request received to the data processing instance with the highest load score in the pre-filling component 20, where a higher number of data processing requests being processed by the data processing instance corresponds to a lower load score. The length of the data request is not considered at this time.

[0091] Alternatively, when the pre-filling component 20 and the decoding service component 30 are separate, the forwarding component 250 can, after receiving the total hit rate score, total load score, and the sum of the two scores from the scoring component 240, and considering both the hit rate and load balancing for a comprehensive score before request delivery, determine the length of the request based on the total hit rate score. It can directly add the two scores, or use a weighted sum based on priority given to the hit rate score, priority given to the load score, or no priority given to any other score, to select the processing instance to be routed. That is, if the first processing instance has a hit rate score of 70 and a load score of -30, or a hit rate score of 60 and a load score of -10, and since the first processing instance has a comprehensive score of 40 while the second processing instance has a comprehensive score of 50, the current data processing request will be routed by the forwarding component 250 to the second processing instance for data processing.

[0092] Alternatively, when the pre-filling component 20 and the decoding service component 30 are separate, a SmoothQuant W8A8 smoothing quantization component (not shown) can be deployed in association with the pre-filling component 20. This component uses a smoothing factor to rescale the activation values ​​and weights accessed by the pre-filling component 20, and then performs quantization processing on the smoothed weights and activation values, converting them into INT8 numerical format before using them for pre-filling processing in the pre-filling component 20. Correspondingly, when the pre-filling component 20 and the decoding service component 30 are separate, a WeightOnly quantization component (not shown) can be deployed in association with the decoding service component 30. This component performs quantization processing on the weight values ​​to be accessed by the decoding service component 30, converting them into INT4 numerical format before using them for decoding processing in the decoding service component 30.

[0093] In traditional large language model inference, quantization of weights can significantly reduce model memory overhead, correspondingly reducing the overhead of reading weights and thus accelerating inference. Traditional quantization schemes for large models mainly fall into two categories: WeightOnly quantization, which quantizes only the model's weights without quantizing activation values, reducing model storage overhead. During model inference, the weights are first dequantized, and then normal-precision calculations are performed with the activation values; this scheme can typically quantize weights to 8 bits or 4 bits. SmoothQuant W8A8 quantization rescales activations and weights using a smoothing factor, reducing the scale of outliers in activations and performing fixed-point operations during inference, while simultaneously reducing storage and computational overhead. However, in traditional single-instance deployments (i.e., traditional unified deployments of pre-filling and decoding), it's difficult to coordinate the quantization of the pre-filling and decoding scenarios. Specifically, the pre-filling instance, due to processing a large number of tokens, has a correspondingly high computational load, making it a computationally constrained scenario. From an optimization perspective, a more computationally efficient quantization scheme should be chosen, such as the SmoothQuant W8A8 quantization strategy. On the other hand, the decoding instance processes fewer tokens, has a lower computational load, and needs to frequently read from the KVCache in memory, making it a memory-constrained scenario. A more memory-efficient quantization scheme should be chosen, such as the WeightOnly quantization scheme. Moreover, the WeightOnly scheme requires dequantizing the weights back to FP16 data type and performing FP16 matrix multiplication, which is often less computationally efficient than SmoothQuant in the pre-filling stage. Therefore, in a unified deployment of pre-filling and decoding, it's difficult to reconcile the two scenarios and achieve satisfactory quantization results for both, leading to more work for later adjustments and optimizations, thus negating the purpose of later optimizations.

[0094] Therefore, the pre-filling and decoding scenarios of this disclosure are separated to provide separate quantization strategies for each scenario and achieve their respective quantization optimization effects. On one hand, by using the SmoothQuant W8A8 quantization component to rescale the activation values ​​and weights used to access the pre-filling component 20, and then quantizing the smoothed weights and activation values ​​into INT8 numerical format, the pre-filling component 20 can fully utilize the INT8 tensor core within the hardware to perform INT8 matrix operations. Its computational intensity is often twice that of unquantized FP16 matrix operations, accelerating the pre-filling calculation process. On the other hand, since the decoding service component 30 processes a smaller number of tokens, its computational load is not high, and it frequently needs to read from the KVCache in memory. By using the WeightOnly quantization component to quantize the weight values ​​to be accessed by the decoding service component 30 into INT4 numerical format for the decoding service component 30 to perform decoding, its weight reading weight is half that of the SmoothQuant W8A8 quantization strategy, resulting in better decoding performance.

[0095] Therefore, by separating the prefilling and decoding scenarios, it becomes possible to freely choose appropriate quantization schemes for each scenario. For the prefilling component 20, a more computationally efficient SmoothQuant W8A8 quantization component is deployed in conjunction to achieve better prefill performance. For the decoding service component 30, a WeightOnly 4-bit quantization component is deployed in conjunction to achieve higher memory access efficiency, thereby making the overall inference system performance better than that of the inference system in a unified hybrid deployment scenario.

[0096] Figure 3 This is a flowchart of a first embodiment of a large-scale load balancing routing method 300 according to an exemplary embodiment. Figure 3 As shown, in the load balancing routing method 300 of the large model, after the user inputs the current data processing request, in step S310, the request parsing component 110 or 210 receives the user's data processing request to obtain the token information of the data processing request. In a large language model (LLM), a token is the basic unit of text, which can be a word, subword, or character. The model decomposes the input text into tokens for processing and understanding. The number of tokens determines the input length and computational complexity of the model. The text data input by the user is first converted into tokens and input into the model. The request parsing component 110 parses the current data processing request into one or more tokens.

[0097] Next, in step S320, the subset segmentation component 120 or 220, for each data processing request's one or more tokens, divides a subset of tokens from the header of the received data processing request's token information, using a predetermined number of tokens as a base, until the end of the token information is reached. A hash value is generated for each of the divided token subsets using a hash function, thereby forming one or more hash value sets for the data processing request. When the subset segmentation component 120 divides the token subsets, in two adjacent divisions, the number of token subsets in the later division is twice the number of token subsets in the previous division, or the number of token subsets in the later division is increased by a base number relative to the number of token subsets in the previous division.

[0098] After the segmentation, in step S330, the scoring component 140 or 240 scores each destination processing instance based on the matching degree between the hash value set corresponding to the current data processing request and the hash value set corresponding to the data processing requests received by each destination processing instance collected by the load information collection component, as well as the load of each destination processing instance. In step S350, the load information collection component collects the hash value set of each data processing request forwarded to it by the forwarding component and the load of the data processing request being processed by each destination processing instance. Although the description here shows step S350 after step S330, due to the cyclical nature of the entire data processing, step S350 precedes step S330 for any new data processing request. Finally, in step S340, the forwarding component 150 or 250 routes the current data processing request to the destination processing instance with the highest score or multiple highest-scoring instances. The specific processing procedure for each step can be found in the documentation for... Figure 1 and 2 The details described will not be elaborated upon here.

[0099] Figure 4 This is a flowchart of a second embodiment of a large-scale load balancing routing method 400 according to an exemplary embodiment. Figure 4 and Figure 3 The same steps use essentially the same step numbering, the difference being that "S4" is used as the first letter instead. Figure 3The only difference between the two steps is the addition of step S431 between step S430 and step S440, the division of step S340 into steps S441 and S442, and the division of step S360 into steps S461 and S462. Other identical steps will not be described in detail here.

[0100] like Figure 4 As shown, in step S430, the scoring component 140 or 240 scores each destination processing instance based on the matching degree between the hash value set corresponding to the current data processing request and the hash value set corresponding to the data processing requests received by each destination processing instance collected by the load information collection component for each destination processing instance, and the load of each destination processing instance. Then, in step S431, if the total hit rate score for any destination processing instance is lower than the predetermined threshold for the total hit rate score ("Yes"), the forwarding component 250 directly selects the destination processing instance with the highest total score in the pre-filling component 20 for forwarding. If the total hit rate score for a certain destination processing instance is higher than the predetermined threshold for the total hit rate score, the forwarding component 250 directly selects the destination processing instance with the highest total score in the decoding service component 30 for forwarding.

[0101] In step S431, after directly selecting the destination processing instance with the highest total score in the pre-filling component 20 for forwarding, in step S441, the processing instance with the highest routing score in the pre-filling component 20 is selected as the destination for routing forwarding. It is important to note that, in order to perform the subsequent pre-filling process, the maximum number of tokens for long requests is set to 1. Subsequently, in step S461, the pre-filling component 20, through its processing instance, pre-fills the long request forwarded from the forwarding component 250, generating an initial context representation (KV cache), and its processing instance updates the load information collection component 230 in real time with information including its own cache, load, and liveness status, while simultaneously transmitting the original long request and the generated initial context to the decoding service component 30. Next, in step S462, the decoding service component 30 performs decoding based on the original long request transmitted from the pre-filling component 20 via a communication component (not shown) and the initial context generated for the long request, generating an output token corresponding to the long request, and its processing instance updates the load information collection component 230 with information including its own cache, load, and liveness status. The communication component between the decoding service component 30 and the pre-filling component 20 is an RDMA communication component, which will not be described in detail here.

[0102] Similarly, in step S431, after the forwarding component 250 directly selects the destination processing instance with the highest total score in the decoding service component 30 for forwarding when the total hit rate score of all destination processing instances is higher than a predetermined threshold, then in step S462, the decoding service component 30, through its processing instance, performs decoding processing based on the short request forwarded from the forwarding component 250 and the cached context associated with the short request, generating an output token corresponding to the short request. The fact that the forwarding component 250 directly selects the destination processing instance with the highest total score in the decoding service component for forwarding when the total hit rate score of all destination processing instances is higher than the predetermined threshold means that regardless of whether the new processing request is long or short, most of it has already been processed by a destination processing instance. Therefore, further processing by that destination processing instance saves a lot of computation and memory, significantly reducing the load on the entire data processing system and speeding up the processing of the new request.

[0103] In summary, compared to traditional routing mechanisms that typically adopt globally fixed routes, the system and method of this disclosure comprehensively consider the repeatable computational load of each parallel processing instance and the data request load being processed. By selecting specific parallel processing instances for resource optimization, parallel processing, and load balancing, this significantly improves inference speed and reduces resource consumption to meet the needs of different models and tasks. In other words, this disclosure introduces a cache-aware routing mechanism that polls each node (or processing instance) to record its current load. The routing system automatically distributes requests based on the load of each node, striving to ensure relatively balanced load across nodes and stable operation. This fully utilizes all node resources, enabling load-balanced parallel processing of both filling and decoding computations. Furthermore, this disclosure optimizes the latency of the first token and the latency between tokens by separating the pre-filling and decoding computation stages and executing them on different machines without interference. Based on this separation, a cache-aware routing mechanism is introduced. Each node (or processing instance) is polled to record its current load, and the routing component automatically distributes requests based on the load of each node, aiming for a relatively balanced load across nodes and stable operation. This fully utilizes all node resources, enabling load-balanced parallel processing of the pre-filling and decoding computations. In addition, based on the separated pre-filling and decoding computation structure, this disclosure introduces RDMA high-performance network transmission technology. Network devices can directly access host memory for data transmission without CPU involvement in the data copying process, reducing transmission overhead and significantly lowering transmission latency. Finally, based on the separated pre-filling and decoding computation structure, this disclosure allows pre-filling instances and decoding instances to select appropriate model quantization strategies according to the resource needs of the inference scenario, thereby improving the utilization of computing resources.

[0104] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0105] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0106] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. A load balancing routing system for large models, comprising: a request parsing component configured to receive a data processing request from a user to obtain token information of the data processing request; a subset splitting component configured to split a token subset from a head of the token information of the received data processing request each time based on a predetermined token number until a tail of the token information is reached, and generate a hash value of each split token subset using a hash function, thereby forming one or more hash value sets of the data processing request; a load information collecting component configured to collect and cache, for each destination processing instance, a hash value set of each data processing request forwarded to the destination processing instance by a forwarding component and a load amount of data processing requests being processed by each destination processing instance; a scoring component configured to score each destination processing instance based on a matching degree between a hash value set corresponding to a current data processing request and hash value sets corresponding to data processing requests received by the load information collecting component for each destination processing instance and a load amount of each destination processing instance; and a forwarding component configured to route and forward the current data processing request to a destination processing instance with the highest score or one of destination processing instances with the highest scores.

2. The load balancing routing system for large models of claim 1, wherein the subset splitting component, when splitting the token subsets, doubles a number of token subsets split in a later time from a number of token subsets split in a previous time or increases the number of token subsets split in the later time by one base number relative to the number of token subsets split in the previous time.

3. The load balancing routing system for large models of claim 1 or 2, wherein the load information collecting component allocates a memory with a predetermined length for each destination processing instance, and replaces hash values cached in the memory using a first-in-first-out manner when the memory is filled. ​ 4. The load balancing routing system of a large model according to claim 1, wherein the scoring component respectively scores the matching degree and the load amount, the scoring of the matching degree is that for any destination processing instance, the number of hash values matched by the hash value of the current data processing request and the hash values cached by the destination processing instance is calculated as the matching hit rate, and the corresponding hit rate score is given according to the multiple of the radix contained in the division page corresponding to the matched hash value, so as to accumulate the total hit rate score of the current data processing request with respect to each destination processing instance, and obtain the sum of the hit rate scores as the total hit rate score of the current data processing request with respect to each destination processing instance; and the scoring of the load amount is that for any destination processing instance, the negative number of the multiple of the preset length of each data processing request being processed by the destination data processing instance is calculated as the load amount score of each data processing request, thereby accumulating the load amount scores of all data processing requests being processed by the destination data processing instance as the total load amount score of the destination processing instance, and summing the total hit rate score and the total load amount score of each destination processing instance as the total score of the current data request with respect to each destination processing instance.

5. The load balancing routing system of a large model according to claim 4, wherein the forwarding component directly selects the destination processing instance with the highest total score in the pre-filling component for forwarding when the total hit rate score of any one destination processing instance is lower than the predetermined threshold of the total hit rate score in the case that the destination processing instance exists in the pre-filling component and the decoding service component.

6. The load balancing routing system of a large model according to claim 4, wherein the forwarding component directly selects the destination processing instance with the highest total score in the decoding service component for forwarding when the total hit rate score of any one destination processing instance is higher than the predetermined threshold of the total hit rate score in the case that the destination processing instance exists in the pre-filling component and the decoding service component.

7. The load balancing routing system of a large model according to any one of claims 4-6, wherein the scoring component directly gives a score of -1 when the length of the data processing request being processed by the destination data processing instance is less than the preset length.

8. A load balancing routing method for a large model, comprising: receiving a data processing request of a user by a request parsing component, and parsing and obtaining token information of the data processing request; dividing a token subset from the head of the token information of the received data processing request every time with a predetermined token number as the radix until the end of the token information is reached, and generating a hash value of each divided token subset by using a hash function, thereby forming one or more hash value sets of the data processing request; collecting, by the load information collection component, for each destination processing instance, a set of hash values of each data processing request forwarded to it by the forwarding component and a load amount of data processing requests being processed by each destination processing instance; scoring, by the scoring component, each destination processing instance based on a matching degree between the set of hash values corresponding to the current data processing request and the set of hash values corresponding to the data processing requests received by each destination processing instance and collected by the load information collection component for each destination processing instance and the load amount of each destination processing instance; and summing up the hit rate total score and the total load amount score of each destination processing instance as the total score of the current data processing request for each destination processing instance.

9. The load balancing routing method of a large model according to claim 8, wherein when the subset splitting component splits the token subsets, in adjacent two times of splitting, the number of the token subsets split in the later time is twice the number of the token subsets split in the former time or the number of the token subsets split in the later time increases by one base number relative to the number of the token subsets split in the former time.

10. The load balancing routing method of a large model according to claim 8 or 9, wherein the load information collection component allocates a memory of a predetermined length for each destination processing instance and replaces the hash values cached in the memory in a first-in-first-out manner when the memory is filled.

11. The load balancing routing method of a large model according to claim 8, wherein the scoring component includes the following when scoring: calculating, for any destination processing instance, a matching hit rate as a ratio of the number of hash values matched by the hash value of the current data processing request and the set of hash values cached by the load information collection component for the destination processing instance and the number of the set of hash values of the current data processing request, and assigning a hit rate score corresponding to the number of the base number contained in the split page corresponding to the matched hash value according to the number of the base number contained in the split page corresponding to the matched hash value, thereby accumulating the total hit rate score of the current data processing request with respect to the destination processing instance to obtain the sum of the hit rate scores as the hit rate total score of the current data processing request for each destination processing instance; calculating, for any destination processing instance, a load amount score of each data processing request being processed by the destination data processing instance as a negative number of a multiple of a preset length, thereby accumulating the load amount scores of all data processing requests being processed by the destination data processing instance as the total load amount score of the destination processing instance; and summing up the hit rate total score and the total load amount score of each destination processing instance as the total score of the current data processing request for each destination processing instance.

12. The load balancing routing method of a large model according to claim 11, wherein the forwarding by the forwarding component further includes: ​ ​ In the case where the destination processing instances exist in the pre-filled component and the decoding service component distribution, when the hit rate total score of any one destination processing instance is lower than the hit rate total score predetermined threshold, the destination processing instance with the highest total score in the pre-filled component is directly selected for forwarding.

13. The load balancing routing method of a large model according to claim 11, wherein the forwarding by the forwarding component further comprises: In the case where the destination processing instances exist in the pre-filled component and the decoding service component distribution, when the hit rate total score of any one destination processing instance is lower than the hit rate total score predetermined threshold, the destination processing instance with the highest total score in the pre-filled component is directly selected for forwarding.

14. The load balancing routing method of a large model according to any one of claims 11-13, wherein the scoring component directly assigns a score of -1 when the length of the data processing request being processed by the destination data processing instance is less than the preset length.

Images