Inference system and method for large models

Abstract

The present disclosure relates to an inference system and method for large models. The system comprises a routing component, a pre-filling component, a decoding service component deployed independently of the pre-filling component, and a communication component between the pre-filling component and the decoding service component. The routing component routes long data processing requests to the pre-filling component for pre-filling processing based on token information of the current data processing request, routes short data processing requests to the decoding service component for decoding processing, and performs load balancing routing based on load states of multiple inference capabilities in the pre-filling component or the decoding service component.

Description

Technical Field

[0001] This disclosure relates to the field of computer information processing, and more specifically, to a reasoning system and method for large models. Background Technology

[0002] Deep learning is a very popular machine learning method that uses neural networks for feature extraction to perform tasks such as classification and generation. Deep learning models frequently perform matrix multiplication and addition operations during inference (using a trained model to handle real-world tasks). In recent years, the scale of large models has exploded, from millions to hundreds of millions of parameters in the early days, and now reaching tens of billions, hundreds of billions, or even trillions of parameters. These large models have demonstrated astonishing capabilities in language understanding, code generation, and logical reasoning, but they also bring massive computational and memory requirements. Although hardware resources are constantly developing, even for a medium-sized model, such operations still require a lot of computing power and time; and the current trend is that the growth rate of model size has exceeded the development rate of hardware computing power, often reaching billions, tens of billions, or even hundreds of billions of parameters, making inference increasingly costly and slower.

[0003] In the large language model inference process, it is specifically divided into two stages: the prefill stage and the decoding stage. In the prefill stage, the model first accepts user input tokens and performs preliminary context understanding and inference. This process involves quickly processing the entire input text to generate an initial context representation (KVCache), which is cached to avoid recalculation in subsequent decoding stages. In the decoding stage, the model generates the next token step by step based on previous outputs and context. This process is autoregressive, meaning that each token generated by the model serves as input for the next round of decoding. Therefore, in the prefill stage, the inference engine receives user input tokens and performs computations. Since the number of user input tokens is often large, combined with the number of concurrent requests, the prefill stage, which generates context representations while processing the input sequence, is highly parallelized, involving numerous matrix multiplication operations, resulting in a large computational load and enormous computational cost. Because the parallel computing capabilities of GPUs are fully utilized, the performance bottleneck mainly lies in computing resources (such as GPU computing power), making the prefill stage a typical "computationally constrained" stage. On the other hand, during the decoding phase, the inference engine receives a single token generated in the previous round and interacts with the context representation generated in the pre-filling phase. In this phase, each request has only one token, but a large amount of model context needs to be read. Therefore, the model's computational load is not high, making the decoding phase a memory-constrained scenario. Specifically, the Decode phase is the process of the model generating output tokens one by one, which is serialized. Each token generation requires frequent memory access (such as loading model parameters and KV cache). Due to the relatively small computational load and the underutilization of GPU parallel computing capabilities, the performance bottleneck is mainly memory bandwidth; therefore, the Decode phase is a typical "memory-constrained" phase.

[0004] In real-world applications, large language model inference engines often receive a large number of user requests. Large language model inference typically requires significant computing resources, but hardware resources are limited and cannot process all requests simultaneously. To efficiently and stably handle a large number of concurrent requests while optimizing resource utilization and user experience, traditional large language model inference engines introduce a request scheduling layer to queue requests, wait for the inference engine to have available computing resources, dynamically adjust the processing order based on priority, and merge multiple requests to reuse hardware parallel computing capabilities and improve throughput. This traditional technique is called continuous batching. It combines multiple batches to flexibly handle input sequences of different lengths and updates the batches in real time during inference to reduce resource idleness and inference latency. Traditional batching requires waiting for all requests to be ready, which can easily lead to low hardware utilization and increased response time. Continuous batching allows the model to dynamically merge requests during inference, supports different sequence lengths, and immediately replaces a new request after one request is completed, thus significantly improving throughput and efficiency. This technique is particularly suitable for high-concurrency scenarios, better balancing computing resource utilization and response speed.

[0005] Therefore, in the data inference process of traditional large-scale models, request scheduling strategies are typically prefill-first, decode-first, and Chunked Prefill. The prefill-first strategy prioritizes prefill requests when both prefill and decode requests exist. The decode-first strategy prioritizes decode requests when both exist. The Chunked Prefill strategy, when both exist, splits a prefill request into multiple parts and executes them in multiple rounds, combining them with the decode request in each round to form a complete batch before sending it to the engine.

[0006] Clearly, considering the efficiency of large model inference and the stability of the inference service, using a prefill-first or decode-first strategy will increase the inference service latency. The prefill-first strategy prioritizes prefill requests into the engine, delaying decode requests, which significantly impacts decoding requests. The decode-first strategy prioritizes decode requests into the engine, which can lead to a small number of input tokens, reduced GPU utilization, and a significant increase in the latency of the first token, affecting user experience. Similarly, the Chunked Prefill strategy, by combining prefill and decode requests into a complete batch, increases the number of tokens processed by the engine in each round, improving inference throughput. Since the number of tokens processed by the inference engine in each round is relatively stable, the latency between tokens is relatively stable. However, its total latency is still affected by the insertion of prefill requests, slowing down the processing speed of decode requests, and because prefill requests still occupy GPU memory for a long time, the concurrency of decode requests is limited.

[0007] Therefore, there is a desire to obtain an inference system in which pre-filling and decoding do not mutually diminish each other, and which can improve inference speed and reduce resource consumption to meet the needs of different models and tasks.

[0008] The information disclosed in the background section is only intended to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0009] In view of this, this disclosure provides an efficient large language model inference system for large models, especially large-scale hardware clusters. By splitting the pre-filling computation and decoding computation stages onto different machines for execution without interference, the system optimizes the latency of the first token and the latency between tokens. Furthermore, it enables load-balanced parallel processing of the respective instances of the pre-filling and decoding computations.

[0010] Other features and advantages of this disclosure will become apparent from the following detailed description, or may be learned in part from practice of this disclosure.

[0011] According to one aspect of this disclosure, an inference system for large models is proposed, comprising: a routing component, a pre-filling component, a decoding service component deployed independently of the pre-filling component, and a communication component between the pre-filling component and the decoding service component. The routing component receives a user's current data processing request, obtains the token information of the current data processing request, and matches the token information of the current data processing request with the token information cached in a metadata server. Current data processing requests with a matching hit rate higher than a predetermined threshold are identified as short requests, and current data processing requests with a matching hit rate less than or equal to the predetermined threshold are identified as long requests. The maximum number of tokens for long requests is set to 1, and the token information of the current data processing request is cached in the metadata server. The pre-filling component, through its processing instance, pre-fills the long requests forwarded from the routing component, generates an initial context representation, and its processing instance updates information including its own cache, load, and liveness status to the metadata server in real time. Simultaneously, it transmits the original long request and the generated initial context to the decoding service component. The decoding service component, through its processing instances, performs decoding processing based on short requests forwarded from the routing component and the context associated with the short requests already cached in the metadata server, generating an output token corresponding to the short request. It also performs decoding processing based on the original long requests transmitted from the pre-filling component via the communication component and the initial context generated for the long requests, generating an output token corresponding to the long requests. Its processing instances continuously update the metadata server with information including their own cache, load, and liveness status. The communication component between the decoding service component and the pre-filling component is an RDMA communication component.

[0012] According to the inference system for large models disclosed herein, it further includes: a SmoothQuant W8A8 smoothing quantization component, deployed in association with the pre-filled component, which rescales the activation values ​​and weights accessed by the pre-filled component using a smoothing factor, and performs quantization processing on the smoothed weights and activation values ​​to convert them into INT8 numerical format before using them for pre-filling processing by the pre-filled component.

[0013] According to the inference system for large models disclosed herein, it further includes: a WeightOnly quantization component, deployed in association with the decoding service component, which performs quantization processing on the weight values ​​to be accessed by the decoding service component, converts them into INT4 numerical format, and then uses them for decoding processing by the decoding service component.

[0014] According to the inference system for large models disclosed herein, the token information of the current data processing request is the hash value information of the token value of the current data processing request obtained by: parsing the current data processing request into one or more tokens; dividing a set of tokens from the header of the current data processing request each time, with a predetermined number of tokens as the base, wherein in two adjacent divisions, the number of token sets divided in the latter division is twice the number of token sets divided in the former division, or the number of token sets divided in the latter division is increased by a base relative to the number of token sets divided in the former division; and generating a hash value for each of the divided token sets using a hash function, thereby forming one or more hash values ​​of the current data processing request, wherein the matching hit rate is the ratio of the number of hash values ​​corresponding to the current data processing request that match the hash values ​​cached in the metadata server to the number of hash values ​​corresponding to the current data processing request.

[0015] According to the inference system for large models disclosed herein, after forwarding each received data processing request to a pre-filling component or a decoding service component, the routing component calculates the hash value of the data processing requests routed from the routing component to the destination data processing instance, caches it in a metadata server and / or locally on the routing component, calculates the matching hit rate score between the hash value corresponding to the current data processing request and the hash values ​​of the data processing requests processed by each data processing instance, and based on the calculated matching hit rate score, the routing component routes the received current data processing request to the data processing instance with the highest matching hit rate score in the pre-filling component or the decoding service component.

[0016] According to the inference system for large models disclosed herein, after the routing component forwards each received data processing request to the pre-filling component or the decoding service component, it also forms information about the destination data processing instance routed in the pre-filling component or the decoding service component, so as to calculate the load score of the data processing requests routed from the routing component that the destination data processing instance is processing, and based on the calculated load score, load-balancedly route the received current data processing request to the data processing instance with the highest load score in the pre-filling component or the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load score.

[0017] According to the inference system for large models disclosed herein, the load quantity score of the data processing requests being processed by the data processing instance gradually increases as the tasks arranged therein are completed.

[0018] According to the inference system for large models disclosed herein, after the routing component forwards each received data processing request to the pre-filling component or the decoding service component, it also forms information about the destination data processing instances routed and forwarded in the pre-filling component or the decoding service component, so as to statistically analyze the load number score of the data processing requests being processed by the destination data processing instance from the routing component, and calculate the sum of the statistical load number score and the match hit rate score, and load-balancely route the received current data processing request to the data processing instance with the highest score in the pre-filling component or the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load number score.

[0019] According to another aspect of this disclosure, an inference method for large models is also provided, comprising: a routing step, wherein after a routing component receives a user's current data processing request, it obtains the token information of the current data processing request, matches the token information of the current data processing request with the token information cached in a metadata server, and determines current data processing requests with a matching hit rate higher than a predetermined threshold as short requests and routes them to a decoding service component, and determines current data processing requests with a matching hit rate less than or equal to the predetermined threshold as long requests and routes them to a pre-filling component, and sets the maximum number of tokens for long requests to 1, while caching the token information of the current data processing request in the metadata server; and a pre-filling step, wherein the pre-filling component processes instances of long requests forwarded from the routing component. The process involves pre-filling a short request to generate an initial context representation, and having its processing instance update the metadata server in real time with information including its own cache, load, and liveness status. Simultaneously, the original long request and the generated initial context are transmitted to the decoding service component. The decoding service component's processing instance performs decoding based on the short request forwarded from the routing component and the context associated with the short request already cached in the metadata server, generating an output token corresponding to the short request. It also performs decoding based on the original long request transmitted from the pre-filling component via the communication component and the initial context generated for the long request, generating an output token corresponding to the long request. Its processing instance then updates the metadata server in real time with information including its own cache, load, and liveness status.

[0020] According to the inference method for large models disclosed herein, it further includes: a smoothing quantization step, wherein after the pre-filled component completes training, the activation values ​​and weights accessed by the pre-filled component are rescaled using a smoothing factor through the SmoothQuant W8A8 smoothing quantization component, and the smoothed weights and activation values ​​are quantized and converted into INT8 numerical format for use in the pre-filled component to perform pre-filling processing.

[0021] According to the inference method for large models disclosed herein, it further includes: a weight quantization step, wherein after the decoding service component completes training, the weight values ​​to be accessed by the decoding service component are quantized by the WeightOnly quantization component and converted into INT4 numerical format for the decoding service component to perform decoding processing.

[0022] According to the inference method for large models disclosed herein, obtaining the token information of the current data processing request includes: parsing the current data processing request into one or more tokens; dividing a set of tokens from the header of the current data processing request each time, based on a predetermined number of tokens, wherein in two adjacent divisions, the number of token sets divided in the latter division is twice the number of token sets divided in the former division, or the number of token sets divided in the latter division is increased by a base number relative to the number of token sets divided in the former division; and generating a hash value for each of the divided token sets using a hash function, thereby forming one or more hash values ​​of the current data processing request, wherein the matching hit rate is the ratio of the number of hash values ​​corresponding to the current data processing request that match the hash values ​​cached in the metadata server to the number of hash values ​​corresponding to the current data processing request.

[0023] According to the inference method for large models disclosed herein, it further includes: after forwarding each received data processing request to the pre-filling component or the decoding service component, the routing component calculates the hash value of the data processing request routed from the routing component to the destination data processing instance and caches it in the metadata server and / or locally on the routing component; calculates the matching hit rate score between the hash value corresponding to the current data processing request and the hash values ​​of the data processing requests processed by each data processing instance; and based on the calculated matching hit rate score, routes the received current data processing request to the data processing instance with the highest matching hit rate score in the pre-filling component or the decoding service component.

[0024] According to the inference method for large models disclosed herein, it further includes: after the routing component forwards each received data processing request to the pre-filling component or the decoding service component, it calculates the load number score of the data processing requests being processed by the destination data processing instance from the data processing requests routed and forwarded by the routing component; and based on the calculated load number score, it load-balances the current received data processing request to the data processing instance with the highest load number score in the pre-filling component or the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load number score.

[0025] According to the inference method for large models disclosed herein, it further includes: after forwarding each received data processing request to a pre-filling component or a decoding service component, the routing component calculates a load score of the number of data processing requests being processed by the destination data processing instance from the data processing requests routed and forwarded by the routing component; and calculates the sum of the calculated load score and the match hit rate score, and load-balances the current received data processing request to the data processing instance with the highest score in the pre-filling component or the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load score.

[0026] The inference system and method for large models disclosed herein focus on accelerating inference of large models on large inference clusters. It retains the functional characteristics of the original inference engine while significantly improving inference speed and reducing resource consumption through resource optimization, parallel processing, and load balancing strategies to meet the needs of different models and tasks. In other words, this disclosure optimizes the latency of the first token and the latency between tokens by separating the pre-filling computation and decoding computation stages and executing them on different machines without interference. Furthermore, based on the separation of pre-filling and decoding computation, a cache-aware routing mechanism is introduced. This mechanism polls each node (or processing instance) to record the current load, and the routing component automatically distributes requests based on the load of each node, striving to ensure relatively balanced load across nodes and stable operation. This fully utilizes all node resources, enabling load-balanced parallel processing of the respective processing instances for pre-filling and decoding computation. Moreover, based on the separation structure of pre-filling and decoding computation, this disclosure introduces RDMA high-performance network transmission technology, allowing network devices to directly access host memory for data transmission without CPU involvement in the data copying process, reducing transmission overhead and significantly lowering transmission latency. Furthermore, this disclosure, based on the separate structure of pre-filled computation and decoding computation, supports pre-filled instances and decoding computation instances to select appropriate model quantization strategies according to the resource needs of the inference scenario, so as to improve the utilization of computing resources.

[0027] It should be understood that the above general description and the following detailed description are merely exemplary and do not limit this disclosure. Attached Figure Description

[0028] The above and other objects, features, and advantages of this disclosure will become more apparent from the detailed description of exemplary embodiments with reference to the accompanying drawings. The drawings described below are merely some embodiments of this disclosure, and those skilled in the art will be able to obtain other drawings based on these drawings without any inventive effort.

[0029] Figure 1 This is a block diagram illustrating a first embodiment of a large-scale inference system according to an exemplary embodiment.

[0030] Figure 2 This is a block diagram illustrating a second embodiment of a large-scale inference system according to an exemplary embodiment.

[0031] Figure 3 This is a flowchart of a first embodiment of a reasoning method for a large model, illustrated according to an exemplary embodiment.

[0032] Figure 4 This is a flowchart of a second embodiment of a reasoning method for a large model, illustrated according to an exemplary embodiment.

[0033] Figure 5 This is a flowchart of a third embodiment of a reasoning method for a large model, illustrated according to an exemplary embodiment. Detailed Implementation

[0034] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the embodiments set forth herein; rather, they are provided so that this disclosure will be thorough and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted.

[0035] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0036] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0037] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0038] It should be understood that although the terms first, second, third, etc., may be used herein to describe various components, these components should not be limited by these terms. These terms are used to distinguish one component from another. Therefore, the first computing device discussed below may be referred to as the second computing device without departing from the teachings of this disclosure. As used herein, the term "and / or" includes all combinations of any and more of the associated listed items.

[0039] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of exemplary embodiments, and the modules or processes in the drawings are not necessarily necessary for implementing this disclosure, and therefore cannot be used to limit the scope of protection of this disclosure.

[0040] Figure 1 This is a block diagram illustrating a first embodiment of a large-model inference system according to an exemplary embodiment. Figure 1 As shown, the inference system 100 for large models includes: a routing component 110, a pre-filling component 120, a decoding service component 130 deployed independently of the pre-filling component 120, and a communication component 140 between the pre-filling component and the decoding service component.

[0041] As mentioned earlier, pre-filling computation is computationally intensive and can be executed on machines with higher computing power (metric: tflops). Decoding, on the other hand, is memory-bandwidth limited and can be performed on machines with greater video memory bandwidth. Therefore, to optimize the first-token latency and inter-token latency by separating the pre-filling and decoding computation stages and executing them on different machines without interference, the pre-filling component 120 and the decoding service component 130 are deployed separately. For example, the pre-filling component 120 can be deployed on the first computing device, and the decoding service component 130 on the second computing device. Alternatively, they can be deployed on different physical parts of the same computing device. Thus, from the perspective of the overall model, the traditionally monolithic model is divided into a sub-model for performing pre-filling inference and a sub-model for performing decoding inference. This separate deployment eliminates the situation where prioritizing pre-filling inference and decoding inference leads to one being neglected for the other. Specifically, it eliminates the drawback of a pre-filling priority strategy having a greater impact on decoding requests, while a decoding priority strategy results in an insufficient number of input tokens, reduced GPU utilization efficiency, and a significant increase in first-token latency, thus affecting the user experience. Therefore, it also eliminates the drawback that prefill-first or decode-first strategies would increase inference service latency. Similarly, it also eliminates the drawback that the total latency of the ChunkedPrefill strategy is still affected by the insertion of prefill requests, which slows down the processing speed of decoding requests, and that the concurrency of decoding requests is limited because prefill requests still occupy video memory for a long time.

[0042] Specifically, after receiving the user's current data processing request, the routing component 110 determines the length of the request. If it's a long request, it's usually a relatively recent one, and the cached context identifiers related to long requests are relatively few. Since the decoding phase involves the model generating output tokens one by one, it's serialized, requiring frequent memory accesses each time a token is generated (e.g., loading model parameters and KV cache). Due to the relatively small computational load and the underutilization of GPU parallel computing capabilities, the performance bottleneck is mainly memory bandwidth; therefore, the decoding phase is a typical "memory-constrained" phase. Therefore, routing long requests to the pre-filling component 120 for pre-filling before decoding is a better approach, while routing short requests directly to the decoding service component 130 is also a good solution.

[0043] In Large Language Models (LLMs), tokens are the basic units of text, which can be words, subwords, or characters. The model breaks down the input text into tokens for processing and understanding. The number of tokens determines the input length and computational complexity of the model. User-input text data is first converted into tokens and input into the model. To determine the length of a request, in addition to the length of the request text itself, it is also necessary to consider whether the context representation of the request exists or is mostly present relative to the cached context representation of the trained model. If most of it is present, it means that even a long request is relatively short relative to the large model, because only the remaining tokens need to be decoded. Therefore, the relative length of a request with the existing data in the large model can be determined based on the tokens cached by the model. Therefore, the routing component 110 obtains the token information of the current data processing request and matches it with the token information cached in the metadata server. Requests with a match rate higher than a predetermined threshold are identified as short requests, while requests with a match rate less than or equal to the predetermined threshold are identified as long requests. The maximum number of tokens for long requests is set to 1, and the token information of the current data processing request is cached in the metadata server. The predetermined hit rate threshold can be 50%, 60%, or 70%. Setting it to 60% typically achieves a compromise in efficiency.

[0044] like Figure 1As shown, the pre-filling component 120 pre-fills long requests forwarded from the routing component 110 using its processing instance, generating an initial context representation (KV cache). Its processing instance then updates the metadata server 150 in real-time with information including its own cache, load, and liveness status. Simultaneously, it transmits the original long request and the generated initial context to the decoding service component 130. The decoding service component 130, based on a short request forwarded from the routing component 110 and the context associated with the short request already cached in the metadata server, performs decoding to generate an output token corresponding to the short request. It also performs decoding on the original long request transmitted from the pre-filling component via the communication component 140 and the initial context generated for the long request, generating an output token corresponding to the long request. Its processing instance then updates the metadata server 150 in real-time with information including its own cache, load, and liveness status. The communication component 140 between the decoding service component 130 and the pre-filling component 120 is an RDMA communication component. The KV cache is a caching mechanism used to store intermediate states generated during the inference process of the Transformer model. In the Transformer model, each inference step (whether in the pre-filling or decoding phase) requires generating a corresponding Key and Value based on the current input. These intermediate states are reused in subsequent inference steps. The role of KVCache is to cache these intermediate states, avoiding repeated calculations during each inference, thereby significantly improving inference efficiency. In the pre-filling and decoding separation system disclosed herein, after the pre-filling instance completes the pre-filling phase, it needs to transfer the KVCache to the decoding instance. The core objective of the KVCache transfer module is to achieve high-speed, low-overhead data transmission to ensure that the performance advantages of the inference system in the pre-filling and decoding separation mode are fully realized; therefore, RDMA is adopted. RDMA is a high-performance network transmission technology that allows network devices to directly access the memory of remote hosts without CPU involvement in data copying, significantly reducing transmission latency and CPU overhead. This RDMA communication can be implemented using the network hardware of the computing devices deployed by the pre-filling component and the decoding service component using conventional RDMA settings. Alternatively, zero-copy technology can be used to directly transfer the KV Cache from the memory of the Prefill node to the memory of the Decode node by avoiding multiple copies of data between the kernel space and user space, thereby further reducing transmission latency. Alternatively, an efficient network protocol can be used, such as RoCEv2 or InfiniBand, to ensure low-latency, high-bandwidth data transmission in a high-speed network environment.Although the communication component 140 is shown separately here, it can also be implemented using the RDMA communication ports of the computing devices deployed by the decoding service component 130 and the pre-filling component 120, respectively.

[0045] Furthermore, the routing component 110 obtains the token information of the current data processing request as the hash values ​​of multiple tokens to reduce the computational and access overhead of token matching. The token information of the current data processing request is obtained as the hash value information of the token value of the current data processing request through the following steps: parsing the current data processing request into one or more tokens; dividing a set of tokens from the header of the current data processing request each time, based on a predetermined number of tokens, wherein in two adjacent divisions, the number of token sets in the later division is twice the number of token sets in the previous division; and generating a hash value for each of the divided token sets using a hash function, thereby forming one or more hash values ​​of the current data processing request, wherein the matching hit rate is the ratio of the number of hash values ​​corresponding to the current data processing request that match the hash values ​​cached in the metadata server to the number of hash values ​​corresponding to the current data processing request.

[0046] For example, the routing component 110 divides the tokens generated from the current data processing request according to the number of tokens per page of page attention, and sets the dividing line according to the rule of doubling each time. Assuming that the number of tokens per page in the system is 32, and a request has 555 tokens, then the dividing line is [32, 64, 128, 256, 512]. Usually, the small number of tokens that are more than 512 in 555 are ignored because they will not affect the accuracy of the final pre-filling or decoding. Thus, the token set generated each time starts from the 0th token of the request and is named tokens[0:32], tokens[0:64], tokens[0:128], tokens[0:256], tokens[0:512], where tokens[0:i] represents from the 0th token to the i-th token. These token sets (or token pages) can be converted into 5-hash values ​​using a fixed hash function. The advantage of this partitioning method is that it maintains the overall linear complexity. For example, the complexity above is only 32 + 64 + 128 + 256 + 512 = 992 <= 2 * 555. Alternatively, each page can be partitioned once, that is, a set of tokens is partitioned each time, starting from the header of the current data processing request, based on a predetermined number of tokens. In two adjacent partitions, the number of tokens in the later partition increases by the base number relative to the number of tokens in the previous partition. For example, 32, 64, 96, 128, 160, ..., 512, 544. However, its complexity will reach n^2: 32 + 64 + 96 + 128 + 160 + ... + 512 + 544 = 32 * (1 + 17) * 17 / 2. Here, 17 or n will increase linearly with the total number of tokens, so the complexity is quadratic. For routing convenience, after the partition line is given, the router component uses a dict and a FIFO list to store the hash values ​​of existing requests from 0 to the partition line for each processing instance used by the decoding service component 130 and the pre-filling component 120 for data parallel inference. The hash values ​​in this dict are somewhat like indices of the token set corresponding to the hash values. In other words, the dict is a mapping from the hash value to the number of times that hash value appears in the processing instance. In the first partitioning, maintaining the mapping only requires O(1) complexity.

[0047] A FIFO list, or First In First Out, is used because the number of pages is finite. Once the cache is full, new requests will push older requests' caches out. Therefore, the FIFO list has a size. Before the hash value reaches its maximum, the FIFO list stores the hash values ​​of all incoming requests, separated by dividing lines. Each request may store multiple hash values, with the number of dividing lines (hash values) determined by the token length. Once the hash value reaches its maximum, the FIFO list will pop the oldest hash value and add a new one, repeating this process multiple times. Multiple hash values ​​from a single request are entered in ascending order of page number. Therefore, the hash values ​​from the same request also have a specific order.

[0048] Examples of such FIFO lists are as follows:

[0049] For the first processing instance, it was recorded that the routing component forwarded 3 requests.

[0050] I ate an apple and a banana.

[0051] I ate an apple and a pineapple.

[0052] "I ate apples and bananas, they were so delicious I couldn't stop eating them."

[0053] For ease of explanation, let's assume there's only 1 token per page, although this is practically impossible; it's usually 32 or 128. Here, we simply assume each character and each punctuation mark represents one token. Also, for simplicity, let's assume the corresponding hash value is 4 digits (hash values ​​are very long; 4 digits are used here for clarity). The hash value of the token set after the aforementioned request's token split (for simplicity, let's assume it conforms to a predetermined splitting rule; this rule can be manually set and doesn't necessarily have to be the one exemplified in this disclosure) is:

[0054] "Me" → 1234

[0055] "I eat" → 2222

[0056] "I ate an apple" → 3214

[0057] "I ate an apple and a banana" → 4444

[0058] "I ate an apple and a pineapple" → 8888

[0059] "I ate apples and bananas, they were so delicious I couldn't stop eating!" → 5555

[0060] Assuming the FIFO list has a length of 7 (for example, it can only store 7 hash values. In real-world scenarios, the length can be set to any finite value depending on the data processing volume), then both the FIFO list and the dictionary are empty before the first request arrives.

[0061] FIFO_list = [], dict = {}

[0062] After the first request "I ate an apple and a banana" comes in, its data structure is as follows:

[0063] FIFO_list=[1234, 2222, 3214, 4444]

[0064] Dict = {1234:1, 2222:1, 3214:1, 4444:1}, where 1 indicates that the hash value exists only once in FIFI_list.

[0065] If the second request "I ate apples and pineapples" comes in, the data structure becomes:

[0066] FIFO_list = [2222, 3214, 4444, 1234, 2222, 3214, 8888]

[0067] Dict = {1234: 1, 2222: 2, 3214: 2, 4444: 1, 8888: 1}. Here, 1234 appears only once; 2222 appears twice, in positions 1 and 5; 3214 appears twice, in positions 2 and 6; 4444 appears only once; and 8888 appears only once. It can be seen that in the FIFO_list, the oldest hash 1234 was pushed out before the last hash 8888 even entered.

[0068] If the third request, "I ate apples and bananas, and they were so delicious I couldn't stop eating them," comes in, the data structure becomes:

[0069] FIFO_list = [3214, 8888, 1234, 2222, 3214, 4444, 5555]

[0070] Dict = {8888: 1, 1234: 1, 2222: 1, 3214: 2, 4444: 1, 5555: 1}

[0071] Because the FIFO list was already full before the third request came in, for every new hash value that was added, an old hash value was pushed out.

[0072] Using these two data structures, we can not only confirm the existence of a certain hash value in the corresponding processing instance in O(1) time, but also squeeze out the oldest hash value in chronological order after the FIFO_list is full, perfectly simulating the key-value cache process, so that the routing shoulder can restore the real load of each processing instance as much as possible. If the simulation is not good, for example, it is thought that there are two requests a and b in the first processing instance, but in fact, there are two requests c and d in the first processing instance, or the first processing instance has 100 requests, while the second processing instance is empty. When a follow-up problem of request a is encountered, because the simulation is not good and the real load is not clear, it is still placed in the first processing instance, which will increase the load of the first processing instance.

[0073] Although the request token can be directly used to evaluate the hit rate, using the hash value of the segmented token set obtained from the above-mentioned request segmentation to evaluate the length of the request by matching the hit rate makes the hit rate evaluation faster and more efficient. It also allows for load balancing within the pre-filling component 120 or the decoding service component 130. Specifically, after forwarding each received data processing request to the pre-filling component 120 or the decoding service component 130, the routing component 110 calculates the hash value of the data processing requests routed from the routing component to the destination data processing instance, caches it in the metadata server and / or locally on the routing component, and calculates the matching hit rate score between the hash value corresponding to the current data processing request and the hash values ​​of the data processing requests processed by each data processing instance. Based on the calculated matching hit rate score, the routing component 110 routes the received current data processing request to the data processing instance with the highest matching hit rate score in the pre-filling component 120 or the decoding service component 130. In layman's terms, this means allocating new requests to processing instances that have previously processed highly similar requests, thereby making full use of the computational results already performed during the processing of new requests and reducing the amount of computation.

[0074] For example, a delivery request can be made after scoring based solely on the hit rate, or after scoring based solely on the load balance, or a delivery request can be made after considering both the hit rate and the load balance for a comprehensive score.

[0075] When requesting delivery is scored based solely on hit rate, a single request generates multiple hash values ​​along dividing lines. Since different hashes are assigned scores, the score is determined by the page number. One point is awarded for each page that is hit.

[0076] For example, for a single processing instance, the first processing instance has calculated the request "I ate an apple and a banana".

[0077] Dict = {1234: 1, 2222: 1, 3214: 1, 4444: 1}

[0078] The second processing instance calculated a request: "I ate an apple and a pineapple".

[0079] Dict = {1234: 1, 2222: 1, 3214: 1, 8888: 1}

[0080] Then there's a current data processing request: "I ate apples and bananas, they were so delicious I couldn't stop eating them." The hash value of each split page is: Hash: 1234, 2222, 3214, 4444, 5555

[0081] As mentioned above, multiple hash values ​​are calculated for a single request based on the dividing line. Since different hash values ​​are assigned scores, the score is based on the number of pages. That is, one point is awarded for each page hit. The score for hitting different hash values ​​will vary. The score is assigned as a multiple of a predetermined number of tokens (e.g., 32 tokens). Therefore, the scores for hitting the 1st to 5th hash values ​​are: 1, 2, 4, 8, 16 respectively. As previously mentioned, assuming the dividing line is obtained by doubling the number of tokens per page, hitting the first hash value (one page of tokens) means that in actual data inference, one page of tokens can be counted less; hitting the second hash value means two pages less; hitting the third hash value means four pages less; hitting the fourth hash value means eight pages less, and so on. Therefore, the hash value set for the current data processing request, "I ate apples and bananas, and they were so delicious I couldn't stop eating them," is: Hash: 1234, 2222, 3214, 4444, 5555. These hash values ​​are matched against the hash values ​​in the Dict cached by the first and second processing instances, respectively. This assigns a hit score to the hash value of the current data processing request. For the first processing instance, hitting the first four hashes results in a total hit rate of 15, while for the second processing instance, hitting the first three hashes results in a total hit rate of 7. Therefore, the routing component 110 routes the current data processing request to the data processing instance with the highest hit rate score in the pre-filling component 120 or the decoding service component 130, i.e., the first processing instance, thereby reducing the computation and / or access volume of the first processing instance performing data filling or decoding services.

[0082] When request delivery is performed after scoring based on load balancing, the routing component 110, after forwarding each received data processing request to the pre-filling component or decoding service component, also forms information about the destination data processing instance routed in the pre-filling component or decoding service component. This is to calculate the load quantity score of the data processing requests being processed by the destination data processing instance from the routing component, and based on the calculated load quantity score, to load balance the received current data processing request to the data processing instance with the highest load quantity score in the pre-filling component or decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load quantity score.

[0083] For example, load balancing requires considering both computation time and memory load. Simply put, a request with 100,000 tokens will have the same memory footprint as a request with 20 tokens. However, at the router, only the prefill is generated, not the decoding part. A request with 100,000 tokens might only generate one token and then stop. A request with 20 tokens might generate 8,000 tokens (a simple example is a request like "Write me an 8,000-word essay"). The generated length is unpredictable, but fortunately, based on statistics, the average length of a typical request can be known. The same logic applies to computation time. Request A's prefill length is 5,000 times that of request B, but A's actual computation time might only be 50 times that of request B. In the LLM model, the attention component increases linearly with the number of tokens for each decoded token, but the MLP's time remains constant. It's important to note that generating a single token for a request with 100,000 tokens will definitely take longer than generating a single token for a request with 20 tokens, but it's certainly not 5000 times longer. Therefore, for specific models, an empirical token value can be set for each request to evaluate the load score. To show that higher load results in a lower score, negative points are applied to the increased load on a specific processing instance; that is, -1 point is assigned for each additional basic unit (empirical token value) of load request, and -1 point is assigned to any token set, even if its length is less than one basic unit. Therefore, the score can be calculated using the formula: Score = - (token_len / / 2048 + 1), where / / represents the quotient (without remainder or decimal part), token_len is the length of the token set, and 2048 is the empirical length. Of course, for simplicity, scores can be directly assigned based on the number of load requests, without considering the length of the load requests, i.e., without using empirical values, because from the perspective of normal distribution, the overall average length does not change much when there are a large number of requests. Therefore, directly using a counting method, assigning -1 point to each additional load request for each processing instance, can also achieve the purpose of this disclosure. Of course, any method that can achieve the result of linearly decreasing scores as the load increases can achieve the technical effect of this disclosure. Alternatively, for specific purposes, a non-linear decreasing or changing method can be used to assign negative scores. In this way, when a current data processing request arrives, without considering the hit rate, it can be routed to the processing instance with the highest load score among multiple parallel processing instances.

[0084] It's important to note that the load score of the data processing requests being processed by a data processing instance gradually increases as the tasks within it are completed. That is, each time a request finishes processing an instance, the corresponding score is deducted from the instance's score data. (Because the score is negative, the instance's score increases when the request ends). Additionally, due to certain special requests, a request might not finish normally from the instance (e.g., it's canceled). Therefore, the instance cleans up requests exceeding a certain duration (currently set at 80 seconds, sufficient for short requests. For scenarios requiring long requests, or where the model runs slowly, such as Deepseek-R1, this could be set to 10 minutes or 30 minutes). In other words, the router receives information about the completion or cancellation of data processing requests from each processing instance, or in other words, it receives the number and length of data processing requests being processed by each instance. The router also receives the liveness status of each processing instance.

[0085] When considering both match hit rate and load balancing for a comprehensive score before request delivery, the two scores can be directly added together. Alternatively, a weighted sum can be obtained by prioritizing either the match hit rate score or the load balancing score to select the processing instance to be routed. That is, if the first processing instance has a hit rate score of 70 and a load balancing score of -30, or a hit rate score of 60 and a load balancing score of -10, and since the first processing instance has a comprehensive score of 40 while the second processing instance has a comprehensive score of 50, the current data processing request will be routed by routing component 110 to the second processing instance for data processing.

[0086] Figure 2 This is a block diagram illustrating a second embodiment of a large-model inference system 200 according to an exemplary embodiment. Figure 1 Compared to the large model inference system 100 shown, the addition of the SmoothQuant W8A8 smoothing quantization component 260 and the WeightOnly quantization component 270, with other identical parts having the same reference markers, is as follows: Figure 1 The similarities are only in that they begin with the number "2". Therefore, the descriptions of the same parts adopt a targeted approach. Figure 1 The descriptions are not detailed here.

[0087] like Figure 2As shown, the SmoothQuant W8A8 smoothing quantization component 260 is deployed in association with the pre-filling component 220. It uses a smoothing factor to rescale the activation values ​​and weights accessed by the pre-filling component 220, and then performs quantization processing on the smoothed weights and activation values, converting them into INT8 numerical format before using them for pre-filling processing in the pre-filling component 220. Correspondingly, the WeightOnly quantization component 270 is deployed in association with the decoding service component 230. It performs quantization processing on the weight values ​​to be accessed by the decoding service component 230, converting them into INT4 numerical format before using them for decoding processing in the decoding service component 230.

[0088] In traditional large language model inference, quantization of weights can significantly reduce model memory overhead, correspondingly reducing the overhead of reading weights and thus accelerating inference. Traditional quantization schemes for large models mainly fall into two categories: WeightOnly quantization, which quantizes only the model's weights without quantizing activation values, reducing model storage overhead. During model inference, the weights are first dequantized, and then normal-precision calculations are performed with the activation values; this scheme can typically quantize weights to 8 bits or 4 bits. SmoothQuant W8A8 quantization rescales activations and weights through a smoothing factor, reducing the scale of outliers in activations and performing fixed-point operations during inference, while simultaneously reducing storage and computational overhead. However, in traditional single-instance deployments (i.e., traditional unified deployments of pre-filling and decoding), it's difficult to coordinate the quantization of the pre-filling and decoding scenarios. Specifically, the pre-filling instance, due to processing a large number of tokens, has a correspondingly high computational load, making it a computationally constrained scenario. From an optimization perspective, a more computationally efficient quantization scheme should be chosen, such as the SmoothQuant W8A8 quantization strategy. On the other hand, the decoding instance processes fewer tokens, has a lower computational load, and needs to frequently read from the KVCache in memory, making it a memory-constrained scenario. A more memory-efficient quantization scheme should be chosen, such as the WeightOnly quantization scheme. Moreover, the WeightOnly scheme requires dequantizing the weights back to FP16 data type and performing FP16 matrix multiplication, which is often less computationally efficient than SmoothQuant in the pre-filling stage. Therefore, in a unified deployment of pre-filling and decoding, it's difficult to reconcile the two scenarios and achieve satisfactory quantization results for both, leading to more work for later adjustments and optimizations, thus negating the purpose of later optimizations.

[0089] Therefore, the separate pre-filling and decoding scenarios disclosed in this disclosure provide separate quantization strategies for each scenario and achieve their respective quantization optimization effects, thus achieving the goal. On one hand, by using the SmoothQuant W8A8 quantization component 260 to rescale the activation values ​​and weights accessed by the pre-filling component 220, and then quantizing the smoothed weights and activation values ​​into INT8 numerical format, the pre-filling component 220 can fully utilize the INT8 tensor core within the hardware to perform INT8 matrix operations. Its computational intensity is often twice that of unquantized FP16 matrix operations, accelerating the pre-filling calculation process. On the other hand, since the decoding service component 230 processes a smaller number of tokens, its computational load is not high, and it frequently needs to read from the KVCache in memory. By using the WeightOnly quantization component 270 to quantize the weight values ​​accessed by the decoding service component 230 into INT4 numerical format for the decoding service component 230 to perform decoding, its weight reading weight is half that of the SmoothQuant W8A8 quantization strategy, resulting in better decoding performance.

[0090] As can be seen, the separate deployment of prefilling and decoding scenarios in this disclosure allows for the flexible selection of appropriate quantization schemes for each scenario. For the prefilling component 220, a more computationally efficient SmoothQuant W8A8 quantization component 260 is deployed in conjunction to achieve superior prefill performance. For the decoding service component 230, a WeightOnly 4-bit quantization component 270 is deployed in conjunction to achieve higher memory access efficiency. This results in overall inference system performance that is superior to that of an inference system deployed in a unified hybrid configuration.

[0091] Figure 3 This is a flowchart of a first embodiment of a large-model reasoning method 300, illustrated according to an exemplary embodiment. (See attached flowchart.) Figure 3As shown, in the inference method 300 for large models, in the routing step, after the routing component 110 or 210 receives the user's current data processing request, at step S310, the token information of the current data processing request is obtained, and matched with the token information of the current data processing request cached in the metadata server 150 or 250. Current data processing requests with a matching hit rate higher than a predetermined threshold are identified as short requests and routed to the decoding service component 130 or 230. At step S320, current data processing requests with a matching hit rate less than or equal to the predetermined threshold are identified as long requests and routed to the pre-filling component 120 or 220. The maximum number of tokens for long requests is set to 1, and the token information of the current data processing request is cached in the metadata server. Related information can also be cached at the pre-filling component 120 or 220.

[0092] Next, in the pre-filling step S330, the processing instance of the pre-filling component 120 or 220 pre-fills the long request forwarded from the routing component 110 or 210 to generate an initial context representation. In step S360, its processing instance updates information including its own cache, load and liveness status to the metadata server 150 or 250 in real time. In step S340, the original long request and the generated initial context are transmitted to the decoding service component.

[0093] Finally, at the decoding service step S350, the processing instance of the decoding service component 130 or 230 performs decoding processing based on the short request forwarded from the routing component 110 or 210 and the context associated with the short request already cached in the metadata server 150 or 250, generating an output token corresponding to the short request. It also performs decoding processing based on the original long request transmitted from the pre-filling component 120 or 220 via the communication component 140 or 240 and the initial context generated for the long request, generating an output token corresponding to the long request. At step S370, its processing instance updates information including its own cache, load, and liveness status to the metadata server in real time.

[0094] Figure 4 This is a flowchart of a second embodiment of a large model reasoning method 400 illustrated according to an exemplary embodiment. Figure 4 and Figure 3 The same steps use essentially the same step numbering, the difference being that "S4" is used as the starting number. Figure 3 The only difference between the two methods is the addition of step S480 before step S420 and step S450. The other identical steps will not be detailed here. Figure 4In step S420 shown, with Figure 3 The difference in step S320 is that a balanced selection of pre-populated instances is also performed. To this end, after forwarding each received data processing request to the pre-population component, routing component 110 or 210 calculates the hash value of the data processing requests routed from the routing component to the destination data processing instance and caches it in the metadata server and / or locally on the routing component for use when routing to a specific processing instance. The current data processing request is parsed into one or more tokens. Using a predetermined number of tokens as a base, a set of tokens is partitioned from the header of the current data processing request each time. In two consecutive partitions, the number of token sets in the later partition is twice the number of token sets in the previous partition, or the number of token sets in the later partition is increased by a base number relative to the number of token sets in the previous partition. A hash function is then used to generate a hash value for each partitioned token set, thus forming one or more hash values ​​for the current data processing request. The matching hit rate is the ratio of the number of hash values ​​corresponding to the current data processing request that match the hash values ​​cached in the metadata server to the number of hash values ​​corresponding to the current data processing request. Finally, based on the calculated matching hit rate score, the received current data processing request is routed to the data processing instance with the highest matching hit rate score in the pre-populated component, achieving load balancing based on the matching hit rate.

[0095] In step S420, during the load balancing process of the current data processing request to the pre-filling component, when request delivery is performed after scoring based on load balancing alone, the routing component 110, after forwarding each received data processing request to the pre-filling component, also forms information about the destination data processing instance routed and forwarded in the decoding service component. This is to calculate the load quantity score of the data processing requests being processed by the destination data processing instance from the routing component, and based on the calculated load quantity score, to load balance the received current data processing request to the data processing instance with the highest load quantity score in the decoding service component. The higher the number of data processing requests being processed by the data processing instance, the lower the load quantity score.

[0096] Optionally, in the load balancing process of the current data processing request to the pre-filling component in step S420, after forwarding each received data processing request to the pre-filling component, the routing component calculates the load number score of the data processing requests being processed by the destination data processing instance from the data processing requests routed and forwarded by the routing component; and calculates the sum of the calculated load number score and the match hit rate score, and load-balances the received current data processing request to the data processing instance with the highest score in the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load number score.

[0097] Similarly, in step S480, after routing component 110 or 210 forwards each received data processing request to the decoding service component, it counts the hash values ​​of the data processing requests routed from the routing component to the destination data processing instance and caches them in the metadata server and / or locally in the routing component for use when routing to a specific processing instance. For the current data processing request, it is parsed into one or more tokens, and a set of tokens is partitioned from the header of the current data processing request each time, based on a predetermined number of tokens. In two adjacent partitions, the number of tokens in the later partition is twice the number of tokens in the previous partition, or the number of tokens in the later partition is increased by a base number relative to the number of tokens in the previous partition. A hash value is generated for each partitioned token set using a hash function, thereby forming one or more hash values ​​for the current data processing request. The matching hit rate is the ratio of the number of hash values ​​corresponding to the current data processing request that match the hash values ​​cached in the metadata server to the number of hash values ​​corresponding to the current data processing request. Finally, based on the calculated match hit rate score, the received current data processing request is routed and forwarded to the data processing instance with the highest match hit rate score in the decoding service component, thereby achieving load balancing based on the match hit rate.

[0098] In step S480, during the load balancing process of the current data processing request to the decoding service component, when request delivery is performed after scoring based on load balancing alone, the routing component 110, after forwarding each received data processing request to the decoding service component, also forms information about the destination data processing instance that is routed and forwarded in the decoding service component. This is to calculate the load quantity score of the data processing requests being processed by the destination data processing instance from the data processing requests routed and forwarded by the routing component, and based on the calculated load quantity score, to load balance the received current data processing request to the data processing instance with the highest load quantity score in the decoding service component. The higher the number of data processing requests being processed by the data processing instance, the lower the load quantity score.

[0099] Optionally, in the load balancing process of the current data processing request to the decoding service component in step S480, after forwarding each received data processing request to the decoding service component, the routing component calculates the load number score of the data processing requests being processed by the destination data processing instance from the data processing requests routed and forwarded by the routing component; and calculates the sum of the calculated load number score and the match hit rate score, and load-balances the current data processing requests received to the data processing instance with the highest score in the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load number score.

[0100] Although the load balancing steps in step S420 and S480 are described separately, they can actually be combined. The hit rate score can be used to directly determine whether the routing is to a specific processing instance of the pre-filled component or the decoding service component. For ease of understanding, the aforementioned separate description is used, but they can also be described together. Therefore, the separate description does not mean they must be separated. The separate description also implies their combination, and those skilled in the art can directly combine them within the scope of this disclosure.

[0101] Figure 5 This is a flowchart of a third embodiment of a reasoning method for a large model, illustrated according to an exemplary embodiment. Figure 5 and Figure 4 The same steps use essentially the same step numbering, the difference being that "S5" is used as the starting number. Figure 4The only difference between the two steps is the addition of steps S561 and S571. Other identical steps will not be detailed here. Specifically, different quantization processes are performed on the intermediate data or output data generated by the pre-filled component 120 or 220 and the decoding service component 130 or 230. Specifically, in the smoothing quantization step S561, after the pre-filled component 120 or 220 completes training, the SmoothQuant W8A8 smoothing quantization component uses a smoothing factor to rescale the activation values ​​and weights accessed by the pre-filled component, and then performs quantization processing on the smoothed weights and activation values, converting them into INT8 numerical format before using them for pre-filling processing in the pre-filled component 120 or 220. The specific smoothing process can be implemented using existing techniques.

[0102] And at the weight quantization step S571, after the decoding service component 130 or 230 has completed training, the weight values ​​to be accessed by the decoding service component are quantized by the WeightOnly quantization component and converted into INT4 numerical format for the decoding service component to perform decoding processing.

[0103] In summary, compared to traditional routing mechanisms that typically adopt globally fixed routes without distinguishing between requests of different lengths, which can easily lead to excessive load on some inference nodes, unstable inference service latency, and underutilization of the computing resources of all nodes, this disclosure optimizes the latency of the first token and the latency between tokens by splitting the pre-filling computation and decoding computation stages onto different machines, ensuring they do not interfere with each other. Furthermore, based on the separation of pre-filling computation and decoding computation, a cache-aware routing mechanism is introduced. This mechanism polls each node (or processing instance) to record the current load, and the routing component automatically distributes requests based on the load of each node, striving to make the load of each node relatively balanced and stable, thus fully utilizing the resources of all nodes. This enables the separate processing instances of pre-filling computation and decoding computation to achieve load balancing and parallel processing. Furthermore, compared to traditional KVCache transmission modules that utilize the TCP / IP network protocol for KVCache transmission, which introduces complex synchronization mechanisms to ensure transmission reliability (adding overhead and reducing transmission efficiency), and whose reliability mechanisms require data packets to wait for acknowledgment before transmission (leading to high latency), this disclosure employs a pre-fill computation and decoding computation separation structure and introduces RDMA high-performance network transmission technology. Network devices can directly access host memory for data transmission without CPU involvement in the data copying process, reducing transmission overhead and significantly lowering latency. Moreover, unlike existing technologies where single-node instance deployment only allows for a single model quantization strategy, failing to simultaneously meet the effective resource requirements of both Prefill and Decode inference scenarios, this disclosure, based on a pre-fill computation and decoding computation separation structure, allows pre-fill instances and decoding computation instances to select appropriate model quantization strategies according to the resource needs of the inference scenario, thereby improving computational resource utilization.

[0104] Those skilled in the art will understand that the above modules can be distributed in the device as described in the embodiments, or they can be modified accordingly and placed in one or more devices that are unique to this embodiment. The modules in the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.

[0105] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, mobile terminal, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0106] Exemplary embodiments of this disclosure have been specifically shown and described above. It should be understood that this disclosure is not limited to the detailed structures, arrangements, or implementations described herein; rather, this disclosure is intended to cover various modifications and equivalent arrangements contained within the spirit and scope of the appended claims.

Claims

1. An inference system for large models, comprising: The components include a routing component, a pre-filling component, a decoding service component deployed independently of the pre-filling component, and a communication component between the pre-filling component and the decoding service component. The routing component receives the user's current data processing request, obtains the token information of the current data processing request, and matches the token information of the current data processing request with the token information cached in the metadata server. The current data processing request with a matching hit rate higher than a predetermined threshold is identified as a short request, and the current data processing request with a matching hit rate less than or equal to the predetermined threshold is identified as a long request. The maximum number of tokens for long requests is set to 1, and the token information of the current data processing request is cached in the metadata server. The pre-filling component, through its processing instance, pre-fills long requests forwarded from the routing component, generates an initial context representation, and updates the metadata server in real time with information including its own cache, load, and liveness status. At the same time, it transmits the original long request and the generated initial context to the decoding service component. The decoding service component, through its processing instances, performs decoding processing based on short requests forwarded from the routing component and the context associated with the short requests already cached in the metadata server, generating an output token corresponding to the short request. It also performs decoding processing based on the original long requests transmitted from the pre-filling component via the communication component and the initial context generated for the long requests, generating an output token corresponding to the long requests. Furthermore, its processing instances update the metadata server in real time with information including their own cache, load, and liveness status.

2. The inference system for large models as described in claim 1, wherein, The communication component between the decoding service component and the pre-filling component is an RDMA communication component.

3. The inference system for large models as described in claim 1, further comprising: The SmoothQuant W8A8 smoothing quantization component is deployed in association with the pre-filled component. It uses a smoothing factor to rescale the activation values ​​and weights accessed by the pre-filled component, and then performs quantization processing on the smoothed weights and activation values ​​to convert them into INT8 numerical format before using them for pre-filling processing in the pre-filled component.

4. The inference system for large models as described in claim 1 or 3, further comprising: The WeightOnly quantization component is deployed in association with the decoding service component. It performs quantization processing on the weight values ​​that the decoding service component needs to access, converts them into INT4 numerical format, and then uses them for decoding processing by the decoding service component.

5. The inference system for large models as described in claim 1, wherein the token information of the current data processing request is the hash value information of the token value of the current data processing request obtained as follows: The current data processing request is parsed into one or more tokens; Using a predetermined number of tokens as a base, each time a set of tokens is partitioned from the header of the current data processing request, the number of tokens in the later partition is either twice the number of tokens in the previous partition, or the number of tokens in the later partition is increased by a base number relative to the number of tokens in the previous partition; and A hash function is used to generate a hash value for each of the partitioned token sets, thereby forming one or more hash values ​​for the current data processing request. The matching hit rate is the ratio of the number of hash values ​​corresponding to the current data processing request that match the hash values ​​cached in the metadata server to the number of hash values ​​corresponding to the current data processing request.

6. The inference system for large models as described in claim 5, wherein after forwarding each received data processing request to the pre-filling component or the decoding service component, the routing component calculates the hash value of the data processing request routed from the routing component to the destination data processing instance, caches it in the metadata server and / or locally on the routing component, calculates the matching hit rate score between the hash value corresponding to the current data processing request and the hash values ​​of the data processing requests processed by each data processing instance, and based on the calculated matching hit rate score, the routing component routes the received current data processing request to the data processing instance with the highest matching hit rate score in the pre-filling component or the decoding service component.

7. The inference system for large models as described in claim 1 or 5, wherein after forwarding each received data processing request to the pre-filling component or the decoding service component, the routing component further forms information about the destination data processing instance routed in the pre-filling component or the decoding service component, so as to calculate the load score of the data processing requests being processed by the destination data processing instance from the data processing requests routed by the routing component, and based on the calculated load score, load-balancedly route the received current data processing request to the data processing instance with the highest load score in the pre-filling component or the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load score.

8. The inference system for large models as described in claim 7, wherein the load quantity score of the data processing requests being processed by the data processing instance gradually increases as the load quantity score of the tasks arranged therein is completed.

9. The inference system for large models as described in claim 6, wherein after forwarding each received data processing request to the pre-filling component or the decoding service component, the routing component further generates information about the destination data processing instances routed and forwarded in the pre-filling component or the decoding service component, so as to calculate the load number score of the data processing requests being processed by the destination data processing instance from the data processing requests routed and forwarded by the routing component, and calculate the sum of the calculated load number score and the match hit rate score, and load-balancely route the received current data processing requests to the data processing instance with the highest score in the pre-filling component or the decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load number score.

10. An inference method for large models, including The routing step involves the following steps: After the routing component receives the user's current data processing request, it obtains the token information of the current data processing request and matches it with the token information cached in the metadata server. Current data processing requests with a matching hit rate higher than a predetermined threshold are identified as short requests and routed to the decoding service component. Current data processing requests with a matching hit rate less than or equal to the predetermined threshold are identified as long requests and routed to the pre-filling component. The maximum number of tokens for long requests is set to 1, and the token information of the current data processing request is cached in the metadata server. The pre-filling step involves the pre-filling component processing instance pre-filling the long request forwarded from the routing component to generate an initial context representation. The processing instance then updates information including its own cache, load, and liveness status to the metadata server in real time. At the same time, the original long request and the generated initial context are transmitted to the decoding service component. as well as The decoding service step involves a processing instance of the decoding service component performing decoding processing based on a short request forwarded from the routing component and the context associated with the short request already cached in the metadata server, generating an output token corresponding to the short request. It also performs decoding processing based on the original long request transmitted from the pre-filling component via the communication component and the initial context generated for the long request, generating an output token corresponding to the long request. Furthermore, its processing instance updates information including its own cache, load, and liveness status to the metadata server in real time.

11. The reasoning method for large models as described in claim 10, further comprising: The smoothing quantization step involves using the SmoothQuant W8A8 smoothing quantization component after the pre-filled component has completed training. The activation values ​​and weights accessed by the pre-filled component are rescaled using a smoothing factor. The smoothed weights and activation values ​​are then quantized and converted into INT8 numerical format for use in the pre-filled component for pre-filling.

12. The reasoning method for large models as described in claim 10 or 11, further comprising: In the weight quantization step, after the decoding service component completes training, the weight values ​​to be accessed by the decoding service component are quantized by the WeightOnly quantization component and converted into INT4 numerical format for the decoding service component to perform decoding processing.

13. The inference method for large models as described in claim 10, wherein obtaining the token information of the current data processing request includes: The current data processing request is parsed into one or more tokens; Using a predetermined number of tokens as a base, a set of tokens is divided from the header of the current data processing request each time. In two adjacent divisions, the number of tokens in the second division is twice the number of tokens in the previous division, or the number of tokens in the second division is increased by a base number relative to the number of tokens in the previous division. as well as A hash function is used to generate a hash value for each of the partitioned token sets, thereby forming one or more hash values ​​for the current data processing request. The matching hit rate is the ratio of the number of hash values ​​corresponding to the current data processing request that match the hash values ​​cached in the metadata server to the number of hash values ​​corresponding to the current data processing request.

14. The reasoning method for large models as described in claim 13, further comprising: After forwarding each received data processing request to the pre-filling component or the decoding service component, the routing component calculates the hash value of the data processing requests routed from the routing component to the destination data processing instance and caches it in the metadata server and / or locally in the routing component. as well as Calculate the matching hit rate score between the hash value corresponding to the current data processing request and the hash values ​​of the data processing requests processed by each data processing instance; as well as Based on the calculated match hit rate score, the received current data processing request is routed to the data processing instance with the highest match hit rate score in either the pre-filled component or the decoding service component.

15. The reasoning method for large models as described in claim 10 or 13, further comprising: After forwarding each received data processing request to the pre-filling component or the decoding service component, the routing component calculates a load score of the number of data processing requests routed and forwarded from the routing component that the destination data processing instance is currently processing. as well as Based on the statistical load count score, the load balancer routes and forwards the received current data processing request to the data processing instance with the highest load count score in the pre-filled component or decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load count score.

16. The reasoning method for large models as described in claim 14, further comprising: After forwarding each received data processing request to the pre-filling component or the decoding service component, the routing component calculates a load score of the number of data processing requests routed and forwarded from the routing component that the destination data processing instance is currently processing. as well as The sum of the statistical load quantity score and the match hit rate score is calculated, and the received current data processing requests are load-balancedly routed to the data processing instance with the highest sum of scores in the pre-filled component or decoding service component, wherein the higher the number of data processing requests being processed by the data processing instance, the lower the load quantity score.

Images