Deployment method and system of large language model

Abstract

The invention relates to the field of artificial intelligence, in particular to a deployment method and system for a large language model, and the method comprises the steps: (1) a service request end receives at least one piece of service request information, and stores the service request information in a service request message queue; (2) responding to a pre-filling stage calculation unit request and a load condition, and distributing service request information; (3) finishing the processing of the pre-filling stage to obtain at least one operation result; (4) the load balancer determines a task operation calculation unit in the stage according to the load condition of the calculation unit in the decoding stage; and (5) inputting the operation result of the pre-filling stage into the task operation calculation unit of the decoding stage, and outputting the result. The method has the advantages that the pre-filling stage and the decoding stage are deployed on a machine with high-performance computing power and a large memory respectively, load tasks are balanced, maximum hardware utilization is achieved, idle computing power is reduced, overall delay is reduced, throughput is improved, and expansibility and fault tolerance of the system are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence, and particularly to a method and system for deploying a large language model. Background Art

[0002] With the wide application of large language models (LLMs) in the field of natural language processing, the scale and complexity of the models have been continuously increasing. Especially in generative tasks (such as text generation, dialogue systems), the computational and memory requirements of the models have increased significantly. The inference process of an LLM generally can be divided into two main stages: the prefill stage and the decode stage. In the prefill stage: mainly processes the input sequence, performs forward propagation on the entire input sequence, and generates the initial hidden state. This stage needs to process a batch of tokens. In the decode stage: gradually generates the output sequence, generates one or several tokens each time, and updates the context. This stage processes one token each time.

[0003] In the prior art, usually these two stages are deployed on the same machine, which may cause the following problems:

[0004] 1. Unbalanced utilization of hardware resources. The two stages have different hardware requirements. The prefill stage is computationally intensive and requires high-performance computing resources (such as GPUs), but has relatively low memory requirements. The decode stage is memory intensive, needs to maintain the generated context, requires high video memory bandwidth, but has low requirements for computing resources. If deployed on the same physical device, it may lead to unbalanced utilization of hardware resources, resulting in some resources being overused while other resources are idle.

[0005] 2. Increased latency. When processing the prefill stage and the decode stage on the same physical device, the latency may increase due to resource contention. For example, the decode stage occupies a large amount of memory, resulting in a decrease in the throughput of the prefill stage and affecting the generation speed of the first token. The computationally intensive prefill stage may occupy a large amount of CPU / GPU resources, resulting in an increase in the waiting time of the tasks in the decode stage, thus affecting the word output speed of each character.

[0006] 3. Unbalanced load. For a computer cluster, placing the two stages on the same physical device makes it impossible to achieve secondary load balancing between the two stages, and it may also occur that due to the bottleneck of a single computer's resources, it is impossible to better achieve overall load balancing.

[0007] 4. Limited scalability. Placing the two stages on the same physical device will limit the scalability of the system. For example, when the length of the input sequence increases, the memory requirement will rise sharply, while the computing resources may not be able to scale as needed, resulting in a decline in system performance.

[0008] 5. Poor fault tolerance. Both the pre-filling stage and the decoding stage run on the same machine. Once this machine fails, the entire inference process will be interrupted. Summary of the Invention

[0009] The purpose of the present invention is to overcome the above deficiencies, and to provide a deployment method and system for large language models, which distribute the pre-filling stage and the decoding stage on different node machines, and each stage performs load balancing allocation through a load balancer, thereby improving the overall operation efficiency of the system.

[0010] The present invention achieves the above object through the following solutions: A deployment method for a large language model, comprising the following steps:

[0011] (1) The service request end receives at least one service request message, and stores the service request message in the service request message queue;

[0012] (2) Respond to the request of the pre-filling stage calculation unit, and allocate service request messages according to the load status of the calculation unit in this stage;

[0013] (3) Complete the processing of the pre-filling stage to obtain at least one operation result;

[0014] (4) The load balancer determines the task running calculation unit in this stage according to the load of the decoding stage calculation unit;

[0015] (5) Input the operation result of the pre-filling stage into the task running calculation unit of the decoding stage, enter the decoding stage for loop operation, and output the result.

[0016] Preferably, the pre-filling stage and the decoding stage are separately deployed on different physical devices.

[0017] Preferably, one or more GPUs are divided into a calculation unit, which is used as the unit of an independent inference task. The number of GPUs included in one calculation unit is adjusted according to the deployment requirements.

[0018] Preferably, in step (2), the task is allocated by the EntryPoint according to the load of the pre-filling stage calculation unit, so that the computing power of the calculation unit can be fully utilized.

[0019] Preferably, the specific steps implemented in step (2) include:

[0020] (2.1) The pre-filling stage calculation unit sends a request to the Entrypoint to obtain the requested task;

[0021] (2.2) Obtain the current calculation quota of the calculation unit.

[0022] (2.3) Allocate the service request information in the message queue of requests.

[0023] Preferably, during the process of a single request task in step (2), it is expected to obtain as many request tasks as possible, so that the current computing unit can reach or even slightly exceed the maximum computing quota.

[0024] Preferably, the load balancing strategy in the decoding stage is implemented by the load balancer to allocate the next KV Cache to the computing unit with the least load as much as possible, while taking into account the load balance of the overall nodes as much as possible.

[0025] Preferably, the implementation steps of step (4) specifically include:

[0026] (4.1) Allocate to the computing unit with the least load without exceeding the maximum load of the corresponding node + the overload value.

[0027] (4.2) Ensure that after the allocation is completed, on the premise that the load of the corresponding computing unit is lower than 88%, allocate to the computing unit with the least excess over the max of the corresponding node.

[0028] (4.3) Allocate to the computing unit with the least load among all computing units.

[0029] A deployment system for a large language model, comprising: a service request end entrance, a plurality of prefill phase nodes, a load balancer, a plurality of decoding phase nodes, and an output device; the service request end entrance is connected to the prefill phase nodes, the prefill phase nodes are connected to the decoding phase nodes, and are scheduled through the load balancer in the middle, and the output device is connected to the decoding phase nodes.

[0030] Preferably, the prefill phase nodes are computers with high computing performance configurations.

[0031] Preferably, the decoding phase nodes are computers configured with a relatively high video memory bandwidth.

[0032] The beneficial effects of the present invention are as follows: For large-scale generative tasks, by separately deploying the prefill phase and the decoding phase on different devices, the Prefill phase can be deployed on the physical devices of machines with high computing performance (such as GPU clusters), and the corresponding Decode phase can be deployed on machines with large memory. According to the resource requirements of the corresponding phases, configure computing and memory resources, improve the utilization rate of hardware resources, and try to achieve the maximization of hardware utilization, avoid processing computing and memory-intensive tasks on the same machine at the same time, thereby reducing the overall latency, effectively improving the throughput, and enhancing the scalability and fault tolerance of the system while improving the user experience.

[0033] Meanwhile, a load balancer is used to connect the machines that execute the prefill phase and the decoding phase, and the load balancing of the inference task allocation in the two phases is achieved through the EntryPoint and the KVLB (Key-Value Load Balancer). In the Prefill phase, as many request tasks as possible are allocated during a single task allocation process, so that the computing power can be fully utilized and the idle computing power can be minimized. In the Decode phase, the adopted load balancing strategy is achieved by allocating the next KV Cache to the computing unit with the least load as much as possible, while also considering that the load of the overall node is as balanced as possible, so as to improve the overall operation efficiency of the system, reduce the first output latency of the large language model, and ensure the uniform pronunciation of subsequent output latency. Description of the Drawings

[0034] Figure 1 is a schematic diagram of the step flow of the method of the present invention;

[0035] Figure 2 is a schematic diagram of the specific step flow of step (2) of the method of the present invention;

[0036] Figure 3 is a schematic diagram of the specific step flow of step (3) of the method of the present invention;

[0037] Figure 4 is a schematic diagram of the method step flow of the load balancing strategy of the present invention;

[0038] Figure 5 is a schematic diagram of the structure of the system of the present invention. Detailed Embodiments

[0039] The present invention will be further described below in conjunction with specific implementation examples, but the protection scope of the present invention is not limited thereto:

[0040] To better understand the method of the present invention, some technical contents involved in the present invention will be described below first, and then each step will be described in detail.

[0041] The present invention is directed to large-scale generative tasks. By distributing the prefill stage and the decoding stage on different nodes and connecting them through a load balancer. The service request side is responsible for receiving and processing at least one service request message sent by the client. After being processed by this system, the corresponding processing result is returned to the client. The service request side accepts the request message through the EntryPoint. The EntryPoint is the entry of the large language model system of the service request side, responsible for coordinating and controlling the startup of the entire system, initializing each component, such as the prefill stage, the decoding stage, the KVLB (Key-Value LoadBalancer), etc., loading the configuration file, implementing the distribution of service requests, data stream management and resource scheduling, to ensure that the system runs as expected. The EntryPoint needs to implement dynamic scheduling of tasks according to the system load and resource availability, and decide the allocation of the nodes where the service request message runs.

[0042] The service request message can be an inference request. For example, it can include requests for natural language processing services such as text generation, translation, and question answering.

[0043] In one embodiment, the large language model service system can provide multiple access interfaces and support multiple access protocols, so as to receive different types of service requests. For example, it can be service requests of the HTTP or GRPC type. In one embodiment, users or application programs, etc. can send service requests to the server request side through an interactive interface, an interface, etc. For example, in the scenario of a dialogue robot, the user can input the question raised in the form of voice or input text through the interactive interface, and trigger the service request to answer the question. After the server request side receives the service request triggered by the above user, it further executes the corresponding steps of the present invention.

[0044] The cluster for executing service requests includes multiple nodes, and the nodes refer to computer servers. One node includes multiple GPUs. The nodes are divided into two categories according to different performance configurations and execution task stages. One category of nodes is only used to execute the prefill stage tasks. This stage can be configured with computers with more high-performance computing resources (such as GPUs) specifically, but has relatively low memory requirements. Another category of nodes is only used to execute the decoding stage tasks. The decoding stage is memory-intensive and needs to maintain the generated context. This stage can be configured with computers with a relatively high video memory bandwidth according to the requirements.

[0045] To improve the utilization rate of computing power resources, according to the characteristics of task operation, one or more GPUs are divided into a computing unit, which serves as a unit for an independent inference task. A node may include one or more computing units. The number of GPUs included in the computing unit can be adjusted according to actual needs and computing characteristics. For example, in one embodiment, 4 GPUs are used as a computing unit in the pre-filling stage, and 1 GPU is used as a computing unit in the decoding stage; in another embodiment, the number of computing units in both stages is adjusted to the same number, both being 8 GPUs. This system can select other several numbers of GPUs as a computing unit according to deployment requirements.

[0046] Communication between two types of nodes is achieved through a load balancer. The load balancer can adopt a Key-Value (KV) database based on multi-value hashing, which is responsible for managing and distributing KV Cache data. In a specific embodiment, the load balancer can be KVLB. The KV Cache is a collection of Key and Value matrices, which stores all the Key and Value matrices generated during the inference process of the model. It is a dynamically updated data structure, and as new tokens are generated, the KV Cache will continuously expand.

[0047] Data communication between the pre-filling stage and the decoding stage is achieved through KVLB, and the multi-value hashing mechanism and load balancing function of KVLB are used to optimize data transmission and processing efficiency. For the KVCache generated in the pre-filling stage, the load balancer instructs the KV Cache to be sent to the corresponding decoding stage computing unit according to the load balancing strategy.

[0048] In one embodiment, a node computer server includes one or more GPUs. Most of the operations involved in deep learning are vectorized matrix operations. The GPU is used to execute the inference task of the large language model. The corresponding algorithm model is loaded into the GPU video memory according to the task request information to execute the inference task.

[0049] A deployment method for a large language model, as Figure 1 shown, includes the following steps:

[0050] S1: The service request end receives at least one service request information and stores the service request information in the service request message queue. As the entry point of the system, EntryPoint stores the service request information in the request message queue after receiving the service request information.

[0051] The access order of the request message queue can be various, including but not limited to priority access, sequential access, service type access, First In First Out (FIFO), and Last In First Out (LIFO), etc.

[0052] In one embodiment, the request message queue access order includes priority access, service type access, and first-in-first-out.

[0053] In another embodiment, the request message queue access order includes sequential access, service type access, and last-in-first-out.

[0054] S2: Respond to the requests of the pre-filling phase computing unit, and allocate service request information according to the load status of the computing unit in this phase. As Figure 2 shown, the specific steps include:

[0055] S21: The pre-filling phase computing unit sends a request to the EntryPoint to obtain the request task.

[0056] The request task is an inference task for further processing the service request information in the request message queue, and is used to implement natural language processing such as text generation, translation, and question answering.

[0057] S22: Obtain the current computing quota (ComputeQuota) of the computing unit.

[0058] The computing quota refers to the computing power that can be used to complete one or more inference tasks. The maximum computing power that each computing unit can provide for a single run is the maximum computing quota (CQ_max) of the computing unit.

[0059] When the pre-filling phase computing unit needs to send a request to the EntryPoint to obtain the request task, it also needs to send the current computing quota of this computing unit to the EntryPoint. The current computing quota refers to the computing power currently available for corresponding consumption to complete one or more inference tasks, reflecting the current carrying capacity of this computing unit.

[0060] To ensure that the computing power of the computing unit is fully utilized and idle computing power is minimized, during a single request task, it is expected to obtain as many request tasks as possible, so that the current computing unit can reach or slightly exceed the maximum computing quota (CQ_max). After exceeding the maximum computing quota, the request task will be truncated, and the remaining request tasks will be calculated in the second round. After the request task is truncated, the computing power required for the tasks that need to be completed in the second round is the remaining computing quota (CQ_rest). If the computing power of the request task is just fully consumed during the first-round request, the remaining computing quota (CQ_rest) = 0. Therefore, in each request of the computing power unit, the current computing quota of this computing unit sent to the EntryPoint is equal to the difference between the maximum computing quota (CQ_max) and the remaining computing quota (CQ_rest). Specifically, the calculation method of the current computing quota is expressed by the formula ComputeQuota = A * sum(seq_len Q) + B * sum(seq_len Q * seq_len K - (seq_len Q)^2 / 2 + 409.6 * seq_len K), where A and B are physical attribute parameters corresponding to the computing unit, obtained by actual measurement, seq_len Q is the length of the request task that has not been calculated, and seq_len K is the total length of the request task.

[0061] S23: Allocate the service request information in the request message queue.

[0062] After receiving the service request information and the current computing quota, the EntryPoint allocates the corresponding service request information from the service request message queue according to the current computing quota, and distributes the request information to the corresponding computing units.

[0063] S3: Complete the processing of the pre-filling stage to obtain at least one running result.

[0064] In one embodiment, the request task is allocated to the computing unit in the pre-filling stage to complete the corresponding processing. The pre-filling stage is the first stage of large language model inference. As Figure 3 shown, it mainly includes the following steps:

[0065] S31: Process the input sequence. Processing the input sequence means receiving the input sequence provided by the user or the system and performing preprocessing. The input sequence is usually the input data of the model, such as the user's question, conversation history, or sentence to be translated. The preprocessing of the input sequence includes Tokenization, Padding, Masking, etc.

[0066] S32: Calculate the initial hidden state. Calculating the hidden state refers to performing forward propagation calculation on the input sequence to generate the initial hidden state. It includes inputting the input sequence into the embedding layer of the model to generate the embedding representation of the token; calculating the hidden state layer by layer through multiple Transformer layers of the model; in each layer, calculating the self-attention and feed-forward neural network, and calculating the hidden state layer by layer.

[0067] S33: Generate KV Cache.

[0068] KV Cache is the core output of the pre-filling phase. These matrices will be used to gradually generate outputs in the decoding phase to accelerate the reasoning process in the decoding phase. In a specific implementation, the generated KV Cache optimizes the allocation of the Decode phase through KVLB to achieve load balancing.

[0069] S4: The load balancer determines the computing unit to run the task in this stage according to the load of the computing unit in the decoding stage.

[0070] In a specific embodiment, KVLB, as a load balancer, schedules the KV Cache generated in the pre-filling phase according to the load balancing strategy, and specifies that the KV Cache is sent to the corresponding decoding phase computing unit. In the decoding phase, the output sequence is generated step by step, that is, a token is generated each time and added to the output sequence. Therefore, KVLB optimizes the allocation of the decoding phase, processes and allocates tokens one by one, and ensures that the KV Cache is distributed as evenly as possible to achieve load balancing.

[0071] The load balancing strategy is implemented by allocating the next KV Cache to the computing unit with the least load as much as possible, while also considering that the load of the entire node is as balanced as possible, such as Figure 4 As shown, the specific implementation method is:

[0072] S41: Allocate to the computing unit with the least load without exceeding the maximum load of the corresponding node + the load excess value.

[0073] First, the existing load of each node is obtained as X, the KVCache calculation load to be allocated is N, and the node with the maximum load among all nodes is determined as max, and the load excess value is L. For the allocated node, if X+N≤max+L is satisfied, the KV Cache to be allocated is allocated to the computing unit with the least load among the computing units that meet the conditions after loading N. If not satisfied, go to step 42.

[0074] The definition of the load excess value is the maximum value of the excess part that allows exceeding the maximum load of the node. The significance of this value is to reduce the cases entering the next step determination. In this step, exceeding the maximum load value of the node is allowed, but there is an upper limit for the excess part. This upper limit can be selected according to the situation in different deployment environments. In this embodiment, the load excess value is 16K tokens. In other embodiments, the load excess value can also be adjusted according to the actual situation.

[0075] S42: After ensuring the allocation is completed, under the premise that the load of the corresponding computing unit is lower than 88%, allocate the computing unit with the least excess over the corresponding node max.

[0076] If the previous one cannot be satisfied, continue the determination. Determine limit as the load upper limit of the computing unit, and Y as the load of each computing unit. If Y + N <= limit * 88% is satisfied, then select the computing unit with the least load among the nodes with the least excess over max after allocation. That is, under the premise of satisfying Y + N <= limit * 88%, first select the node with the least X + N - max, and then select the computing unit with the least load among the aforementioned nodes. The purpose is that if the GPU usage of the node exceeds 90%, then the node needs to queue in the queue.

[0077] If the condition is not satisfied, enter step S43.

[0078] S43: Allocate to the computing unit with the least load among all computing units. If all the above determinations are not satisfied, directly allocate the next KV Cache to the computing unit with the least load among all computing units.

[0079] S5: Input the operation result of the pre-filling stage into the task operation computing unit of the decoding stage, enter the decoding stage to run in a loop, and output the result.

[0080] The decoding stage is the second stage of the large language model inference, mainly responsible for gradually generating the output sequence. Different from the pre-filling stage, the decoding stage is generated step by step, that is, generating one or several tokens each time and updating the context.

[0081] This stage receives the KVCache output from the pre-fill stage, initializes the context, and gradually generates the output. The process of generating each token is as follows: First, calculate the probability distribution of the next token through self-attention calculation and feed-forward neural network calculation, etc.; Second, according to the probability distribution, select the next token through a selection algorithm, and the selection algorithm includes greedy search, beam search, sampling algorithm, etc.; Third, add the generated token to the context, and update the hidden state and KVCache; Finally, output the result through the EntryPoint. Repeat the above process, allocate and process the next token until the end.

[0082] Figure 5 It is a deployment system of a large language model disclosed according to an embodiment of the present invention. For large-scale generative tasks, the system includes: an EntryPoint, several pre-fill stage nodes, a load balancer, several decoding stage nodes, and an output device; the service request end entrance is connected to the pre-fill stage nodes, the pre-fill stage nodes are connected to the decoding stage nodes, and are scheduled through the load balancer in the middle, and the output device is connected to the decoding stage nodes.

[0083] The EntryPoint is the entrance of the large language model system at the service request end, responsible for coordinating and controlling the startup of the entire system, initializing each component (such as the pre-fill stage, decoding stage, KVLB, etc.), loading the configuration file, realizing the distribution of service requests, data flow management and resource scheduling, and ensuring that the system runs as expected. The EntryPoint needs to implement dynamic scheduling tasks according to the system load situation and resource availability, and decide the allocation of the service request running nodes.

[0084] The load balancer uses a Key-Value (KV) database based on multi-value hashing, which is responsible for managing and distributing KVCache data, such as KVLB. The KV Cache is a collection of Key and Value matrices, which stores all the Key and Value matrices generated by the model during the inference process. Data communication between the pre-fill stage and the decoding stage is achieved through KVLB, and the multi-value hashing mechanism and load balancing function of KVLB are used to optimize data transmission and processing efficiency. For the KVCache generated in the pre-fill stage, the load balancer instructs the KV Cache to send it to the corresponding decoding stage computing unit according to the load balancing strategy.

[0085] The pre-fill stage node refers to a computer dedicated to the pre-fill stage. This stage mainly processes the input sequence, performs forward propagation on the entire input sequence, and generates an initial hidden state. This stage needs to process a batch of tokens. The preprocessing stage is computationally intensive, so a computer configured with high-performance computing resources (such as GPUs) is used.

[0086] The decoding stage node refers to a computer used in the decoding stage. In this stage, the output sequence is gradually generated, one or several tokens are generated each time, and the context is updated. Each time, one token is processed. This stage is memory-intensive and requires maintaining the generated context. A computer with a relatively high video memory bandwidth can be configured according to requirements.

[0087] The output device is the output terminal of the system, used to output the tokens generated in the decoding stage. In one embodiment, the output can also be achieved through the EntryPoint.

[0088] A node includes multiple GPUs. To improve the utilization rate of computing power resources, according to the characteristics of task operation, one or more GPUs are divided into a computing unit, which serves as a unit for an independent inference task. A node may include multiple computing units. The number of GPUs included in the computing unit can be adjusted according to the actual deployment strategy.

[0089] By deploying the prefill stage and the decoding stage on different machines and performing load balancing on the KV Cache transmission process, it is possible to configure computing and memory resources according to the resource requirements of the prefill stage and the decoding stage, improve the utilization rate of hardware resources, avoid processing computationally intensive and memory-intensive tasks on the same machine simultaneously, thereby reducing the overall latency, effectively improving the throughput, enhancing the user experience, and at the same time, enhancing the scalability and fault tolerance of the system.

[0090] The above are the specific embodiments of the present invention and the technical principles applied. If changes made according to the concept of the present invention do not exceed the spirit covered by the description and the drawings in terms of the functions and effects produced, they should still fall within the protection scope of the present invention.

Claims

1. A deployment method for a large language model, characterized in that It includes the following steps: (1) The service request end receives at least one service request message and stores the service request message in the service request message queue; (2) Respond to the request of the pre-filling stage calculation unit and allocate service request messages according to the load condition of the calculation unit in this stage; (3) Complete the processing of the pre-filling stage to obtain at least one operation result; (4) The load balancer determines the calculation unit for task operation in this stage according to the load condition of the calculation unit in the decoding stage; (5) Input the operation result of the pre-filling stage into the task operation calculation unit in the decoding stage, enter the decoding stage for loop operation, and output the result.

2. The deployment method of a large language model according to claim 1, wherein The pre-filling stage and the decoding stage are separately deployed on different physical devices.

3. The deployment method of a large language model according to claim 2, characterized in that One or more GPUs are divided into one calculation unit, which is used as the unit of an independent inference task. The number of GPUs included in one calculation unit is adjusted according to the deployment requirements.

4. A deployment method of a large language model according to any one of claims 1-3, characterized in that In step (2), the EntryPoint allocates tasks according to the load condition of the calculation unit in the pre-filling stage, so that the computing power of the calculation unit can be fully utilized.

5. A method for deploying a large language model according to any one of claims 1-4, characterized in that The specific steps implemented in step (2) include: (2.1) The calculation unit in the pre-filling stage sends a request to the EntryPoint to obtain the requested task; (2.2) Obtain the current calculation quota of the calculation unit. (2.3) Allocate the service request messages in the request message queue.

6. A deployment method of a large language model according to any one of claims 1-5, characterized in that In the process of a single request task in step (2), it is expected to obtain as many requested tasks as possible, so that the current calculation unit can reach or slightly exceed the maximum calculation quota.

7. A method for deploying a large language model according to any one of claims 1-3, characterized in that In step (4), the load balancing strategy of the decoding stage is implemented by the load balancer by allocating the next KV Cache to the calculation unit with the least load as much as possible, while taking into account the overall load balance of the nodes as much as possible.

8. A deployment method for a large language model according to claim 7, wherein The implementation steps of step (4) specifically include: (4.1) Allocate to the calculation unit with the least load without exceeding the maximum load of the corresponding node + the load excess value; (4.2) Ensure that after the allocation is completed, allocate to the calculation unit with the least excess over the max of the corresponding node on the premise that the load of the corresponding calculation unit is lower than 88%; (4.3) Allocate to the calculation unit with the least load among all calculation units.

9. A deployment system for a large language model, characterized in that It includes: The service request end entrance, several pre-filling stage nodes, a load balancer, several decoding stage nodes and an output device; the service request end entrance is connected to the pre-filling stage nodes, the pre-filling stage nodes are connected to the decoding stage nodes, and are scheduled through the load balancer in the middle, and the output device is connected to the decoding stage nodes.

10. The deployment system of a large language model according to claim 9, wherein The pre-filling stage nodes are computers with high computing performance configurations.

11. The deployment system of a large language model according to claim 9, characterized in that The decoding stage nodes are computers with relatively high video memory bandwidth configurations.

Images