Large language model system and request response method thereof

Abstract

The invention discloses a large language model system and a request response method thereof, and relates to the technical field of large language models. According to the specific implementation mode of the method, a large language model system comprises a pre-filling instance and a decoding instance, the pre-filling instance and the decoding instance are deployed on first hardware and second hardware respectively, the first hardware comprises at least one first processor, and the second hardware comprises at least one second processor; the computing power of the first processors and the total number of the first processors are determined according to the load characteristics of the pre-filled instances, the computing power of the second processors and the total number of the second processors are determined according to the load characteristics of the decoded instances, and the total computing power of the first processors is higher than the total computing power of the second processors. According to the implementation mode, flexible configuration of hardware resources and full utilization of the hardware resources are realized, and the hardware cost is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of large language models, and in particular to a large language model system and a request response method thereof. Background Art

[0002] Large language model reasoning can be divided into two stages: pre-population stage and decoding stage. Currently, the pre-population stage and decoding stage use the same service instance and are deployed on the same hardware. However, most hardware clusters use homogeneous configurations. Currently, a service instance of a model can only be deployed on the same hardware, and different hardware can only deploy different instances. This hardware architecture is not flexible enough and will result in insufficient resource utilization. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a large language model system and a request response method thereof, wherein the large language model system includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are respectively deployed on a first hardware and a second hardware, wherein the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor. Thus, by deploying the pre-filled instance and the decoding instance on different hardware, and adjusting the resources of the first hardware and the second hardware according to the respective load characteristics of the pre-filled instance and the decoding instance, flexible configuration of hardware resources and full utilization of hardware resources are achieved, thereby reducing hardware costs.

[0004] To achieve the above objective, according to one aspect of an embodiment of the present invention, a large language model system is provided.

[0005] A large language model system according to an embodiment of the present invention includes: a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are deployed on a first hardware and a second hardware, respectively; the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor.

[0006] Optionally, the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, using the following method: predicting the first computing resources required to run the pre-filled instance according to the load characteristics of the pre-filled instance, predicting the second computing resources required to run the decoding instance according to the load characteristics of the decoding instance, and the first computing resources are greater than the second computing resources; and screening out at least one of the first processors and at least one of the second processors according to the computing power of each processor in the processor resource pool so that the total computing power of the first processor is higher than the first computing resource, and the total computing power of the second processor is higher than the second computing resource.

[0007] Optionally, at least one of the first processors includes an image processing unit, a neural network processing unit and / or a tensor processing unit; at least one of the second processors includes an image processing unit, a neural network processing unit and / or a tensor processing unit.

[0008] Optionally, when there are multiple first processors, the computing powers of the multiple first processors are not completely consistent, and / or the types of the multiple first processors are not completely consistent; when there are multiple second processors, the computing powers of the multiple second processors are not completely consistent, and / or the types of the multiple second processors are not completely consistent.

[0009] To achieve the above objective, according to another aspect of an embodiment of the present invention, a request response method based on a large language model system is provided.

[0010] An embodiment of the present invention provides a request response method based on a large language model system, wherein the large language model system includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are respectively deployed on a first hardware and a second hardware, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor; the method includes: pre-processing a received user request through the pre-filled instance to obtain a pre-processing result, and sending the pre-processing result to the decoding instance; generating reply information for the user request according to the pre-processing result through the decoding instance.

[0011] Optionally, the first hardware includes at least one first processor, which preprocesses the received user request through the pre-filled instance to obtain a preprocessing result, including: splitting the preprocessing task into at least one subtask through the pre-filled instance, and correspondingly assigning it to at least one first processor, so that at least one subtask is correspondingly executed by the at least one first processor to obtain at least one calculation result output by the at least one first processor; aggregating the at least one calculation result to obtain the preprocessing result.

[0012] Optionally, the second hardware includes multiple second processors, and the total computing power of the first processor is higher than the total computing power of the multiple second processors. The decoding instance generates reply information for the user request according to the preprocessing results, including: for multiple batches of user requests: the decoding instance calculates one of the batches of user requests through the current second processor to obtain sub-reply information, sends the sub-reply information to the next second processor, and calculates the next batch of user requests through the current second processor.

[0013] Optionally, the received user requests are preprocessed by pre-filling the instance to obtain a preprocessing result, including: determining the number of user requests to be processed in this batch and the length of context information corresponding to each user request by pre-filling the instance; the context information includes each user request, the request previously input by the user and its corresponding reply information; constructing an input matrix using the context information corresponding to the user requests to be processed; each row of the input matrix corresponds to the context information of a user request to be processed; the number of rows of the input matrix is ​​the number of user requests to be processed in this batch, and the number of columns of the input matrix is ​​the maximum length of the context information corresponding to the request to be processed; according to the dimension of the input matrix, determining a first target computing unit from multiple first computing units; the multiple first computing units are generated by code blocks running on the first hardware; the multiple first computing units can be used to calculate input matrices of different dimensions; using the first target computing unit to calculate the input matrix to obtain the preprocessing result.

[0014] Optionally, a decoding instance generates reply information for a user request according to a preprocessing result, including: the decoding instance calculates a matrix corresponding to the preprocessing result through a second computing unit, and generates reply information for the user request according to the calculation result; the second computing unit is generated by a code block running on second hardware; the dimension of the matrix that can be processed by the second unit is smaller than the dimension of the matrices that can be processed by multiple first computing units.

[0015] Optionally, the preprocessing of the received user request by the pre-filled instance to obtain a preprocessing result also includes: converting the parameters corresponding to the context information and the model parameters required for preprocessing the user request into first target parameters and second target parameters by the pre-filled instance, and performing calculations using the first target parameters and the second target parameters to obtain the preprocessing result; the accuracy of the first target parameters and the second target parameters are respectively lower than the accuracy of the parameters corresponding to the context information and the model parameters required for preprocessing the user request.

[0016] Optionally, the preprocessing task of the current preprocessing layer in the large language model is performed through the pre-filled instance to obtain a current preprocessing result corresponding to the current preprocessing layer, and the current preprocessing result is sent to the current decoding layer corresponding to the current preprocessing layer in the large language model, and the current preprocessing result is sent to the next preprocessing layer; the current reply information is generated according to the current preprocessing result through the decoding instance, and the current reply information is sent to the next decoding layer.

[0017] Optionally, the decoding instance generates reply information for the user request according to the preprocessing result, including: the decoding instance performs precision conversion on the model parameters acquired for generating the reply information to obtain converted model parameters, wherein the precision of the converted model parameters is higher than the precision of the model parameters before the conversion; and the reply information is generated according to the converted model parameters, context information, user request and the preprocessing result.

[0018] To achieve the above objective, according to another aspect of an embodiment of the present invention, an electronic device for request response based on a large language model system is provided.

[0019] An electronic device for request response based on a large language model system according to an embodiment of the present invention includes: one or more processors; a storage device for storing one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement a request response method based on a large language model system according to an embodiment of the present invention.

[0020] To achieve the above objective, according to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided.

[0021] A computer-readable storage medium according to an embodiment of the present invention stores a computer program, which, when executed by a processor, implements a request response method based on a large language model system according to an embodiment of the present invention.

[0022] One embodiment of the above invention has the following advantages or beneficial effects: the large language model system includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are respectively deployed on the first hardware and the second hardware, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor. Thus, by deploying the pre-filled instance and the decoding instance on different hardware, and adjusting the resources of the first hardware and the second hardware according to the load characteristics of the pre-filled instance and the decoding instance, flexible configuration of hardware resources and full utilization of hardware resources are achieved, thereby reducing hardware costs.

[0023] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0025] Figure 1 is a schematic diagram of the architecture of a large language model system according to an embodiment of the present invention;

[0026] Figure 2 It is a logical diagram of large language model reasoning;

[0027] Figure 3 This is a schematic diagram of the architecture of a large language model reasoning in the prior art;

[0028] Figure 4 is a schematic diagram of a large language model reasoning architecture according to an embodiment of the present invention;

[0029] Figure 5 is a flowchart of a request response method based on a large language model system according to an embodiment of the present invention;

[0030] Figure 6 is a schematic diagram of transmitting KVcache between a pre-filled instance and a decoded instance according to an embodiment of the present invention;

[0031] Figure 7 It is a software framework intention of a large language model system according to an embodiment of the present invention;

[0032] Figure 8 is an exemplary system architecture diagram to which embodiments of the present invention may be applied;

[0033] Fig. 9 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0035] It should be noted that the collection, collection, updating, analysis, processing, use, transmission, storage and other aspects of user personal information involved in the technical solution of this disclosure are in compliance with the provisions of relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken for user personal information to prevent illegal access to user personal information data and maintain the security of user personal information, network security and national security.

[0036] It should be pointed out that the embodiments of the present invention and the technical features therein may be combined with each other without conflict.

[0037] Figure 1 Schematic diagram of the main architecture of a large language model system according to an embodiment of the present invention.

[0038] like Figure 1 As shown, the large language model system of an embodiment of the present invention mainly includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are deployed on a first hardware and a second hardware respectively, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor.

[0039] The reasoning process of a large language model can be divided into two stages: the pre-filling stage and the decoding stage. The pre-filling stage is the process of pre-processing a piece of user input text, which is the user request; the decoding stage is the process of generating a reply message for the user input text. The decoding stage includes multiple iterations, each of which generates a token, that is, a word or a text fragment, and finally generates a reply message through continuous autoregression. In a typical question-and-answer product, the input of the pre-filling stage is the question asked by the user, and the output of the decoding stage is the answer given by the large language model. The challenge is that pre-filling and decoding are two tasks with completely different load characteristics. The input of the pre-filling stage is usually a long context, which can be tens to hundreds of megabytes of tokens. It is a computationally intensive task that relies on high computing power; the input of the decoding stage is only a token generated in the previous iteration, which is an I / O intensive task that relies on high memory access bandwidth. The interaction between the pre-filling stage and the decoding stage requires the transmission of KV cache data of the historical long context. The decoding stage needs to calculate the correlation between the token generated last time and the KV cache of the historical long context. The KV cache refers to the key-value set corresponding to all tokens in the context information. The reasoning process of the above large language model is as follows Figure 2 shown.

[0040] In the prior art, the pre-filling stage and the decoding stage are usually deployed in the same instance. The schematic diagram of the large language model inference architecture that uses the same instance for the current pre-filling stage and the decoding stage is as follows: Figure 3 shown. Figure 3 The large language model inference architecture shown has the following disadvantages:

[0041] First, the pre-filling stage and the decoding stage are implemented with the same hardware configuration and software. That is, computationally intensive tasks and I / O intensive tasks are executed together. As a result, when optimizing the inference performance, various optimization strategies are usually only applicable to one of the tasks but not the other, making it difficult to achieve optimal performance.

[0042] Second, since most hardware clusters use homogeneous configurations and rely on high-end communication facilities, a service instance of a model can usually only be deployed on the same hardware. Different hardware can only deploy different instances. In other words, even if the hardware requirements for the decoding stage are not high, since the two are mixed in the same instance, high-computing hardware must be used to meet the pre-filling stage with higher hardware requirements. This inference architecture makes the hardware configuration inflexible and unable to fully utilize hardware resources.

[0043] Third, the software reasoning framework corresponding to this reasoning structure only supports mixed scheduling execution of the pre-filling stage and the decoding stage. A service instance is responsible for responding to multiple requests and alternately calculating the pre-filling and decoding processes of different requests. Since the context length (corresponding calculation amount) of different requests is completely random, the mixed calculation mode easily leads to uncontrollable service quality.

[0044] In view of the shortcomings of the large language model reasoning structure in the prior art, this application proposes a large language model system. The core idea of ​​the system is to split the pre-filling stage and the decoding stage included in the reasoning process. After the split, each of the two is implemented by an independent instance. The schematic diagram of the split reasoning architecture is as follows: Figure 4 As shown. After the split, the hardware deployment scheme is involved according to the load characteristics of the pre-filling stage and the decoding stage, as follows: use the first hardware, that is, high-computing hardware to deploy the pre-filling instance, and use the second hardware, that is, low-computing hardware to deploy the decoding instance. In addition, the pre-filling instance pre-processes multiple user requests, and the decoding instance generates reply information for multiple user requests, separating computing-intensive tasks from I / O-intensive tasks, improving the utilization of hardware resources, increasing the response speed to user requests, and improving the user experience.

[0045] In an optional embodiment of the present invention, the computing power resources owned by the first hardware are higher than the computing power resources owned by the second hardware, that is, the total computing power of the first processor is higher than the total computing power of the second processor. Since the pre-filling stage is a computationally intensive task, and the decoding stage is an I / O intensive task, the total computing power of the first hardware for deploying the pre-filling instance is higher than the total computing power of the second hardware for deploying the decoding instance.

[0046] In an optional embodiment of the present invention, the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, and the following method is adopted: according to the load characteristics of the pre-filled instance, the first computing resources required to run the pre-filled instance are predicted, and according to the load characteristics of the decoding instance, the second computing resources required to run the decoding instance are predicted, and the first computing resources are greater than the second computing resources; according to the computing power of each processor in the processor resource pool, at least one of the first processors and at least one of the second processors are screened out, so that the total computing power of the first processor is higher than the first computing resource, and the total computing power of the second processor is higher than the second computing resource.

[0047] Specifically, for example, if the first computing resource is A, three processors are selected from the multiple processors in the processor resource pool, and the computing power of the three processors is A1, A2 and A3 respectively, and the computing power of the three processors needs to satisfy the formula: A1+A2+A3>A. It is understandable that in order to reduce costs, the difference between the total computing power of the first processor and the first computing resource A can be made within a preset range, that is, the total computing power of the first hardware is higher than the first computing resource to a certain extent, but it does not need to be too much higher. Similarly, for example, if the second computing resource is B, three processors are selected from the multiple processors in the processor resource pool, and the computing power of the three processors is B1, B2 and B3 respectively, and the computing power of the three processors needs to satisfy the formula: B1+B2+B3>B. It is understandable that in order to reduce costs, the difference between the total computing power of the second processor and the second computing resource B can be made within a preset range, that is, the total computing power of the second hardware is higher than the second computing resource to a certain extent, but it does not need to be too much higher.

[0048] In an optional embodiment of the present invention, at least one of the first processors includes an image processing unit, a neural network processing unit and / or a tensor processing unit; at least one of the second processors includes an image processing unit, a neural network processing unit and / or a tensor processing unit. The first processor may include one or more of a GPU (Graphic Processing Unit), an NPU (Neural Processing Unit), a TPU (Tensor Processing Unit), etc. The second processor may also include one or more of a GPU, an NPU, a TPU, etc.

[0049] In an optional embodiment of the present invention, when there are multiple first processors, the computing power of the multiple first processors is not completely consistent, and / or the types of the multiple first processors are not completely consistent; when there are multiple second processors, the computing power of the multiple second processors is not completely consistent, and / or the types of the multiple second processors are not completely consistent. The same type of processing unit can be used in the first hardware and the second hardware, such as the first hardware and the second hardware both use a graphics processing unit, or different types of processing units can be used, such as the first hardware uses a graphics processing unit and the second hardware uses a neural network processing unit. According to the computing power requirements of the pre-filled instance, a first processor with suitable computing power is selected from the optional processor resources, and according to the computing power requirements of the decoding instance, a second processor with suitable computing power is selected from the optional processor resources. In this way, high computing power and high cost processor products can be used in the first hardware and / or the second hardware respectively with some low computing power and low cost processor products, so as to reduce the processor cost while ensuring computing power. Since the first hardware supports computing-intensive tasks and the second hardware supports I / O-intensive tasks, the total computing power of the first hardware is higher than the total computing power of the second hardware. In addition, for the first hardware with high computing power, some processor products of the large computing power series can be used, and for the second hardware, some processor products of the small computing power series can be used, so that the hardware configuration can be flexibly adjusted, the range of optional processor products is wider, and the hardware configuration cost is reduced as a whole. In addition, since the first hardware adopts the processor products of the large computing power series, and the second hardware adopts the processor products of the small computing power series, the interaction between the pre-filled instance and the decoding instance no longer depends on high-configuration network communication equipment, such as NVLink or HCCS, etc., and only the most conventional Ethernet communication is required. Therefore, in an optional embodiment of the present invention, the first hardware and the second hardware communicate via Ethernet, thereby reducing the configuration cost of communication resources.

[0050] The large language model system of the embodiment of the present invention includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are respectively deployed on the first hardware and the second hardware, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor. Thus, by deploying the pre-filled instance and the decoding instance on different hardware, and adjusting the resources of the first hardware and the second hardware according to the respective load characteristics of the pre-filled instance and the decoding instance, flexible configuration of hardware resources and full utilization of hardware resources are achieved, thereby reducing hardware costs.

[0051] like Figure 5 As shown, in an embodiment of the present invention, a request response method based on a large language model system, the large language model system includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are respectively deployed on a first hardware and a second hardware, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor; the method mainly includes steps S501-S502:

[0052] Step S501: Preprocess the received user request through the pre-filling instance to obtain the preprocessing result, and send the preprocessing result to the decoding instance. Among them, in the large language model reasoning process, the pre-filling stage is the first important stage in the reasoning process, which is mainly used to process the initial input sequence and calculate the model state related to it (such as key-value pairs, attention weights, etc.). The preprocessing at this stage is crucial. By converting the input sequence into tokens, calculating the cache required by the attention mechanism, and other operations, it is prepared for subsequent incremental reasoning (i.e., gradually generating new tokens).

[0053] When responding to a user request, the pre-filled instance receives the user request. If the request is the first request from the user, the pre-filled instance only pre-processes the user request to obtain a token corresponding to the user request. If the request is not the first request from the user, the pre-filled instance needs to pre-process the current request, the user's previous request, and the reply information corresponding to the previous request to obtain a pre-processing result, which is the KV cache.

[0054] In an optional embodiment of the present invention, the first hardware includes at least one first processor, and preprocesses the received user request through the pre-filled instance to obtain the preprocessing result, including: splitting the preprocessing task into at least one subtask through the pre-filled instance, and correspondingly assigning it to at least one first processor, so that at least one subtask is correspondingly executed by at least one first processor to obtain at least one calculation result output by at least one first processor; aggregating at least one calculation result to obtain the preprocessing result. The first processor can be a GPU, an NPU and / or a TPU. The first processor can be one or more, but usually multiple first processors are used to speed up the calculation speed. In order to improve the response speed, the distributed strategy of the pre-filling stage adopts a tensor parallel strategy. Specifically, the parameters of each layer of the model are evenly split onto multiple processors, each processor is responsible for completing part of the calculation of each layer, and multiple processors can perform calculation tasks in parallel. After each layer is calculated, the calculation results of all processors are aggregated and then the next layer is calculated. This is equivalent to splitting the preprocessing tasks of each layer of the large language model into multiple subtasks, which are executed in parallel by multiple processors, summarizing the preprocessing calculation results output by multiple processors, and passing them to the next layer. The next layer performs the same operation until the summary result of the last layer is obtained, which is used as the preprocessing result of the pre-filled instance.

[0055] In order to improve the response speed, the pre-filling stage also uses multiple different large-shape kernels to optimize the computing performance. In an optional embodiment of the present invention, the received user request is pre-processed by the pre-filling instance to obtain the pre-processing result, including: determining the number of user requests to be processed in this batch and the length of the context information corresponding to each user request by the pre-filling instance; the context information includes each user request, the request previously input by the user and its corresponding reply information; constructing the input matrix using the context information corresponding to the user request to be processed; each row of the input matrix corresponds to the context information of a user request to be processed; the number of rows of the input matrix is ​​the number of user requests to be processed in this batch, and the number of columns of the input matrix is ​​the maximum length of the context information corresponding to the request to be processed; according to the dimension of the input matrix, determining the first target computing unit from multiple first computing units; multiple first computing units are generated by code blocks running on the first hardware; multiple first computing units can be used to calculate input matrices of different dimensions; using the first target computing unit to calculate the input matrix to obtain the pre-processing result.

[0056] Since the data volume of context information corresponding to a batch of user requests is different, that is, the number of user requests in each batch and the number of tokens of context information are different, if the same computing unit is used for batches of user requests of various data volumes, it is difficult to achieve the optimal response performance for each batch of user requests. Therefore, after determining the user request to be processed, the data volume of the user request to be processed can be first determined. Specifically, the dimension of the input matrix is ​​determined according to the number of rows and columns of the input matrix corresponding to the user request to be processed, and the target computing unit corresponding to the dimension of the input matrix is ​​selected from multiple computing units according to the dimension of the input matrix. The multiple computing units are all code blocks with computing functions, and the code blocks run on the first hardware. The dimensions of the input matrices that can be processed by multiple computing units are different. For example, the dimension of the input matrix that can be processed by some computing units is less than 256, the dimension of the input matrix that can be processed by some computing units is greater than or equal to 256 and less than 1024, and the dimension of the input matrix that can be processed by some computing units is greater than or equal to 1024. Which computing unit is selected specifically needs to be determined according to the size of the input matrix.

[0057] The input matrix corresponding to the above-mentioned pending user requests is composed of the context information corresponding to the pending user requests. The context information corresponding to each user request includes all user requests input by the user and all reply information generated for it. Each row of the input matrix is ​​a token of the context information corresponding to a user request, and the number of columns of the input matrix is ​​determined according to the number of tokens of the longest context information; the number of rows of the input matrix is ​​the number of pending user requests in this batch.

[0058] In order to reduce the computational overhead, the pre-filling instance adopts a method of reducing the precision of parameters through a quantization algorithm. In an optional embodiment of the present invention, the received user request is pre-processed by the pre-filling instance to obtain a pre-processing result, which also includes: converting the parameters corresponding to the context information and the model parameters required for pre-processing the user request into the first target parameter and the second target parameter through the pre-filling instance, and using the first target parameter and the second target parameter for calculation to obtain the pre-processing result; the precision of the first target parameter and the second target parameter are respectively lower than the precision of the parameters corresponding to the context information and the model parameters required for pre-processing the user request. Specifically, high-precision model parameters and intermediate activation values ​​are converted into low-precision parameters, such as quantizing 16-bit floating point numbers into 8-bit integers through the Smoothquant W8A8 quantization algorithm, so as to improve the throughput of the pre-filling stage and thus accelerate the calculation process.

[0059] Step S502: Generate reply information for the user request based on the preprocessing result by decoding the instance.

[0060] The decoding instance is an I / O intensive task, and the optimization focus is to reduce the amount of data accessed. This stage adopts a pipeline parallel strategy. In an optional embodiment of the present invention, the second hardware includes multiple second processors, and the total computing power of the first processor is higher than the total computing power of the multiple second processors. The decoding instance generates reply information for the user request according to the preprocessing results, including: for multiple batches of user requests: the decoding instance calculates one batch of user requests through the current second processor, obtains sub-reply information, sends the sub-reply information to the next second processor, and calculates the next batch of user requests through the current second processor. Specifically, the model is split into different second processors by layer, and each processor calculates one or several layers of the model. Multiple batches of user requests are constructed into a pipeline, the current processor calculates the user requests of the current batch, and passes the calculation results to the next processor, which then calculates the user requests of the current batch, and the current processor starts to calculate the user requests of the next batch. This pipeline strategy prevents each processor from being idle and waiting, which can improve service throughput.

[0061] The decoding instance is an I / O intensive task, and the optimization focus is to reduce the amount of data accessed. In this embodiment, the decoding instance uses a small-shape kernel to generate reply information. In an optional embodiment of the present invention, the decoding instance generates reply information for the user request according to the preprocessing result, including: the decoding instance calculates the matrix corresponding to the preprocessing result through the second computing unit, and generates reply information for the user request according to the calculation result; the second computing unit is generated by a code block running on the second hardware; the dimension of the matrix that can be processed by the second unit is smaller than the dimension of the matrices that can be processed by multiple first computing units. The input of the decoding stage is only a token generated in the previous iteration, that is, in the previous iteration, one user request only generates one token, then the dimension of each matrix in the model is the number of user requests to be processed in this batch, usually in the hundreds, so the dimension of the matrix that can be processed by the computing unit used in the decoding stage is relatively small.

[0062] In order to reduce storage space, the decoding instance stores the model parameters after reducing the precision through a quantization algorithm, such as quantizing the model parameters from 16-bit floating point numbers to 4-bit integers. Since the scale of the model parameters is relatively large, the storage space required for storing the model parameters is greatly reduced, and the communication overhead is also greatly saved when reading and writing the model parameters. In addition, when the decoding instance performs matrix multiplication to generate reply information, high-precision data is still required. Therefore, when performing matrix multiplication, the model parameters of the precision 4-bit integers need to be dequantized first, such as dequantizing the model parameters of the 4-bit integers to 16-bit floating point numbers. Therefore, in an optional embodiment of the present invention, the reply information for the user request is generated according to the preprocessing result by the decoding instance, including: the decoding instance performs precision conversion on the model parameters obtained for generating the reply information to obtain the converted model parameters, and the precision of the converted model parameters is higher than the precision of the model parameters before the conversion; the reply information is generated according to the converted model parameters, context information, user request and preprocessing results. Based on this embodiment, the decoding instance achieves the purpose of reducing the amount of memory access data.

[0063] The interaction between the pre-filled instance and the decoding instance is the KV cache of the input context, which is generated by the pre-filled instance, consumed by the decoding instance, and participates in the subsequent token generation process. After the pre-filled instance and the decoding instance are split, they do not share the same storage space, so the KV Cache needs to be transmitted through communication. In order to hide the communication overhead as much as possible, in an optional embodiment of the present invention, the pre-processing task of the current pre-processing layer in the large language model is executed by the pre-filled instance, and the current pre-processing result corresponding to the current pre-processing layer is obtained, and the current pre-processing result is sent to the current decoding layer corresponding to the current pre-processing layer in the large language model, and the current pre-processing result is sent to the next pre-processing layer; the current reply information is generated according to the current pre-processing result by the decoding instance, and the current reply information is sent to the next decoding layer.

[0064] Specifically, a layer-by-layer transmission strategy is adopted, with the model layer as the smallest unit. After each layer completes the pre-filled calculation, the KV cache of the current layer is immediately started to be transmitted, which can overlap with the calculation of the next layer; the KV Cache of the current layer is transmitted to the decoding instance, and the decoding instance can start the calculation of the current layer, which can also overlap with the KV Cache transmission of the subsequent layer. Figure 6 As shown in FIG. 1 , the KV cache communication transmission of layer-0 can be overlapped with the pre-filling calculation of layer-1. In this way, the KV cache transmission communication overhead caused by the split can be hidden as much as possible.

[0065] In a specific embodiment of the present invention, in order to improve the response speed of the large language model system, different optimization strategies are adopted for the pre-filled instance and the decoded instance. Figure 7As shown in the figure, the SmoothquantW8A8 quantization strategy, tensor parallel strategy, and different large-shape kernels are used for the pre-filling instance; the AWQW4A16 quantization strategy, pipeline parallel strategy, and small-shape kernels are used for the decoding instance, thereby achieving real-time targeted optimization measures for the different load characteristics of the pre-filling instance and the decoding instance, so that the response performance of the pre-filling instance and the decoding instance can be greatly improved. This is also the beneficial effect of splitting the pre-filling instance and the decoding instance.

[0066] The request response method based on the large language model system of the embodiment of the present invention preprocesses the received user request through the pre-filling instance to obtain the pre-processing result, and sends the pre-processing result to the decoding instance; the decoding instance generates reply information for the user request according to the pre-processing result, wherein the large language model system includes a pre-filling instance and a decoding instance, the pre-filling instance and the decoding instance are respectively deployed on the first hardware and the second hardware, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filling instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor. Thus, by deploying the pre-filling instance and the decoding instance on different hardware, and adjusting the resources of the first hardware and the second hardware according to the load characteristics of the pre-filling instance and the decoding instance, the flexible configuration of hardware resources and the full utilization of hardware resources are achieved, and the hardware cost is reduced.

[0067] Figure 8 An exemplary system architecture 800 is shown to which a large language model system or a request response method based on a large language model system according to an embodiment of the present invention can be applied.

[0068] like Figure 8 As shown, the system architecture 800 may include a first server 801 for deploying a pre-populated instance, networks 802 and 803, a second server 804 for deploying a decoding instance, and user terminals 805, 806, and 807. The networks 802 and 803 are used to provide a medium for a communication link between the first server 801, the second server 804, and the user terminals 805, 806, and 807. The networks 802 and 803 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0069] The first server 801 and the second server 804 interact with the user terminals 805, 806, and 807 through the networks 802 and 803 to receive or send messages, etc. The first server 801 receives the user request input by the user terminals 805, 806, and 807, pre-processes the user request, obtains the pre-processing result, and sends it to the second server 804. The second server 804 receives the pre-processing result from the first server 801, generates reply information according to the pre-processing result, and sends the reply information to the user terminals 805, 806, and 807.

[0070] It should be noted that the request response method provided in the embodiment of the present invention is generally performed jointly by the first server 801 and the second server 804 .

[0071] It should be understood that Figure 8 The number of the first server, network, second server and user terminal in the embodiment is only for illustration. According to the implementation requirements, there may be any number of the first server, network, second server and user terminal.

[0072] Reference below Fig. 9 , which shows a schematic diagram of the structure of a computer system 900 of an electronic device suitable for implementing an embodiment of the present invention. Fig. 9 The first server shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0073] like Fig. 9 As shown, the computer system 900 includes a central processing unit (CPU) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage part 908 into a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the computer system 900 are also stored. The CPU 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0074] The following components are connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 99 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 910 as needed, so that a computer program read therefrom is installed into the storage section 908 as needed.

[0075] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909, and / or installed from the removable medium 911. When the computer program is executed by the central processing unit (CPU) 901, the above-mentioned functions defined in the system of the present invention are executed.

[0076] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0077] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0078] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be set in a processor, for example, it may be described as: a processor includes a pre-filling instance and a decoding instance. The names of these modules do not constitute a limitation on the modules themselves in some cases, for example, the decoding instance may also be described as "an instance of generating reply information for a user request based on a pre-processing result".

[0079] As another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes: the large language model system includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are respectively deployed on a first hardware and a second hardware, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor; the method includes: pre-processing the received user request through the pre-filled instance to obtain a pre-processing result, and sending the pre-processing result to the decoding instance; through the decoding instance, according to the pre-processing result, generating a reply message for the user request.

[0080] According to the technical solution of the embodiment of the present invention, the large language model system includes a pre-filling instance and a decoding instance, wherein the pre-filling instance and the decoding instance are respectively deployed on the first hardware and the second hardware, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processor are determined according to the load characteristics of the pre-filling instance, and the computing power of the second processor and the total number of the second processor are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor; the method includes: pre-processing the received user request through the pre-filling instance to obtain a pre-processing result, and sending the pre-processing result to the decoding instance; through the decoding instance, according to the pre-processing result, generating a reply message for the user request. Thus, by deploying the pre-filling instance and the decoding instance on different hardware, and adjusting the resources of the first hardware and the second hardware according to the load characteristics of the pre-filling instance and the decoding instance, the flexible configuration of hardware resources and the full utilization of hardware resources are achieved, and the hardware cost is reduced.

[0081] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A large language model system, characterized in that: include: Prefilled instances and decoded instances, where The pre-filling instance and the decoding instance are deployed on a first hardware and a second hardware respectively; The first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor.

2. The system according to claim 1, characterized in that The computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filling instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, using the following method: Predicting, according to the load characteristics of the pre-filling instance, a first computing resource required to run the pre-filling instance, and predicting, according to the load characteristics of the decoding instance, a second computing resource required to run the decoding instance, wherein the first computing resource is greater than the second computing resource; At least one of the first processors and at least one of the second processors are selected according to the computing power of each processor in the processor resource pool, so that the total computing power of the first processor is higher than that of the first computing resource, and the total computing power of the second processor is higher than that of the second computing resource.

3. The system according to claim 1, characterized in that At least one of said first processors comprises an image processing unit, a neural network processing unit and / or a tensor processing unit; At least one of the second processors comprises an image processing unit, a neural network processing unit and / or a tensor processing unit.

4. The system according to claim 1, characterized in that In the case where there are multiple first processors, the computing powers of the multiple first processors are not completely consistent, and / or the types of the multiple first processors are not completely consistent; When there are multiple second processors, the computing powers of the multiple second processors are not completely consistent, and / or the types of the multiple second processors are not completely consistent.

5. A request response method based on a large language model system, characterized in that: The large language model system includes a pre-filled instance and a decoding instance, wherein the pre-filled instance and the decoding instance are deployed on first hardware and second hardware respectively, the first hardware includes at least one first processor, and the second hardware includes at least one second processor; the computing power of the first processor and the total number of the first processors are determined according to the load characteristics of the pre-filled instance, and the computing power of the second processor and the total number of the second processors are determined according to the load characteristics of the decoding instance, wherein the total computing power of the first processor is higher than the total computing power of the second processor; the method includes: Preprocessing the received user request through the pre-filling instance to obtain a preprocessing result, and sending the preprocessing result to the decoding instance; Through the decoding instance, reply information for the user request is generated according to the preprocessing result.

6. The method according to claim 5, characterized in that The first hardware includes at least one first processor, and the preprocessing of the received user request by the prefilled instance to obtain a preprocessing result includes: Splitting the preprocessing task into at least one subtask through the prefilled instance, and correspondingly allocating the subtask to at least one first processor, so that at least one subtask is correspondingly executed by at least one first processor to obtain at least one calculation result output by at least one first processor; At least one of the calculation results is aggregated to obtain the preprocessing result.

7. The method according to claim 6, characterized in that The second hardware includes a plurality of second processors, the total computing power of the first processor is higher than the total computing power of the plurality of second processors, and the generating, through the decoding instance and according to the preprocessing result, reply information for the user request includes: For multiple batches of user requests: The decoding instance calculates one of the batches of user requests through the current second processor to obtain sub-reply information, sends the sub-reply information to the next second processor, and calculates the next batch of user requests through the current second processor.

8. The method according to claim 5, characterized in that The preprocessing of the received user request by the prefilled instance to obtain a preprocessing result includes: Determine the number of user requests to be processed in this batch and the length of context information corresponding to each user request through the pre-filled instance; the context information includes each user request, a request previously input by the user and its corresponding reply information; An input matrix is ​​constructed using the context information corresponding to the pending user request; each row of the input matrix corresponds to the context information of a pending user request; the number of rows of the input matrix is ​​the number of pending user requests in this batch, and the number of columns of the input matrix is ​​the maximum length of the context information corresponding to the pending request; Determine a first target computing unit from a plurality of first computing units according to the dimension of the input matrix; the plurality of first computing units are generated by a code block running on the first hardware; the plurality of first computing units can be used to calculate input matrices of different dimensions; The input matrix is ​​calculated using the first target calculation unit to obtain the preprocessing result.

9. The method according to claim 8, characterized in that The step of generating reply information for the user request according to the preprocessing result through the decoding instance includes: The decoding instance calculates the matrix corresponding to the preprocessing result through a second computing unit, and generates reply information for the user request according to the calculation result; the second computing unit is generated by a code block running on the second hardware; the dimension of the matrix that can be processed by the second unit is smaller than the dimension of the matrix that can be processed by multiple first computing units.

10. The method according to claim 5, characterized in that The preprocessing of the received user request by the prefilled instance to obtain a preprocessing result also includes: The parameters corresponding to the context information and the model parameters required for preprocessing the user request are converted into first target parameters and second target parameters through the pre-filled instance, and the first target parameters and the second target parameters are used for calculation to obtain a preprocessing result; the precision of the first target parameter and the second target parameter is lower than the precision of the parameters corresponding to the context information and the model parameters required for preprocessing the user request, respectively.

11. The method according to claim 5, characterized in that Also includes: Execute the preprocessing task of the current preprocessing layer in the large language model through the prefilled instance, obtain the current preprocessing result corresponding to the current preprocessing layer, send the current preprocessing result to the current decoding layer corresponding to the current preprocessing layer in the large language model, and send the current preprocessing result to the next preprocessing layer; The current reply information is generated according to the current preprocessing result through the decoding instance, and the current reply information is sent to the next decoding layer.

12. The method according to claim 8, characterized in that The step of generating reply information for the user request according to the preprocessing result through the decoding instance includes: The decoding instance performs precision conversion on the model parameters obtained for generating reply information to obtain converted model parameters, wherein the precision of the converted model parameters is higher than the precision of the model parameters before conversion; and generates reply information according to the converted model parameters, context information, user request and preprocessing results.

13. An electronic device for requesting a response, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 5 to 12.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 5 to 12 is implemented.

15. A computer program product comprising a computer program, characterized in that When the program is executed by a processor, the method according to any one of claims 5 to 12 is implemented.

Images