A fast reasoning method and system for large-scale generative language models supporting high concurrency

Abstract

The present invention discloses a method and system for fast reasoning of a large-scale generative language model supporting high concurrency, the method comprising: obtaining the attention median value of the preceding text of the i-th step and the predicted text of the i-th step, and saving the attention median value of the preceding text of the i-th step; obtaining the attention median value corresponding to the predicted text of the i-th step, obtaining the attention output result corresponding to the preceding text of the i+1th step according to the attention median value corresponding to the predicted text of the i-th step and the attention median value corresponding to the preceding text of the i-th step; generating the predicted text of the i+1th step according to the attention output result corresponding to the preceding text of the i+1th step. The present invention can accelerate the reasoning speed of a large-scale generative language model and shorten the time for users to wait for model output.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a method and system for fast reasoning of a large-scale generative language model supporting high concurrency. Background Art

[0002] More and more language models with huge numbers of parameters are emerging. When deploying such large-scale generative language model online inference services, the model size limits the inference speed. The inference speed is slow, and users often have to wait a long time to get the model's inference results.

[0003] Therefore, the prior art still needs to be improved and enhanced. Summary of the invention

[0004] In view of the above-mentioned defects of the prior art, the present invention provides a method and system for fast reasoning of a large-scale generative language model supporting high concurrency, aiming to solve the problem of slow reasoning speed of large-scale generative language models in the prior art.

[0005] In order to solve the above technical problems, the technical solution adopted by the present invention is as follows:

[0006] A first aspect of the present invention provides a method for fast reasoning of a large-scale generative language model supporting high concurrency, the method comprising:

[0007] Get the median attention value of the previous text of step i and the predicted text of step i, and save the median attention value of the previous text of step i;

[0008] Obtain the middle attention value corresponding to the predicted text of step i, and obtain the attention output result corresponding to the preceding text of step i+1 according to the middle attention value corresponding to the predicted text of step i and the middle attention value corresponding to the preceding text of step i;

[0009] Generate the predicted text of step i+1 according to the attention output result corresponding to the previous text of step i+1;

[0010] Wherein, i is a positive integer.

[0011] The fast reasoning method for a large-scale generative language model supporting high concurrency, wherein the intermediate attention value of the preceding text of the i-th step includes the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step.

[0012] The fast reasoning method for a large-scale generative language model supporting high concurrency, wherein the step of saving the intermediate attention value of the preceding text of the i-th step includes:

[0013] Obtaining the first intermediate matrix and the second intermediate matrix of the i-th step according to the K and V vectors of the preceding text of the i-th step, wherein the sizes of the first intermediate matrix and the second intermediate matrix are both preset sizes, and the preset size is M*N, where M is the dimension of each vector in the matrix, and N is the number of vectors;

[0014] The first m vectors in the first intermediate matrix and the second intermediate matrix are the K and V vectors corresponding to the words in the preceding text of the i-th step, respectively, and the vectors after the m-th vector are 0;

[0015] The first intermediate matrix and the second intermediate matrix of the i-th step are stored in the video memory.

[0016] The fast reasoning method for a large-scale generative language model supporting high concurrency, wherein the step of obtaining the attention output result corresponding to the preceding text of the i+1th step according to the attention intermediate value corresponding to the predicted text of the i-th step and the attention intermediate value corresponding to the preceding text of the i-th step, comprises:

[0017] Calculate the Q, K, and V vectors corresponding to the predicted text of the i-th step, and store the K and V vectors corresponding to the predicted text of the i-th step as a third intermediate matrix and a fourth intermediate matrix, respectively. The sizes of the third intermediate matrix and the fourth intermediate matrix are the preset sizes. The m+1th vector of the third intermediate matrix is ​​the K vector corresponding to the predicted text of the i-th step, the m+1th vector of the fourth intermediate matrix is ​​the V vector corresponding to the predicted text of the i-th step, and the remaining vectors are 0;

[0018] Update the first intermediate matrix and the second intermediate matrix of the i-th step according to the third intermediate matrix and the fourth intermediate matrix to obtain the first intermediate matrix and the second intermediate matrix of the i+1-th step;

[0019] The attention output result corresponding to the preceding text of the i+1th step is calculated according to the first intermediate matrix and the second intermediate matrix of the i+1th step and the Q vector corresponding to the predicted text of the i-th step.

[0020] The fast reasoning method for a large-scale generative language model supporting high concurrency, wherein the updating of the first intermediate matrix and the second intermediate matrix of the i-th step according to the third intermediate matrix and the fourth intermediate matrix to obtain the first intermediate matrix and the second intermediate matrix of the i+1-th step includes:

[0021] An addition operation is performed on the third intermediate matrix and the first intermediate matrix of the i-th step to obtain the first intermediate matrix of the i+1-th step, and an addition operation is performed on the fourth intermediate matrix and the second intermediate matrix of the i-th step to obtain the second intermediate matrix of the i+1-th step.

[0022] The fast reasoning method for a large-scale generative language model supporting high concurrency, wherein the step of obtaining the intermediate attention value of the preceding text of the i-th step includes:

[0023] When i=1, the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step are calculated according to the preset first weight matrix, second weight matrix, and third weight matrix.

[0024] The fast reasoning method for a large-scale generative language model supporting high concurrency, wherein the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step are calculated according to the preset first weight matrix, the second weight matrix, and the third weight matrix, including:

[0025] Get the embedding vector of each word in the preceding text of step i;

[0026] The embedding vector of each word is multiplied by the first weight matrix, the second weight matrix and the third weight matrix respectively to obtain the Q, K, and V vectors corresponding to each word.

[0027] The fast reasoning method for a large-scale generative language model supporting high concurrency, wherein after generating the predicted text of the i+1th step according to the attention output result corresponding to the preceding text of the i+1th step, the method further includes:

[0028] When the predicted text of step i+1 includes a preset terminator, or the total length of the predicted text of step i+1 and the preceding text of step i+1 reaches a preset threshold, the reasoning is stopped and the reasoning result is output, which includes the preceding text of step i+1 and the predicted text of step i+1, otherwise the reasoning of step i+2 is performed.

[0029] According to a second aspect of the present invention, a terminal is provided, comprising a processor and a computer-readable storage medium communicatively connected to the processor, wherein the computer-readable storage medium is suitable for storing a plurality of instructions, and the processor is suitable for calling the instructions in the computer-readable storage medium to execute the steps of implementing any one of the above-mentioned methods for fast reasoning of a large-scale generative language model supporting high concurrency.

[0030] A third aspect of the present invention provides a large-scale generative language model fast reasoning system supporting high concurrency, the system comprising: the system comprising: a front-end device and at least one terminal provided by the second aspect of the present invention;

[0031] The front-end device is used to receive an inference request, and send a preamble text corresponding to the inference request to a target terminal among at least one of the terminals according to the load of each of the terminals;

[0032] The terminal is used to perform reasoning according to the preceding text corresponding to the reasoning request, and return the reasoning result to the front-end device.

[0033] The large-scale generative language model fast reasoning system supporting high concurrency, wherein the front-end device includes a request receiving module and a queue management module;

[0034] The request receiving module is used to receive the reasoning request and pre-process the text in the reasoning request to obtain the preceding text corresponding to the reasoning request;

[0035] The queue management module is used to determine whether the current task queue has reached a preset length. If so, a prompt message is returned to the sender of the inference request. Otherwise, the preceding text corresponding to the inference request is sent to the target terminal with the lowest load according to the load of each terminal, and the task queue length is increased by one.

[0036] The large-scale generative language model fast reasoning system supporting high concurrency, wherein the front-end device also includes a post-processing module;

[0037] The post-processing module is used to receive the inference result output by the terminal, post-process the inference result and then send it to the sender of the inference request;

[0038] The queue management module is further configured to reduce the length of the task queue by one after the post-processing module receives the inference result of the terminal.

[0039] A fourth aspect of the present invention provides a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to implement the steps of any of the above-mentioned methods for fast reasoning of large-scale generative language models supporting high concurrency.

[0040] Compared with the prior art, the present invention provides a fast reasoning method and system for a large-scale generative language model that supports high concurrency. In the method provided by the present invention, when calculating the attention output result of the i+1 step, the attention intermediate value of the i-th step is reused, and only the attention intermediate value of the predicted text generated in the i-th step needs to be newly calculated, and the attention intermediate value of the preceding text of the i-th step does not need to be calculated, thereby reducing the parameter amount of the attention calculation in the i+1 step, being able to accelerate the reasoning speed of the large-scale generative language model and shortening the time users wait for model output. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A flowchart of an embodiment of a method for fast reasoning of a large-scale generative language model supporting high concurrency provided by the present invention;

[0042] Figure 2 An improved schematic diagram of an embodiment of a method for fast reasoning of a large-scale generative language model supporting high concurrency provided by the present invention;

[0043] Figure 3 A schematic diagram of the effect of an embodiment of the method for fast reasoning of a large-scale generative language model supporting high concurrency provided by the present invention;

[0044] Figure 4 A schematic diagram of the principles of an embodiment of a terminal provided by the present invention;

[0045] Figure 5 A schematic diagram of an embodiment of a large-scale generative language model fast reasoning system supporting high concurrency provided by the present invention;

[0046] Figure 6 A schematic diagram of the processing flow of a reasoning task proposed by a user in an embodiment of the large-scale generative language model fast reasoning system supporting high concurrency provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0048] The large-scale generative language model fast reasoning method supporting high concurrency provided by the present invention can be applied to a terminal with computing capability. The terminal can execute the large-scale generative language model fast reasoning method supporting high concurrency provided by the present invention to process the text to be processed. The terminal can be but is not limited to various computers, mobile terminals, smart home appliances, wearable devices, etc.

[0049] Embodiment 1

[0050] like Figure 1 As shown, in one embodiment of the method for fast reasoning of a large-scale generative language model supporting high concurrency, the steps include:

[0051] S100, obtaining the median attention value of the preceding text of the i-th step and the predicted text of the i-th step, and saving the median attention value of the preceding text of the i-th step.

[0052] Specifically, the attention intermediate value referred to in this embodiment refers to the Q, K, and V vectors in the attention mechanism. The attention intermediate value of the pre-order text of the i-th step includes the Q, K, and V vectors corresponding to each word in the pre-order text of the i-th step. In the generative language model, most of them use the attention mechanism for reasoning. In each step of the generative language task, a text is input, and then the next word of the text is predicted, and then a new text is generated as the input of the next step, and then the next word is predicted. In this embodiment, the text input in each step is called the pre-order text of the step, and the word predicted based on the input text is called the predicted text of the step. That is to say, the pre-order text of the i-th step and the predicted text of the i-th step are combined to form the pre-order text of the i+1 step. Specifically, in this embodiment, the K and V vectors corresponding to each word in the pre-order text of the i-th step are saved for the calculation of the i+1 step, and i is a positive integer.

[0053] When i>1, the median attention value of the preceding text of the i-th step can be calculated based on the median attention value of the preceding text and the predicted text of the i-1th step. For the specific process, please refer to the description of the process of obtaining the median attention value of the preceding text of the i+1th step in the following text. When i=1, the median attention value of the preceding text of the i-th step is calculated by the word vector of each word of the preceding text of the i-th step. That is, obtaining the median attention value of the preceding text of the i-th step includes:

[0054] When i=1, the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step are calculated according to the preset first weight matrix, second weight matrix, and third weight matrix.

[0055] The step of calculating the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step according to the preset first weight matrix, the second weight matrix, and the third weight matrix includes:

[0056] Get the embedding vector of each word in the preceding text of step i;

[0057] The embedding vector of each word is multiplied by the first weight matrix, the second weight matrix and the third weight matrix respectively to obtain the Q, K, and V vectors corresponding to each word.

[0058] The storing of the intermediate attention value of the preceding text of step i includes:

[0059] Obtaining the first intermediate matrix and the second intermediate matrix of the i-th step according to the K and V vectors of the preceding text of the i-th step, wherein the sizes of the first intermediate matrix and the second intermediate matrix are both preset sizes, and the preset size is M*N, where M is the dimension of each vector in the matrix, and N is the number of vectors;

[0060] The first m vectors in the first intermediate matrix and the second intermediate matrix are the K and V vectors corresponding to the words in the preceding text of the i-th step, respectively, and the vectors after the m-th vector are 0;

[0061] The first intermediate matrix and the second intermediate matrix of the i-th step are stored in the video memory.

[0062] In this embodiment, the intermediate attention value corresponding to the preceding text of the i-th step is reused for calculating the reasoning of the i+1-th step. In the reasoning of the i+1-th step, only the intermediate attention value corresponding to the predicted text of the i-th step needs to be calculated.

[0063] In this embodiment, the K and V vectors corresponding to all words of a sentence in each step of reasoning are stored in a corresponding matrix of preset size, so that the model does not need to define calculation methods for different numbers of K and V vectors corresponding to different preceding texts, but only needs to define a calculation method for a matrix of fixed size. In this embodiment, the matrices corresponding to the K vector and the V vector are respectively the first intermediate matrix and the second intermediate matrix, and the sizes of the first intermediate matrix and the second intermediate matrix are both preset sizes, and the preset size is M*N, where M is the dimension of each vector in the matrix, and N is the number of vectors. The K and V vectors of the preceding text of the i-th step are stored in the first intermediate matrix and the second intermediate matrix of the i-th step respectively. Specifically, if there are m words in the preceding text of the i-th step, then the first m vectors in the corresponding first intermediate matrix of the i-th step are the K vectors corresponding to each word in the preceding text of the i-th step, and the first m vectors in the second intermediate matrix of the i-th step are the V vectors corresponding to each word in the preceding text of the i-th step, and the other vectors of the first intermediate matrix and the second intermediate matrix are set to 0. The first intermediate matrix and the second intermediate matrix of the i-th step are saved in the video memory, so that the data synchronization time of host2device can be avoided. In the i+1th step, the first intermediate matrix and the second intermediate matrix can be directly read to obtain the K and V vectors required for calculating the attention result, saving time.

[0064] Please refer again Figure 1 , the method provided in this embodiment also includes the steps of:

[0065] S200. Obtain the median attention value corresponding to the predicted text of the i-th step, and obtain the attention output result corresponding to the predecessor text of the i+1-th step based on the median attention value corresponding to the predicted text of the i-th step and the median attention value corresponding to the predecessor text of the i-th step.

[0066] like Figure 2As shown, in the existing generation model, after the prediction text of step i is generated based on the previous text of step i, the previous text of step i and the prediction text of step i are combined and then input into the model to generate the prediction text of step i+1, for example, Figure 2 In the process, "Pangu is a" is input into the model, attention is calculated, and the predicted word "language" is generated. Then "Pangu is a language" is input into the model to calculate attention and generate the predicted word "model". In this process, the K and V vectors of the preceding text of the i-th step are repeatedly calculated. In the present embodiment, when calculating the attention of the i+1-th step, the K and V vectors of the preceding text of the i-th step are reused. For example, "Pangu is a" is input into the model to calculate attention and generate the predicted word "language". Then "language" is input into the model to calculate the intermediate vector of attention, and the intermediate vector of attention corresponding to "Pangu is a" is input into it, and the predicted word "model" is calculated together.

[0067] Specifically, obtaining the attention output result corresponding to the preceding text of the i+1th step according to the attention intermediate value corresponding to the predicted text of the i-th step and the attention intermediate value corresponding to the preceding text of the i-th step includes:

[0068] Calculate the Q, K, and V vectors corresponding to the predicted text of the i-th step, store the K and V vectors corresponding to the predicted text of the i-th step as a third intermediate matrix and a fourth intermediate matrix, respectively, the sizes of the third intermediate matrix and the fourth intermediate matrix are the preset sizes, the m+1th vector of the third intermediate matrix is ​​the K vector corresponding to the predicted text of the i-th step, the m+1th vector of the fourth intermediate matrix is ​​the V vector corresponding to the predicted text of the i-th step, and the remaining vectors are set to 0;

[0069] Update the first intermediate matrix and the second intermediate matrix of the i-th step according to the third intermediate matrix and the fourth intermediate matrix to obtain the first intermediate matrix and the second intermediate matrix of the i+1-th step;

[0070] The attention output result corresponding to the preceding text of the i+1th step is calculated according to the first intermediate matrix and the second intermediate matrix of the i+1th step and the Q vector corresponding to the predicted text of the i-th step.

[0071] In the autoregressive generative language model, a word is generated in each step, that is, the predicted text of the i-th step is a word, the predicted text of the i-th step is input, and after obtaining the Q, K, and V vectors corresponding to the predicted text of the i-th step, the Q vector corresponding to the predicted text of the i-th step is kept unchanged, and the K and V vectors of the predicted text of the i-th step and the K and V vectors of the preceding text of the i-th step are combined to obtain the intermediate value of attention of the i+1-th step. Specifically, first, the K and V vectors calculated for the predicted text of the i-th step are saved as the third intermediate matrix and the fourth intermediate matrix, and the sizes of the third intermediate matrix and the fourth intermediate matrix are both the preset sizes, that is, M*N. The m+1th vector of the third intermediate matrix is ​​the K vector corresponding to the predicted text of the i-th step, the m+1th vector of the fourth intermediate matrix is ​​the V vector corresponding to the predicted text of the i-th step, and the remaining vectors are 0. The third intermediate matrix and the first intermediate matrix of the i-th step are added to update the first intermediate matrix of the i-th step to obtain the first intermediate matrix of the i+1-th step. The fourth intermediate matrix and the second intermediate matrix of the i-th step are added to update the second intermediate matrix of the i-th step to obtain the second intermediate matrix of the i+1-th step.

[0072] The attention output result corresponding to the i+1th preceding text is calculated according to the first intermediate matrix and the second intermediate matrix of the i+1th step and the Q vector corresponding to the predicted text of the ith step, and the preceding text of the i+1th step is a combination of the preceding text of the ith step and the predicted text of the ith step. The formula of the attention output result can be expressed as:

[0073] S300, generating the predicted text of the i+1th step according to the attention output result corresponding to the previous text of the i+1th step.

[0074] After generating the predicted text of step i+1 according to the attention output result corresponding to the preceding text of step i+1, the method provided in this embodiment further includes the steps of:

[0075] When the predicted text of step i+1 includes a preset terminator, or the total length of the predicted text of step i+1 and the preceding text of step i+1 reaches a preset threshold, the reasoning is stopped and the reasoning result is output, which includes the preceding text of step i+1 and the predicted text of step i+1, otherwise the reasoning of step i+2 is performed.

[0076] The preset terminator may be a period, a space, or other preset symbols, etc. The inference result is a combination of the preceding text and the predicted text in the last step.

[0077] The method provided in this embodiment is used to conduct experiments on an existing large-scale generative language model (Pengcheng Pangu model). The experimental results are as follows: Figure 3 As shown, it is found that the single word generation delay of the model with 2.6 billion parameters using the fast reasoning method provided in this embodiment is about 30ms, and the original reasoning delay is about 120ms, while the single word generation delay of the fast reasoning method provided in this embodiment on the model with 13 billion parameters is about 50ms, and the original reasoning delay is about 285ms. The fast reasoning method provided in this embodiment can shorten the field input to the language model from a fixed sequence length to one word through the mechanism of intermediate state reuse without losing the accuracy of the model, effectively reducing the number of adoptions in the model reasoning process, and significantly improving the reasoning speed of large-scale generative language models. In addition, the method provided in this embodiment is a language model for various decoder architectures, which is universal, suitable for optimizing the reasoning engine of large-scale online reasoning services, improving the user experience of reasoning services, and has good scalability.

[0078] It should be understood that, although the various steps in the flowcharts given in the accompanying drawings of the present specification are displayed in sequence according to the indications of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0079] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0080] Embodiment 2

[0081] Based on the method provided in the above embodiment, the present invention also provides a fast reasoning system for large-scale generative language models that supports high concurrency, such as Figure 5 As shown, the system includes: a front end and at least one terminal, the terminal includes: a processor, a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium is suitable for storing multiple instructions, and the processor is suitable for calling the instructions in the computer-readable storage medium to execute the steps of the fast reasoning method for large-scale generative language models supporting high concurrency described in the above-mentioned embodiment 1.

[0082] The front-end device is used to receive an inference request, and send a preamble text corresponding to the inference request to a target terminal among at least one of the terminals according to the load of each of the terminals;

[0083] The terminal is used to perform reasoning according to the preceding text corresponding to the reasoning request, and return the reasoning result to the front-end device.

[0084] Specifically, Figure 6As shown, the front-end device can receive requests issued by each user terminal (client) in the online user pool. The user sends an inference request to the front-end device through the user terminal. The front-end device includes a request receiving module and a queue management module. The request receiving module is used to receive the inference request and pre-process the text in the inference request to obtain the preceding text corresponding to the inference request. The pre-processing can be to filter sensitive words from the text in the user's inference request. The request receiving module sends the preceding text corresponding to the inference request to the queue management module. The queue management module is used to determine whether the current task queue has reached a preset length. If so, a prompt message is returned to the issuing end of the inference request. Otherwise, the preceding text corresponding to the inference request is sent to the target terminal with the lowest load according to the load request of each terminal, and the length of the task queue is increased by one. When the current task queue reaches the preset length, it means that the current inference request to be processed has reached a predetermined processing upper limit. At this time, a prompt message can be returned, such as returning an [abnormal response] string, to the issuing end of the inference request. When the current task queue does not reach the preset length, the inference request can be sent to one of the multiple terminals for processing. Specifically, the queue management module obtains the load conditions of each terminal, determines that the terminal with the lowest current load is the target terminal, sends the preamble text corresponding to the reasoning request to the target terminal, and increases the length of the task queue by one. In this way, high-concurrency load balancing configuration can be supported, and the number of simultaneous online users of large-scale model reasoning service systems can be increased. After receiving the request, the target terminal performs reasoning according to the method provided in Embodiment 1, and returns the output reasoning result to the front-end device, which then returns to the issuing end of the reasoning request.

[0085] The front-end device also includes a post-processing module, which is used to receive the inference result output by the terminal, post-process the inference result and send it to the sender of the inference request. The post-processing of the inference result can be filtering sensitive words, deduplication and other operations. After the post-processing module receives the inference result of the terminal, the length of the task queue is reduced by one. Preprocessing the inference request and post-processing the inference result output by the terminal can effectively control the reasonable and compliant input content and model generation results.

[0086] The system provided by the present invention is implemented based on the python-flask framework, and the code is easy to transplant and lightweight.

[0087] Embodiment 3

[0088] Based on the method provided in the above embodiment, the present invention also provides a terminal accordingly, such as Figure 4As shown, the terminal includes a processor 10 and a memory 20. Figure 4 Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0089] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a large-scale generative language model fast reasoning program 30 supporting high concurrency is stored on the memory 20, and the large-scale generative language model fast reasoning program 30 supporting high concurrency can be executed by the processor 10, thereby realizing the large-scale generative language model fast reasoning method supporting high concurrency in the present application.

[0090] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other chip, used to run the program code or process data stored in the memory 20, such as executing the fast reasoning method for large-scale generative language models that supports high concurrency.

[0091] In one embodiment, when the processor 10 executes the high-concurrency large-scale generative language model fast reasoning program 30 in the memory 20, the following steps are implemented:

[0092] Get the median attention value of the previous text of step i and the predicted text of step i, and save the median attention value of the previous text of step i;

[0093] Obtain the middle attention value corresponding to the predicted text of step i, and obtain the attention output result corresponding to the preceding text of step i+1 according to the middle attention value corresponding to the predicted text of step i and the middle attention value corresponding to the preceding text of step i;

[0094] Generate the predicted text of step i+1 according to the attention output result corresponding to the previous text of step i+1;

[0095] Wherein, i is a positive integer.

[0096] Among them, the attention intermediate value of the preceding text of the i-th step includes the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step.

[0097] The step of saving the intermediate attention value of the preceding text of step i includes:

[0098] Obtaining the first intermediate matrix and the second intermediate matrix of the i-th step according to the K and V vectors of the preceding text of the i-th step, wherein the sizes of the first intermediate matrix and the second intermediate matrix are both preset sizes, and the preset size is M*N, where M is the dimension of each vector in the matrix, and N is the number of vectors;

[0099] The first m vectors in the first intermediate matrix and the second intermediate matrix are the K and V vectors corresponding to the words in the preceding text of the i-th step, respectively, and the vectors after the m-th vector are 0;

[0100] The first intermediate matrix and the second intermediate matrix of the i-th step are stored in the video memory.

[0101] The step of obtaining the attention output result corresponding to the preceding text of the i+1th step according to the attention intermediate value corresponding to the predicted text of the i-th step and the attention intermediate value corresponding to the preceding text of the i-th step includes:

[0102] Calculate the Q, K, and V vectors corresponding to the predicted text of the i-th step, and store the K and V vectors corresponding to the predicted text of the i-th step as a third intermediate matrix and a fourth intermediate matrix, respectively. The sizes of the third intermediate matrix and the fourth intermediate matrix are the preset sizes. The m+1th vector of the third intermediate matrix is ​​the K vector corresponding to the predicted text of the i-th step, the m+1th vector of the fourth intermediate matrix is ​​the V vector corresponding to the predicted text of the i-th step, and the remaining vectors are 0;

[0103] Update the first intermediate matrix and the second intermediate matrix of the i-th step according to the third intermediate matrix and the fourth intermediate matrix to obtain the first intermediate matrix and the second intermediate matrix of the i+1-th step;

[0104] The attention output result corresponding to the preceding text of the i+1th step is calculated according to the first intermediate matrix and the second intermediate matrix of the i+1th step and the Q vector corresponding to the predicted text of the i-th step.

[0105] The updating of the first intermediate matrix and the second intermediate matrix of the i-th step according to the third intermediate matrix and the fourth intermediate matrix to obtain the first intermediate matrix and the second intermediate matrix of the i+1-th step includes:

[0106] An addition operation is performed on the third intermediate matrix and the first intermediate matrix of the i-th step to obtain the first intermediate matrix of the i+1-th step, and an addition operation is performed on the fourth intermediate matrix and the second intermediate matrix of the i-th step to obtain the second intermediate matrix of the i+1-th step.

[0107] The step of obtaining the intermediate attention value of the preceding text of step i includes:

[0108] When i=1, the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step are calculated according to the preset first weight matrix, second weight matrix, and third weight matrix.

[0109] The step of calculating the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step according to the preset first weight matrix, second weight matrix, and third weight matrix includes:

[0110] Get the embedding vector of each word in the preceding text of step i;

[0111] The embedding vector of each word is multiplied by the first weight matrix, the second weight matrix and the third weight matrix respectively to obtain the Q, K, and V vectors corresponding to each word.

[0112] After generating the predicted text of step i+1 according to the attention output result corresponding to the preceding text of step i+1, the method further includes:

[0113] When the predicted text of step i+1 includes a preset terminator, or the total length of the predicted text of step i+1 and the preceding text of step i+1 reaches a preset threshold, the reasoning is stopped and the reasoning result is output, which includes the preceding text of step i+1 and the predicted text of step i+1, otherwise the reasoning of step i+2 is performed.

[0114] Embodiment 4

[0115] The present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the high-concurrency large-scale generative language model fast reasoning method as described above.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A fast reasoning method for large-scale generative language models supporting high concurrency, characterized in that: The method comprises: Get the median attention value of the previous text of step i and the predicted text of step i, and save the median attention value of the previous text of step i; Obtain the middle attention value corresponding to the predicted text of step i, and obtain the attention output result corresponding to the preceding text of step i+1 according to the middle attention value corresponding to the predicted text of step i and the middle attention value corresponding to the preceding text of step i; When i>1, the attention median value of the preceding text of the i-th step is calculated by the attention median values ​​corresponding to the preceding text of the i-1-th step and the predicted text of the i-1-th step; Generate the predicted text of step i+1 according to the attention output result corresponding to the previous text of step i+1; Wherein, i is a positive integer; The attention intermediate value of the preceding text of the i-th step includes the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step; The storing of the intermediate attention value of the preceding text of step i includes: Obtaining the first intermediate matrix and the second intermediate matrix of the i-th step according to the K and V vectors of the preceding text of the i-th step, wherein the sizes of the first intermediate matrix and the second intermediate matrix are both preset sizes, and the preset size is M*N, where M is the dimension of each vector in the matrix, and N is the number of vectors; The first m vectors in the first intermediate matrix and the second intermediate matrix are the K and V vectors corresponding to the words in the preceding text of the i-th step, respectively, and the vectors after the m-th vector are 0; The first intermediate matrix and the second intermediate matrix of the i-th step are stored in the video memory; The step of obtaining the attention output result corresponding to the preceding text of the i+1th step according to the attention intermediate value corresponding to the predicted text of the ith step and the attention intermediate value corresponding to the preceding text of the ith step includes: Calculate the Q, K, and V vectors corresponding to the predicted text of the i-th step, and store the K and V vectors corresponding to the predicted text of the i-th step as a third intermediate matrix and a fourth intermediate matrix, respectively. The sizes of the third intermediate matrix and the fourth intermediate matrix are the preset sizes. The m+1th vector of the third intermediate matrix is ​​the K vector corresponding to the predicted text of the i-th step, the m+1th vector of the fourth intermediate matrix is ​​the V vector corresponding to the predicted text of the i-th step, and the remaining vectors are 0; Update the first intermediate matrix and the second intermediate matrix of the i-th step according to the third intermediate matrix and the fourth intermediate matrix to obtain the first intermediate matrix and the second intermediate matrix of the i+1-th step; The attention output result corresponding to the preceding text of the i+1th step is calculated according to the first intermediate matrix and the second intermediate matrix of the i+1th step and the Q vector corresponding to the predicted text of the i-th step.

2. The method for fast reasoning of a large-scale generative language model supporting high concurrency according to claim 1, characterized in that: The updating of the first intermediate matrix and the second intermediate matrix of the i-th step according to the third intermediate matrix and the fourth intermediate matrix to obtain the first intermediate matrix and the second intermediate matrix of the i+1-th step includes: An addition operation is performed on the third intermediate matrix and the first intermediate matrix of the i-th step to obtain the first intermediate matrix of the i+1-th step, and an addition operation is performed on the fourth intermediate matrix and the second intermediate matrix of the i-th step to obtain the second intermediate matrix of the i+1-th step.

3. The method for fast reasoning of a large-scale generative language model supporting high concurrency according to claim 1, characterized in that: The step of obtaining the intermediate attention value of the preceding text of step i includes: When i=1, the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step are calculated according to the preset first weight matrix, second weight matrix, and third weight matrix.

4. The method for fast reasoning of a large-scale generative language model supporting high concurrency according to claim 3, characterized in that: The step of calculating the Q, K, and V vectors corresponding to each word in the preceding text of the i-th step according to the preset first weight matrix, the second weight matrix, and the third weight matrix includes: Get the embedding vector of each word in the preceding text of step i; The embedding vector of each word is multiplied by the first weight matrix, the second weight matrix and the third weight matrix respectively to obtain the Q, K, and V vectors corresponding to each word.

5. The method for fast reasoning of a large-scale generative language model supporting high concurrency according to claim 1, characterized in that: After generating the predicted text of step i+1 according to the attention output result corresponding to the preceding text of step i+1, the method further includes: When the predicted text of step i+1 includes a preset terminator, or the total length of the predicted text of step i+1 and the preceding text of step i+1 reaches a preset threshold, the reasoning is stopped and the reasoning result is output, which includes the preceding text of step i+1 and the predicted text of step i+1, otherwise the reasoning of step i+2 is performed.

6. A terminal, characterized in that: The terminal includes: a processor, a computer-readable storage medium communicatively connected to the processor, the computer-readable storage medium being suitable for storing a plurality of instructions, and the processor being suitable for calling the instructions in the computer-readable storage medium to execute the steps of the method for fast reasoning of a large-scale generative language model supporting high concurrency as described in any one of claims 1 to 5 above.

7. A large-scale generative language model fast reasoning system supporting high concurrency, characterized in that: The system comprises: a front-end device and at least one terminal according to claim 6; The front-end device is used to receive an inference request, and send a preamble text corresponding to the inference request to a target terminal among at least one of the terminals according to the load of each of the terminals; The terminal is used to perform reasoning according to the preceding text corresponding to the reasoning request, and return the reasoning result to the front-end device.

8. The large-scale generative language model fast reasoning system supporting high concurrency according to claim 7, characterized in that: The front-end device includes a request receiving module and a queue management module; The request receiving module is used to receive the reasoning request and pre-process the text in the reasoning request to obtain the preceding text corresponding to the reasoning request; The queue management module is used to determine whether the current task queue has reached a preset length. If so, a prompt message is returned to the sender of the inference request. Otherwise, the preceding text corresponding to the inference request is sent to the target terminal with the lowest load according to the load of each terminal, and the task queue length is increased by one.

9. The large-scale generative language model fast reasoning system supporting high concurrency according to claim 8, characterized in that: The front-end device also includes a post-processing module; The post-processing module is used to receive the inference result output by the terminal, post-process the inference result and then send it to the sender of the inference request; The queue management module is further configured to reduce the length of the task queue by one after the post-processing module receives the inference result of the terminal.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the method for fast reasoning of a large-scale generative language model supporting high concurrency as described in any one of claims 1-5.

Images