A large model compilation reasoning method, device and medium

Abstract

The present application discloses a large model compilation reasoning method, device and medium, which relates to the field of electrical digital data processing technology. The method includes: obtaining a pre-compiled large model; adjusting the input token length and data buffer in the pre-filled model and the decoding model to obtain the adjusted large model; for the adjusted large model, constructing the dependency mapping relationship between its corresponding input and output tensors and the data buffer, and compiling the adjusted large model to obtain the compiled large model; receiving the question and answer request sent by the user, converting the question corresponding to the question and answer request into an input token sequence, and inputting the input token sequence into the compiled large model; based on the compiled large model, realizing the reasoning of the input token sequence to obtain the output token sequence; according to the output token sequence, constructing the answer to the question.

Description

Technical Field

[0001] The present application relates to the technical field of electronic digital data processing, and in particular to a large model compilation reasoning method, device and medium. Background Art

[0002] The application of large language models is inseparable from the deployment of models. For popular artificial intelligence computing chips, there are many large model inference frameworks to choose from for the deployment and inference of large models. However, for new artificial intelligence computing chips, if you want to deploy large models through open source large model inference frameworks, you need a lot of adaptation work, which involves rewriting drivers, runtimes, operator libraries, etc. The workload is huge and highly professional, requiring a lot of manpower and material resources.

[0003] For traditional deep neural network models, AI compilers can be used to compile neural networks into static data flow calculation graphs, and the graph compiler can be used to fuse and optimize the operators in the calculation graph to improve the running efficiency of the model on specific hardware. However, large models often contain massive parameters, resulting in large amounts of computation and high memory usage during the inference process, which seriously affects response speed and scalability. Summary of the invention

[0004] In order to solve the above problems, this application proposes a large model compilation reasoning method, including:

[0005] Obtain a precompiled large model; wherein the large model includes a pre-filled model and a decoding model;

[0006] Adjusting the input token length and the data buffer in the pre-filled model and the decoding model to obtain an adjusted large model;

[0007] For the adjusted large model, construct a dependency mapping relationship between its corresponding input and output tensors and the data buffer, and compile the adjusted large model to obtain a compiled large model;

[0008] Receive a question-and-answer request sent by a user, convert the question corresponding to the question-and-answer request into an input token sequence, and input the input token sequence into the compiled large model;

[0009] Based on the compiled large model, according to the adjusted input token length, the initial calculation amount corresponding to each vector in the pre-filling stage is reduced, and according to the dependency mapping relationship, the specified vectors stored in the historical batch are obtained through the data buffers corresponding to the pre-filling model and the decoding model respectively, and the attention calculation in the current batch is realized through the specified vectors to realize the reasoning of the input token sequence and obtain the output token sequence;

[0010] Construct an answer to the question based on the output token sequence.

[0011] In one implementation of the present application, adjusting the input token length and the data buffer in the pre-filled model and the decoding model specifically includes:

[0012] Get the maximum input length corresponding to the input token specified by the large model, and the maximum output length corresponding to the attention range;

[0013] Adjusting the input token length in the pre-filled model from the maximum output length to the maximum input length, and modifying the input token length in the decoding model from the maximum output length to 1;

[0014] The specified vector obtained by self-attention calculation in each batch is stored in an external data buffer, so that when executing the inference of the next batch, the specified vector is obtained from the external data buffer, and the attention score corresponding to the next batch is calculated by the specified vector; wherein the specified vector is a key vector and a value vector.

[0015] In one implementation of the present application, adjusting the input token length and the data buffer in the pre-filled model and the decoding model specifically includes:

[0016] Get the maximum input length corresponding to the input token specified by the large model, and the maximum output length corresponding to the attention range;

[0017] Adjusting the input token length in the pre-filled model from the maximum output length to the maximum input length, and modifying the input token length in the decoding model from the maximum output length to 1;

[0018] The specified vector obtained by self-attention calculation in each batch is stored in an external data buffer, so that when executing the inference of the next batch, the specified vector is obtained from the external data buffer, and the attention score corresponding to the next batch is calculated by the specified vector; wherein the specified vector is a key vector and a value vector.

[0019] In one implementation of the present application, after inputting the input token sequence into the compiled large model, the method further includes:

[0020] For the input token sequence, determine a sequence length corresponding to the input token sequence;

[0021] According to the sequence length, the position encoding matrix and the mask matrix corresponding to the input token sequence are reduced.

[0022] In one implementation of the present application, after adjusting the input token length and the data buffer in the pre-filled model and the decoding model to obtain the adjusted large model, the method further includes:

[0023] For the input token input by the decoding model in each batch, a corresponding position encoding variable is generated to obtain the token id corresponding to the input token through the position encoding vector to locate the input token.

[0024] In one implementation of the present application, after compiling the adjusted large model to obtain the compiled large model, the method further includes:

[0025] Obtain a first computation graph and a second computation graph obtained by compiling the pre-filled model and the decoding model;

[0026] Merging the first computation graph and the second computation graph to obtain a merged computation graph, so as to load the pre-filled model and the decoding model into a memory at the same time through the merged computation graph;

[0027] Adjust the execution path between the first computation graph and the second computation graph according to the input token lengths corresponding to the pre-filled model and the decoding model respectively.

[0028] In one implementation of the present application, the method further includes:

[0029] In the case where there are multiple question-and-answer requests, determining the question type to which the question corresponding to the question-and-answer request belongs;

[0030] For at least one question in each question type, performing similarity analysis on the questions to obtain similarities between the questions;

[0031] When the similarity is greater than a preset threshold, it is determined that there is an association relationship between the questions, and after obtaining an answer corresponding to any question through the compiled large model, the answer is added to the answer example library corresponding to the question type, and an association identifier is generated between the question and its associated questions;

[0032] When reasoning is performed on the associated question, the corresponding answer is retrieved from the answer example library according to the associated identifier to achieve reuse of the answer.

[0033] In one implementation of the present application, the reasoning of the input token sequence is implemented, specifically including:

[0034] Inputting the input token sequence into the pre-filled model, and performing reasoning on the input token sequence through the pre-filled model to obtain a first output token;

[0035] Inputting the first output token into the decoding model, and performing decoding reasoning on the first output token through the decoding model to obtain a second output token;

[0036] The second output token is used as the input of the decoding model in the next batch until the output token sequence obtained by the decoding model meets the maximum output length.

[0037] The embodiment of the present application provides a large model compilation and reasoning device, the device comprising:

[0038] at least one processor;

[0039] and, a memory communicatively coupled to the at least one processor;

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a large model compilation reasoning method as described in any one of the above items.

[0041] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0042] A large model compilation reasoning method as described in any of the above items.

[0043] The large model compilation reasoning method proposed in this application can bring the following beneficial effects:

[0044] By adjusting the input token length in the pre-filling model and the decoding model, the initial calculation amount corresponding to each vector in the pre-filling stage can be significantly reduced. During the inference process, the large model only needs to process the necessary calculation steps, avoiding unnecessary calculation waste, thereby effectively reducing the overall calculation amount.

[0045] The data buffer is used to store the specified vectors calculated in the historical batches, and these vectors are retrieved when needed through dependency mapping relationships, avoiding the recalculation of these vectors during each inference process and significantly reducing memory usage. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0047] Figure 1 A flowchart of a large model compilation reasoning method provided in an embodiment of the present application;

[0048] Figure 2 An example diagram of data buffer adjustment provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of the structure of a large model compilation and reasoning device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0051] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0052] like Figure 1 As shown, a large model compilation reasoning method provided in an embodiment of the present application includes:

[0053] S101: Obtain a precompiled large model; wherein the large model includes a pre-filled model and a decoding model.

[0054] The large model is divided into two stages during inference. According to the different inference stages, the large model can be divided into two types of modes, namely the pre-filled model and the decoding model. The input and output relationships of the two stages are slightly different, but most of the model parameters are shared. If they are compiled into two calculation graphs according to the inference stage, when the weights are loaded, it is bound to cause a waste of memory and involve the problem of KVcache. Therefore, it is necessary to make targeted modifications to the calculation graph of the large model to be compiled.

[0055] S102: Adjust the input token length and data buffer in the pre-filled model and the decoding model to obtain an adjusted large model.

[0056] When rewriting the large model, the main adjustments are made to the input token length limit and data caching mechanism, so that the rewritten large model can adapt to input token sequences of different lengths and reduce computational redundancy and resource waste during reasoning. When the original model first calculates K (key vector), V (value vector), and Q (query vector), in order to ensure that the output token sequence length of the large model can meet the batch_matmul operator dimension required by the model, it is necessary to limit the input token length of the pre-filled model according to the maximum output length of the attention range. However, the output token sequence length of the large model during reasoning is significantly higher than the input token sequence length. This method will lead to a large waste of computing resources and affect the reasoning speed of the large model.

[0057] Therefore, it is necessary to first determine the maximum length allowed by the large model when processing input tokens (maximum input length) and the maximum range that the attention mechanism may involve when generating outputs (maximum output length). These length limits are the basis for model design and performance optimization. For the pre-filled model, its input token length needs to be adjusted from the maximum output length to the maximum input length. In this way, regardless of the length of the actual input sequence, the pre-filled model will be processed according to the maximum input length, which may involve padding or truncation operations. For the decoding model, its input token length is modified from the maximum output length to 1, because the decoding process is usually autoregressive, that is, only one token is generated each time and used as the input of the next time step.

[0058] In the self-attention mechanism, the Q vector of each token needs to be dot-producted with the K vector of all time steps in the sequence to calculate the attention score. Once the K vector and V vector are calculated, they can be reused in subsequent calculations without recalculation. In particular, during the decoding process, when the model gradually generates the output token sequence, it can use the previously calculated K vector and V vector to speed up the calculation. Therefore, in order to avoid repeated calculations of the above vectors, so that the large model can significantly improve the reasoning speed when processing long sequences, it is necessary to store the specified vectors obtained by self-attention calculation in each batch into the external data buffer. The specified vectors here refer to the K vector and the V vector. In this way, when performing the reasoning of the next batch, these already calculated specified vectors can be directly obtained from the external data buffer instead of recalculating. In this way, the calculation results in the historical batch can be used to calculate the corresponding attention scores in the next batch, speeding up the reasoning process of the next batch.

[0059] It should be noted that modifying the input of the pre-filled model can reduce the amount of calculation of the K, V, and Q vectors during the first inference, but the attention range of the batch_matmul operator still needs to maintain the maximum output length, so an external data buffer needs to be introduced to solve the problem that the output of K and V is not completely consistent with the input of batch_matmul. At this time, it is necessary to determine whether the dimension corresponding to the specified vector stored in the external data buffer is less than the maximum output length. If so, the shape of the specified vector needs to be padded so that its dimension meets the maximum output length. This process can be achieved by adding a padding vector at the end of the specified vector. The padding vector is usually a vector composed of zeros or specific placeholder values. Through the above padding operation, the large model can ensure that when processing each element in the input token sequence, there are K vectors and V vectors of corresponding lengths available for calculation, thereby maintaining the coherence and consistency of the calculation.

[0060] like Figure 2 The figure shows an example of data buffer adjustment. The left side shows the original model, and the right side shows the modified large model. d represents batch_size, n represents max_output_length (maximum output length), and m represents max_input_length (maximum input length). The input token length of the model is changed from m to n, and the corresponding dimensions of the K, Q, and V vectors calculated are also m-dimensional. In order to keep the attention range of the batch_matmul operator at max_output_length, it is necessary to introduce an external data buffer (external K_Cache and external V_Cache) to obtain the K and V vectors calculated in the historical batches. At the same time, it is necessary to fill the K and V vectors so that their dimensions meet the maximum output length n. The corresponding attention score can be calculated by performing a dot product operation on the K and Q vectors. It should be noted that the K and V vectors of each batch will be stored in the new K_Cache and V_Cache after participating in reasoning.

[0061] In addition, due to the change in the length of the input token sequence, the position encoding matrix and mask matrix related to the input token sequence length also need to be resized accordingly. Specifically, it is necessary to determine the sequence length corresponding to the input token sequence, and then reduce the size of the position encoding matrix and mask matrix to match the current sequence length, further improving the efficiency and performance of the model.

[0062] For the decoding model, the adjustment mechanism of the input token length and data buffer is similar to that of the pre-filled model, but there are differences in adjusting the position encoding matrix and the mask matrix. Since the decoder generates sequences token by token, the position id of each token is a variable that increments as the decoding process progresses. Therefore, the decoder needs to generate a position encoding variable to input the position id of the current token in order to generate the corresponding position encoding. This position encoding will be used to provide position information for the current token to help the model understand the sequential relationship in the sequence. The mask matrix is ​​used to implement autoregressive constraints, that is, to ensure that the model can only see previous tokens but not future tokens when generating the current token. As the decoding process progresses, the size of the mask matrix will gradually increase.

[0063] S103: For the adjusted large model, construct a dependency mapping relationship between its corresponding input and output tensors and the data buffer, and compile the adjusted large model to obtain a compiled large model.

[0064] Before using the adjusted large model for reasoning, it is necessary to establish a dependency mapping relationship between the input and output tensors of the large model and its data buffer. In this way, according to the dependency mapping relationship, the large model can obtain the pre-stored K vector and V vector from the dependent data buffer when performing reasoning, reducing the amount of reasoning calculation. The large model needs to be compiled before performing reasoning to convert the model definition into executable code so that it can run efficiently on specific hardware.

[0065] S104: Receive a question-and-answer request sent by a user, convert the question corresponding to the question-and-answer request into an input token sequence, and input the input token sequence into the compiled large model.

[0066] When a user sends a question-and-answer request, the question corresponding to the question is obtained, and the compiled big model can infer the corresponding answer based on the question. Before that, the question needs to be preprocessed, such as word segmentation, removal of special characters, etc. The preprocessed question text needs to be converted into an input token sequence that the big model can understand. After the input token sequence is passed to the compiled big model, the big model can execute the inference logic and make a corresponding response.

[0067] When dealing with large models, especially when two models have most of the same weights and resources are limited (such as insufficient computing card memory), merging the data flow calculation graphs of the two models can reduce memory usage and accelerate calculations by using weight sharing. Therefore, it is necessary to obtain the first calculation graph and the second calculation graph obtained after the pre-filled model and the decoded model are compiled, and then merge the first calculation graph and the second calculation graph to obtain a merged calculation graph. When merging the calculation graphs, it is necessary to ensure the correctness and consistency of the model weights. After obtaining the merged calculation graph, load the merged calculation graph into memory. Since the merged calculation graph optimizes memory usage by sharing weights and possible calculation steps, it can use memory resources more efficiently than loading two calculation graphs separately.

[0068] It should be noted that when running inference on a large model, the execution path between the first calculation graph and the second calculation graph needs to be adjusted according to the input token lengths corresponding to the pre-filled model and the decoding model. This allows the corresponding model part to be loaded when needed, rather than loading the entire model at once, which helps reduce memory usage and avoid unnecessary computational waste.

[0069] S105: Based on the compiled large model, according to the adjusted input token length, the initial calculation amount corresponding to each vector in the pre-filling stage is reduced, and according to the dependency mapping relationship, the specified vectors stored in the historical batch are obtained through the data buffers corresponding to the pre-filling model and the decoding model respectively, and the attention calculation in the current batch is realized through the specified vector to realize the reasoning of the input token sequence and obtain the output token sequence.

[0070] Based on the compiled large model, according to the adjusted input token length, for shorter inputs, unnecessary calculation steps can be reduced or the vector dimension can be reduced to reduce the initial calculation amount corresponding to each vector in the pre-filling stage. During the reasoning process, according to the dependency mapping relationship, the specified vectors stored in the historical batch are retrieved from the data buffer corresponding to the pre-filled model and the decoding model. If matching vectors are found, these vectors can be directly used for the attention calculation of the current batch without recalculation.

[0071] When the large model is inferring, it first needs to input the input token sequence into the pre-filled model, and then infer the input token sequence through the pre-filled model to obtain the first output token. The first output token is input into the decoding model, and the decoding model decodes and infers the first output token to obtain the second output token. The second output token is used as the input of the decoding model in the next batch, and the above decoding process is repeated. A new token is generated in each iteration until the output token sequence obtained by the decoding model meets the maximum output length. At this time, the output token sequence generated by the decoding model is the final output.

[0072] S106: Construct an answer to the question based on the output token sequence.

[0073] Once the model generates an output token sequence, it is usually necessary to combine these tokens into a meaningful sentence or paragraph, that is, to construct a specific answer to the input question. This process may include converting the tokens into readable text form, adjusting the word order, adding punctuation, etc. to ensure that the generated answer is grammatically and semantically correct.

[0074] In one embodiment, the server may receive multiple question-and-answer requests. In order to achieve rapid feedback on the question-and-answer requests and reduce the amount of reasoning calculations, it is necessary to perform similarity analysis on the question-and-answer requests received at the same time to extract common questions from them, and then perform reasoning on these questions in a unified manner to speed up the reasoning speed. First, in the case of multiple question-and-answer requests, it is necessary to determine the question type to which the question corresponding to the question-and-answer request belongs. The question type can be divided according to characteristics such as question content, purpose or structure, such as "weather query", "definition explanation", "factual question and answer", etc. For at least one question in each question type, a similarity analysis is performed. Similarity analysis is to evaluate the degree of similarity between questions by comparing semantic, lexical, structural and other features between them. Similarity analysis can be performed using methods such as word embedding, TF-IDF, semantic role labeling, dependency syntax analysis, etc. The specific method depends on actual needs and is not limited in this application. The purpose of similarity analysis is to identify questions that have the same or very close meanings but different expressions.

[0075] If the similarity between two or more questions is greater than a preset threshold, they are considered to be associated. This means that these questions are equivalent, but they are expressed differently. Once the association is determined, one of the questions is processed by the compiled large model and its answer is obtained. The answer is added to the answer example library corresponding to the question type, and an association identifier is generated between this question and its associated questions. The association identifier is used to quickly retrieve and reuse the answer in the subsequent reasoning process. In this way, when it is necessary to perform reasoning on a new question that has an association relationship with the processed question (i.e., an associated question), the previously generated association identifier is used to directly retrieve the corresponding answer from the answer example library without re-executing the entire reasoning process. The reuse of answers can significantly reduce the amount of calculation and improve the response speed and efficiency of the model. At the same time, since the answer is retrieved from the verified example library, its accuracy is also guaranteed.

[0076] In addition, the response speed of the model can be further improved by concurrent reasoning between different questions. First, a thread pool needs to be initialized for each question type. When multiple question-and-answer requests are received, the questions are classified according to the question type, and then idle sub-threads equal to the number of currently pending questions are called from the thread pool of the corresponding question type. If the thread pool is full, you may need to wait or queue. Each sub-thread is assigned one or more questions for reasoning. The sub-threads assigned questions will independently perform reasoning tasks and generate answers. When the sub-thread completes the reasoning task, it will return the generated answer to the main thread or the corresponding processing logic. The main thread or processing logic is responsible for collecting the answers of all sub-threads and feeding back the answers to the user as needed.

[0077] The above are embodiments of the method proposed in this application. Based on the same idea, some embodiments of this application also provide devices and non-volatile computer storage media corresponding to the above methods.

[0078] Figure 3 A schematic diagram of the structure of a large model compilation and reasoning device provided in an embodiment of the present application. Figure 3 As shown, including:

[0079] at least one processor; and,

[0080] at least one processor is communicatively connected to a memory; wherein,

[0081] The memory stores instructions executable by at least one processor, the instructions being executed by at least one processor to enable the at least one processor to:

[0082] Obtain a precompiled large model; wherein the large model includes a pre-filled model and a decoding model;

[0083] Adjust the input token length and data buffer in the pre-filled model and the decoding model to obtain an adjusted large model;

[0084] For the adjusted large model, a dependency mapping relationship between its corresponding input and output tensors and the data buffer is constructed, and the adjusted large model is compiled to obtain a compiled large model;

[0085] Receive the question-and-answer request sent by the user, convert the question corresponding to the question-and-answer request into an input token sequence, and input the input token sequence into the compiled large model;

[0086] Based on the compiled large model, according to the adjusted input token length, the initial calculation amount corresponding to each vector in the pre-filling stage is reduced, and according to the dependency mapping relationship, the specified vectors stored in the historical batch are obtained through the data buffers corresponding to the pre-filling model and the decoding model respectively, and the attention calculation in the current batch is realized through the specified vectors to realize the reasoning of the input token sequence and obtain the output token sequence;

[0087] Based on the output token sequence, construct the answer to the question.

[0088] The embodiment of the present application provides a non-volatile computer storage medium storing computer executable instructions, wherein the computer executable instructions are configured as follows:

[0089] Obtain a precompiled large model; wherein the large model includes a pre-filled model and a decoding model;

[0090] Adjust the input token length and data buffer in the pre-filled model and the decoding model to obtain an adjusted large model;

[0091] For the adjusted large model, a dependency mapping relationship between its corresponding input and output tensors and the data buffer is constructed, and the adjusted large model is compiled to obtain a compiled large model;

[0092] Receive the question-and-answer request sent by the user, convert the question corresponding to the question-and-answer request into an input token sequence, and input the input token sequence into the compiled large model;

[0093] Based on the compiled large model, according to the adjusted input token length, the initial calculation amount corresponding to each vector in the pre-filling stage is reduced, and according to the dependency mapping relationship, the specified vectors stored in the historical batch are obtained through the data buffers corresponding to the pre-filling model and the decoding model respectively, and the attention calculation in the current batch is realized through the specified vectors to realize the reasoning of the input token sequence and obtain the output token sequence;

[0094] Based on the output token sequence, construct the answer to the question.

[0095] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0096] The devices and media provided in the embodiments of the present application correspond one-to-one to the methods. Therefore, the devices and media also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the devices and media will not be repeated here.

[0097] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0098] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0099] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0100] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0101] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0102] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0103] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0104] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0105] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A large model compilation reasoning method, characterized in that: The method comprises: Obtain a precompiled large model; wherein the large model includes a pre-filled model and a decoding model; Adjusting the input token length and the data buffer in the pre-filled model and the decoding model to obtain an adjusted large model; For the adjusted large model, construct a dependency mapping relationship between its corresponding input and output tensors and the data buffer, and compile the adjusted large model to obtain a compiled large model; Receive a question-and-answer request sent by a user, convert the question corresponding to the question-and-answer request into an input token sequence, and input the input token sequence into the compiled large model; Based on the compiled large model, according to the adjusted input token length, the initial calculation amount corresponding to each vector in the pre-filling stage is reduced, and according to the dependency mapping relationship, the specified vectors stored in the historical batch are obtained through the data buffers corresponding to the pre-filling model and the decoding model respectively, and the attention calculation in the current batch is realized through the specified vectors to realize the reasoning of the input token sequence and obtain the output token sequence; Constructing an answer to the question based on the output token sequence; The input token length and data buffer in the pre-filled model and the decoding model are adjusted, specifically including: Get the maximum input length corresponding to the input token specified by the large model, and the maximum output length corresponding to the attention range; Adjusting the input token length in the pre-filled model from the maximum output length to the maximum input length, and modifying the input token length in the decoding model from the maximum output length to 1; The specified vector obtained by self-attention calculation in each batch is stored in an external data buffer, so that when executing the inference of the next batch, the specified vector is obtained from the external data buffer, and the attention score corresponding to the next batch is calculated by the specified vector; wherein the specified vector is a key vector and a value vector.

2. A large model compilation reasoning method according to claim 1, characterized in that: After acquiring the specified vector stored in the historical batch through the data buffers corresponding to the pre-filled model and the decoding model respectively according to the dependency mapping relationship, the method further includes: Determine whether the dimension corresponding to the specified vector is less than the maximum output length; If so, a corresponding padding vector is added at the end of the specified vector so that the dimension of the specified vector satisfies the maximum output length.

3. A large model compilation reasoning method according to claim 1, characterized in that: After inputting the input token sequence into the compiled large model, the method further includes: For the input token sequence, determine a sequence length corresponding to the input token sequence; According to the sequence length, the position encoding matrix and the mask matrix corresponding to the input token sequence are reduced.

4. A large model compilation reasoning method according to claim 1, characterized in that: After adjusting the input token length and the data buffer in the pre-filled model and the decoding model to obtain the adjusted large model, the method further includes: For the input token input by the decoding model in each batch, a corresponding position encoding variable is generated to obtain the token id corresponding to the input token through the position encoding vector to locate the input token.

5. A large model compilation reasoning method according to claim 1, characterized in that: After compiling the adjusted large model to obtain the compiled large model, the method further includes: Obtain a first computation graph and a second computation graph obtained by compiling the pre-filled model and the decoding model; Merging the first computation graph and the second computation graph to obtain a merged computation graph, so as to load the pre-filled model and the decoding model into a memory at the same time through the merged computation graph; Adjust the execution path between the first computation graph and the second computation graph according to the input token lengths corresponding to the pre-filled model and the decoding model respectively.

6. A large model compilation reasoning method according to claim 1, characterized in that: The method further comprises: In the case where there are multiple question-and-answer requests, determining the question type to which the question corresponding to the question-and-answer request belongs; For at least one question in each question type, performing similarity analysis on the questions to obtain similarities between the questions; When the similarity is greater than a preset threshold, it is determined that there is an association relationship between the questions, and after obtaining an answer corresponding to any question through the compiled large model, the answer is added to the answer example library corresponding to the question type, and an association identifier is generated between the question and its associated questions; When reasoning is performed on the associated question, the corresponding answer is retrieved from the answer example library according to the associated identifier to achieve reuse of the answer.

7. A large model compilation reasoning method according to claim 1, characterized in that: Implementing reasoning on the input token sequence, specifically including: Inputting the input token sequence into the pre-filled model, and performing reasoning on the input token sequence through the pre-filled model to obtain a first output token; Inputting the first output token into the decoding model, and performing decoding reasoning on the first output token through the decoding model to obtain a second output token; The second output token is used as the input of the decoding model in the next batch until the output token sequence obtained by the decoding model meets the maximum output length.

8. A large model compilation and reasoning device, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute a large model compilation reasoning method as described in any one of claims 1-7.

9. A non-volatile computer storage medium storing computer executable instructions, characterized in that: The computer executable instructions are configured to: A large model compilation reasoning method as described in any one of claims 1-7.

Images