Method and apparatus for inference in large language model

Abstract

Disclosed in the embodiments of the present application are a method and apparatus for inference in a large language model, which are used for reducing the bandwidth consumption of inference in a large language model. The method in the embodiments of the present application comprises: a computing device acquiring input data of a large language model running on a plurality of processors, wherein the input data comprises input text of a user and reference information generated on the basis of the input text, and the reference information comprises domain-related knowledge corresponding to the input text and timeliness information corresponding to the input text; on the basis of the input data, matching historical key-value cache data in a shared storage pool, so as to determine key-value cache data corresponding to the input data, wherein the historical key-value cache data is used for indicating intermediate data generated when the large language model processes different historical input data, the shared storage pool is used for storing historical key-value cache data generated by the plurality of processors, and the different historical input data comprises historical input text and reference information corresponding to the historical input text; and on the basis of the key-value cache data, generating an output result corresponding to the input data.

Description

A large language model reasoning method and device

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on January 18, 2024, with application number 202410077095.0 and application name “A method, device and other equipment for data processing”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on February 28, 2024, with application number 202410223760.2 and application name “A reasoning method and device for a large language model”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The embodiments of the present application relate to the field of storage, and in particular to a method and device for reasoning about a large language model. Background Art

[0003] With the development of machine learning and other related technologies, large language models (LLMs) based on machine learning have gained widespread application. For example, large language models can automatically generate text or responses, and can be used for tasks such as machine translation, speech recognition, question-answering systems, and dialogue generation.

[0004] In the current large language model inference process, the computing device first needs to preprocess the input text and input the preprocessed data into the large language model. During the preprocessing process, the computing device can perform retrieval enhancement on the input text based on prompt engineering to obtain some reference information corresponding to the input text. This retrieval enhancement information contains more timely information. The computing device inputs this input text and the corresponding reference information into the large language model, thereby improving the quality of the large language model's answers.

[0005] However, when a large language model processes the input text and reference information corresponding to different requests, the input text and reference information will make the request sequence longer, resulting in a large amount of computation in the prefill phase of the large language model and prolonged inference time.

[0006] Summary of the Invention

[0007] Embodiments of the present application provide a method for reasoning with a large language model. A computing device establishes a shared storage pool to store pre-calculated intermediate data corresponding to different requests, enabling the computing device to reuse this intermediate data for the next inference, thereby reducing the computational workload of the large language model and further reducing the inference latency of the large language model. Embodiments of the present application also provide a large language model reasoning apparatus, computing device, computing device cluster, computer-readable storage medium, and computer program product corresponding to the large language model reasoning method.

[0008] In the first aspect, an embodiment of the present application provides an inference method for a large language model, which can be executed by a computing device, or by a component of the computing device, such as a processor, chip or chip system of the computing device, or by a logic module or software that can realize all or part of the functions of the computing device. The method provided in the first aspect includes: the computing device obtains input data of a large language model running on multiple processors, the input data includes the user's input text and reference information generated based on the input text, and the reference information includes the domain-related knowledge corresponding to the input text and the timeliness information corresponding to the input text. The computing device matches the historical key-value cache data in the shared storage pool based on the input data, and determines the key-value cache data corresponding to the input data. The historical key-value cache data is used to indicate the intermediate data generated by the large language model when processing different historical input data. The shared storage pool is used to store the historical key-value cache data generated by multiple processors. The different historical input data includes historical input text and reference information corresponding to the historical input text. The computing device generates an output result corresponding to the input data based on the key-value cache data.

[0009] In the embodiment of the present application, the computing device can store the intermediate data calculated during the reasoning process of historical input text based on a shared storage pool, so that the computing device can reuse these intermediate data for the next reasoning, reducing the computational complexity of the large language model and further reducing the reasoning latency of the large language model.

[0010] In one possible implementation, in the process of matching historical key-value cache data in a shared storage pool based on input data, the computing device segments the input data to obtain segmented input data, and determines a universally unique identification code corresponding to the segmented input data. The computing device matches the historical key-value cache data in the shared storage pool according to the universally unique identification code to determine the key-value cache data corresponding to the segmented input data.

[0011] In the embodiment of the present application, the computing device can match historical key-value cache data in the shared storage pool based on the unique identification code of the segmented input data, thereby improving the matching efficiency of the key-value cache data and further improving the reasoning efficiency of the large language model.

[0012] In one possible implementation, the computing device performs a prefix tree search on the segmented input data in a shared storage pool to determine the key-value cache data corresponding to the segmented input data. Specifically, the computing device performs prefix matching based on the segmented input data and the key-value cache data in the shared storage, thereby first achieving an approximate match. If the approximate match is successful, the computing device may further calculate a hash value for the segmented input data and the key-value cache data in the shared storage, performing a hash match, and thereby achieving a complete match.

[0013] In the embodiment of the present application, the computing device may also perform a prefix tree search on the segmented input data to determine the key-value cache data corresponding to the segmented input data, thereby improving the input data matching accuracy.

[0014] In one possible implementation, when generating an output result corresponding to input data based on key-value cache data, the computing device concatenates the key-value cache data corresponding to the segmented input data to generate the full-segment key-value cache data corresponding to the input data. The computing device processes the full-segment key-value cache data based on the large language model to generate the output result.

[0015] In the embodiment of the present application, the computing device can splice the key-value cache data corresponding to the segmented input data to generate the full-segment key-value cache data corresponding to the input data, thereby improving the feasibility of the solution.

[0016] In one possible implementation, when the computing device assembles the segmented input data, the large language model may leave blank key-value cache data that was not matched in the shared storage pool during the first round of inference as missing morphemes in the full segment key-value cache data. During subsequent inference, the large language model fills in the missing morphemes based on historical key-value cache data in the shared storage pool, or directly calculates these missing morphemes.

[0017] In the embodiment of the present application, the historical key-value cache data in the shared storage pool of the computing device is used to fill in the missing morphemes, or the missing morphemes are calculated based on a large language model, thereby improving the richness of the solution.

[0018] In one possible implementation, the computing device may use a cost model to calculate the cost of searching for key-value cache data and directly calculating the key-value cache data from shared cache data during inference, thereby determining a method for obtaining the key-value cache data. For example, in the process of filling missing morphemes, the computing device may use a cost model to calculate the cost of filling the missing morphemes using the search history key-value cache data in the shared storage pool. If the cost of filling the missing morphemes using the search history key-value cache data in the shared storage pool is higher than the cost of directly calculating the missing morphemes based on the already filled key-value cache data, the computing device will directly calculate the missing morphemes based on the already filled key-value cache data.

[0019] In the embodiment of the present application, the computing device can calculate the cost of obtaining key-value cache data in different ways through a cost model, and thus determine the method of obtaining the key-value cache data based on the acquisition cost, thereby improving the reasoning efficiency of the large language model.

[0020] In one possible implementation, when input data does not match historical key-value cache data in the shared storage pool, the computing device infers the input data based on the large language model, generates key-value cache data corresponding to the input data, and stores the key-value cache data in the shared storage pool. Specifically, the computing device infers and calculates the segmented input data, generates key-value cache data corresponding to the segmented input data, and stores the key-value cache data in the shared storage pool.

[0021] In an embodiment of the present application, when input data does not match historical key-value cache data in the shared storage pool, the computing device can infer the input data based on the large language model to generate key-value cache data corresponding to the input data, and store these key-value cache data in the shared storage pool for subsequent key-value cache data matching of input data, thereby improving the reasoning efficiency of the large language model.

[0022] In one possible implementation, before obtaining input data for the large language model, the computing device receives input text from a user, including the user's question text. The computing device performs retrieval enhancement on the input text based on a prompt engineering system and generates reference information corresponding to the input text. This reference information can provide the large language model with domain-related knowledge corresponding to the input text and timeliness information corresponding to the input text. Because different input texts may correspond to the same reference information, the reference information corresponding to the input text may overlap with the reference information corresponding to historical input texts.

[0023] In the embodiment of the present application, the computing device can perform retrieval enhancement on the input text based on the prompt engineering system and generate reference information corresponding to the input text. This reference information can provide the large language model with domain-related knowledge corresponding to the input text and timeliness information corresponding to the input text, thereby improving the reasoning accuracy of the large language model.

[0024] In a possible implementation, the historical key-value cache data includes historical input texts of different users, historical input texts at different times, and historical key-value cache data corresponding to historical input texts in different sessions.

[0025] In the embodiment of the present application, the historical key-value cache data in the shared storage pool includes historical key-value cache data corresponding to historical input texts of different users, historical input texts at different times, and historical input texts in different sessions, thereby realizing the reuse of historical key-value cache data across users, requests, and sessions, further improving the inference efficiency of large language models.

[0026] In the second aspect, an embodiment of the present application provides an inference device for a large language model, which includes an acquisition unit and a processing unit. The acquisition unit is used to acquire input data of a large language model running on multiple processors, and the input data includes the user's input text and reference information generated based on the input text, and the reference information includes the field-related knowledge corresponding to the input text and the timeliness information corresponding to the input text. The processing unit is used to match historical key-value cache data in a shared storage pool based on the input data, and determine the key-value cache data corresponding to the input data. The historical key-value cache data is used to indicate the intermediate data generated by the large language model when processing different historical input data. The shared storage pool is used to store historical key-value cache data generated by multiple processors, and different historical input data includes historical input text and reference information corresponding to the historical input text. The processing unit is also used to generate output results corresponding to the input data based on the key-value cache data.

[0027] In one possible implementation, the processing unit is specifically configured to segment the input data, obtain the segmented input data, and determine a universally unique identifier corresponding to the segmented input data. The processing unit is specifically configured to match historical key-value cache data in a shared storage pool based on the universally unique identifier to determine the key-value cache data corresponding to the segmented input data.

[0028] In one possible implementation, the processing unit is specifically configured to concatenate key-value cache data corresponding to the segmented input data to generate full-segment key-value cache data corresponding to the input data, and process the full-segment key-value cache data based on the large language model to generate an output result.

[0029] In one possible implementation, the entire key-value cache data includes missing morphemes, and the processing unit is further configured to fill the missing morphemes based on historical key-value cache data in the shared storage pool, or to calculate the missing morphemes based on the large language model.

[0030] In one possible embodiment, the processing unit is further configured to, when the input data does not match historical key-value cache data in the shared storage pool, infer the input data based on the large language model to generate key-value cache data corresponding to the input data, and store the key-value cache data in the shared storage pool.

[0031] In a possible implementation, the historical key-value cache data includes historical input texts of different users, historical input texts at different times, and historical key-value cache data corresponding to historical input texts in different sessions.

[0032] In a third aspect, an embodiment of the present application provides a computing device, comprising a processor coupled to a memory, the processor being used to store instructions. When the instructions are executed by the processor, the computing device executes the method described in the first aspect or any possible implementation of the first aspect.

[0033] In a fourth aspect, an embodiment of the present application provides a computing device cluster, which includes one or more computing devices, each of which includes a processor coupled to a memory, and the processor is used to store instructions. When the instructions are executed by the processor, the computing device cluster executes the method described in the first aspect or any possible implementation method of the first aspect.

[0034] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having instructions stored thereon. When the instructions are executed, the computer executes the method described in the first aspect or any possible implementation method of the first aspect.

[0035] In a sixth aspect, an embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed, the computer implements the method described in the first aspect or any possible implementation method of the first aspect.

[0036] It can be understood that the beneficial effects that can be achieved by any of the above-mentioned inference devices, computing devices, computing device clusters, computer-readable media or computer program products of the large language model can be referred to the beneficial effects in the corresponding methods and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] FIG1 is a schematic diagram of the system architecture of a large language model inference system provided in an embodiment of the present application;

[0038] FIG2 is a flow chart of a large language model inference method provided in an embodiment of the present application;

[0039] FIG3 is a flow chart of another large language model inference method provided in an embodiment of the present application;

[0040] FIG4 is a flow chart of another large language model inference method provided in an embodiment of the present application;

[0041] FIG5 is a schematic diagram of the structure of a large language model inference device provided in an embodiment of the present application;

[0042] FIG6 is a schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0043] FIG7 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application;

[0044] FIG8 is a schematic diagram of the structure of another computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application provide a large language model inference method and apparatus for reducing the inference latency during the large language model inference process.

[0046] The terms "first," "second," "third," "fourth," and the like (if any) in the specification and claims of this application and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, e.g., a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0047] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0048] First, some terms involved in the embodiments of the present application are introduced to facilitate those skilled in the art to understand the technical solutions.

[0049] A large language model (LLM) is a natural language processing model based on deep learning technology. Trained using large amounts of text data, it possesses the ability to understand, generate, and process natural language. LLMs treat natural language text as sequential data, such as word or character sequences, and use deep learning models to model the statistical patterns and underlying semantic information in these sequential data. LLMs can perform a wide range of tasks, including text summarization, translation, sentiment analysis, question-answering, and conversation.

[0050] Prompt engineering is a technique used to improve the input of large language models. Prompt engineering adjusts input prompts to guide large language models to more accurately understand and answer relevant questions. For example, prompt engineering can optimize prompts to emphasize key information, reduce the model's misunderstanding of context, and provide task-specific guidance, enabling the model to better adapt to the needs of specific domains or tasks.

[0051] A morpheme is the smallest semantic unit in a language after word segmentation. It can also be called a segmentation or lexical marker. A morpheme can be one or more Chinese characters, an English word, or half an English word. For example, "apple" is one morpheme, while "apple tree" can be composed of two morphemes: "apple" and "tree." However, "apple" alone does not convey any meaning.

[0052] Context refers to the contextual information that a large language model relies on when processing text. That is, the contextual environment in which the large language model generates or understands text. For a large language model, context can be a paragraph of text, a conversation, a question, or any other form of text input.

[0053] In order to make the technical solution of the present application clearer and easier to understand, the system architecture of the present application is introduced below with reference to the accompanying drawings.

[0054] Please refer to Figure 1, which is a schematic diagram of the system architecture of a large language model inference system provided as an example in this application. In the system architecture shown in Figure 1, the large language model inference system 10 includes a prompt engineering module 101, a large language model inference module 102, and a shared storage pool module 103. The large language model inference module 102 includes an embedding layer 1021, a self-attention layer 1022, and a feedforward network layer 1023. The specific functions of each module of the large language model inference system 10 are described in detail below.

[0055] The prompt engineering module 101 is used to generate reference information corresponding to the input text based on the user's input text, and input the input text and reference information into the large language model inference module 102. These input text and reference information can also be called enhanced text. The prompt engineering module 101 stores relevant domain knowledge and timeliness information. Therefore, the prompt engineering module 101 can perform search enhancement based on the input text within the relevant domain knowledge and timeliness information to obtain the relevant domain knowledge and timeliness information corresponding to the input text.

[0056] It should be noted that, since the relevant domain knowledge and timeliness of different input texts may be the same, different input texts may be retrieved and enhanced to obtain the same reference information, that is, the reference information after the engineering module 101 retrieves and enhances different input texts may overlap.

[0057] The large language model inference module 102 includes a built-in large language model that can perform inference calculations on input text and reference information to generate corresponding output text. For example, the large language model inference module 102 can receive a natural language text as input text. The input text can be a sentence, a paragraph, or a longer text sequence. The large language model inference module 102 can perform inference calculations on the input text and reference information based on the large language model to generate output text. The output text can be an answer to a question, a text summary, or a translation result corresponding to the input text.

[0058] The model structure of the large language model built into the large language model inference module 102 is a multi-layer structure. For example, the multi-layer structure includes an embedding layer 1021, a self-attention layer 1022, and a feedforward network layer 1023. Among them, the embedding layer 1021 is used to convert each morpheme in the user's input text and reference information into a vector representation. The self-attention layer 1022 is used to determine the contextual relationship between the input text and the reference information. For example, the self-attention layer 1022 performs self-attention calculations on the vector representations corresponding to the input text and the reference information to generate contextual representations corresponding to the input text and the reference information. The feedforward network layer 1023 is used to perform nonlinear transformations after the self-attention layer 1022 to extract high-level semantic features in the input text and generate feature representations.

[0059] The inference phase of the large language model inference module 102 includes a prefill phase and a decode phase. The prefill phase, also known as the first-word inference phase or the full-word inference phase, involves the computing device inputting the entire input text and reference information into the large language model for the first round of inference, generating the first output word of the output text. The decode phase, also known as the incremental inference phase, involves the computing device continuously generating the next output word from the generated output text using an autoregressive loop.

[0060] The shared storage pool module 103 is used to store the key-value cache (KV cache) data generated by the large language model during the inference process. The KV cache data is intermediate data generated by the large language model during the inference process. Because the reference information after the hint engineering module 101 performs retrieval enhancement on different input texts can overlap, the KV cache data generated by the large language model during the inference process for different input texts can also be duplicated. The large language model can reuse this KV cache data, and the computing device can store this KV cache data through the shared storage pool module 103.

[0061] It should be noted that the various modules of the above-mentioned large language model inference system 10 can be deployed on a single computing device or a computing device cluster consisting of multiple computing devices, without specific limitation.

[0062] Based on the large language model inference system 10 shown in Figure 1, the present application also provides a large language model inference method. The following describes the large language model inference method provided in the present application embodiment in conjunction with the embodiment.

[0063] Please refer to Figure 2, which is a flow chart of a method for reasoning a large language model provided in an embodiment of the present application. In the example shown in Figure 2, the method includes the following steps:

[0064] Step 201. The computing device obtains input data of the large language model, where the input data includes user input text and reference information generated based on the input text, where the reference information includes domain-related knowledge corresponding to the input text and timeliness information corresponding to the input text.

[0065] The computing device obtains input data of the large language model, and the input data includes the user's input text and reference information generated based on the input text, wherein the input text refers to the text input by the user into the language model, for example, the input text can be a sentence or paragraph asked by the user, and the reference information refers to the text after retrieval enhancement by the prompt engineering.

[0066] Specifically, the computing device receives input text from a user into the large language model and enhances the user's input text based on the hint project to obtain reference information for the input text. The user's input text and the reference information corresponding to the input text serve as input data for the large language model. Because the input data is text that has been search-enhanced based on the hint project, it can also be called augmented text.

[0067] It is understood that the input data enhanced by the hint engineering includes the input text and reference information corresponding to the input text. This reference information can provide the large language model with domain-related knowledge and timeliness information corresponding to the input text. During the process of enhancing the input text based on the hint engineering, the computing device can retrieve the reference information through components such as a vector database (vector DB) component or a search engine.

[0068] Please refer to Figure 3, which is a flowchart of a large language model reasoning method provided by an embodiment of the present application. In the example shown in Figure 3, the user inputs text such as "What are the new features of the Askjie M9?" The computing device enhances the input text based on the prompt engineering and generates reference information for the input text, where the reference information includes time-sensitive information such as "Askjie M9 press conference news" and related knowledge in the field of "Askjie M9 manual".

[0069] For another example, the user inputs text such as "What was discussed at the press conference of Q&A M9?" The computing device enhances the input text based on the prompt engineering and generates reference information for the input text, wherein the reference information is time-sensitive information such as "Q&A M9 press conference news." For two different user input texts, "What are the new features of Q&A M9?" and "What was discussed at the press conference of Q&A M9?", the reference information generated by the computing device based on the prompt engineering may overlap. For example, for these two input texts, the prompt engineering module 101 can generate reference information for both input texts from the time-sensitive information "Q&A M9 press conference news."

[0070] In the example shown in FIG3 , after the computing device generates reference information for the input text, it inputs the input text and the reference information as input data into the large language model.

[0071] In one possible implementation, the reference information includes, in addition to the domain-related knowledge corresponding to the input text and the timeliness information corresponding to the input text, instructions and examples of the large language model, where the instructions are used to indicate the reasoning limitations and reasoning requirements of the large language model, and the examples are used to indicate the format of the output text of the large language model.

[0072] Step 202: The computing device matches the historical key-value cache data in the shared storage pool based on the input data to determine the key-value cache data corresponding to the input data.

[0073] After the computing device obtains the input data of the large language model, during the process of large language model inference, it matches the historical key-value cache data in the shared storage pool based on the input data to determine the key-value cache data corresponding to the input data, wherein the historical key-value cache data is the intermediate data generated by the large language model when processing different historical input data. These intermediate data are stored in the form of multi-dimensional vectors, and these historical key-value cache data can be stored in the shared storage pool.

[0074] It should be noted that when performing inference on a large language model, a computing device can also perform inference calculations directly based on the input data, without having to retrieve key-value cache data from a shared storage pool. This means that the input data is processed based on the multi-layer structure of the large language model to generate output text corresponding to the input data. This processing includes converting the input data into a vector representation through an embedding layer, then using a self-attention layer to mine and integrate key information in the vector representation, and performing further calculations and optimization through a feedforward network layer, ultimately converting the generated vector representation into output text.

[0075] In one possible implementation, in the process of matching historical key-value cache data in a shared storage pool based on input data, the computing device first segments the input data based on a segmentation algorithm to obtain segmented input data. The segmentation algorithm may be a special character-based segmentation algorithm or an adaptive segmentation algorithm. After obtaining the segmented input data, the computing device matches the segmented input data with historical key-value cache data in the shared storage pool to determine the key-value cache data corresponding to each segmented input data. Specifically, the computing device needs to first convert the segmented input data into a vector identifier, and match the historical key-value cache data in the shared storage pool based on the vector representation corresponding to the segmented input data.

[0076] Please continue to refer to Figure 3. In the example shown in Figure 3, after the computing device receives the input data, the large language model performs inference calculations based on the input data. During the inference calculation process of the large language model, the computing device first segments the input data based on the segmentation algorithm to obtain segmented input data, and converts the segmented input data into a vector representation. Then, the large language model performs multiple rounds of inference calculations based on the converted vector representation, including calculations of multiple transmission layers such as the self-attention layer and the feedforward network layer, and finally generates output text corresponding to the input data.

[0077] In the example shown in Figure 3, in each round of inference calculation, the large language model can match and search historical key-value cache data in the shared storage pool based on the vector representation corresponding to the segmented input data. If the shared storage pool contains historical key-value cache data corresponding to the segmented input data, the large language model does not need to perform inference calculation again based on the vector representation of the segmented input data, and can directly obtain the corresponding key-value cache data from the shared storage pool for the next round of inference calculation, thereby improving inference efficiency.

[0078] In one possible implementation, when input data does not match historical key-value cache data in the shared storage pool, the computing device performs inference on the input data based on the large language model, generates key-value cache data corresponding to the input data, and stores the key-value cache data in the shared storage pool. Specifically, the computing device performs inference calculations on the segmented input data, generates key-value cache data corresponding to the segmented input data, and stores the key-value cache data in the shared storage pool.

[0079] Please continue to refer to Figure 3. In the example shown in Figure 3, if the historical key-value cache data corresponding to the segmented input data does not exist in the shared storage pool, the large language model can perform inference calculations based on the segmented input data to obtain the key-value cache data corresponding to the segmented input data, and store the key-value cache data corresponding to the segmented input data in the shared storage pool.

[0080] It should be noted that the historical key-value cache data stored in the shared storage pool includes historical key-value cache data corresponding to historical input text from different users, historical input text from different times, and historical input text from different sessions. Because historical key-value cache data corresponding to historical input text from different users, historical input text from different times, and historical input text from different sessions may contain duplication, the key-value cache data stored in the shared storage pool needs to be deduplicated.

[0081] For example, historical key-value cache data corresponding to historical input text from different users includes key-value cache data generated based on user A's input text and enhanced text, and key-value cache data generated based on user B's input text and enhanced text. Historical key-value cache data corresponding to historical input text at different times includes key-value cache data generated based on the user's input text at 09:50:00 on January 19, 2024, and key-value cache data generated based on the user's input text at 09:57:00 on January 19, 2024. Historical key-value cache data corresponding to historical input text in different sessions includes key-value cache data corresponding to input text for questions asked by the user in the first session, and key-value cache data corresponding to input text for questions asked by the user in the second session.

[0082] In one possible implementation, after the computing device obtains the segmented input data, it further determines a universally unique identifier (UUID) corresponding to the segmented input data, and matches historical key-value cache data in a shared storage pool based on the universally unique identifier to determine the key-value cache data corresponding to the segmented input data.

[0083] Please refer to Figure 4, which is a schematic diagram of the reasoning process of another large language model provided in an embodiment of the present application. In steps a to f of the example in Figure 4, after the large language model obtains the input data, it segments the input data to obtain the segmented input data, and further determines the universal unique identification code corresponding to the segmented input data. The computing device matches the key-value cache data in the shared storage pool based on the universal unique identification code. If the key-value cache data is matched, the key-value cache data is read from the shared storage pool. If the key-value cache data is not matched, the position corresponding to the segmented input data is left as a vacant morpheme for the next round of reasoning.

[0084] In one possible implementation, the computing device may also perform a prefix tree search on the segmented input data to determine the key-value cache data corresponding to the segmented input data. Specifically, the computing device performs prefix matching on the segmented input data and the key-value cache data in the shared storage to achieve an approximate match. If the match is successful, the computing device may further perform a hash match on the segmented input data and the key-value cache data in the shared storage to achieve a complete match.

[0085] Please continue to refer to Figure 3. In the example shown in Figure 3, the large language model matches the search history key-value cache data in the shared storage pool. The history key-value cache data can be indexed based on the universal unique identifier UUID of the segmented input data, or a prefix tree search can be performed in the shared storage pool based on the vector representation corresponding to the segmented input data.

[0086] For example, when the input data is the input text "What are the new features of Wenjie M9?" and the reference information "Wenjie M9 press conference news" and "Wenjie M9 instruction manual", the large language model segments the input data during the process of reasoning based on these input data and matches the historical key-value cache data in the shared storage pool based on the vector representation corresponding to the segmented input data. For example, the large language model matches "segmented input data 1" to "key-value cache data 1" in the shared storage pool, and the output text corresponding to the key-value cache data 1 is, for example, a paragraph of "Wenjie M9 technical parameter introduction content".

[0087] Please continue to refer to Figure 4. In steps k to m and steps e to f in the example of Figure 4, the large language model segments the input data. After obtaining the segmented input data, the computing device can also directly match the key-value cache data in the shared storage pool based on the vector representation corresponding to the segmented input data. Specifically, the computing device can also perform prefix matching based on the vector representation corresponding to the segmented input data and the historical key-value cache data in the shared storage pool. If the prefix match is successful, it means that there is key-value cache data in the shared storage pool that approximately matches the segmented input data. The computing device further performs hash matching on the approximately matching key-value cache data, that is, matching the segmented input data with the hash value of the number of approximately matching key-value cache data. If the hash match is successful, the successfully matched key-value cache data is read from the shared storage pool. If the prefix match is unsuccessful or the hash match is unsuccessful, the position corresponding to the segmented input data is left as a missing morpheme for the next round of reasoning.

[0088] Step 203: The computing device generates an output result corresponding to the input data based on the key-value cache data.

[0089] The computing device generates an output result corresponding to the input data based on the key-value cache data. Specifically, after obtaining the key-value cache data corresponding to each segmented input data, the computing device concatenates the key-value cache data corresponding to these segmented input data to generate the full-segment key-value cache data corresponding to the input data, and processes the full-segment key-value cache data based on the large language model to generate the output result.

[0090] Continuing with Figure 3, in the example shown in Figure 3, after the computing device obtains the key-value cache data of the segmented input data, it concatenates the key-value cache data corresponding to the segmented input data to obtain the full-segment key-value cache data corresponding to the input data. For example, the computing device concatenates key-value cache data 1, key-value cache data 2, key-value cache data 3, and key-value cache data 4, and performs inference calculations on the concatenated full-segment key-value cache data to generate output text.

[0091] In one possible implementation, when the computing device assembles the segmented input data, the large language model may leave blank key-value cache data that was not matched in the shared storage pool during the first round of inference as missing morphemes in the full segment key-value cache data. During subsequent inference, the large language model fills in the missing morphemes based on historical key-value cache data in the shared storage pool, or directly calculates these missing morphemes.

[0092] It is understood that the computing device can use a cost model to calculate the cost of searching for key-value cache data from shared cache data and directly calculating the key-value cache data during the inference process, thereby determining the method for obtaining the key-value cache data. For example, in the process of filling missing morphemes, the computing device can use a cost model to calculate the cost of filling the missing morphemes using the search history key-value cache data in the shared storage pool, and directly calculate the cost of these missing morphemes based on the filled key-value cache data, thereby determining the method for filling the missing morphemes.

[0093] Please refer to Figure 3. In the example shown in Figure 3, during the first round of inference, the large language model splices the key-value cache data corresponding to the segmented input data. This may result in some missing morphemes. For example, the large language model obtains key-value cache data 1, key-value cache data 2, and key-value cache data 4 from the shared storage pool and splices them together. There are missing morphemes between key-value cache data 2 and key-value cache data 4. The large language model needs to fill in the missing morphemes in the next round of inference calculations.

[0094] In the example shown in Figure 3, the large language model obtains the key-value cache data 3 corresponding to the missing morpheme in the next round of inference calculation. The computing device can obtain the key-value cache data 3 by searching in the shared storage, or it can obtain the key-value cache data 3 by performing inference calculation based on the key-value cache data 1 and the key-value cache data 2. There is no specific limitation.

[0095] Continuing with Figure 4, in steps g through j of the example shown in Figure 4, during the next round of inference calculations, the large language model determines how to fill missing morphemes based on the cost model. The computing device can fill the missing morphemes using the search history key-value cache data in the shared storage pool, or directly calculate these missing morphemes based on the already filled key-value cache data. After filling the missing morphemes, the computing device can perform the next round of inference based on the large language model engine, including inference in the pre-filling phase and inference in the decoding phase.

[0096] It should be noted that when the computing device directly calculates these missing morphemes based on the populated key-value cache data, the missing morphemes are calculated only based on the key-value cache data preceding the missing morpheme. Furthermore, when calculating the missing morphemes, the computing device may preload the key-value cache data following the missing morpheme in parallel, without further limitation.

[0097] Please continue to refer to Figure 3. In the example shown in Figure 3, when the large language model fills the key-value cache data 3 corresponding to the missing morpheme, it can preload the key-value cache data 4 after the missing morpheme, but the large language model only infers and calculates the key-value cache data 3 based on the key-value cache data 1 and the key-value cache data 2.

[0098] It can be seen from the above embodiments that the computing device in the embodiments of the present application can store the intermediate data calculated during the reasoning process of the historical input text based on a shared storage pool. Since the intermediate data corresponding to the user's input text and the intermediate data corresponding to the historical input text are repeated, the computing device can reuse these intermediate data for the next reasoning, thereby reducing the computational complexity of the large language model and reducing the reasoning delay of the large language model.

[0099] Based on the above method embodiment, the embodiment of the present application also provides an inference device for a large language model. The following specifically introduces the inference device for a large language model provided by the embodiment of the present application.

[0100] Please refer to Figure 5, which is a schematic diagram of the structure of an inference device for a large language model provided in an embodiment of the present application. In the example shown in Figure 5, the inference device 500 for a large language model is used to implement the various steps performed by the end-side shared module in the above embodiments. The inference device 500 for a large language model includes an acquisition unit 501 and a processing unit 502.

[0101] Among them, the acquisition unit 501 is used to obtain the input data of the large language model running on multiple processors, and the input data includes the user's input text and the reference information generated based on the input text. The reference information includes the field-related knowledge corresponding to the input text and the timeliness information corresponding to the input text. The processing unit 502 is used to match the historical key-value cache data in the shared storage pool based on the input data, and determine the key-value cache data corresponding to the input data. The historical key-value cache data is used to indicate the intermediate data generated by the large language model when processing different historical input data. The shared storage pool is used to store the historical key-value cache data generated by multiple processors. The different historical input data includes historical input text and reference information corresponding to the historical input text. The processing unit 502 is also used to generate an output result corresponding to the input data based on the key-value cache data.

[0102] In one possible implementation, processing unit 502 is specifically configured to segment input data, obtain segmented input data, and determine a universally unique identifier corresponding to the segmented input data. Processing unit 502 is specifically configured to match historical key-value cache data in a shared storage pool based on the universally unique identifier to determine the key-value cache data corresponding to the segmented input data.

[0103] In one possible implementation, the processing unit 502 is specifically configured to concatenate the key-value cache data corresponding to the segmented input data to generate the full-segment key-value cache data corresponding to the input data, and process the full-segment key-value cache data based on the large language model to generate an output result.

[0104] In a possible implementation, the entire key-value cache data includes missing morphemes, and the processing unit 502 is further configured to fill the missing morphemes based on historical key-value cache data in the shared storage pool, or calculate the missing morphemes based on the large language model.

[0105] In one possible embodiment, the processing unit 502 is further configured to, when the input data does not match the historical key-value cache data in the shared storage pool, infer the input data based on the large language model, generate key-value cache data corresponding to the input data, and store the key-value cache data in the shared storage pool.

[0106] In a possible implementation, the historical key-value cache data includes historical input texts of different users, historical input texts at different times, and historical key-value cache data corresponding to historical input texts in different sessions.

[0107] It can be understood that the acquisition unit 501 and the processing unit 502 in the large language model reasoning device 500 can be mapped as functional modules to the various modules in the large language model reasoning system 10 in Figure 1, thereby realizing the functions of the various modules in the large language model reasoning system 10.

[0108] It should be understood that the division of units in the above device is merely a division of logical functions. In actual implementation, they can be fully or partially integrated into one physical entity, or they can be physically separated. Moreover, the units in the device can all be implemented in the form of software called through processing elements; or they can all be implemented in the form of hardware; or some units can be implemented in the form of software called through processing elements, and some units can be implemented in the form of hardware. For example, each unit can be a separately established processing element, or it can be integrated into a certain chip of the device. In addition, it can also be stored in the memory in the form of a program, called by a certain processing element of the device and perform the function of the unit. In addition, all or part of these units can be integrated together, or they can be implemented independently. The processing element described here can also be a processor, which can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each unit above can be implemented by the integrated logic circuit of the hardware in the processor element or in the form of software called through the processing element.

[0109] It is worth noting that, for the sake of simplicity of description, the above method embodiments are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited to the order of the actions described. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required for this application.

[0110] Other reasonable step combinations that can be thought of by those skilled in the art based on the above description also fall within the scope of protection of this application. Secondly, those skilled in the art should also be familiar with that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by this application.

[0111] Please refer to Figure 6, which is a schematic diagram of the structure of a computing device provided in an embodiment of the present application. As shown in Figure 6, computing device 600 includes: a processor 601, a memory 602, a communication interface 603, and a bus 604. Processor 601, memory 602, and communication interface 603 are coupled via a bus (not labeled in the figure). Memory 602 stores instructions. When the execution instructions in memory 602 are executed, computing device 600 performs the method performed by the computing device in the above method embodiment.

[0112] The computing device 600 may be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), one or more digital signal processors (DSPs), one or more field programmable gate arrays (FPGAs), or a combination of at least two of these integrated circuit forms. For example, when a unit in the apparatus can be implemented in the form of a processing element scheduler, the processing element may be a general-purpose processor, such as a central processing unit (CPU) or other processor that can call a program. For example, these units may be integrated together and implemented in the form of a system-on-a-chip (SOC).

[0113] The processor 601 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0114] Memory 602 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable ROM (PROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0115] Memory 602 stores executable program code, which processor 601 executes to implement the functions of the aforementioned units or modules, thereby implementing the aforementioned large language model reasoning method. That is, memory 602 stores instructions for executing the aforementioned large language model reasoning method.

[0116] The communication interface 603 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 600 and other devices or a communication network.

[0117] In addition to the data bus, bus 604 may also include a power bus, a control bus, and a status signal bus. The bus may be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a unified bus (Ubus or UB), a Compute Express Link (CXL), or a Cache Coherent Interconnect for Accelerators (CCIX). Buses can be categorized as address buses, data buses, and control buses.

[0118] Please refer to FIG7 , which is a schematic diagram of a computing device cluster provided in an embodiment of the present application. As shown in FIG7 , the computing device cluster 700 includes at least one computing device 600 .

[0119] 7 , the computing device cluster 700 includes at least one computing device 600. The memory 602 in one or more computing devices 600 in the computing device cluster 700 may store the same instructions for executing the above-mentioned inference method for the large language model.

[0120] In some possible implementations, the memory 602 of one or more computing devices 600 in the computing device cluster 700 may also store partial instructions for executing the aforementioned large language model reasoning method. In other words, the combination of one or more computing devices 600 can collectively execute instructions for executing the aforementioned large language model reasoning method.

[0121] It should be noted that the memory 602 in different computing devices 600 in the computing device cluster 700 can store different instructions, each for executing a portion of the functions of the inference apparatus for the large language model. In other words, the instructions stored in the memory 602 in different computing devices 600 can implement the functions of one or more modules in the acquisition unit and the processing unit.

[0122] In some possible implementations, one or more computing devices 600 in the computing device cluster 700 may be connected via a network, which may be a wide area network or a local area network.

[0123] Please refer to Figure 8, which is a schematic diagram of computer devices in a computer cluster provided by an embodiment of the present application connected via a network. As shown in Figure 8, two computing devices 600A and 600B are connected via a network. Specifically, the connection to the network is through a communication interface in each computing device.

[0124] In one possible implementation, the memory of the computing device 600A stores instructions for executing the functions of the acquisition unit, while the memory of the computing device 600B stores instructions for executing the functions of the processing unit and the display unit.

[0125] It should be understood that the functions of the computing device 600A shown in Figure 8 may also be completed by multiple computing devices. Similarly, the functions of the computing device 600B may also be completed by multiple computing devices.

[0126] In another embodiment of the present application, a computer-readable storage medium is provided, in which computer-executable instructions are stored. When the processor of the device executes the computer-executable instructions, the device executes the method executed by the computing device in the above method embodiment.

[0127] In another embodiment of the present application, a computer program product is provided, the computer program product including computer-executable instructions stored in a computer-readable storage medium. When a processor of a device executes the computer-executable instructions, the device performs the method performed by the computing device in the above method embodiment.

[0128] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0129] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0130] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0131] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0132] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. An inference method for a large language model, characterized in that, Including: Obtain the input data of the large language model running on multiple processors, where the input data includes the user's input text and reference information generated based on the input text, and the reference information includes domain-related knowledge corresponding to the input text and timeliness information corresponding to the input text; Match historical key-value cache data in the shared storage pool based on the input data, and determine the key-value cache data corresponding to the input data. The historical key-value cache data is used to indicate the intermediate data generated by the large language model when processing different historical input data. The shared storage pool is used to store the historical key-value cache data generated by the multiple processors. The different historical input data includes historical input text and reference information corresponding to the historical input text; Generate the output result corresponding to the input data according to the key-value cache data.

2. The method according to claim 1, wherein The matching of the historical key-value cache data in the shared storage pool based on the input data includes: Segment the input data to obtain the segmented input data, and determine the universally unique identifier corresponding to the segmented input data; Match the historical key-value cache data in the shared storage pool according to the universally unique identifier, and determine the key-value cache data corresponding to the segmented input data.

3. The method according to claim 2, wherein The generating the output result corresponding to the input data according to the key-value cache data includes: Concatenate the key-value cache data corresponding to the segmented input data to generate the full-segment key-value cache data corresponding to the input data; Process the full-segment key-value cache data based on the large language model to generate the output result.

4. The method according to claim 3, characterized in that, The full-segment key-value cache data includes missing morphemes, and the method further includes: Fill the missing morphemes based on the historical key-value cache data in the shared storage pool, or calculate the missing morphemes based on the large language model.

5. The method according to any one of claims 1 to 4, characterized in that, The method further includes: When the input data does not match the historical key-value cache data in the shared storage pool, perform inference on the input data based on the large language model to generate the key-value cache data corresponding to the input data; Store the key-value cache data in the shared storage pool.

6. The method according to any one of claims 1 to 5, characterized in that, The historical key-value cache data includes the historical key-value cache data corresponding to the historical input text of different users, the historical input text at different times, and the historical input text in different sessions.

7. An inference device for a large language model, characterized in that, Including: An acquisition unit for acquiring the input data of the large language model running on multiple processors, where the input data includes the user's input text and reference information generated based on the input text, and the reference information includes domain-related knowledge corresponding to the input text and timeliness information corresponding to the input text; A processing unit for matching historical key-value cached data in a shared storage pool based on the input data to determine the key-value cached data corresponding to the input data, where the historical key-value cached data is used to indicate intermediate data generated by the large language model when processing different historical input data, the shared storage pool is used to store the historical key-value cached data generated by the multiple processors, and the different historical input data includes historical input texts and reference information corresponding to the historical input texts; The processing unit is further configured to generate an output result corresponding to the input data according to the key-value cached data.

8. The device according to claim 7, characterized in that, Specifically, the processing unit is configured to: Segment the input data to obtain segmented input data, and determine a universally unique identifier corresponding to the segmented input data; Match the historical key-value cached data in the shared storage pool according to the universally unique identifier to determine the key-value cached data corresponding to the segmented input data.

9. The device according to claim 8, wherein Specifically, the processing unit is configured to: Concatenate the key-value cached data corresponding to the segmented input data to generate full-segment key-value cached data corresponding to the input data; Process the full-segment key-value cached data based on the large language model to generate the output result.

10. The device according to claim 9, characterized in that, The full-segment key-value cached data includes missing morphemes, and the processing unit is further configured to: Fill the missing morphemes based on the historical key-value cached data in the shared storage pool, or calculate the missing morphemes based on the large language model.

11. The device according to any one of claims 7 to 10, characterized in that, The processing unit is further configured to: When no historical key-value cached data is matched for the input data in the shared storage pool, perform inference on the input data based on the large language model to generate key-value cached data corresponding to the input data; Store the key-value cached data in the shared storage pool.

12. The device according to any one of claims 7 to 11, characterized in that, The historical key-value cached data includes historical key-value cached data corresponding to historical input texts of different users, historical input texts at different times, and historical input texts in different sessions.

13. A computing device, characterized in that, Comprising a processor, the processor is coupled to a memory, and the processor is configured to store instructions, which when executed by the processor, cause the electronic device to execute the method according to any one of claims 1 to 6.

14. A cluster of computing devices, characterized in that, Comprising at least one computing device, the computing device includes a processor, the processor is coupled to a memory, and the processor is configured to store instructions, which when executed by the processor, cause the computing device cluster to execute the method according to any one of claims 1 to 6.

15. A computer-readable storage medium having instructions stored thereon, characterized in that, When the instructions are executed, cause the computer to execute the method according to any one of claims 1 to 6.

16. A computer program product comprising instructions, characterized in that, When the instructions are executed, cause the computer to implement the method according to any one of claims 1 to 6.

Images