Inference method and apparatus for large language model

Abstract

An inference method and apparatus for a large language model, which method and apparatus are used for improving the inference calculation efficiency of a large language model. The method comprises: receiving an inference task, which carries a morpheme number, wherein the morpheme number is used for identifying a morpheme to be subjected to inference calculation, the morpheme comprising one or more sub-words determined on the basis of input text of a large language model (201); on the basis of the morpheme number corresponding to the inference task, querying a global index tree to determine a target inference node, wherein the global index tree comprises sub-trees corresponding to a plurality of inference nodes in a distributed inference cluster, and a sub-tree corresponding to the target inference node is a sub-tree in the global index tree, the matching degree of similarity of which sub-tree with the morpheme number is greater than a threshold, the matching degree of similarity indicating the number of pieces of reusable key-value cache data during inference calculation (202); and on the basis of the target inference node, executing the inference task (203).

Description

Inference method and device of large language model

[0001] The present application claims priority to the Chinese Patent Application No. 202410637806.5, filed on May 20, 2024, entitled "A data processing method, device and other equipment", and to the Chinese Patent Application No. 202410773361.3, filed on June 14, 2024, entitled "Inference method and device of large language model", the contents of which are incorporated herein by reference in their entirety. TECHNICAL FIELD

[0002] Embodiments of the present application relate to the field of cloud computing, and in particular to an inference method and device of a large language model. BACKGROUND

[0003] With the development of machine learning and other related technologies, large language models (LLM) based on machine learning technology have been widely applied. For example, a large language model can automatically generate language text or generate a reply, and can be used for tasks such as machine translation, speech recognition, question and answer systems, and dialogue generation.

[0004] In the current inference scheme of the large language model, a computing device implements inference of the large language model through a processor, such as a graphics processing unit (GPU) and a neural processing unit (NPU). In the process of the processor processing the inference of the large language model, since the key-value cache data generated by the large language model in the inference process of different input texts may be repeated, the processor can reuse the key-value cache data generated in the previous inference process, thereby reducing the inference calculation amount and improving the inference efficiency of the large language model.

[0005] However, in the process of the processor reusing the key-value cache data generated in the previous inference process to improve the inference efficiency, the processor needs to reuse as much key-value cache data generated in the previous inference process as possible to improve the inference efficiency. However, since the key-value cache data generated in the previous inference process is stored in different inference nodes, the processor randomly schedules the inference task to a single inference node in the inference calculation process. However, the inference node may not be the inference node that matches the most key-value cache data, resulting in that the key-value cache data is not maximally reused, and the inference efficiency of the large language model is reduced. SUMMARY

[0006] Embodiments of the present application provide a large language model inference method for improving the inference efficiency of the large language model. Embodiments of the present application also provide a large language model inference method corresponding to a language model inference device, a computing device, a computing device cluster, a computer-readable storage medium, and a computer program product.

[0007] In a first aspect, embodiments of the present application provide a large language model inference method. The method can be executed by a distributed inference cluster, or by a component of a distributed inference system, such as a processor, a chip, or a chip system of the distributed inference system, or by a logic module or software that can implement all or part of the functions of the distributed inference system. The method provided in the first aspect includes: receiving, by the distributed inference system, an inference task, the inference task being an inference task generated based on user input text of a large language model, the inference task carrying a morpheme number, the morpheme number being used to identify a morpheme to be inferred, the morpheme including one or more subwords determined based on the input text of the large language model. The distributed inference system queries a global index tree based on the morpheme number corresponding to the inference task to determine a target inference node, the global index tree including sub-trees corresponding to a plurality of inference nodes in the distributed inference cluster, wherein the sub-tree corresponding to the target inference node is a sub-tree in the global index tree that has a similarity matching degree greater than a threshold value with the morpheme number, for example, the sub-tree corresponding to the target inference node is a sub-tree with the highest similarity matching degree, and the similarity matching degree can be used to indicate the amount of reusable key-value cache data in inference calculation. The distributed inference system schedules the inference task to the target inference node and executes the inference task based on the target inference node.

[0008] In the embodiments of the present application, the distributed inference system can construct a global index tree, and in the scheduling process of the inference task, the inference scheduling system can perform similarity matching degree calculation in the global index tree based on the morpheme number in the inference task, so as to schedule the inference task to the target inference node with high similarity matching degree for execution. Compared with the random scheduling technology based on load condition in the prior art, because the scheduling in the embodiments of the present application is based on similarity matching degree, the distributed inference system can schedule the inference task to the target inference node that can reuse more key-value cache data for inference calculation, thereby reducing the inference calculation amount of the distributed inference system and improving the inference efficiency of the large language model.

[0009] In a possible implementation, in the process of querying the global index tree based on the morpheme number corresponding to the reasoning task, the distributed reasoning system calculates a similar matching degree of each sub-tree according to a proportion of a matching prefix length of the morpheme number in each sub-tree and a total length of the morpheme number, queries the sub-trees with a similar matching degree greater than a threshold based on the similar matching degrees of each sub-tree, and determines the reasoning nodes corresponding to the sub-trees with a similar matching degree greater than the threshold as target reasoning nodes. For example, the distributed reasoning system sorts the sub-trees according to the similar matching degrees, and determines the reasoning node corresponding to the sub-tree with the highest similar matching degree as the target reasoning node.

[0010] In the embodiments of the present application, the distributed reasoning system can calculate the similar matching degree of each sub-tree according to the proportion of the matching prefix length of the morpheme number in each sub-tree and the total length of the morpheme number, thereby improving the calculation feasibility of the similar matching degree.

[0011] In a possible implementation, the similar matching degree is positively correlated with the number of reusable key-value cache data in the reasoning process, that is, the higher the similar matching degree of the morpheme number and the reasoning node in the reasoning task, the higher the data amount of the reusable key-value cache data in the reasoning node, thereby reducing the reasoning calculation.

[0012] In the embodiments of the present application, the higher the similar matching degree of the morpheme number and the reasoning node in the reasoning task, the higher the data amount of the reusable key-value cache data in the reasoning node, and the distributed reasoning system performs reasoning calculation on the large language model based on the similar matching degree, thereby improving the reasoning efficiency of the large language model.

[0013] In a possible implementation, before the distributed reasoning system queries the global index tree based on the morpheme number corresponding to the reasoning task, when the reasoning node generates key-value cache data when processing the historical input text, the distributed reasoning system can create the global index tree based on the key data of the key-value cache data, wherein the key data includes a hash value corresponding to the input text.

[0014] In the embodiments of the present application, before the distributed reasoning system queries the global index tree based on the morpheme number corresponding to the reasoning task, the distributed reasoning system can create the global index tree based on the key data of the historical key-value cache data. Since the global index tree only contains the key data, the query efficiency of the global index tree is improved.

[0015] In a possible implementation, the reasoning node is deployed with a relational vector cache RTC, and the reasoning node deployed with the relational vector cache can also be referred to as an RTC node. The relational vector cache is used to store the key-value cache data generated in the reasoning process of the large language model.

[0016] In the implementation of the present application, a relational vector cache RTC is deployed in the inference node. The relational vector cache can store key-value cache data generated in the inference process of the large language model, and the stored key-value cache data can be reused in the subsequent inference process, thereby improving the inference efficiency of the large language model.

[0017] In a possible implementation, in the process of executing the inference task by the target inference node based on the distributed inference system, the distributed inference cluster determines the key-value cache data in the target inference node that matches the inference task, that is, the distributed inference system performs accurate matching of the key-value cache data, and reuses the value data of the key-value cache data in the target inference node that matches the inference task to perform inference calculation, wherein the value data includes hidden layer output data of the large language model.

[0018] In the implementation of the present application, the distributed inference system can reuse the value data of the key-value cache data stored in the target inference node to perform inference calculation in the process of executing the inference task, thereby reducing the amount of inference calculation and improving the inference efficiency of the large language model.

[0019] In a possible implementation, in the process of determining the key-value cache data in the target inference node that matches the inference task by the distributed inference system, the distributed inference system matches the morpheme number in the inference task with the key-value cache data in the target inference node, determines the key-value cache data that matches the inference task, and the key-value cache data that matches the inference task is the key-value cache data that can be reused in the inference process.

[0020] In the implementation of the present application, the distributed inference system matches the key-value cache data in the target inference node with the morpheme number in the inference task, determines the matched key-value cache data, and reuses the key-value cache data for inference calculation, thereby reducing the amount of inference calculation of the distributed cluster and improving the inference efficiency of the large language model.

[0021] In a possible implementation, before receiving the inference task by the distributed inference system, the distributed inference system receives an input text, and the input text includes a word or a sentence input by a user to the large language model, for example, a question asked by the user. The distributed inference system first performs a word segmentation operation on the input text to determine one or more morphemes contained in the input text, and then performs inference calculation based on the one or more morphemes to generate one or more inference tasks, which are used to schedule to multiple inference nodes in the distributed inference cluster for execution.

[0022] In the implementation of the present application, before receiving the inference task by the distributed inference system, the distributed inference system performs a word segmentation operation on the input text of the large language model to determine one or more morphemes, and performs inference calculation based on the one or more morphemes to generate the inference task, thereby improving the realizability of generating the inference task.

[0023] In a possible implementation, the inference node includes one or more of the following: a graphics processing unit (GPU), a neural processing unit (NPU), and a data processing unit (DPU). The GPU, the NPU, and the DPU can be deployed in a single computing device or in multiple different computing devices.

[0024] In the embodiments of the present application, the inference node can be a GPU, an NPU, and a DPU deployed in one or more computing devices, and various implementations of the inference node improve the richness of the scheme.

[0025] In a possible implementation, in the process of constructing the global index tree, the inference scheduling system can construct a subtree of the global index tree according to an index range of key data in the key-value cache data, and after constructing the subtree of the global index tree according to the index range, when the inference scheduling system receives an inference task, the inference scheduling system can determine a subtree corresponding to the index range based on a morpheme number of the inference task, and determine an inference node corresponding to the subtree as a target inference node. Then, the inference scheduling system schedules the inference task to the target inference node.

[0026] In the embodiments of the present application, the inference scheduling system can construct a subtree of the global index tree based on an index range of key-value cache data, so that the inference scheduling system can determine a target inference node based on an index range to which a morpheme number of an inference task belongs, improve the query efficiency of the morpheme number, and further improve the inference efficiency of the large language model.

[0027] In a second aspect, the embodiments of the present application provide an inference device of a large language model, which includes a transceiver unit and a processing unit. The transceiver unit is configured to receive an inference task, the inference task carrying a morpheme number, the morpheme number being used to identify a morpheme to be inferred, and the morpheme including one or more subwords determined based on an input text of the large language model. The processing unit is configured to query a global index tree based on a morpheme number corresponding to the inference task, determine a target inference node, the global index tree including subtrees corresponding to a plurality of inference nodes in a distributed inference cluster, and the target inference node corresponding to a subtree in the global index tree that has a similarity matching degree greater than a threshold with the morpheme number, the similarity matching degree being used to indicate a number of key-value cache data that can be reused in an inference process. The processing unit is further configured to execute the inference task based on the target inference node.

[0028] In a possible implementation, the processing unit is specifically configured to calculate a similarity matching degree of each sub-tree according to the prefix length of each sub-tree and the total length of the morpheme number, query the sub-trees with the similarity matching degree greater than a threshold based on the similarity matching degree of each sub-tree, and determine the inference nodes corresponding to the sub-trees with the similarity matching degree greater than the threshold as the target inference nodes.

[0029] In a possible implementation, the similarity matching degree is positively correlated with the number of key-value cache data that can be reused in the inference process.

[0030] In a possible implementation, the processing unit is further configured to generate key-value cache data when the inference node generates the key-value cache data, and create a global index tree based on key data of the key-value cache data, the key data including a hash value corresponding to the input text.

[0031] In a possible implementation, the inference node is deployed with a relational vector cache (RTC) for storing key-value cache data generated in the inference process of the large language model.

[0032] In a possible implementation, the processing unit is specifically configured to determine the key-value cache data matching the inference task in the target inference node, and reuse the key-value cache data matching the inference task in the target inference node to perform inference calculation, the value data including a hidden layer output of the large language model.

[0033] In a possible implementation, the processing unit is specifically configured to match the morpheme number in the inference task with the key-value cache data in the target inference node, and determine the matched key-value cache data, the matched key-value cache data being the key-value cache data that can be reused in the inference process.

[0034] In a possible implementation, the transceiving unit is further configured to receive the input text, the input text including words or sentences input by a user to the large language model. The processing unit is further configured to perform a word segmentation operation on the input text, and determine one or more morphemes included in the input text. The processing unit is further configured to perform inference calculation based on the one or more morphemes, and generate one or more inference tasks, the one or more inference tasks being used to schedule execution to one or more inference nodes in the distributed inference cluster.

[0035] In a possible implementation, the inference node includes one or more of a graphics processing unit (GPU), a neural network processing unit (NPU), and a data processing unit (DPU).

[0036] In a third aspect, an embodiment of the present application provides a computing device, including a processor, the processor being coupled with a memory, and the processor being configured to store instructions, when the instructions are executed by the processor, to cause the computing device to perform the method in the first aspect or any possible implementation of the first aspect.

[0037] In a fourth aspect, an embodiment of the present application provides a computing device cluster, the computing device cluster comprising one or more computing devices, the computing device comprising a processor coupled with a memory, the processor configured to store instructions that, when executed by the processor, cause the computing device cluster to perform the method of the first aspect or any possible implementation of the first aspect.

[0038] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium having stored thereon instructions that, when executed by a computer, cause the computer to perform the method of the first aspect or any possible implementation of the first aspect.

[0039] In a sixth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising instructions that, when executed by a computer, cause the computer to implement the method of the first aspect or any possible implementation of the first aspect.

[0040] It can be understood that the beneficial effects achievable by any of the above-provided inference apparatuses of large language models, computing devices, computing device clusters, computer-readable media, or computer program products can refer to the beneficial effects in the corresponding methods, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0041] FIG. 1 is a schematic diagram of a system architecture of an inference system of a large language model according to an embodiment of the present application;

[0042] FIG. 2 is a schematic diagram of a flow of an inference method of a large language model according to an embodiment of the present application;

[0043] FIG. 3 is a schematic diagram of a flow of another inference method of a large language model according to an embodiment of the present application;

[0044] FIG. 4 is a schematic diagram of a flow of another inference method of a large language model according to an embodiment of the present application;

[0045] FIG. 5 is a schematic diagram of a flow of another inference method of a large language model according to an embodiment of the present application;

[0046] FIG. 6 is a schematic diagram of a structure of an inference apparatus of a large language model according to an embodiment of the present application;

[0047] FIG. 7 is a schematic diagram of a structure of a computer device according to an embodiment of the present application;

[0048] FIG. 8 is a schematic diagram of a structure of a computing device cluster according to an embodiment of the present application;

[0049] FIG. 9 is a schematic diagram of another structure of a computing device cluster according to an embodiment of the present application. DETAILED DESCRIPTION

[0050] The embodiments of the present application provide a reasoning method and device of a large language model, which are used to improve the reasoning scheduling efficiency of the large language model.

[0051] The terms "first", "second", "third", "fourth" and the like in the description and claims of the present application and above-mentioned drawings, if any, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so construed herein is merely for convenience and brevity and that one of skill in the art will appreciate that steps illustrated and / or described can be carried out in any order and / or simultaneously, except where order or sequence is specifically dictated by functional aspects of the embodiments. Furthermore, the term "comprising" and variations thereof, as used in describing and claiming the embodiments of the present application, are intended to be given their broadest interpretation to include both active and inactive steps or components. In other words, the term "comprising" should not be interpreted as being restricted to the listed steps or components or their variants.

[0052] In the embodiments of the present application, the word "exemplary" or "for example" is used to mean serving as an example, instance, or illustration. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the embodied words are used to present concepts in a concrete manner.

[0053] First, some terms involved in the embodiments of the present application are introduced, which are convenient for the skilled in the art to understand the technical solutions.

[0054] A large language model (LLM) is a natural language processing model based on deep learning technology, which is trained using a large amount of text data and has the ability to understand, generate and process natural language. The large language model regards natural language text as a sequence of data, such as a sequence of words or characters, and models the statistical rules and potential semantic information of these sequence data through a deep learning model. The large language model can perform a wide range of tasks, including text summarization, translation, sentiment analysis, question answering, and dialogue.

[0055] A token refers to the smallest semantic unit after the tokenization of language, which can also be referred to as tokenization or morphological tagging. A token can be a Chinese character or multiple Chinese characters, an English word or half an English word. For example, "apple" is a token, and "apple tree" can be two tokens "apple" + "tree", but "ping" alone does not represent any meaning.

[0056] A relational tensor cache (RTC) is a mechanism used in large language models to manage and store relational tensors. In large neural networks, caches are often used to store intermediate results or key information in order to improve computational efficiency and save memory space. The RTC serves to store frequently used relational tensors during model training, thereby avoiding repeated calculations. Relational tensors typically represent relationships between entities or associations between contexts, such as semantic or grammatical relationships between words in natural language processing tasks.

[0057] Context refers to the contextual information relied upon by a large language model when processing text, i.e., the context environment upon which the large language model relies when generating or understanding text. For a large language model, context can be a piece of text, a conversation, a question, or any other form of text input.

[0058] To make the technical solutions of the present application clearer and easier to understand, the system architecture of the present application will be introduced below in conjunction with the drawings.

[0059] Please refer to FIG. 1, which is a system architecture diagram of a large language model inference system provided by an example of the present application. In the system architecture shown in FIG. 1, the inference system 10 of the large language model includes a global scheduling layer 101 and a relational tensor cache layer 102, wherein the global scheduling layer 101 includes a global scheduler 1011, and the relational tensor cache layer 102 includes one or more inference nodes 1021, and the plurality of inference nodes 1021 can form a distributed inference cluster. The specific functions of each part of the inference system 10 of the large language model will be introduced below.

[0060] The global scheduling layer 101 is used to schedule inference tasks of the large language model, for example, the global scheduler 1011 of the global scheduling layer 101 schedules inference tasks of the large language model to different inference nodes 1021 (1022) in the relational tensor cache layer 102 for inference calculation, thereby achieving load balancing of the large language model inference calculation. The global scheduling layer 101 in the embodiment of the present application can also create and store a global index tree, which can match an inference task with an inference node 1021 that executes the inference task, and the determined inference node is called a target inference node.

[0061] The global index tree created in the global scheduling layer 101 includes a subtree corresponding to each inference node 1021, and each subtree stores key data in the key value cache (KV cache) data generated by the inference node 1021 in executing a historical inference task. The global scheduling layer 101 can determine which inference node 1021 corresponding to a subtree executes an inference task based on the similarity matching degree of the morpheme number in the inference task and the key data in each subtree, thereby achieving scheduling of the inference task.

[0062] The relational vector cache layer 102 is configured to store the key value cache data generated by the distributed inference cluster in executing a historical inference task of the large language model. The relational vector cache layer 102 includes one or more inference nodes 1021, which can also be referred to as relational vector cache nodes, and each inference node 1021 stores the key value cache data generated by the inference node in executing a historical inference task. The one or more inference nodes 1021 form a distributed inference cluster, and the large language model is deployed in the distributed inference cluster. Therefore, the inference system 10 of the large language model can also be referred to as a distributed inference system, and the large language model can perform inference calculation on the input text of the user to generate corresponding output text.

[0063] For example, the distributed inference cluster can receive a natural language text as input text, and the input text can be a sentence, a paragraph, or a longer text sequence. The distributed inference cluster performs inference calculation on the input text and reference information based on the large language model to generate output text, and the output text can be an answer to the question corresponding to the input text, a text summary, or a translation result, etc.

[0064] The model structure of the large language model deployed in the distributed inference cluster is a multi-layer structure, for example, the multi-layer structure includes an embedding layer, a self-attention layer, and a feedforward network layer. The embedding layer is configured to convert each morpheme in the input text and the reference information of the user into a vector representation. The self-attention layer is configured to determine the context relationship of the input text, for example, the self-attention layer performs self-attention calculation on the vector representation corresponding to the input text to generate a context representation corresponding to the input text. The feedforward network layer is configured to perform nonlinear transformation after the self-attention layer to extract high-level semantic features in the input text and generate a feature representation.

[0065] The inference stage of the large language model includes a prefill stage and a decode stage. The prefill stage, also known as the first word inference stage or the full inference stage, is a stage in which the computing device inputs the complete input text into the large language model for a first round of inference to generate the first output word of the output text. The decode stage, also known as the incremental inference stage, is a stage in which the computing device can iteratively infer and calculate the next output word based on the generated output text through a self-recurrent cycle.

[0066] It should be noted that the inference nodes 1021 in the distributed inference cluster described above can be different graphics processing units (GPUs), neural network processing units (NPUs), or data processing units (DPUs) of a single device, or can be graphics processing units (GPUs), neural network processing units (NPUs), or data processing units (DPUs) distributed in different devices, and the specific implementation is not limited.

[0067] Based on the inference system 10 of the large language model shown in FIG. 1, the present application also provides an inference method of a large language model. The inference method of the large language model provided by the embodiments of the present application will be introduced below in combination with the embodiments.

[0068] Referring to FIG. 2, FIG. 2 is a flowchart of a scheduling method of a large language model according to an embodiment of the present application. In the example shown in FIG. 2, the method includes the following steps:

[0069] Step 201. The inference system receives an inference task, and the inference task carries a morpheme number, which is used to identify a morpheme to be inferred.

[0070] In the process of processing the input text of the user, the inference scheduling system 10 receives an inference task, which is an inference task generated based on the input text of the large language model. The inference task carries a morpheme number, which is used to identify a morpheme to be inferred. The morpheme number can also be referred to as a token ID. The morpheme includes one or more subwords determined based on the input text of the large language model.

[0071] In one possible implementation, before receiving the inference task, the inference system 10 receives an input text of the large language model, and then performs a tokenization operation on the input text to determine one or more morphemes contained in the input text, wherein the input text includes a word or a sentence input by a user to the large language model, and the morpheme includes a sub-word after the tokenization of the input text. The inference system 10 performs inference calculation based on the one or more morphemes after the tokenization to generate one or more inference tasks, and the one or more inference tasks are used to schedule the execution to one or more inference nodes 1021 in the distributed inference cluster 10. Wherein the inference task contains a morpheme number, and the morpheme number is used to identify the morpheme to be inferred.

[0072] Specifically, in the process of generating the inference task by the inference system 10 based on the inference calculation of the one or more morphemes, the inference system 10 first encodes the one or more morphemes after the tokenization to obtain a morpheme number corresponding to the morpheme, and the morpheme number is a digitized morpheme. The inference system 10 further performs inference calculation based on the morpheme number to generate a plurality of inference tasks, and the plurality of inference tasks are scheduled to be executed to one or more inference nodes 1021 in the distributed inference cluster 10. The inference calculation includes embedding layer calculation, self-attention layer calculation, and feedforward network layer calculation, etc.

[0073] For example, in one example of step 201, the input text of the large language model input by the user is "today how is the weather", and the inference system 10 performs a tokenization operation on the input text to determine that the morphemes contained in the input text include "today", "weather" and "how". The inference system 10 further encodes the morphemes to obtain morpheme numbers, for example, the morpheme number corresponding to "today" is 100, the morpheme number corresponding to "weather" is 200, and the morpheme number corresponding to "how" is 300. The inference task generated by the inference system 10 contains the above morpheme numbers.

[0074] It should be noted that in addition to carrying the morpheme numbers of the morphemes in the current input text of the user in the inference task, the inference task can also contain morpheme numbers corresponding to the context, which can also be referred to as context sequence. The context includes the historical input text of the user.

[0075] Step 202. The inference system queries the global index tree based on the morpheme number corresponding to the inference task to determine the target inference node, and the global index tree includes a plurality of sub-trees corresponding to the inference nodes in the distributed inference cluster. The sub-tree corresponding to the target inference node is a sub-tree in the global index tree with a similarity matching degree greater than a threshold value with the morpheme number, and the similarity matching degree is used to indicate the number of key-value cache data that can be reused in the inference calculation.

[0076] In the process of scheduling the inference task to the plurality of inference nodes 1021 of the distributed inference cluster by the inference system 10, the inference system 10 queries the global index tree based on the morpheme number corresponding to the inference task, and determines the target inference node according to the query result of the global index tree. The global index tree includes the plurality of sub-trees corresponding to the plurality of inference nodes in the distributed inference cluster, and the plurality of sub-trees corresponding to the plurality of inference nodes are also called relationship vector cache (RTC) sub-trees. The sub-tree corresponding to the target inference node is the sub-tree in the global index tree with a similarity matching degree greater than a threshold value with the morpheme number. The similarity matching degree is used to indicate the number of key-value cache data that can be reused in inference calculation. The query result of the global index tree is the similarity matching degree of each sub-tree in the global index tree.

[0077] Referring to FIG. 3, FIG. 3 is a schematic diagram of another inference method of a large language model provided by an embodiment of the present application. In the example shown in FIG. 3, the global index tree includes sub-trees corresponding to a plurality of inference nodes in the distributed inference cluster, and the inference system 10 needs to schedule the inference task to the inference nodes corresponding to the sub-trees based on the global index tree. For example, the global index tree shown in FIG. 3 includes four sub-trees corresponding to four inference nodes, namely sub-tree 1, sub-tree 2, sub-tree 3, and sub-tree 4. Each sub-tree corresponds to an inference node, and the global index tree stores the key data in the key-value cache data. The key data includes the morpheme number of the morpheme in the historical input text of the user.

[0078] In the example shown in FIG. 3, the inference system 10 can divide the sub-trees by index ranges, for example, the index range 1-10k of the global index tree. The index range 1-2k is a sub-tree, namely sub-tree 1. The index range 2-4k is a sub-tree, namely sub-tree 2. The index range 4-6k is a sub-tree, namely sub-tree 3. The index range 8-10k is a sub-tree, namely sub-tree 4.

[0079] In a possible implementation, when the inference system 10 calculates the similarity matching degree of the morpheme number with each sub-tree, the inference system 10 calculates the similarity matching degree of each sub-tree according to the matching prefix length of the morpheme number in each sub-tree and the total length of the morpheme number. The inference system 10 queries the sub-tree with a similarity matching degree greater than a threshold value based on the similarity matching degree of each sub-tree, and determines the inference node corresponding to the sub-tree with the similarity matching degree greater than the threshold value as the target inference node. For example, the inference system 10 determines the inference node corresponding to the sub-tree with the highest similarity matching degree as the target inference node.

[0080] When the inference system 10 calculates the similarity matching degree of the morpheme number with each sub-tree based on the global index tree in the embodiment of the present application, the similarity matching degree satisfies the following formula:

[0081] The similarity matching degree = L1 / L2.

[0082] Wherein, L1 represents the matching length of the morpheme number in each subtree of the global index tree, and L2 represents the total length of the morpheme number. Wherein, the similarity matching degree ranges from 0 to 100%.

[0083] In the example shown in FIG. 3, during the process of querying the global index tree, the inference system 10 respectively calculates the similarity matching degrees of the morpheme number and each subtree of the global index tree, and determines the subtree corresponding to the target inference node according to the similarity matching degrees of each subtree.

[0084] For example, the inference system 10 calculates the similarity matching degree of the morpheme number and the subtree 1 as 81%, the similarity matching degree of the morpheme number and the subtree 2 as 79%, the similarity matching degree of the morpheme number and the subtree 3 as 83%, and the similarity matching degree of the morpheme number and the subtree 4 as 70%, and then the inference system 10 determines that the inference node corresponding to the subtree 3 is the target inference node.

[0085] In a possible implementation, before the inference system 10 queries the global index tree based on the morpheme number corresponding to the inference task, the inference system 10 needs to construct the global index tree first, that is, after the inference node 1021 generates the key-value cache data by performing the inference calculation of the history, the inference system 10 creates the global index tree based on the key data of the key-value cache data of each inference node, and the key data includes the hash value corresponding to the input text.

[0086] It can be understood that, since the inference system 10 creates the global index tree based on the key data in the history key-value cache data of each inference node, that is, the global index tree only stores the key data of the key-value cache data of each inference node, and does not contain the value data in the key-value cache data, therefore, the global index tree can realize the fast matching of the morpheme number, and the global index tree can also be called a global affinity index tree.

[0087] In another possible implementation of the embodiments of the present application, during the process of constructing the global index tree, the inference system 10 can construct the subtree of the global index tree according to the index range of the key data in the key-value cache data, and after the inference system 10 constructs the subtree of the global index tree according to the index range, when the inference system 10 receives the inference task, the inference system 10 can also determine the corresponding subtree based on the index range to which the morpheme number carried in the inference task belongs, and schedule the inference task to the inference node corresponding to the subtree.

[0088] For example, in the example shown in FIG. 3, the index range of the global index tree is 1-10k, wherein the index range of 1-2k is a subtree, i.e., subtree 1, the index range of 2-4k is a subtree, i.e., subtree 2, the index range of 4-6k is a subtree, i.e., subtree 3, and the index range of 8-10k is a subtree, i.e., subtree 4. If the morpheme number carried in the inference task received by the inference system 10 is 3004, then the inference system 10 determines that the corresponding subtree is subtree 2 based on the index range to which the morpheme number carried in the inference task belongs, and the inference system 10 schedules the inference task to the inference node corresponding to the subtree 2.

[0089] It can be understood that, after the inference system 10 constructs the subtrees of the global index tree based on the index range, the inference system 10 determines the subtree based on the index range to which the morpheme number belongs, at this time, the inference system 10 does not need to perform prefix matching and calculate the similarity matching degree based on the morpheme number, thereby improving the query efficiency of the morpheme number.

[0090] In a possible implementation, the inference node 1021 is deployed with a relational vector cache RTC, which can also be referred to as a relational vector cache node, and the relational vector cache is used to store key-value cache data generated in the large language model inference process.

[0091] Step 203. The inference system executes the inference task based on the target inference node.

[0092] After the inference system 10 determines the target inference node, the inference system 10 schedules the inference task to the target inference node, and the target inference node executes the inference task. In the process of executing the inference task based on the target inference node, the inference system 10 determines the key-value cache data in the target inference node that matches the inference task, and reuses the value data of the key-value cache data in the target inference node that matches the inference task to execute inference calculation, wherein the value data includes the hidden layer output of the large language model.

[0093] The process of the inference system 10 executing the inference task in the embodiments of the present application includes a pre-filling stage and a decoding stage, the pre-filling stage is also referred to as a full-inference stage and an incremental stage, and the decoding stage is also referred to as an incremental-inference stage. In the pre-filling stage, the computing device inputs the complete input text to the large language model at one time to perform the first round of inference, and generates the first output word of the output text. In the decoding stage, the computing device can continuously generate the next output word based on the self-recurrence cycle iteration of the generated output text.

[0094] Referring to FIG. 4, FIG. 4 is a flow diagram of performing an inference task according to an embodiment of the present disclosure. In the example shown in FIG. 4, after the target inference node receives the inference task, the processing of the inference task goes through a full-inference stage and an incremental-inference stage. In the full-inference stage, the target inference node can perform embedding layer calculation on the morpheme number in the inference task to obtain a vector representation corresponding to the morpheme number, and then the target inference node performs self-attention layer calculation on the vector representation to obtain key-value cache data,

[0095] In the example shown in FIG. 4, the target inference node can also reuse the key-value cache data generated in the full-inference stage in the incremental-inference stage, thereby reducing the inference calculation amount in the incremental-inference stage. In the calculation process of the self-attention layer, the target inference node mines and integrates the key information in the vector representation, and further calculates and optimizes through the feedforward network layer, and finally converts the generated vector representation into an output text.

[0096] In a possible implementation, in the process of determining the key-value cache data in the target inference node that matches the inference task by the inference system 10, the inference system 10 matches the morpheme number in the inference task with the key-value cache data in the target inference node, which is an exact match between the morpheme number and the key-value cache data, so as to determine the matched key-value cache data, which is the reusable key-value cache data in the inference calculation.

[0097] Referring to FIG. 5, FIG. 5 is another schematic diagram of inference scheduling of a large language model according to an embodiment of the present disclosure. In diagram (a) of the example shown in FIG. 5, after the inference system 10 determines the target inference node by querying the global index tree, the target inference node determines the reusable key-value cache data based on the morpheme encoding in the inference task. In the process of determining the reusable key-value cache data by the target inference node, since the target inference node stores the key-value cache data based on a tree structure, when the target inference node performs exact matching based on the morpheme number and the cached key-value cache data, the target inference node retrieves the reusable key-value cache data based on the tree structure.

[0098] In diagram (b) of the example shown in FIG. 5, for example, when the target inference node processes the historical input text to obtain the output text “I love playing basketball”, and when the target inference node processes the input text again to obtain the output text “I love China”, since “I love playing basketball” and “I love China” have the same morpheme, that is, there is reusable key-value cache data in the inference process corresponding to the two output texts, therefore, when the target inference node generates the output text “I love China”, the target inference node can reuse the key-value cache data corresponding to the previous output text “I love playing basketball” to perform inference calculation, and the reused key-value cache data is, for example, the calculation output of the self-attention layer corresponding to “I love”.

[0099] As can be seen from the above embodiments, in the scheduling process of the inference task, the distributed inference system in the embodiments of the present application calculates the similarity matching degree based on the morpheme number in the inference task, so as to schedule the inference task to the target inference node with high similarity matching degree, so that the distributed inference system can reuse more key-value cache data for inference calculation, thereby reducing the inference calculation amount of the distributed inference system and improving the inference efficiency of the large language model.

[0100] Based on the above method embodiments, the embodiments of the present application also provide an inference scheduling device of a large language model. The inference scheduling device of a large language model provided by the embodiments of the present application is described in detail below.

[0101] Referring to FIG. 6, FIG. 6 is a structural schematic diagram of an inference device of a large language model provided by the embodiments of the present application. In the example shown in FIG. 6, the inference device of a large language model 600 is used to implement each step performed by the inference system 10 in each of the above embodiments, and the inference device of a large language model 600 includes a transceiver unit 601 and a processing unit 602.

[0102] The transceiver unit 601 is configured to receive an inference task, and the inference task carries a morpheme number, which is used to identify a morpheme to be inferred, and the morpheme includes one or more subwords determined based on an input text of a large language model. The processing unit 602 is configured to query a global index tree based on the morpheme number corresponding to the inference task, and determine a target inference node. The global index tree includes sub-trees corresponding to a plurality of inference nodes in a distributed inference cluster, and the sub-tree corresponding to the target inference node is a sub-tree with a similarity matching degree greater than a threshold in the global index tree, and the similarity matching degree is used to indicate the number of reusable key-value cache data in the inference calculation. The processing unit 602 is further configured to execute the inference task based on the target inference node.

[0103] In a possible implementation, the processing unit 602 is specifically configured to calculate the similarity matching degree of each sub-tree according to the matching prefix length of each sub-tree and the total length of the morpheme number, query the sub-tree with a similarity matching degree greater than a threshold based on the similarity matching degree of each sub-tree, and determine the inference node corresponding to the sub-tree with a similarity matching degree greater than a threshold as the target inference node.

[0104] In a possible implementation, the similarity matching degree is positively correlated with the number of reusable key-value cache data in the inference process.

[0105] In a possible implementation, the processing unit 602 is further configured to create a global index tree based on the key data of the key-value cache data when the inference node generates the key-value cache data, and the key data includes a hash value corresponding to the input text.

[0106] In a possible implementation, the inference node is deployed with a relational vector cache RTC, which is configured to store key-value cache data generated in a large language model inference process.

[0107] In a possible implementation, the processing unit 602 is specifically configured to determine key-value cache data in the target inference node that matches the inference task, and reuse value data of the key-value cache data stored in the target inference node to perform inference calculation, the value data including hidden layer output of the large language model.

[0108] In a possible implementation, the processing unit 602 is specifically configured to match the morpheme number in the inference task with the key-value cache data in the target inference node, and determine the matched key-value cache data, the matched key-value cache data being key-value cache data that can be reused in the inference calculation.

[0109] In a possible implementation, the transceiving unit 601 is further configured to receive input text, the input text including words or sentences input by a user to the large language model. The processing unit 602 is further configured to perform a word segmentation operation on the input text, and determine one or more morphemes included in the input text. The processing unit 602 is further configured to perform inference calculation based on the one or more morphemes, and generate one or more inference tasks, the one or more inference tasks being configured to be scheduled to a plurality of inference nodes in a distributed inference cluster for execution.

[0110] In a possible implementation, the inference node includes a graphics processing unit (GPU) and a neural network processing unit (NPU).

[0111] It can be understood that the transceiving unit 601 and the processing unit 602 in the inference scheduling apparatus 600 of the large language model can be mapped to the modules in the inference scheduling system 10 of the large language model in FIG. 1 as functional modules, thereby realizing the functions of the modules in the inference scheduling system 10 of the large language model.

[0112] It should be understood that the division of units in the above apparatus is only a logical functional division, and all or part of them can be integrated into a physical entity or physically separated when actually implemented. The units in the apparatus can all be implemented in the form of software invoked by a processing element; all in the form of hardware; or some units in the form of software invoked by a processing element and some units in the form of hardware. For example, each unit can be a separately established processing element, or can be integrated in a chip of the apparatus, in addition, it can also be stored in the form of a program in a memory, and the function of the unit is invoked and executed by a processing element of the apparatus. In addition, all or part of these units can be integrated together or independently implemented. The processing element described herein can be a processor, which can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each unit can be implemented by integrated logic circuits of hardware in the processing element or in the form of software invoked by the processing element.

[0113] It should be noted that, for the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, and those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.

[0114] Other reasonable step combinations that those skilled in the art can think of based on the above description are also within the protection scope of the present application. In addition, those skilled in the art should also be familiar with the fact that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the present application.

[0115] Please refer to FIG. 7, which is a structural schematic diagram of a computing device provided by an embodiment of the present application. As shown in FIG. 7, the computing device 700 includes a processor 701, a memory 702, a communication interface 703 and a bus 704, and the processor 701, the memory 702 and the communication interface 703 are coupled through a bus (not labeled in the figure). The memory 702 stores instructions, and when the execution instructions in the memory 702 are executed, the computing device 700 executes the method executed by the computing device in the above method embodiments.

[0116] The computing device 700 can be one or more integrated circuits (ICs) configured to implement one or more of the above methods, e.g., one or more application specific integrated circuits (ASICs), or, one or more digital signal processors (DSPs), or, one or more field programmable gate arrays (FPGAs), or a combination of at least two of these forms of ICs. As another example, when units of the apparatus can be implemented by way of a processing element scheduler, the processing element can be a general purpose processor, e.g., a central processing unit (CPU) or other processor that can invoke a program. As another example, these units can be integrated together in a system-on-a-chip (SOC) form.

[0117] The processor 701 can be a central processing unit (CPU), and can also be other general purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, transistor logic devices, hardware components that perform a special function, or any combination thereof. The general purpose processor can be a microprocessor or any conventional processor.

[0118] The memory 702 can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous dynamic RAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0119] The executable program code is stored in the memory 702, and the processor 701 executes the executable program code to respectively realize the functions of the foregoing units or modules, thereby realizing the inference scheduling method of the large language model. That is, the instructions for executing the inference scheduling method of the large language model are stored on the memory 702.

[0120] The communication interface 703 uses a transceiving module such as, but not limited to, a network interface card, a transceiver, to realize the communication between the computing device 700 and other devices or communication networks.

[0121] The bus 704 can include, in addition to a data bus, a power bus, a control bus, and a state signal bus, etc. The bus can be a peripheral component interconnect express (PCIe) bus, or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0122] Referring to FIG. 8, FIG. 8 is a schematic diagram of a computing device cluster provided by an embodiment of the present application. As shown in FIG. 8, the computing device cluster 800 includes at least one computing device 700.

[0123] As shown in FIG. 8, the computing device cluster 800 includes at least one computing device 700. The memory 702 in one or more computing devices 700 in the computing device cluster 800 can have the same instructions for performing the inference scheduling method of the large language model described above.

[0124] In some possible implementations, the memory 702 of one or more computing devices 700 in the computing device cluster 800 can also respectively have partial instructions for performing the inference scheduling method of the large language model described above. In other words, the combination of one or more computing devices 700 can collectively execute the instructions for performing the inference scheduling method of the large language model described above.

[0125] It should be noted that the memories 702 in different computing devices 700 in the computing device cluster 800 can store different instructions, respectively used to perform part of the functions of the inference scheduling apparatus of the large language model described above. That is, the instructions stored in the memories 702 in different computing devices 700 can implement the functions of one or more modules in the obtaining unit and the processing unit.

[0126] In some possible implementations, one or more computing devices 700 in the computing device cluster 800 can be connected through a network. The network can be a wide area network or a local area network, etc.

[0127] Referring to FIG. 9, FIG. 9 is a schematic diagram of the connection of the computer devices in a computer cluster through a network according to an embodiment of the present application. As shown in FIG. 9, two computer devices 700A and 700B are connected through a network. Specifically, the communication interface in each computer device is connected to the network.

[0128] In a possible implementation, the memory in the computer device 700A stores instructions for performing the function of the obtaining unit. Meanwhile, the memory in the computer device 700B stores instructions for performing the functions of the processing unit and the display unit.

[0129] It should be understood that the functions of the computer device 700A shown in FIG. 9 can also be completed by multiple computer devices. Similarly, the functions of the computer device 700B can also be completed by multiple computer devices.

[0130] In another embodiment of the present application, a computer readable storage medium is also provided, which stores computer execution instructions. When the processor of a device executes the computer execution instructions, the device performs the method performed by the inference scheduling system in the above method embodiments.

[0131] In another embodiment of the present application, a computer program product is also provided, which includes computer execution instructions stored in a computer readable storage medium. When the processor of a device executes the computer execution instructions, the device performs the method performed by the inference scheduling system in the above method embodiments.

[0132] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the above method embodiments, which will not be described here.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely schematic, for example, the division of the units is merely a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0134] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0135] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0136] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present application essentially or the part that contributes to the prior art or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, read-only memory), a random access memory (RAM, random access memory), a magnetic disk or an optical disk, and various program code storage media.

Claims

1. An inference method of a large language model, characterized by, The method is applied to a distributed reasoning cluster including a plurality of reasoning nodes, and the method comprises: receiving a reasoning task carrying a morpheme number, wherein the morpheme number is used to identify a morpheme to be reasoned, and the morpheme comprises one or more subwords determined based on an input text of a large language model; querying a global index tree based on the morpheme number corresponding to the reasoning task to determine a target reasoning node, wherein the global index tree comprises sub-trees corresponding to a plurality of reasoning nodes in the distributed reasoning cluster, the sub-tree corresponding to the target reasoning node is a sub-tree in the global index tree having a similarity matching degree greater than a threshold with the morpheme number, and the similarity matching degree is used to indicate a number of key-value cache data reusable in a reasoning process; executing the reasoning task based on the target reasoning node.

2. The method of claim 1, wherein, The querying of the global index tree based on the morpheme number corresponding to the reasoning task comprises: calculating a similarity matching degree of each sub-tree according to a matching prefix length of each sub-tree and a total length of the morpheme number; querying a sub-tree having a similarity matching degree greater than a threshold based on the similarity matching degree of each sub-tree, and determining a reasoning node corresponding to the sub-tree having the similarity matching degree greater than the threshold as the target reasoning node.

3. The method according to claim 1 or 2, characterized in that, The similarity matching degree is positively correlated with the number of key-value cache data reusable in the reasoning process.

4. The method according to any one of claims 1 to 3, characterized in that, Before the querying of the global index tree based on the morpheme number corresponding to the reasoning task, the method further comprises: when the reasoning node generates key-value cache data, creating the global index tree based on key data of the key-value cache data, wherein the key data comprises a hash value corresponding to the input text.

5. The method according to any one of claims 1 to 4, characterized in that, The reasoning node is deployed with a relational vector cache, and the relational vector cache is used to store key-value cache data generated in a reasoning process of the large language model.

6. The method according to any one of claims 1 to 5, characterized in that, The execution of the reasoning task based on the target reasoning node comprises: determining key-value cache data matching the reasoning task in the target reasoning node, and reusing value data of the key-value cache data matching the reasoning task to execute reasoning, wherein the value data comprises hidden layer output data of the large language model.

7. The method of claim 6, wherein, The determination of the key-value cache data matching the reasoning task in the target reasoning node comprises: matching the morpheme number in the reasoning task with the key-value cache data in the target reasoning node to determine the key-value cache data matching the reasoning task, wherein the key-value cache data matching the reasoning task is key-value cache data reusable in a reasoning process.

8. The method according to any one of claims 1 to 7, characterized in that, Before the receiving of the reasoning task, the method further comprises: receiving the input text, wherein the input text comprises words or sentences input by a user to a large language model; performing a word segmentation operation on the input text to determine one or more morphemes contained in the input text; executing reasoning based on the one or more morphemes to generate one or more reasoning tasks, wherein the one or more reasoning tasks are used to schedule execution to a plurality of reasoning nodes in the distributed reasoning cluster.

9. The method according to any one of claims 1 to 8, characterized in that, The reasoning node comprises one or more of a graphics processing unit (GPU), a neural network processing unit (NPU), and a data processing unit (DPU).

10. An inference apparatus of a large language model, characterized by, The device comprises: a transceiver unit configured to receive an inference task, the inference task carrying a morpheme number, wherein the morpheme number is used to identify a morpheme to be inferred, and the morpheme comprises one or more subwords determined based on an input text of a large language model; a processing unit configured to query a global index tree based on the morpheme number corresponding to the inference task to determine a target inference node, wherein the global index tree comprises sub-trees corresponding to a plurality of inference nodes in the distributed inference cluster, the sub-tree corresponding to the target inference node is a sub-tree in the global index tree having a similarity matching degree greater than a threshold with the morpheme number, and the similarity matching degree is used to indicate a number of key-value cache data reusable in an inference process; the processing unit is further configured to execute the inference task based on the target inference node.

11. The apparatus of claim 10, wherein, The processing unit is specifically configured to: calculate a similarity matching degree of each sub-tree according to a matching prefix length of each sub-tree and a total length of the morpheme number; and query a sub-tree having a similarity matching degree greater than a threshold based on the similarity matching degree of each sub-tree, and determine the inference node corresponding to the sub-tree having the similarity matching degree greater than the threshold as the target inference node.

12. The apparatus of claim 10 or 11, wherein, The similarity matching degree is positively correlated with the number of key-value cache data reusable in the inference process.

13. The apparatus of any one of claims 10-12, wherein, The processing unit is further configured to: create the global index tree based on key data of the key-value cache data when the inference node generates the key-value cache data, wherein the key data comprises a hash value corresponding to the input text.

14. The apparatus of any one of claims 10-13, wherein, A relational vector cache is deployed in the inference node, and the relational vector cache is used to store key-value cache data generated in the inference process of the large language model.

15. The apparatus of any one of claims 10-14, wherein, The processing unit is specifically configured to: determine key-value cache data matching the inference task in the target inference node, and reuse value data of the key-value cache data matching the inference task to perform inference, wherein the value data comprises hidden layer output data of the large language model.

16. The apparatus of claim 15, wherein, The processing unit is specifically configured to: match the morpheme number in the inference task with the key-value cache data in the target inference node to determine matching key-value cache data, wherein the matching key-value cache data is key-value cache data reusable in inference calculation.

17. The apparatus of any one of claims 10-16, wherein, The transceiver unit is further configured to: receive the input text, wherein the input text comprises words or sentences input by a user to the large language model; The processing unit is further configured to perform a word segmentation operation on the input text to determine one or more morphemes contained in the input text; The processing unit is further configured to perform inference based on the one or more morphemes to generate one or more inference tasks, and the one or more inference tasks are used to be scheduled to a plurality of inference nodes in the distributed inference cluster for execution.

18. The apparatus of any one of claims 10-17, wherein, The inference node comprises one or more of a graphics processing unit (GPU), a neural network processing unit (NPU), and a data processing unit.

19. A computing device, comprising: The device comprises a processor coupled with a memory, and the processor is configured to store instructions, and when the instructions are executed by the processor, the electronic device performs the method of any one of claims 1 to 9.

20. A cluster of computing devices, characterized in that, comprising at least one computing device comprising a processor coupled with a memory, the processor to store instructions that, when executed by the processor, cause the cluster of computing devices to perform the method of any of claims 1-9.

21. A computer-readable storage medium having stored thereon instructions, The instructions, when executed, cause a computer to perform the method of any of claims 1-9.

22. A computer program product, comprising instructions therein, characterised in that, The instructions, when executed, cause a computer to perform the method of any of claims 1-9.

Images