Model reasoning method and device, electronic equipment and storage medium

Abstract

The invention provides a model reasoning method and device, electronic equipment and a storage medium, and the method comprises the steps: responding to a lexical element sparse instruction, obtaining lexical elements to be discarded from the key value cache information of each layer of a large model, the lexical elements to be discarded of each layer being determined after lexical element sparse processing; and carrying out interlayer combination on the lexical elements to be discarded of each layer, and executing model reasoning operation based on an interlayer combination result. According to the method, the purposes of saving the memory, accelerating the attention calculation and improving the model reasoning effect can be achieved in the whole model reasoning process at the same time, in addition, the influence on hardware is mainly reflected in the aspect of memory saving, the memory or on-chip storage of an artificial intelligence chip can be reduced, and therefore most existing large models can be directly supported, and the model reasoning efficiency is improved. And extra training cost is not needed.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model reasoning method, device, electronic device and storage medium. Background Art

[0002] At present, in the process of reasoning, each layer of the large model with a transformer structure needs to perform attention calculations, and the attention calculations of each layer need to read all the historical tokens information in the key-value cache (KV cache) for calculation. Therefore, when the large model processes long sequences of tokens, the historical tokens information stored in the KV cache increases, the attention calculation speed is very slow, the memory burden increases, and the reasoning speed decreases. Therefore, how to save memory and improve the reasoning speed of large models has become a key issue that needs to be solved urgently.

[0003] In related technologies, in order to save memory and speed up attention calculation, a dilution strategy, a token selection strategy or a similarity analysis strategy is usually adopted to select some important historical tokens from the KV cache of each layer in the large model to participate in the attention calculation, and the remaining tokens that are not selected are merged as tokens to be discarded, and the multiple tokens before and after the merger are saved in the KV cache for the next reasoning operation.

[0004] However, since each layer in the large model uses the same merging method to merge multiple different tokens, and the KV cache of each layer only stores some tokens that have participated in the attention calculation and a small number of merged tokens, it cannot guarantee that the next attention calculation will select the truly important historical tokens, resulting in poor model reasoning effect. Summary of the invention

[0005] The present invention provides a model reasoning method, device, electronic device and storage medium to solve the defect that the prior art cannot simultaneously save memory, accelerate attention calculation and improve model reasoning effect. It can not only save memory, accelerate attention calculation and improve model reasoning effect in the whole model reasoning process, but also has an impact on hardware mainly in terms of memory saving. It can also reduce the memory or on-chip storage of artificial intelligence chips, so it can directly support most existing large models without additional training costs.

[0006] The present invention provides a model reasoning method, comprising the following steps.

[0007] In response to the word-meta sparse instruction, the words to be discarded are obtained from the key-value cache information of each layer of the large model. The words to be discarded in each layer are determined after word-meta sparse processing. The words to be discarded in each layer are merged between layers, and the model inference operation is performed based on the inter-layer merging results.

[0008] According to a model inference method provided by the present invention, the word elements to be discarded are obtained from the key-value cache information of each layer of the large model, including: when the word element sparse strategy of each layer in the large model is carried in the word element sparse instruction, based on the word element sparse strategy of each layer, the word elements to be discarded determined after the word element sparse processing of the corresponding layer are obtained from the key-value cache information of each layer.

[0009] According to a model inference method provided by the present invention, the word-meta sparse strategy of each layer is based on obtaining the word-meta to be discarded determined after the word-meta sparse processing of the corresponding layer from the key-value cache information of each layer, including: when the word-meta sparse strategy of each layer adopts the same first word-meta sparse strategy for each layer, based on the pre-set number of identical calculated word-meta for each layer, respectively obtaining the word-meta to be discarded determined after the first word-meta sparse processing from the key-value cache information of each layer.

[0010] According to a model inference method provided by the present invention, the word-meta sparse strategy based on each layer is used to obtain the word-meta to be discarded determined after the sparse processing of the corresponding layer of word-meta from the key-value cache information of each layer, and also includes: when the word-meta sparse strategy of each layer adopts a different second word-meta sparse strategy for each layer, the word-meta to be discarded determined after the sparse processing of the second word-meta is obtained from the key-value cache information of each layer.

[0011] According to a model inference method provided by the present invention, the inter-layer merging of the to-be-discarded words in each layer includes: determining the number of groups corresponding to all layers in the large model, each group consisting of at least one layer of the large model; and inter-layer merging of the to-be-discarded words in each layer based on an inter-layer merging method corresponding to the number of groups and the total number of layers of the large model.

[0012] According to a model inference method provided by the present invention, the inter-layer merging of the to-be-discarded words in each layer is performed based on the inter-layer merging method corresponding to the number of groups and the total number of layers of the large model, including: when the number of groups is the same as the total number of layers, the same words in the to-be-discarded words in each layer are merged between layers; when the number of groups is less than the total number of layers, for each group, the same words in the to-be-discarded words in at least two layers within the group are merged between layers.

[0013] According to a model inference method provided by the present invention, the model inference operation is performed based on the inter-layer merging result, including: when the inter-layer merging result is the first shared key value cache information of each layer, the next inference operation is performed based on the first target key value cache information used for the current step attention calculation in each layer and the first shared key value cache information; when the inter-layer merging result is the second shared key value cache information of each group, the next inference operation is performed based on the second target key value cache information used for the current step attention calculation in each layer contained in each group and the second shared key value cache information corresponding to the group.

[0014] The present invention also provides a model reasoning device, comprising the following word-unit sparse unit and model reasoning unit.

[0015] The word-element sparse unit is used to respond to the word-element sparse instruction and obtain the word-element to be discarded determined after the key-value cache information of each layer in the large model is processed by the word-element sparse processing.

[0016] The model inference unit is used to perform inter-layer merging of the to-be-discarded words in each layer and perform model inference operations based on the inter-layer merging results.

[0017] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the model inference method described above is implemented.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the model reasoning methods described above.

[0019] The model reasoning method, device, electronic device and storage medium provided by the present invention, wherein the model reasoning method includes: when the artificial intelligence chip responds to the word-element sparse instruction, the key-value cache information of each layer of the large model is respectively obtained to be discarded, and the word-element to be discarded of each layer is determined after the word-element sparse processing, and then the word-element to be discarded of each layer is merged between layers, so as to perform the model reasoning operation based on the inter-layer merging result later. In this way, by making full use of the inter-layer characteristics of the large model and the word-element sparse strategy of each layer, the same tokens between layers are merged, which can not only speed up the model reasoning speed and ensure that the merged tokens can be shared by all layers, but also retain the token information, reduce the degree of information loss of the merged tokens, and reduce the memory overhead. At the same time, it can also ensure that the attention calculation can select the really important tokens, thereby achieving the purpose of saving memory, accelerating the attention calculation and improving the model reasoning effect in the whole model reasoning process. In addition, the impact on hardware is mainly reflected in the memory saving aspect, which can reduce the memory or on-chip storage of the artificial intelligence chip, so that most existing large models can be directly supported without additional training costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a flow chart of the model reasoning method provided by the present invention.

[0022] Figure 2 It is one of the schematic diagrams of the inter-layer merging process provided by the present invention.

[0023] Figure 3 This is the second schematic diagram of the inter-layer merging process provided by the present invention.

[0024] Figure 4 This is the third schematic diagram of the inter-layer merging process provided by the present invention.

[0025] Figure 5 It is a structural schematic diagram of the model reasoning device provided by the present invention.

[0026] Figure 6 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0029] In the embodiments of the present invention, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B may be singular or plural. In the textual description of the present invention, the character " / " generally indicates that the previous and next associated objects are in an "or" relationship. In addition, it should be noted that the serial numbers themselves, such as "first", "second", etc., used to distinguish the objects described in the present invention are only used to distinguish the objects described, and do not have any order or technical meaning.

[0030] At present, during the reasoning process of a large model with a Transformer structure, each layer needs to perform attention calculations, and the attention calculations of each layer need to read all the historical tokens information in the KV cache for calculation. Therefore, when a large model processes long sequences of tokens, the historical word meta-tokens information stored in the KV cache increases, the attention calculation speed is very slow, the memory burden increases, and the reasoning speed decreases. Therefore, how to save memory and improve the reasoning speed of large models has become a key issue that needs to be solved urgently.

[0031] In related technologies, in order to save memory and speed up attention calculation, a dilution strategy, a token selection strategy or a similarity analysis strategy is usually adopted to select some important historical tokens from the KV cache of each layer in the large model to participate in the attention calculation, and the remaining tokens that are not selected are merged as tokens to be discarded, and the multiple tokens before and after the merger are saved in the KV cache for the next reasoning operation.

[0032] However, since each layer in the large model uses the same merging method to merge multiple different tokens, and the KV cache of each layer only stores some tokens that have participated in the attention calculation and a small number of merged tokens, it cannot guarantee that the next attention calculation will select the truly important historical tokens, resulting in poor model reasoning effect.

[0033] For example, large models of the Transformer architecture, due to their large number of parameters and complex structures, are able to capture and learn complex patterns and relationships in the data, such as the Bidirectional Encoder Representations from Transformers (BERT) model, the third-generation Transformer (GPT-3) model of the Generative Pre-trained Transformer‌ series, and the Large Language Model Meta AI (Llama). For large models of the Transformer architecture, the BERT model, GPT-3 model, and Llama not only have millions or even billions of parameters, but also perform well in a variety of tasks such as language translation, image recognition, natural language understanding, and natural language generation.

[0034] For large models of the Transformer architecture, the attention mechanism is one of the main computing modules of each layer in the large model, which allows the large model to dynamically assign weights according to the contextual information of each position when processing sequence data. The attention calculation in the Transformer architecture mainly includes three parts: query, key, and value; that is, first, the input sequence (the input sequence is obtained by encoding the token, and each token is encoded to obtain a sequence vector) is converted into a query vector, a key vector, and a value vector through a linear transformation (usually the input matrix is ​​multiplied by the weight matrix WQ, WK, and WV respectively); further, the large model calculates the dot product of the query vector and all the key vectors, normalizes the dot product result through the normalized exponential function (softmax function), and then performs the weighted sum with the value vector to obtain the output. Since the input query vector needs to be calculated with all the past key-value (KV) information in the key-value cache (KV cache) each time the large model is inferred, when the large model processes the reasoning of long sequence tokens (such as thousands of tokens or even tens of thousands of tokens), it inevitably faces the challenges of complexity and memory limitations; this is because when the large model infers a longer sequence of tokens, the historical tokens information that needs to be stored will also be correspondingly long. At this time, the attention calculation speed is very slow, resulting in a decrease in the reasoning speed.

[0035] For the application scenarios of the above-mentioned large models, when the context length is relatively long, more historical tokens information needs to be stored, the attention calculation speed slows down, and the reasoning speed of the large model slows down; for example, in the document summary scenario, when a long document is input, all document information needs to be traversed when generating the summary, resulting in a very slow summary generation speed; in the code analysis scenario, a code project is analyzed with a very long context, resulting in a very slow code analysis speed; in the multi-round dialogue scenario, during multiple dialogues with the large model, each dialogue needs to be replied based on the historical dialogue information, resulting in a very slow reply speed of the large model; the KV cache used in the above scenarios stores a large amount of historical token information, resulting in a decrease in reasoning speed.

[0036] In order to speed up the attention calculation, it can be implemented through the KV cache sparse mode. This is because the attention calculation needs to read all the historical tokens in the KV cache for calculation; when the inference length is long, the historical tokens information stored in the KV cache, at this time, some important historical tokens can be selected through the preset strategy to participate in the attention calculation; for example, window attention, each time only selects the latest generated tokens in the KV cache to participate in the calculation, or uses the token selection strategy and similarity analysis strategy to select some historical tokens to participate in the calculation, and the remaining unselected historical tokens are directly discarded, which saves memory and speeds up the attention calculation; in addition, it can also be implemented through the KV cache non-sparse mode, continuously saving all historical tokens in the KV cache, and only selecting important historical tokens for calculation during attention calculation, but this method only speeds up the attention calculation but does not save memory.

[0037] In order to save memory while speeding up attention calculation, KV cache sparse + token merging can be used. This is because the historical tokens discarded each time in the attention sparse mode may affect the subsequent reasoning tasks, and directly discarding them will undoubtedly lead to poor subsequent reasoning results. Also, since KV cache does not discard tokens in the non-sparse mode, although important historical tokens can be selected from the complete historical tokens for attention calculation at each reasoning, which speeds up the attention calculation, it cannot save memory. Especially when the context length is long, the memory overhead required by KVcache also increases.

[0038] In order to save memory and retain the discarded historical tokens information, you can choose to merge the discarded historical tokens; for example, there are 1000 historical tokens stored in the KV cache, select 500 important tokens for the current reasoning calculation, and merge the remaining 500 tokens into 50 tokens. In this way, only 550 tokens need to be saved in the KV cache (including the 500 tokens involved in the reasoning calculation and the merged 50 tokens). In the subsequent reasoning steps, the merged 50 tokens can replace the previous 500 tokens. This not only saves memory, but also retains multiple discarded tokens to a certain extent.

[0039] It can be understood that in the existing KV cache sparse implementation method, the purpose of saving content and accelerating attention calculation is achieved by discarding unselected historical tokens, but this method will affect the subsequent reasoning effect; in the existing KV cache non-sparse implementation method, the reasoning effect is improved by performing sparse calculations and not discarding tokens, but it cannot save memory. When the context length is long, the KV cache will occupy more memory; in order to balance the KV cache sparse implementation method and the KV cache non-sparse implementation method, the historical tokens to be discarded can be merged so that a small number of tokens can be used to represent multiple historical tokens to be discarded, which not only saves the information of the tokens to be discarded but also saves memory.

[0040] Considering that large models are usually multi-layer structures, and the existing merging implementation methods are consistent for operations in large models, for example, if two tokens to be discarded are merged into one token, they are merged in the KV cache of all layers; in addition, this method of directly merging multiple tokens is prone to information loss, and a small number of tokens are needed to represent multiple tokens with different semantics, which has limited expression ability and cannot improve the model reasoning effect. In other words, in order to speed up the reasoning of large models, some important tokens are selected from the KV cache for calculation during attention calculation, which can reduce the amount of calculation and speed up the calculation; at the same time, in order to save memory, unselected tokens are usually discarded or merged, which results in serious information loss and poor model reasoning effect.

[0041] In order to solve the above technical problems, the present invention provides a model reasoning method, device, electronic device and storage medium. Figure 1-Figure 6The model reasoning method, device, electronic device and storage medium of the present invention are described, wherein the execution subject of the model reasoning method is an artificial intelligence chip that runs an artificial intelligence algorithm, and the artificial intelligence chip can be built into an electronic device or a server. The artificial intelligence chip can specifically be a chip that carries computing tasks in large model reasoning, such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit) and a TPU (Tensor Processing Unit), etc.; the electronic device can be a personal computer (Personal Computer, PC), a portable device, a laptop, a smart phone, a tablet computer and other devices such as a portable wearable device; the server can refer to a server, or a server cluster composed of multiple servers, a cloud server, etc.; the present invention does not specifically limit the specific form of the electronic device or server. Furthermore, the model reasoning method can also be applied to a model reasoning device set in an artificial intelligence chip, and the model reasoning device can be implemented by software, hardware or a combination of the two. The following describes the model reasoning method by taking the execution subject of the model reasoning method as an artificial intelligence chip as an example.

[0042] In order to facilitate understanding of the model reasoning method provided by the embodiment of the present invention, the model reasoning method provided by the present invention will be described in detail below through the following exemplary embodiments. It can be understood that the following exemplary embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0043] Reference Figure 1 , which is a flow chart of the model reasoning method provided by the present invention, such as Figure 1 As shown, the model reasoning method includes the following steps 110 and 120.

[0044] Step 110, in response to the word-unit sparse instruction, obtain the words to be discarded from the key-value cache information of each layer of the large model, and the length of time to be discarded of each layer is determined after word-unit sparse processing.

[0045] Among them, the word-meta sparse instructions can be automatically triggered based on the word-meta sparse strategies of each layer input by the user for the large model; for example, the word-meta sparse instructions using the same word-meta sparse strategy at each layer can be automatically generated according to the same word-meta sparse strategy input by the user; or, the word-meta sparse instructions using different word-meta sparse strategies at each layer can be automatically generated according to the different word-meta sparse strategies input by the user.

[0046] The way in which the user inputs the same word-meta sparse strategy or different word-meta sparse strategies may include but is not limited to input on an electronic device, input in a terminal application, voice input, and image input. For example, the user may input the same word-meta sparse strategy or different word-meta sparse strategies on an electronic device with a built-in computing chip; or, the user may also input the same word-meta sparse strategy or different word-meta sparse strategies in a terminal device application connected to the electronic device; or, the user may also input the same word-meta sparse strategy or different word-meta sparse strategies to the electronic device by voice; or, the user may also transmit or copy an image containing the same word-meta sparse strategy or different word-meta sparse strategies to the electronic device, and then the electronic device may identify the same word-meta sparse strategy or different word-meta sparse strategies from the image. The present invention does not specifically limit the input method in which the user inputs the same word-meta sparse strategy or different word-meta sparse strategies.

[0047] The number of words to be discarded determined for each layer can be one or more; the words to be discarded of each layer can be the same or different; it depends on whether the word sparse strategies adopted by each layer are the same.

[0048] For the same word-meta sparse strategy adopted by each layer or different word-meta sparse strategies adopted by each layer, the word-meta sparse strategy adopted by each layer may include but is not limited to other existing sparse implementation methods such as window attention sparse strategy, Key attention sparse strategy and attention block sparse strategy.

[0049] It should be noted that the window attention sparse strategy is used to select the latest tokens generated in the KV cache for calculation each time, and the tokens generated earlier are all treated as words to be discarded; the key attention sparse strategy is used to find the most relevant tokens for the current task among all the tokens cached in the KV cache for attention calculation, and the remaining tokens that do not participate in the attention calculation are all treated as words to be discarded; the attention block sparse strategy is used to divide the KV cache into multiple KV blocks according to a fixed length, and select the most relevant tokens contained in the KV blocks from multiple KV blocks according to similarity for attention calculation, and the tokens in the remaining KV blocks are all treated as words to be discarded.

[0050] Specifically, the artificial intelligence chip is built into the electronic device and a large model is pre-deployed in the electronic device. The artificial intelligence chip running the large model can obtain the key-value cache information of each layer in the large model based on the sparse strategy of each layer of word elements input by the user, and determine the words to be discarded after the corresponding word element sparse processing.

[0051] Exemplarily, the key-value cache information of each layer in the large model is sparsely processed by window attention to determine the words to be discarded; or, the key-value cache information of each layer in the large model is sparsely processed by Key attention to determine the words to be discarded; or, the key-value cache information of each layer in the large model is sparsely processed by attention blocks to determine the words to be discarded.

[0052] Step 120: perform inter-layer merging on the to-be-discarded words of each layer, and perform model inference operations based on the inter-layer merging results.

[0053] Specifically, at least one to-be-discarded word element in each layer is merged between layers to ensure that the word element obtained after the inter-layer merging can be used by all layers in the large model.

[0054] It should be noted that, considering that large models are usually multi-layer structures and each layer has a KV cache, when each layer uses the corresponding word-meta sparse strategy for sparse processing, the tokens to be discarded can be obtained and used as at least one word-meta to be discarded in the corresponding layer. At this time, at least one word-meta to be discarded in each layer can be merged between layers; since the information of the same token is stored between layers, the information after merging is still the same token, which will not cause information loss and also reduce memory overhead.

[0055] The model reasoning method provided by the present invention, when responding to the word-element sparse instruction, the artificial intelligence chip obtains the word-element to be discarded from the key-value cache information of each layer of the large model, and the word-element to be discarded of each layer is determined after the word-element sparse processing, and then the word-element to be discarded of each layer is merged between layers, so as to perform the model reasoning operation based on the inter-layer merging result later. In this way, by making full use of the inter-layer characteristics of the large model and the word-element sparse strategy of each layer, the same tokens between layers are merged, which can not only speed up the model reasoning speed and ensure that the merged tokens can be shared by all layers, but also retain the token information, reduce the degree of information loss of the merged tokens, and reduce the memory overhead. At the same time, it can also ensure that the attention calculation can select the really important tokens, thereby achieving the purpose of saving memory, accelerating the attention calculation and improving the model reasoning effect in the whole model reasoning process. In addition, the impact on hardware is mainly reflected in the memory saving aspect, which can reduce the memory or on-chip storage of the artificial intelligence chip, so it can directly support most existing large models without additional training costs.

[0056] Based on the above Figure 1 In the model reasoning method shown, in an exemplary embodiment, the specific process of step 110 is implemented by the following steps.

[0057] When the word-meta sparse strategy of each layer in the large model is carried in the word-meta sparse instruction, the word-meta to be discarded determined after the word-meta sparse processing of the corresponding layer is obtained from the key-value cache information of each layer based on the word-meta sparse strategy of each layer.

[0058] Specifically, when the user inputs the word-meta sparse strategy for each layer, each layer in the large model can adopt the word-meta sparse strategy of the corresponding layer for sparse processing, and then obtain at least one word-meta to be discarded determined by the KV cache of each layer after the corresponding sparse processing.

[0059] In one embodiment, when each layer in a large model adopts the same word-meta sparse strategy for sparse processing, based on the word-meta sparse strategy of each layer, the words to be discarded determined after the word-meta sparse processing of the corresponding layer are obtained from the key-value cache information of each layer. This can be specifically achieved through the following steps.

[0060] When the word-meta sparse strategy of each layer adopts the same first-word-meta sparse strategy for each layer, based on the preset same number of calculated words for each layer, the words to be discarded determined after the first-word-meta sparse processing are obtained from the key-value cache information of each layer.

[0061] It should be noted that each first-word sparse strategy can be used to represent that each layer of the KV cache in the large model needs to perform the same first-word sparse processing.

[0062] For example, the current reasoning sentence of the large model is "Today is Monday, tomorrow is" and each layer of the large model needs to select 4 tokens to participate in the attention calculation; each layer's KV cache stores 6 tokens, namely "Today", "is", "Monday", "," tomorrow" and "is", and each layer's KV cache uses the window attention sparse strategy, which is the first-term sparse strategy, to perform first-term sparse processing. When calculating attention, only the most recent 4 tokens are required to participate in the calculation, that is, the 4 tokens "Monday", "," tomorrow" and "is" participate in the calculation. "Today" and "is" need to be deleted from the KV cache of each layer, that is, "Today" and "is" in the KV cache of each layer are terms to be discarded.

[0063] It should be noted that each layer of the large model adopts the same token sparse strategy, that is, the important tokens selected by each layer to participate in the attention calculation are the same, so the tokens that need to be discarded in each layer are also the same. At this time, the same tokens discarded by all layers can be merged; in this way, the purpose of saving memory, accelerating attention calculation and improving model reasoning effect is achieved.

[0064] In one embodiment, when each layer in a large model adopts a different word-meta sparse strategy for sparse processing, based on the word-meta sparse strategy of each layer, the words to be discarded determined after the word-meta sparse processing of the corresponding layer are obtained from the key-value cache information of each layer. This can be specifically achieved through the following steps.

[0065] In the case that the word-unit sparse strategy of each layer adopts a different second word-unit sparse strategy for each layer, the word-units to be discarded determined after the second word-unit sparse processing are respectively obtained from the key-value cache information of each layer.

[0066] It should be noted that each second-term sparse strategy can be used to represent that the KV cache of the corresponding layer in the large model needs to perform different second-term sparse processing.

[0067] Specifically, when each layer of the large model adopts a separate token sparse strategy, the number and information of tokens that may be discarded in each layer may be inconsistent. At this time, the tokens to be discarded determined after the second token sparse processing of each layer can be obtained.

[0068] For example, the current inference sentence of the large model is "Today is Monday, tomorrow is". The large model has 3 layers and each layer uses the corresponding second-term sparse strategy to perform second-term sparse processing. It can be determined that the to-be-discarded terms in the first layer are "Today" and "is", the to-be-discarded terms in the second layer are "is" and "tomorrow", and the to-be-discarded terms in the third layer are "is" and "Monday".

[0069] Based on the above Figure 1 In the model inference method shown in an exemplary embodiment, in step 120, the to-be-discarded words of each layer are merged between layers, and the specific process can be implemented through the following steps.

[0070] First, the number of groups corresponding to all layers in the large model is determined, and each group is composed of at least one layer of the large model; then, based on the inter-layer merging method corresponding to the number of groups and the total number of layers of the large model, the words to be discarded in each layer are merged between layers.

[0071] Specifically, when the user inputs the word-meta sparse strategy for each layer, he can also input the number of groups for the total number of layers of the large model to automatically generate word-meta sparse instructions. Therefore, by analyzing the word-meta number strategy, it can not only be determined whether each layer adopts the same word-meta sparse strategy or different word-meta sparse strategies, but also the number of groups can be determined, and each group contains at least one layer of the large model.

[0072] For example, when the large model has 32 layers and the number of groups is 2, the first group contains the 1st to 16th layers of the large model, and the second group contains the 17th to 32nd layers of the large model; or, when the large model has 3 layers and the number of groups is 3, the first group contains the 1st layer of the large model, the second group contains the 2nd layer of the large model, and the third group contains the 3rd layer of the large model.

[0073] At this time, the corresponding inter-layer merging method can be determined based on the relationship between the number of groups and the total number of model layers. That is, when the number of groups is equal to the total number of layers, the corresponding inter-layer merging method is to merge at least one to-be-discarded word that is the same in multiple layers into at least one to-be-discarded word; for example, when each layer discards the two tokens "Today" and "is", they can be merged into a KV cache x containing "Today" and "is" for sharing in subsequent reasoning calculations.

[0074] In addition, when the number of groups is not equal to the total number of layers, the corresponding inter-layer merging method is to merge the same words in at least one to-be-discarded word determined by each layer in each group; for example, if the large model has 3 layers and the to-be-discarded words in the 1st layer are "Today" and "is", the to-be-discarded words in the 2nd layer are "is" and "tomorrow", and the to-be-discarded words in the 3rd layer are "is" and "Monday", the same word that needs to be discarded in all 3 layers is "is", and the "is" information in the 3 layers can be merged at this time.

[0075] In one embodiment, based on the inter-layer merging method corresponding to the number of groups and the total number of layers of the large model, the to-be-discarded words of each layer are merged inter-layer, which can be specifically achieved through the following steps.

[0076] When the number of groups is the same as the total number of layers, the same words in the words to be discarded in each layer are merged between layers; or, when the number of groups is less than the total number of layers, for each group, the same words in the words to be discarded in at least two layers within the group are merged between layers.

[0077] It should be noted that the present invention does not specifically limit the method of inter-layer merging, and it can be a simple average pooling operation.

[0078] For example, the process of inter-layer merging for the case where the number of groups is the same as the total number of layers can refer to Figure 2 and Figure 3 , Figure 2 and Figure 3 Each layer of the medium and large models includes at least a feedforward layer and an attention layer.

[0079] Reference Figure 2 One of the schematic diagrams of the inter-layer merging process shown in Figure 2 As shown in the figure, each layer of KV cache stores a total of 6 tokens, namely "Today", "is", "Monday", "," "tomorrow" and "is". After the KV cache of each layer is sparsely processed using the window attention sparse strategy, only the 4 most recent tokens are required to participate in the calculation of attention, that is, the 4 tokens of "Monday", "," "tomorrow" and "is". "Today" and "is" need to be deleted from the KV cache of each layer, that is, "Today" and "is" in each layer of KV cache are both words to be discarded. At this time, the 2 tokens to be deleted in each layer are merged between layers into a KV cachex containing "Today" and "is".

[0080] Reference Figure 3 The second schematic diagram of the inter-layer merging process shown in FIG. Figure 3 As shown in the figure, the large model has 3 layers and the to-be-discarded words in the 1st layer are "Today" and "is", the to-be-discarded words in the 2nd layer are "is" and "tomorrow", and the to-be-discarded words in the 3rd layer are "is" and "Monday". The same word "is" needs to be discarded in all 3 layers. At this time, the common to-be-discarded word "is" in the 3 layers can be merged into a KV cache x containing "is".

[0081] In addition, the process of merging layers can refer to Figure 4 As shown, Figure 4 Each layer of the medium and large models includes at least a feedforward layer and an attention layer.

[0082] Reference Figure 4 The third schematic diagram of the inter-layer merging process shown in FIG. Figure 4As shown in the figure, the large model has a total of 4 layers and the number of groups is 2. Group 1 contains the 1st layer (Layer1) and the 2nd layer (Layer2) of the large model, and Group 2 contains the 3rd layer (Layer3) and the 4th layer (Layer4) of the large model. The words to be discarded in the 1st layer of Group 1 are "Monday" and "tomorrow", and the words to be discarded in the 2nd layer are "is" and "tomorrow". At this time, the same word "tomorrow" in Group 1 is merged into a KV cache x1 containing "tomorrow"; the words to be discarded in the 3rd layer of Group 2 are "is" and "Monday", and the words to be discarded in the 4th layer are "Today" and "is". At this time, the same word "is" in Group 2 is merged into a KV cache x2 containing "is".

[0083] Combination Figure 2~Figure 4 It can be understood from the inter-layer merging process shown that when each layer in the large model adopts the same first-word sparse strategy, the words that need to be discarded in each layer must be the same tokens, so they all need to be merged; when each layer adopts a different second-word sparse strategy, only the commonly discarded tokens are merged.

[0084] It should be noted that for Figure 3 As for the inter-layer merging process shown in the figure, the "is" that need to be discarded in the three layers of the large model are merged and saved in the KV cache x, and the other tokens are discarded. From the perspective of the entire reasoning task, even if some layers are missing some tokens, the missing tokens are retained in other layers, and the commonly missing tokens are shared and calculated using the merged KV.

[0085] for Figure 4 As for the inter-layer merging process shown, in the case where two layers in a large model are one group, the commonly missing tokens in each group can be merged, and the two layers do not affect each other.

[0086] Based on the above Figure 1 In the model inference method shown, in an exemplary embodiment, in step 120, a model inference operation is performed based on the inter-layer merging result, and its specific implementation process can be achieved through the following steps.

[0087] When the inter-layer merging result is the first shared key value cache information of each layer, the next reasoning operation is performed based on the first target key value cache information used for the current step attention calculation in each layer and the first shared key value cache information; or, when the inter-layer merging result is the second shared key value cache information of each group, the next reasoning operation is performed based on the second target key value cache information used for the current step attention calculation in each layer contained in each group and the second shared key value cache information of the corresponding group.

[0088] Specifically, when the inter-layer merging result is the first shared key value cache information of each layer, it can be considered that all layers in the large model use the same first word unit sparse strategy for sparse processing, and then merge the same words of each layer between layers to obtain the shared key value cache information; for example, the aforementioned Figure 2 The KV cache x with "is" shown, or, as mentioned above Figure 3 The KV cache x containing "is" is shown. At this point, the tokens that have participated in the attention calculation in the current step of each layer in the large model are the first target key value cache information, and then combined with the first shared key value cache information used by each layer to perform the next reasoning operation.

[0089] When the inter-layer merging result is the second shared key value cache information of each group, it can be considered that each layer in each group of the large model uses a different second word sparse strategy for sparse processing, and then the same words in each layer contained in each group are merged between layers to obtain the shared key value cache information; for example, the aforementioned Figure 4 As shown, group 1 contains KV cache x1 of "tomorrow", and group 2 contains KV cache x2 of "is".

[0090] For example, Figure 2 As shown in the figure, when the next reasoning operation is performed, important tokens can be selected from the KV cache containing 4 tokens that have participated in the attention calculation at each layer and the shared KV cache x containing "Today" and "is" for the next attention calculation, without losing token information; and so on, until the entire reasoning task is completed.

[0091] At this point, the tokens of each layer in the large model that have participated in the attention calculation in the current step, and the two tokens in the other KVcache x can be shared by all layers; in this way, when performing the next reasoning operation, each layer can select important tokens from the KV cache containing 4 tokens and the shared KV cache x for attention calculation, and there will be no loss of token information; and so on, until the entire reasoning task is completed. In this way, the same token between layers is merged in combination with the inter-layer characteristics of the large model, which not only retains the token information, but also saves memory and improves the model reasoning effect.

[0092] The model reasoning device provided by the present invention is described below. The model reasoning device described below and the model reasoning method described above can be referenced to each other.

[0093] Reference Figure 5 The structural diagram of the model inference device shown in FIG. Figure 5 As shown, the model reasoning device 500 includes: a word sparse unit 510 and a model reasoning unit 520.

[0094] The word-unit sparse unit 510 is used to respond to the word-unit sparse instruction and obtain the words to be discarded from the key-value cache information of each layer of the large model. The words to be discarded in each layer are determined after word-unit sparse processing.

[0095] The model inference unit 520 is used to merge the to-be-discarded words in each layer between layers, and perform model inference operations based on the inter-layer merging results.

[0096] Optionally, the word-meta sparse unit 510 is specifically used to obtain the words to be discarded determined after the word-meta sparse processing of the corresponding layer from the key-value cache information of each layer based on the word-meta sparse strategy of each layer in the large model when the word-meta sparse instruction carries the word-meta sparse strategy of each layer in the large model.

[0097] Optionally, the word-meta sparse unit 510 is specifically used to obtain the words to be discarded determined after the first word-meta sparse processing from the key-value cache information of each layer based on the preset number of identical calculated words for each layer, when the word-meta sparse strategy for each layer adopts the same first word-meta sparse strategy for each layer.

[0098] Optionally, the word-meta sparse unit 510 is specifically used to obtain the words to be discarded determined after the second word-meta sparse processing from the key-value cache information of each layer when the word-meta sparse strategy of each layer adopts a different second word-meta sparse strategy for each layer.

[0099] Optionally, the model inference unit 520 is specifically used to determine the number of groups corresponding to all layers in the large model, each group is composed of at least one layer of the large model; based on the inter-layer merging method corresponding to the number of groups and the total number of layers of the large model, inter-layer merging is performed on the words to be discarded in each layer.

[0100] Optionally, the model inference unit 520 is specifically used to merge the same words in the words to be discarded in each layer between layers when the number of groups is the same as the total number of layers; or, when the number of groups is less than the total number of layers, for each group, merge the same words in the words to be discarded in at least two layers within the group between layers.

[0101] Optionally, the model inference unit 520 is specifically used to perform the next inference operation based on the first target key value cache information used for the current step attention calculation in each layer and the first shared key value cache information when the inter-layer merging result is the first shared key value cache information of each layer; or, when the inter-layer merging result is the second shared key value cache information of each group, perform the next inference operation based on the second target key value cache information used for the current step attention calculation in each layer contained in each group and the second shared key value cache information of the corresponding group.

[0102] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: an artificial intelligence (AI) chip 610, a communication interface (Communications Interface) 620, a memory 630 and a communication bus 640, wherein the AI ​​chip 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The AI ​​chip 610 may call the logic instructions in the memory 630 to execute the model reasoning method, which includes: in response to the word-element sparse instruction, respectively obtaining the word-element to be discarded from the key-value cache information of each layer of the large model, and the word-element to be discarded of each layer is determined after the word-element sparse processing; inter-layer merging of the word-element to be discarded of each layer, and performing the model reasoning operation based on the inter-layer merging result.

[0103] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0104] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the model reasoning method provided by the above methods, which includes: in response to a word-meta sparse instruction, respectively obtaining the words to be discarded from the key-value cache information of each layer of the large model, and the words to be discarded in each layer are determined after word-meta sparse processing; performing inter-layer merging on the words to be discarded of each layer, and performing model reasoning operations based on the inter-layer merging results.

[0105] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the model inference method provided by the above-mentioned methods, the method comprising: in response to a word-meta sparse instruction, respectively obtaining the words to be discarded from the key-value cache information of each layer of the large model, wherein the words to be discarded of each layer are determined after word-meta sparse processing; performing inter-layer merging of the words to be discarded of each layer, and performing model inference operations based on the inter-layer merging results.

[0106] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0107] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0108] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A model reasoning method, It is characterized in that include: In response to the word-element sparse instruction, the word-element to be discarded is obtained from the key-value cache information of each layer of the large model, and the word-element to be discarded in each layer is determined after word-element sparse processing; The to-be-discarded words in each layer are merged between layers, and a model inference operation is performed based on the inter-layer merging result.

2. The model reasoning method according to claim 1, It is characterized in that The step of obtaining the to-be-discarded words from the key-value cache information of each layer of the large model includes: In the case where the word-meta sparse instruction carries the word-meta sparse strategy of each layer in the large model, the word-meta to be discarded determined after the word-meta sparse processing of the corresponding layer is obtained from the key-value cache information of each layer based on the word-meta sparse strategy of each layer.

3. The model reasoning method according to claim 2, It is characterized in that The step of obtaining the to-be-discarded word element determined after the word element sparse processing of the corresponding layer from the key value cache information of each layer based on the word element sparse strategy of each layer includes: In the case that the word-meta sparse strategy of each layer adopts the same first word-meta sparse strategy for each layer, based on the preset same calculated word-meta number of each layer, the word-meta to be discarded determined after the first word-meta sparse processing is obtained from the key-value cache information of each layer.

4. The model reasoning method according to claim 2, It is characterized in that The step of obtaining the to-be-discarded word element determined after the word element sparse processing of the corresponding layer from the key value cache information of each layer based on the word element sparse strategy of each layer further includes: In the case that the word-unit sparse strategy of each layer adopts a different second word-unit sparse strategy for each layer, the word-units to be discarded determined after the second word-unit sparse processing are respectively obtained from the key-value cache information of each layer.

5. The model reasoning method according to any one of claims 1 to 4, It is characterized in that The inter-layer merging of the to-be-discarded words of each layer includes: Determine the number of groups corresponding to all layers in the large model, each group is composed of at least one layer of the large model; Based on the inter-layer merging method corresponding to the number of groups and the total number of layers of the large model, the to-be-discarded words of each layer are merged inter-layer.

6. The model reasoning method according to claim 5, It is characterized in that The inter-layer merging method based on the inter-layer merging method corresponding to the number of groups and the total number of layers of the large model, merging the to-be-discarded words of each layer, comprises: When the number of groups is the same as the total number of layers, merging the same words in the words to be discarded in each layer between layers; When the number of groups is less than the total number of layers, for each group, identical words in the to-be-discarded words in at least two layers within the group are merged between layers.

7. The model reasoning method according to claim 6, It is characterized in that The performing of the model inference operation based on the inter-layer merging result includes: When the inter-layer merging result is the first shared key value cache information of each layer, performing the next reasoning operation based on the first target key value cache information used for the current step attention calculation in each layer and the first shared key value cache information; In the case where the inter-layer merging result is the second shared key value cache information of each group, the next reasoning operation is performed based on the second target key value cache information used for the current step attention calculation in each layer contained in each group and the second shared key value cache information corresponding to the group.

8. A model reasoning device, It is characterized in that include: A word-element sparse unit, for responding to a word-element sparse instruction, and obtaining words to be discarded from the key-value cache information of each layer of the large model, wherein the words to be discarded of each layer are determined after word-element sparse processing; The model inference unit is used to perform inter-layer merging of the to-be-discarded words in each layer and perform model inference operations based on the inter-layer merging results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the model reasoning method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the model reasoning method according to any one of claims 1 to 7 is implemented.

Images