Key-value caching method, text decoding method, device and computer storage medium

Abstract

The present application proposes a key-value caching method, a text decoding method, a text decoding device and a computer storage medium. The key-value caching method includes: based on the current text segment to be decoded, obtaining a number of decoded text segments; obtaining a key-value cache of a number of decoded text segments; retaining part of the key-value cache in all key-value caches according to a preset key-value cache allocation strategy, and discarding the remaining key-value caches; wherein, the number of key-value caches of the first self-attention layer in the retained key-value cache of the number of decoded text segments is less than the key-value cache of the second self-attention layer, and the number of layers of the first self-attention layer is lower than the number of layers of the second self-attention layer. Through the above-mentioned key-value caching method, according to the characteristics of dense semantics of top-level network features and sparse semantics of bottom-level network features, key-value caches of different sizes are retained from top to bottom, so that the retained key-value cache is more effective, the main information required for text segment decoding will not be lost, and the efficiency of the key-value cache is improved.

Description

Technical Field

[0001] The present application relates to the field of key-value caching technology, and in particular to a key-value caching method, a text decoding method, a text decoding device, and a computer storage medium. Background Art

[0002] The language model receives tokens (text fragments). Tokens are the smallest unit for dividing text and are numerical representations of text. Before model inference, the text needs to be converted into a token sequence. The large language model generates tokens one by one during the inference phase. If the current token is the nth token, according to the characteristics of the self-attention structure, it needs to use the key values ​​of the first n-1 tokens for calculation. The key values ​​of the first n-1 tokens have been calculated once when the corresponding tokens are generated. They are saved in advance and called key-value cache. Then, when the nth token is generated, the key-value cache can be used directly to avoid redundant calculations. The key-value cache will continue to grow as tokens are generated, and some key-value caches may not work for subsequent prediction results. In order to reduce the memory burden brought by the key-value cache and ensure the accuracy of large language model inference, key-value cache optimization technology is usually used to reduce memory usage.

[0003] However, the current key-value cache optimization technology does not consider the distribution characteristics of the semantic features of large language models in different network layers, and uses a fixed key-value cache size for each attention layer, which cannot effectively reduce the key-value cache size. Summary of the invention

[0004] In order to solve the above technical problems, the present application proposes a key-value caching method, a text decoding method, a text decoding device and a computer storage medium.

[0005] In order to solve the above technical problems, the present application proposes a key-value caching method, which includes:

[0006] Based on the current text segment to be decoded, obtain several decoded text segments;

[0007] Obtaining key-value caches of the plurality of decoded text segments, wherein the key-value caches are a plurality of key-value caches arranged from high to low according to the self-attention layer of the large language model network;

[0008] According to the preset key value cache allocation strategy, some key value caches are retained in all key value caches, and the rest are discarded;

[0009] Among them, the number of key value caches of the first self-attention layer in the retained key value cache of the several decoded text fragments is less than the key value cache of the second self-attention layer, and the number of layers of the first self-attention layer is lower than the number of layers of the second self-attention layer.

[0010] In order to solve the above technical problems, the present application also proposes a text decoding method, which includes:

[0011] Based on the current text segment to be decoded, obtain a key-value cache of several decoded text segments;

[0012] Decoding the current text segment to be decoded based on the key value cache to obtain a text prediction inference result;

[0013] Wherein, the key-value cache of the plurality of decoded text segments is obtained by the key-value cache method mentioned above.

[0014] In order to solve the above-mentioned technical problems, the present application also proposes a text decoding device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the key-value caching method and / or text decoding method as mentioned above.

[0015] In order to solve the above technical problems, the present application also proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above key-value caching method and / or text decoding method.

[0016] Compared with the prior art, the beneficial effects of the present application are as follows: the text decoding device obtains several decoded text segments based on the current text segment to be decoded; obtains the key value caches of the several decoded text segments, wherein the key value caches are several key value caches arranged from high to low according to the self-attention layer of the large language model network; retains part of the key value caches in all key value caches according to the preset key value cache allocation strategy, and discards the remaining key value caches; wherein, the number of key value caches of the first self-attention layer in the retained key value caches of the several decoded text segments is less than the key value caches of the second self-attention layer, and the number of layers of the first self-attention layer is lower than the number of layers of the second self-attention layer. Through the above-mentioned key value cache method, according to the characteristics of dense semantics of top-level network features and sparse semantics of bottom-level network features, key value caches of different sizes are retained from top to bottom, so that the retained key value caches are more effective, the main information required for text segment decoding will not be lost, and the key value cache efficiency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0018] Figure 1 It is a flowchart of an embodiment of a key-value caching method provided by the present application;

[0019] Figure 2 It is a schematic diagram of the overall process of the key-value caching method provided by this application;

[0020] Figure 3 yes Figure 1 A schematic diagram of the flow chart of the first embodiment of step S13 in the key-value caching method shown;

[0021] Figure 4 This is a schematic diagram of the effect of the first key-value cache allocation strategy provided by this application;

[0022] Figure 5 yes Figure 3 The specific flow chart of step S21 of the key-value caching method shown;

[0023] Figure 6 yes Figure 1 A schematic diagram of a flow chart of a second embodiment of step S13 in the key-value caching method shown;

[0024] Figure 7 This is a schematic diagram of the effect of the second key-value cache allocation strategy provided by this application;

[0025] Figure 8 yes Figure 6 The specific flow diagram of step S33 of the key-value caching method shown;

[0026] Fig. 9 yes Figure 1 A schematic diagram of a flow chart of a third embodiment of step S13 in the key-value caching method shown;

[0027] Fig.10 This is a schematic diagram of the effect of the third key-value cache allocation strategy provided by this application;

[0028] Fig.11 yes Fig. 9 The specific flow chart of step S43 of the key-value caching method shown;

[0029] Fig.12 It is a schematic diagram of the principle of group averaging provided by this application;

[0030] Fig.13 It is a flowchart of another embodiment of the key-value caching method provided by the present application;

[0031] Fig.14 It is a schematic diagram of the principle of an embodiment of a memory pool provided by the present application;

[0032] Fig.15It is a flowchart of an embodiment of a text decoding method provided by the present application;

[0033] Fig.16 It is a structural diagram of an embodiment of a text decoding device provided by the present application;

[0034] Fig.17 It is a structural diagram of an embodiment of a computer storage medium provided by the present application. DETAILED DESCRIPTION

[0035] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0036] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can, for example, be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] Large language models are usually designed with a hierarchical network structure, which stacks self-attention layers. This hierarchical structure design allows the network to show dense semantics in the top-level features, a more generalized semantic information, and sparse semantics in the bottom-level features, a semantic information closer to the target token. Therefore, the key-value cache method of this application tends to retain more bottom-level features.

[0038] Please refer to Figure 1 and Figure 2 , Figure 1 is a flowchart of an embodiment of a key-value cache method provided by the present application. Figure 2 It is a schematic diagram of the overall flow of the key-value caching method provided by this application.

[0039] The key-value caching method of the present application is applied to a text decoding device, wherein the text decoding device of the present application can be a server, a terminal device, or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the text decoding device, such as various units, subunits, modules, and submodules, can all be set in the server, can all be set in the terminal device, or can be set in the server and the terminal device respectively.

[0040] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here.

[0041] like Figure 1 As shown, the specific steps are as follows:

[0042] Step S11: Based on the current text segment to be decoded, obtain a number of decoded text segments.

[0043] In the embodiment of the present application, the text decoding device obtains the current token that needs to be decoded and generated in the large language model inference stage, that is, the current text segment to be decoded. Based on the aforementioned large language model inference principle, the current token needs to be calculated using the key values ​​of the previous n-1 tokens according to the characteristics of the self-attention structure. Therefore, the text decoding device needs to extract the key value cache of several tokens that have been decoded and generated by the large language model.

[0044] Step S12: Obtain key-value caches of several decoded text fragments, wherein the key-value caches are several key-value caches arranged from high to low according to the self-attention layer of the large language model network.

[0045] In the embodiment of the present application, the key-value cache of each token specifically includes the key-value cache of each self-attention layer, and each key-value cache represents the semantic features of the corresponding self-attention layer. The self-attention layers of the large language model network are arranged from high to low, the top-level features store dense semantic information, and the bottom-level features store sparse semantic information.

[0046] Step S13: retaining part of the key value caches in all the key value caches according to the preset key value cache allocation strategy, and discarding the rest of the key value caches.

[0047] In the embodiment of the present application, the text decoding device can use a key-value cache with more tokens at the bottom layer and a key-value cache with fewer tokens at the top layer according to the characteristics that the top-level features of the large language model network store dense semantic information and the bottom-level features store sparse semantic information. Three feasible key-value cache allocation strategies are listed according to this characteristic.

[0048] like Figure 2 As shown, the following three key-value cache allocation strategies are used to determine the total size of the key-value cache that needs to be retained in the current decoding stage. However, the text decoding device then screens all key-value caches of the decoded token according to the self-attention distribution to determine the specific key-value cache of the total size of the key-value cache for reasoning about the current token.

[0049] It should be noted that the three key-value cache allocation strategies provided in this application are used to determine the number of key-value caches that need to be retained in each self-attention layer, and their quantity distribution needs to meet the following conditions: the number of key-value caches of the first self-attention layer in the retained key-value caches of several decoded text fragments is less than the key-value caches of the second self-attention layer, and the number of layers of the first self-attention layer is lower than the number of layers of the second self-attention layer.

[0050] After the text decoding device determines the key-value cache that needs to be retained according to the selected key-value cache allocation strategy, this part of the key-value cache can be saved and the rest of the key-value cache can be discarded, thereby making the retained key-value cache more effective.

[0051] In the present application, a text decoding device obtains a number of decoded text segments based on a current text segment to be decoded; obtains key value caches of the decoded text segments, wherein the key value caches are a number of key value caches arranged from high to low according to the self-attention layer of a large language model network; retains part of the key value caches in all key value caches according to a preset key value cache allocation strategy, and discards the remaining key value caches; wherein, the number of key value caches of the first self-attention layer in the retained key value caches of the decoded text segments is less than the number of key value caches of the second self-attention layer, and the number of layers of the first self-attention layer is lower than the number of layers of the second self-attention layer. Through the above-mentioned key value caching method, according to the characteristics of dense semantics of top-level network features and sparse semantics of bottom-level network features, key value caches of different sizes are retained from top to bottom, so that the retained key value cache is more effective, the main information required for text segment decoding will not be lost, and the efficiency of the key value cache is improved.

[0052] The following describes each key-value cache allocation strategy and its corresponding key-value cache filtering strategy in detail:

[0053] For details about the first key-value cache allocation strategy, please refer to Figure 3 and Figure 4 , Figure 3 yes Figure 1 The flowchart of step S13 in the first embodiment of the key-value caching method shown in FIG. Figure 4 This is a schematic diagram of the effect of the first key-value cache allocation strategy provided by this application.

[0054] like Figure 3 As shown, the specific steps are as follows:

[0055] Step S21: retain a first preset number of key value caches in the key value cache of the first self-attention layer according to the first key value cache allocation strategy.

[0056] In the embodiment of the present application, the text decoding device is based on Figure 4 The key-value cache allocation strategy shown allocates the number of key-value caches.

[0057] Specifically, assuming that the large language model has a total of self-attention layer. If tokens have been generated in the current decoding stage, then The key-value cache of the self-attention layer can be divided into and , where d is the length of the vector of a single key value. Figure 4 As shown, the ones filled with color are retained key-value caches, and the ones without color are discarded key-value caches.

[0058] The feasible key-value cache size allocation strategy provided in the embodiment of the present application is that the cache size reserved for the first layer is , the cache size reserved at the mth layer is , each layer decreases monotonically with equal size, then the The key-value cache size of the layer is:

[0059]

[0060] Therefore, the total size of the key-value cache that needs to be retained in the current decoding phase is .

[0061] In other embodiments, the difference between each self-attention layer may also be different, which are not listed here one by one.

[0062] It should be noted that Figure 4 It is only the result of allocating the number of key-value caches for the first key-value cache allocation strategy provided by this application, and it is not a limitation on the specific key-value cache retention position. Therefore, this application also provides a key-value cache screening strategy corresponding to the first key-value cache allocation strategy, retaining important key-value caches, while the remaining key-value caches can be discarded and do not participate in the token generation of the next decoding stage:

[0063] Please refer to Figure 5 , Figure 5 yes Figure 3 A specific flow chart of step S21 of the key-value caching method is shown.

[0064] like Figure 5 As shown, the specific steps are as follows:

[0065] Step S211: Determine a first preset number of first self-attention layers according to a first key-value cache allocation strategy.

[0066] In an embodiment of the present application, the text decoding device determines the number of key value caches for each self-attention layer according to the first key value cache allocation strategy, and the process will not be repeated here.

[0067] Step S212: Based on the query vector and key vector of each key-value cache of the first self-attention layer, obtain the attention score of each key-value cache.

[0068] In the embodiment of the present application, any layer of self-attention layer will be taken as an example to illustrate the corresponding key-value cache screening strategy:

[0069] Specifically, the self-attention layer uses the attention weight to perform a weighted sum on the value. A high attention score applies a larger proportion to the value, indicating that the key-value cache at the position with the high attention score is more important and needs more attention. Therefore, the key-value cache at the position with the high attention score can be retained. The self-attention used by a typical large language model is usually multi-head attention, with each head The attention weight calculation is as follows:

[0070]

[0071] Thus, the text decoding device obtains the attention score of each key-value cache.

[0072] Step S213: retain a first preset number of key value caches of the first self-attention layer, arranged in descending order according to the attention scores.

[0073] In the embodiment of the present application, the text decoding device uses the attention score to generate Quantify the importance of each token.

[0074] Then, the text decoding device sorts the key value cache of each self-attention layer according to the attention score of each key value cache, such as The layers are sorted by attention score, retaining the top A key-value cache.

[0075] Step S22: retain a second preset number of key value caches in the key value cache of the second self-attention layer.

[0076] Step S23: retain a third preset number of key value caches in the key value cache of the third self-attention layer.

[0077] Step S24: discard the remaining key-value caches.

[0078] The second key-value cache allocation strategy is described in detail in Figure 6 and Figure 7 , Figure 6 yes Figure 1 The flowchart of the second embodiment of step S13 in the key-value caching method shown in FIG. Figure 7 This is a schematic diagram of the effect of the second key-value cache allocation strategy provided by this application.

[0079] like Figure 6 As shown, the specific steps are as follows:

[0080] Step S31: Divide the key value cache of the decoded text segments into a first key value cache and a second key value cache according to the second key value cache allocation strategy.

[0081] In the embodiment of the present application, in order to further maintain the generalization ability of the model, Figure 7 As shown, the second key-value cache size allocation strategy is introduced. The text decoding device divides n tokens into two parts, namely the latest generated The key-value cache of tokens, and the remaining A key-value cache of tokens.

[0082] Step S32: For the first part of the key-value cache, execute: retain all key-value caches.

[0083] In the embodiments of the present application, Figure 7 As shown, the text decoding device caches the first part of the key value, that is, All key-value caches of tokens are retained. The latest generated The token has a stronger contextual relationship with the current token. Therefore, choosing to keep all the key-value caches in the first part can effectively improve the reasoning accuracy of large models.

[0084] Step S33: For the second part of the key value cache, execute: retain the first preset number of key value caches in the key value cache of the first self-attention layer; retain the second preset number of key value caches in the key value cache of the second self-attention layer; retain the third preset number of key value caches in the key value cache of the third self-attention layer; and discard the remaining key value caches.

[0085] In the embodiments of the present application, Figure 7 As shown, the text decoding device caches the second part of the key value, that is, The key-value cache of each token adopts the first key-value cache allocation strategy to determine the number of key-value caches that need to be retained in each self-attention layer.

[0086] Therefore, the total number of key-value caches that need to be retained for each self-attention layer can be expressed as: The key-value cache size retained in the first layer is , No. The size of the key-value cache retained by the layer is , then The key-value cache size of the layer is:

[0087]

[0088] The total size of the key-value cache that needs to be retained in the current decoding phase is .

[0089] In other embodiments, the difference between each self-attention layer may also be different, which are not listed here one by one.

[0090] It should be noted that Figure 7 It is only the result of allocating the number of key-value caches for the second key-value cache allocation strategy provided by this application, and it is not a limitation on the specific key-value cache retention position. Therefore, this application also provides a key-value cache screening strategy corresponding to the second key-value cache allocation strategy, retaining important key-value caches, while the remaining key-value caches can be discarded and do not participate in the token generation of the next decoding stage:

[0091] Please refer to Figure 8 , Figure 8 yes Figure 6 The specific flow chart of step S33 of the key-value caching method is shown.

[0092] like Figure 8 As shown, the specific steps are as follows:

[0093] Step S331: Determine a first preset number of first self-attention layers according to the second key-value cache allocation strategy.

[0094] In an embodiment of the present application, the text decoding device determines the number of key value caches for each self-attention layer according to the second key value cache allocation strategy, and the process will not be repeated here.

[0095] Step S332: Based on the key vector of each key-value cache of the first self-attention layer and the query vector of the first part of the key-value cache, obtain the attention score of each key-value cache.

[0096] In the embodiment of the present application, the second feasible key-value cache screening strategy adopted by the text decoding device is to use the generated token and the most recent token to calculate the sum of the attention scores, and the calculation formula is as follows:

[0097]

[0098] Among them, the attention score of the second part key-value cache is calculated by its own key vector and each query vector of the first part key-value cache according to the attention weight calculation formula, and the calculation results of all the first part key-value caches are accumulated.

[0099] It should be noted that, according to the second key value cache allocation strategy, The key-value cache of each token is retained, so there is no need to calculate the attention score.

[0100] Step S333: retain a first preset number of key value caches of the first self-attention layer, arranged in descending order according to the attention scores.

[0101] In the embodiment of the present application, the text decoding device uses the attention score to generate The importance of the key-value cache of each token is quantified.

[0102] Then, the text decoding device sorts the key value cache of each self-attention layer according to the attention score of each key value cache of the second part of the key value cache, such as Layer by attention score Sort and keep the first A key-value cache.

[0103] For details about the third key-value cache allocation strategy, please refer to Fig. 9 and Fig.10 , Fig. 9 yes Figure 1 The flowchart of step S13 in the third embodiment of the key-value caching method shown in FIG. Fig.10 This is a schematic diagram of the effect of the third key-value cache allocation strategy provided by this application.

[0104] like Fig. 9 As shown, the specific steps are as follows:

[0105] Step S41: Divide the key value cache of the decoded text segments into a first key value cache and a second key value cache according to the third key value cache allocation strategy.

[0106] In the embodiment of the present application, in order to further reduce the key value cache size, such as Fig.10 As shown, the third key-value cache size allocation strategy is introduced. The text decoding device divides n tokens into two parts, namely the latest generated The key-value cache of tokens, and the remaining A key-value cache of tokens.

[0107] Step S42: For the second part of the key value cache, execute: retain the first preset number of key value caches in the key value cache of the first self-attention layer; retain the second preset number of key value caches in the key value cache of the second self-attention layer; retain the third preset number of key value caches in the key value cache of the third self-attention layer; and discard the remaining key value caches.

[0108] In the embodiments of the present application, Fig.10 As shown, the text decoding device caches the second part of the key value, that is, The key-value cache of each token adopts the first key-value cache allocation strategy to determine the number of key-value caches that each self-attention layer needs to retain. Please refer to the above step S33 for details, which will not be repeated here.

[0109] Step S43: For the first part of the key-value cache, execute: divide the key-value caches of all self-attention layers into at least three groups of key-value caches, each group of key-value caches includes at least one key-value cache of the self-attention layer; retain the fourth preset number of key-value caches for each sub-attention layer in the first group of key-value caches; retain the fifth preset number of key-value caches for each sub-attention layer in the second group of key-value caches; retain the sixth preset number of key-value caches for each sub-attention layer in the third group of key-value caches; and discard the remaining key-value caches.

[0110] In the embodiment of the present application, the text decoding device selects the most recent The key-value cache of a token Layer self-attention is divided into The groups are divided into groups, and the groups are divided into groups in descending order according to the equal size. The same key value cache size is adopted in the groups. Layer group number for:

[0111]

[0112] If the key-value cache size of group t is , the key-value cache of group 1 is , then The key-value cache size for a group is:

[0113]

[0114] Combined with the distribution of the key-value cache in the second part, we can get The key-value cache size for the layer is:

[0115]

[0116] The total size of the key-value cache that needs to be retained in the current decoding phase is .

[0117] In other embodiments, the difference between each self-attention layer may also be different, which are not listed here one by one.

[0118] It should be noted that Fig.10 It is only the result of allocating the number of key-value caches for the third key-value cache allocation strategy provided by this application, and it is not a limitation on the specific key-value cache retention position. Therefore, this application also provides a key-value cache screening strategy corresponding to the third key-value cache allocation strategy, retaining important key-value caches, while the remaining key-value caches can be discarded and do not participate in the token generation of the next decoding stage:

[0119] Please refer to Fig.11 , Fig.11 yes Fig. 9 A specific flow chart of step S43 of the key-value caching method is shown.

[0120] like Fig.11 As shown, the specific steps are as follows:

[0121] Step S431: Determine a fourth preset number of the first group of key-value caches according to the third key-value cache allocation strategy.

[0122] Step S432: Obtain the average key value cache of all sub-attention layers in the first group of key value caches, wherein each average key value cache includes an average key vector corresponding to the decoded text segment and an average query vector.

[0123] In the embodiment of the present application, the third feasible key-value cache screening strategy adopted by the text decoding device is tokens adopt the first key-value cache filtering strategy, and the most recent Tokens according to Fig.12 The calculation shown is to average the key cache within the group. , find the average of the query vector in each group .

[0124] Step S433: Based on the average key vector and the average query vector of each decoded text segment, obtain the attention score of each average key value cache.

[0125] In the embodiment of the present application, the text decoding device performs attention weight calculation, and the formula is as follows:

[0126]

[0127] Step S434: retaining a fourth preset number of key value caches of the average key value cache arranged in descending order according to the attention scores.

[0128] In the embodiment of the present application, the text decoding device sorts the key value cache of each self-attention layer according to the attention score of each key value cache of the first part of the key value cache, such as Layer by attention score Sort and keep the first A key-value cache.

[0129] In the present application, the text decoding device retains key-value caches of different sizes from top to bottom based on the characteristics of dense semantics of top-level network features and sparse semantics of bottom-level network features, so that the retained key-value cache is more effective and does not lose the main information required for decoding.

[0130] Based on the key-value cache allocation strategy and key-value cache screening strategy of the above embodiment, this application can pre-create a memory pool to manage the allocation and recycling of key-value cache in large language model calls according to the characteristic that the total size of the key-value cache in the decoding stage remains unchanged. Fig.13 and Fig.14 , Fig.13 is a flowchart of another embodiment of the key-value caching method provided by the present application. Fig.14 It is a schematic diagram of the principles of an embodiment of a memory pool provided by the present application.

[0131] like Fig.13 As shown, the specific steps are as follows:

[0132] Step S51: determining a portion of key-value caches to be saved and a portion of key-value caches to be discarded in all key-value caches according to a preset key-value cache allocation strategy.

[0133] Step S52: Obtain the memory block usage record of the memory pool.

[0134] In the embodiment of the present application, the text decoding device pre-applies for a continuous memory space, and the capacity of the memory space is determined according to the total size of the key-value cache. Then the memory space is evenly divided into memory blocks, each of which stores the key-value cache of a single token single-layer self-attention, and then a record table is created to maintain the usage record of the memory block. Each entry in the record table records the address and usage of the memory block.

[0135] Since the capacity of the memory space is already determined, the total size of the key-value cache in the text decoding stage can be effectively limited not to exceed the capacity of the memory space.

[0136] The text decoding device determines the key value cache that needs to be retained and the key value cache that needs to be discarded in the current decoding stage through the key value cache allocation strategy and the key value cache screening strategy of the above-mentioned embodiment, and compares them with the memory block usage of the memory pool, so that after the memory pool is updated, the key value cache that needs to be retained is saved to the memory block, and the key value cache that needs to be discarded is discarded from the memory pool or not put into the memory pool.

[0137] Step S53: In response to not finding any partial key-value cache to be saved in the usage record, allocating the partial key-value cache to be saved to a free memory block of the memory pool.

[0138] In the embodiment of the present application, when the text decoding device queries the key value cache to be retained and fails to find it in the current usage record, it means that the key value cache is not saved in the memory block of the memory pool. At this time, the text decoding device queries the free memory block in the memory pool for saving the key value cache.

[0139] When the text decoding device queries the key value cache that needs to be retained and can be found in the current usage record, it means that the key value cache is saved in the memory block of the memory pool, and the saved state can be maintained.

[0140] Step S54: in response to finding any partial key value cache to be discarded in the usage record, discard the partial key value cache to be discarded from the memory block of the memory pool, and set the memory block to an idle state.

[0141] In the embodiment of the present application, when the text decoding device queries the key value cache to be discarded and can be found in the current usage record, it means that the key value cache is stored in the memory block of the memory pool. At this time, the text decoding device discards the key value cache from the memory pool and sets the corresponding entry in the record table to an idle state.

[0142] Please continue reading Fig.15 , Fig.15 It is a flowchart of an embodiment of a text decoding method provided by the present application.

[0143] like Fig.15 As shown, the specific steps are as follows:

[0144] Step S61: Based on the current text segment to be decoded, obtain a key value cache of several decoded text segments.

[0145] In the embodiment of the present application, the text decoding device obtains the previous value of the current token according to the key value caching method of the above embodiment. A key-value cache of tokens.

[0146] Step S62: Decode the current text segment to be decoded based on the key value cache to obtain the text prediction inference result.

[0147] In an embodiment of the present application, the text decoding device inputs the above-mentioned key-value cache and the current token into the large language model at the same time, so as to perform inference prediction on the current token.

[0148] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0149] In order to implement the above-mentioned key-value caching method and / or text decoding method, the present application also proposes a text decoding device, for details, please refer to Fig.16 , Fig.16 It is a structural diagram of an embodiment of a text decoding device provided by the present application.

[0150] The text decoding device 700 of this embodiment includes a processor 71 , a memory 72 , an input / output device 73 , and a bus 74 .

[0151] The processor 71, memory 72, and input / output device 73 are respectively connected to a bus 74. The memory 72 stores program data. The processor 71 is used to execute the program data to implement the key-value caching method and / or text decoding method described in the above embodiments.

[0152] In the embodiment of the present application, the processor 71 may also be referred to as a CPU (Central Processing Unit). The processor 71 may be an integrated circuit chip having the ability to process signals. The processor 71 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 71 may also be any conventional processor, etc.

[0153] This application also provides a computer storage medium, please continue to refer to Fig.17 , Fig.17 It is a structural diagram of an embodiment of a computer storage medium provided in the present application. The computer storage medium 800 stores a computer program 81. When the computer program 81 is executed by the processor, it is used to implement the key-value caching method and / or text decoding method of the above-mentioned embodiment.

[0154] When the embodiments of the present application are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0155] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A key-value caching method, characterized in that: The key-value caching method comprises: Based on the current text segment to be decoded, obtain several decoded text segments; Obtaining key-value caches of the plurality of decoded text segments, wherein the key-value caches are a plurality of key-value caches arranged from high to low according to the self-attention layer of the large language model network; According to the preset key value cache allocation strategy, some key value caches are retained in all key value caches, and the rest are discarded; Among them, the number of key value caches of the first self-attention layer in the retained key value cache of the several decoded text fragments is greater than the key value cache of the second self-attention layer, the number of layers of the first self-attention layer is lower than the number of layers of the second self-attention layer, and the first self-attention layer and the second self-attention layer are two adjacent layers.

2. The key-value caching method according to claim 1, characterized in that: The self-attention layer includes at least three layers; According to the preset key-value cache allocation strategy, some key-value caches are retained in all key-value caches, and the rest are discarded, including: Retain a first preset number of key value caches in the key value cache of the first self-attention layer according to the first key value cache allocation strategy; Retaining a second preset number of key-value caches in the key-value cache of the second self-attention layer; Retain a third preset number of key-value caches in the key-value cache of the third self-attention layer; Discard the remaining key-value caches; Among them, the first self-attention layer, the second self-attention layer and the third self-attention layer are arranged from low to high, the first preset number is greater than the second preset number, and the second preset number is greater than the third preset number.

3. The key-value caching method according to claim 2, characterized in that: The step of retaining a first preset number of key value caches in the key value cache of the first self-attention layer according to the first key value cache allocation strategy includes: Determining a first preset number of the first self-attention layers according to the first key-value cache allocation strategy; Based on the query vector and the key vector of each key-value cache of the first self-attention layer, obtaining an attention score of each key-value cache; Retain a first preset number of key-value caches of the first self-attention layer, arranged in descending order of attention scores.

4. The key-value caching method according to claim 1, characterized in that: The self-attention layer includes at least three layers; According to the preset key-value cache allocation strategy, some key-value caches are retained in all key-value caches, and the rest are discarded, including: Dividing the key value cache of the plurality of decoded text segments into a first portion of key value cache and a second portion of key value cache according to a second key value cache allocation strategy; For the first part of key-value cache, execute: retain all key-value caches; For the second part of the key value cache, execute: retain the first preset number of key value caches in the key value cache of the first layer of self-attention layer; retain the second preset number of key value caches in the key value cache of the second layer of self-attention layer; retain the third preset number of key value caches in the key value cache of the third layer of self-attention layer; discard the remaining key value caches; Among them, the first self-attention layer, the second self-attention layer and the third self-attention layer are arranged from low to high, the first preset number is greater than the second preset number, and the second preset number is greater than the third preset number.

5. The key-value caching method according to claim 4, characterized in that: The step of retaining a first preset number of key-value caches in the key-value cache of the first self-attention layer includes: Determining a first preset number of the first self-attention layers according to the second key-value cache allocation strategy; Based on the key vector of each key-value cache of the first self-attention layer and the query vector of the first part of the key-value cache, obtaining an attention score of each key-value cache; Retain a first preset number of key-value caches of the first self-attention layer, arranged in descending order of attention scores.

6. The key-value caching method according to claim 1, characterized in that: The self-attention layer includes at least three layers; According to the preset key-value cache allocation strategy, some key-value caches are retained in all key-value caches, and the rest are discarded, including: Dividing the key value cache of the plurality of decoded text segments into a first portion of key value cache and a second portion of key value cache according to a third key value cache allocation strategy; For the second part of the key value cache, execute: retain the first preset number of key value caches in the key value cache of the first layer of self-attention layer; retain the second preset number of key value caches in the key value cache of the second layer of self-attention layer; retain the third preset number of key value caches in the key value cache of the third layer of self-attention layer; discard the remaining key value caches; Wherein, the first self-attention layer, the second self-attention layer, and the third self-attention layer are arranged from low to high, the first preset number is greater than the second preset number, and the second preset number is greater than the third preset number; For the first part of the key-value cache, execute: divide the key-value caches of all self-attention layers into at least three groups of key-value caches, each group of key-value caches includes at least one key-value cache of the self-attention layer; retain the fourth preset number of key-value caches of each sub-attention layer in the first group of key-value caches; retain the fifth preset number of key-value caches of each sub-attention layer in the second group of key-value caches; retain the sixth preset number of key-value caches of each sub-attention layer in the third group of key-value caches; discard the remaining key-value caches; Among them, the self-attention layers of the first group of key-value caches, the self-attention layers of the second group of key-value caches, and the self-attention layers of the third group of key-value caches are arranged from low to high, the fourth preset number is greater than the fifth preset number, and the fifth preset number is greater than the sixth preset number.

7. The key-value caching method according to claim 6, characterized in that: The retaining a fourth preset number of key-value caches for each sub-attention layer in the first group of key-value caches includes: Determine a fourth preset number of the first group of key-value caches according to the third key-value cache allocation strategy; Obtaining average key value caches of all sub-attention layers in the first group of key value caches, wherein each average key value cache includes an average key vector corresponding to a decoded text segment and an average query vector; Based on the average key vector and the average query vector of each decoded text segment, obtaining an attention score of each average key value cache; A fourth preset number of key value caches of the average key value cache are retained, which are arranged in descending order according to the attention scores.

8. The key-value caching method according to claim 1, characterized in that: The method of retaining part of the key value caches in all the key value caches according to the preset key value cache allocation strategy and discarding the rest of the key value caches includes: Determine, according to a preset key value cache allocation strategy, a portion of key value caches to be saved and a portion of key value caches to be discarded in all key value caches; Get the memory block usage record of the memory pool; In response to not finding any of the partial key-value caches to be saved in the usage record, allocating the partial key-value caches to be saved to a free memory block of the memory pool; In response to finding any one of the partial key-value caches to be discarded in the usage record, the partial key-value cache to be discarded is discarded from a memory block of the memory pool, and the memory block is set to an idle state.

9. A text decoding method, characterized in that: The text decoding method comprises: Based on the current text segment to be decoded, obtain a key-value cache of several decoded text segments; Decoding the current text segment to be decoded based on the key-value cache to obtain a text prediction inference result; Wherein, the key-value cache of the plurality of decoded text segments is obtained by the key-value cache method described in any one of claims 1 to 8.

10. A text decoding device, characterized in that: The text decoding device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the key-value caching method as described in any one of claims 1 to 8, and / or the text decoding method as described in claim 9.

11. A computer storage medium, characterized in that: The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the key-value caching method according to any one of claims 1 to 8, and / or the text decoding method according to claim 9.

Images