Text inference acceleration method applied to large language model and related device

Abstract

The present disclosure provides a text inference acceleration method applied to a large language model and a related device, relating to the technical field of data processing, and in particular to the technical fields of large language models, deep learning, and long text inference. The specific implementation scheme is as follows: selecting, from a target token set stored in a video memory, a core token set that needs to be retained, wherein the core token set at least comprises a first token subset, the first token subset is determined on the basis of attention scores obtained by performing a global query operation on the target token set using a plurality of proxy tokens, and each proxy token is selected from the target token set; and on the basis of the core token set, performing an eviction operation on the target token set in the video memory. In this embodiment of the present disclosure, a core token set that needs to be retained is selected by means of a plurality of proxy tokens, such that the processing burden caused by unnecessary tokens is reduced, thereby effectively freeing video memory space and improving the inference speed of the large language model.

Description

Text reasoning acceleration method and related device applied to large language model Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to technical fields such as large language models, deep learning, long text reasoning, and reasoning acceleration. Background Art

[0002] The "large" nature of large language models is reflected in two aspects: the large scale of model parameters and the large scale of training data. In related technologies, large language models are often built based on transformer models. During the inference phase of large language models, a key-value cache (KV cache) is often used to accelerate the inference process.

[0003] However, as sequence length increases, the GPU memory required to store the key-value pairs (KVs) increases linearly. This results in a significant impact on inference speed in large language model inference acceleration scenarios, especially in inference acceleration of long texts, as GPU memory usage significantly impacts inference speed.

[0004] Summary of the Invention

[0005] The present disclosure provides a text reasoning acceleration method and related devices applied to a large language model.

[0006] According to one aspect of the present disclosure, a method for accelerating text reasoning applied to a large language model is provided, comprising:

[0007] Filtering a core word set to be retained from the target word set stored in the video memory; wherein the core word set includes at least a first word subset, the first word subset being determined based on attention scores obtained by performing a global query operation on the target word set using multiple proxy word sets, and each proxy word set being selected from the target word set;

[0008] Based on the core word set, an eviction operation is performed on the target word set in the video memory.

[0009] According to another aspect of the present disclosure, a text reasoning acceleration device applied to a large language model is provided, comprising:

[0010] a screening module, configured to screen a core word set to be retained from the target word set stored in the video memory; wherein the core word set includes at least a first word subset, the first word subset being determined based on attention scores obtained by performing a global query operation on the target word set by multiple proxy word sets, and each proxy word set being screened from the target word set;

[0011] An eviction module is configured to perform an eviction operation on the target word set in the video memory based on the core word set.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned method.

[0016] According to yet another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the above method.

[0017] According to yet another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the above method when executed by a processor.

[0018] The disclosed embodiment uses multiple proxy word elements to filter out the core word element set that needs to be retained, reducing the processing pressure of unnecessary word elements, thereby effectively freeing up video memory space and improving the inference speed of large language models.

[0019] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] FIG1 is a flow chart of a method for accelerating text reasoning applied to a large language model according to an embodiment of the present disclosure.

[0022] FIG2 is a schematic diagram of a process for determining a first token subset according to an embodiment of the present disclosure.

[0023] FIG3 is a schematic diagram of a process for obtaining attention scores of candidate word units for each attention head according to an embodiment of the present disclosure.

[0024] FIG4 is a schematic diagram of the process of querying each attention head according to an embodiment of the present disclosure.

[0025] FIG5 is a schematic diagram of a random algorithm sampling process according to an embodiment of the present disclosure.

[0026] FIG6 is a schematic diagram of a process of eviction during the encoding phase according to an embodiment of the present disclosure.

[0027] FIG7 is a schematic diagram of a process of eviction during the decoding phase according to an embodiment of the present disclosure.

[0028] FIG8 is a schematic diagram of token eviction combining proxy token eviction and random eviction according to an embodiment of the present disclosure.

[0029] FIG9 is a schematic diagram of the structure of a text reasoning acceleration device applied to a large language model according to an embodiment of the present disclosure.

[0030] FIG10 is a block diagram of an electronic device for implementing the text reasoning acceleration method applied to a large language model according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0031] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0032] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features. Throughout the present disclosure, "plurality" means two or more, unless otherwise specifically defined.

[0033] In addition, in order to better illustrate the solutions provided by the embodiments of the present disclosure, numerous specific details are provided in the following detailed description. Those skilled in the art will understand that the embodiments of the present disclosure can still be implemented without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art are not described in detail in order to highlight the main points of the embodiments of the present disclosure.

[0034] The reasoning generation of a large language model can be divided into two stages:

[0035] Encoding: This phase involves encoding the input prompt. The keys and values ​​calculated at each layer are cached to accelerate the subsequent decoding phase. This can be considered a forward pass of the model, decoding a token. A token is a unit of text, such as a word, character, or phrase.

[0036] Decoding phase: This phase enters the generation process loop, where tokens are decoded, generated, and output one by one. The input is the token decoded by the previous process. Then, the model performs forward calculations. When performing attention (attention mechanism operation), the previously cached key and value are used to complete the calculation in the attention operator. The generated new key and value are cached to provide acceleration for the calculation of the next token.

[0037] In long-text inference scenarios, in addition to model parameters, keys and values ​​are cached to accelerate inference. Generally, graphics memory usage increases linearly with the length of the token sequence. In the Attention calculation module, the computational complexity is quadratic with the sequence length.

[0038] In view of this, the embodiments of the present disclosure propose a method for accelerating inference based on cached KV. This method accelerates inference by evicting less important tokens, that is, by evicting the KV values ​​of unimportant tokens in the video memory. This method is not only applicable to large language models for short texts, but is particularly applicable to large language models for long texts. This method is applicable to large language models with decoder-only structures, or large language models that predict tokens one by one.

[0039] It should be noted that in the embodiments of the present disclosure, the definition of long text does not have a fixed word limit, and the concept of long text will vary according to different models, application scenarios and needs. Generally speaking, long text refers to text length that exceeds the default processing capability of the model. For example, some common NLP (natural language processing) models have a maximum processing length of 2048 "tokens". If the text length exceeds this limit, it can be considered as long text.

[0040] As shown in FIG1 , the text reasoning acceleration method for a large language model provided in an embodiment of the present disclosure includes the following:

[0041] S101, filtering out a core token set to be retained from a target token set stored in a video memory; wherein the core token set includes at least a first token subset, the first token subset being determined based on attention scores obtained by performing a global query operation on the target token set using multiple proxy tokens, and each proxy token being filtered from the target token set;

[0042] In the embodiment of the present disclosure, all tokens currently in the video memory may be assumed to be the target token set.

[0043] Since the tokens stored in the video memory are not of equal importance, the memory usage can be reduced by evicting relatively less important tokens in the video memory, thereby accelerating reasoning. Therefore, in the disclosed embodiment, the core tokens can be understood as relatively important tokens in the video memory, and will be retained. The core token set is a collection of key words for processing text. For example, when the large language model generates a long novel based on a simple description, the large language model will determine the content and direction of the long novel based on the core tokens in the description. These core tokens help the large language model understand the task more accurately, thereby providing more accurate and relevant results.

[0044] In the embodiment of the present disclosure, an attention score can be used to represent the importance of a token. The attention score can describe the degree of correlation between tokens in the target token set from the perspective of a proxy token, which is helpful for screening out key tokens and retaining them.

[0045] S102: Based on the core token set, the target token set in the video memory is evicted.

[0046] In the embodiment of the present disclosure, a global query is performed on all tokens in the video memory through multiple proxy tokens, and the importance of tokens in the target token set can be measured from the perspective of multiple proxy tokens and in multiple dimensions, thereby screening out the core token set that needs to be retained. By performing the eviction operation on the tokens in the video memory, important tokens can be retained as much as possible, thereby reducing the processing pressure of unnecessary tokens and improving the reasoning speed. At the same time, some tokens are expelled to free up video memory space and provide more video memory resources for subsequent reasoning processes. In summary, the embodiment of the present disclosure utilizes multiple proxy tokens, which can effectively free up video memory space and accelerate the reasoning speed of large language models.

[0047] Furthermore, to improve robustness, the disclosed embodiments also propose a randomized algorithm to filter tokens to construct a second token subset. Thus, the core token set in the disclosed embodiments includes not only the aforementioned first token subset, but also the second token subset. Thus, the disclosed embodiments can implement token eviction based on a combination of proxy tokens and a randomized algorithm.

[0048] In order to better understand the text reasoning acceleration method provided by the embodiments of the present disclosure, the following text will explain the long reasoning acceleration method from the aspects of the proxy token-based eviction method, the random eviction method, and the eviction timing.

[0049] 1) Proxy token-based eviction

[0050] To improve the efficiency and quality of text processing by the large language model, the first token subset can be determined based on the following method. The specific steps are shown in Figure 2:

[0051] S201, obtain a target number of tokens from the video memory to obtain a proxy token set including multiple proxy tokens.

[0052] During implementation, a target number of tokens can be randomly selected from the target token set in the video memory to obtain a proxy token set.

[0053] In another possible implementation, in order to increase the speed of expulsion, and considering that the output of the next token is more closely related to the most recent token, in the embodiment of the present disclosure, multiple recently generated tokens are selected as proxy tokens. It can be implemented as follows: from the target token set in the video memory, the target number of tokens are filtered out in order of generation time from late to early, to obtain the multiple proxy tokens. In the embodiment of the present disclosure, multiple recent tokens are selected as proxy tokens, that is, the values ​​of these proxy tokens are the tokens corresponding to the query, that is, the query is used to query the previous context, which is in line with the Attention semantics. In this way, the quality of the filtered core tokens can be improved, thereby improving the timeliness and accuracy of the generated text, and can effectively ensure that the content output by the large model is more in line with the current context and needs. Moreover, selecting the most recent token is simple and easy to implement, which can help improve the reasoning speed.

[0054] S202: Using the query vectors of each proxy token in the proxy token set, a global query is performed on the target token set to obtain the attention score of each candidate token in the candidate token set in the target token set.

[0055] In the disclosed embodiment, the candidate token set can be a subset of the target token set in the video memory, for example, the difference between the target token set and the proxy token set. Alternatively, the candidate token set can be the target token set itself. In the disclosed embodiment, a first token subset to be retained can be screened from the candidate token set. Specifically, in S203, the first token subset to be retained is screened from the candidate token set based on the attention score of each candidate token.

[0056] In the disclosed embodiment, each candidate token in the candidate token set is evaluated from the dimensions of multiple proxy tokens to obtain an attention score for each candidate token, thereby facilitating the screening of core tokens from the candidate token set, and ultimately achieving token expulsion, thereby improving the reasoning speed of the text.

[0057] In the implementation of the present disclosure, as described above, the candidate token set may include two situations. The following describes how to select the first token subset in different situations, including:

[0058] Case (1): When the candidate token set is the target token set, multiple key tokens are screened from the candidate token set according to the size of the attention score to obtain the first token subset; wherein, the larger the attention score, the greater the possibility of being selected as a key token.

[0059] Specifically, a preset number of key tokens may be screened out in descending order of attention scores to construct the first token set.

[0060] The preset number can be determined according to the capacity of the video memory. For example, 20% of the total capacity of the video memory can be selected for reservation.

[0061] In the embodiment of the present disclosure, when the candidate token set is the target token set, the target token set is traversed and the criticality of each token in the target token set is measured using an attention score. Then, based on the size of the attention score, multiple key tokens are screened out from the candidate token set. More important tokens can be screened out from the entire target token set and retained, so as to improve the inference speed by evicting tokens.

[0062] Case (2): When the candidate token set is the difference between the target token set and the multiple proxy tokens, multiple key tokens are screened out from the candidate token set according to the size of the attention score; wherein, the larger the attention score, the greater the possibility of being selected as a key token; then, the union of the multiple key tokens and the proxy token set is determined to obtain the first token subset.

[0063] For example, when the candidate token set is the difference set of the target token set and multiple proxy tokens, multiple key tokens are screened from the candidate token set in descending order of attention scores; the union of the multiple key tokens and the proxy token set is determined to obtain the first token subset.

[0064] It should be noted that the present disclosure does not limit the sorting method, that is, it can be sorted from small to large or from large to small. As long as the sorting can be based on the attention score, multiple tokens with high attention scores can be selected as key tokens.

[0065] In the disclosed embodiment, the difference operation can remove the proxy token and avoid repeated selection of the proxy token as the key token. Therefore, the filtered token can complement the proxy token to jointly construct the first token subset, which helps to more effectively filter out the core token and ensure that the first token subset can better help infer reasonable content as much as possible.

[0066] Regardless of which of the above scenarios the candidate token set falls into, in the disclosed embodiments, each proxy token is queried against the target token set. Each proxy token generates a row vector, resulting in a two-dimensional matrix constructed from multiple row vectors. Each proxy token only queries tokens generated before it. Specifically, the proxy token's query vector q can be used to query the K vectors of each token in the target token set. Each proxy token only queries tokens generated no later than itself.

[0067] Large language models often use multiple attention heads. In the embodiment of the present disclosure, for each attention head, an operation of filtering out a core token set that needs to be retained from the target token set stored in the video memory can be performed separately.

[0068] The core token set is filtered for each attention head separately, so that each attention head can filter out a different core token set based on its own situation. Especially in long-text reasoning scenarios, this can help to search for the core tokens that really need to be retained as much as possible through the independent screening capabilities of different attention heads, thereby achieving reasoning acceleration while better ensuring reasoning quality.

[0069] It is important to emphasize that here, for each attention head, the core token set is selected based on both proxy tokens and random eviction methods. The random eviction method will be explained in detail later, but here we first explain the proxy token-based eviction method.

[0070] Taking the attention module i in the encoding stage as an example, for the nth attention head, its query process can be shown as formula (1):

[0071] In formula (1), x prompt represents the target token set, i.e., the input information of the attention module i; represents the model parameters represented by the query weight corresponding to the nth attention head of the attention module i; represents the model parameter represented by the value weight corresponding to the nth attention head of the attention module i; d represents the size of the model parameter of the attention head, that is, All belong to R d*d A represents the query result. Based on expression (1), A is a two-dimensional matrix, and each element in A is the query result between two tokens.

[0072] For each attention head, it is necessary to use the query vector of each proxy token in the proxy token set to perform a global query on the target token set to obtain the attention score of each candidate token in the candidate token set in the target token set. The specific implementation can be shown in Figure 3, including:

[0073] S301, based on the query matrix constructed by the query vector corresponding to the attention head of each token in the target token set, queries the key-value matrix constructed by the key vector of the attention head of each token in the target token set to obtain the query result; wherein, each token query is generated no later than the token of the token; that is, each token only queries itself and the token generated before it.

[0074] As shown in Figure 4, the Q matrix represents the query matrix constructed by the query vector corresponding to the attention head of each token, the K matrix is ​​the key-value matrix constructed by the key vector of the attention head of each token in the target token set, and the P matrix represents the query result obtained through the query operation. This is only the query result of one attention head. If there are multiple attention heads, each attention head completes the same operation separately to obtain the query result corresponding to each attention head. As shown in Figure 4, a schematic diagram of the P matrix obtained in the case of three attention heads is shown. In Figure 4, the two-dimensional matrix under one attention head is summarized. The same column represents the result after the same token is queried by different tokens, and the same row expresses the query result of the same token for different tokens.

[0075] It should be noted that the proxy tokens selected by each attention head can be the same or different, and this embodiment of the present disclosure does not limit this.

[0076] S302: In the submatrix corresponding to the proxy token in the query result, the query values ​​of the same candidate token in the candidate token set are aggregated to obtain the attention score of each candidate token.

[0077] In this disclosed embodiment, the query values ​​for the same candidate token in the candidate token set are aggregated to obtain an attention score for each candidate token. The aggregated attention score reflects the degree to which the same token is being paid attention to by different proxy tokens, thereby measuring the importance of different tokens from the perspective of multiple proxy tokens. Based on this, key tokens can be accurately screened based on the attention score, facilitating faster inference.

[0078] Continuing with Figure 4 as an example, assume that in a column, from top to bottom, the tokens that perform query operations in the last three rows are proxy tokens. Then the last three rows are the query results based on the proxy token, and the last three rows are the submatrix corresponding to the proxy token. Each value in the submatrix represents a query value. Assuming that the length of the token sequence is S, each row in Figure 4 should have S elements. In the P matrix of Figure 4 (taking only the P matrix of one attention head as an example), the same column corresponds to the same token, indicating the query values ​​of different tokens for the same token. Therefore, the implementation of aggregation is to aggregate the query values ​​of all proxy tokens in the same column.

[0079] The specific aggregation method can be column-by-column summation, that is, summing the query values ​​of all proxy tokens in the same column.

[0080] In some other possible implementations, in order to better evaluate the importance of each candidate token, as shown in FIG3 , step S302 may be implemented as follows:

[0081] S3021, for the same candidate token, obtain at least one query value corresponding to the same candidate token from the submatrix;

[0082] Let’s continue with the query results for the proxy token in the last three rows of an attention head in Figure 4. Assuming that the token corresponding to the first column in Figure 4 is a candidate token, the gray area in the P matrix in Figure 4 represents multiple query values ​​for this candidate token.

[0083] S3022: Map at least one query value corresponding to the same candidate token to a target value range to obtain a mapping value for each query value in the at least one query value;

[0084] In the disclosed embodiment, a nonlinear mapping method can be used to map query values ​​to a value range of 0-1. This can achieve standardization of each query value and better measure the importance of each query value.

[0085] S3023: Determine the cumulative sum of the mapping values ​​of the query values ​​corresponding to the same candidate token to obtain the attention score of the same candidate token.

[0086] The specific summary implementation is shown in formula (2), F score =∑ xb∈R Sofmax(A(xa,xb)) (2)

[0087] In formula (2), F score That is, the attention score obtained by summing the mapping values ​​of each query value corresponding to the same candidate token; Sofmax() represents mapping the query value corresponding to the candidate token to the target value interval, xa represents the candidate token, and xb represents the proxy token. In the case where xa represents the candidate token set and xb represents the proxy token set, F score It is a two-dimensional matrix that includes the attention score of each candidate token.

[0088] In some embodiments, in addition to the direct summation method in Expression (2), the disclosed embodiments may assign corresponding weights to multiple query values ​​corresponding to the same candidate token. After mapping the query values ​​to the target value range, a weighted summation method may be used to calculate the attention score of the candidate token.

[0089] In this disclosed embodiment, by mapping the query values ​​of different proxy tokens to the same candidate token, the query results of different proxy tokens are normalized to better measure the attention levels of different tokens. By aggregating the query results for the same candidate token, the attention score of the candidate token can be comprehensively determined based on the dimensions of different proxy tokens, thereby accurately screening out core tokens.

[0090] It should be noted that when there are multiple attention heads, each attention head selects a core token set. Based on this, the target token set in the video memory is evicted based on the core token set. This can be implemented by: aggregating the core token sets corresponding to each attention head to obtain the total set of tokens to be retained; and evicting tokens outside the total set of tokens in the video memory.

[0091] Therefore, in the case of multiple attention heads, each attention head screens the core token set separately, which increases the diversity of the screened core tokens and can effectively identify the key tokens that need to be retained from the dimensions of different attention heads to facilitate reasoning acceleration.

[0092] Suppose that attention head 1 filters out tokens that need to be retained: token 1, token 3, and token 5. Attention head 2 may filter out token 5, token 8, and token 10. If token 5 and token 10 are more important in reality, this solution can filter out as many core and important tokens as possible and keep them in the video memory. In the case of long text, filtering tokens separately by different attention heads can increase the probability of retaining the true core tokens. When different attention heads are superimposed and a random algorithm is used to filter the tokens that need to be retained, the robustness will be further improved, and the probability of filtering out key tokens will be increased. The random eviction method is explained below.

[0093] 2) Random eviction method

[0094] Relatively important key tokens can be screened out through proxy tokens. However, in long text scenarios, there is a large amount of redundant non-critical information, and proxy tokens may fall into local critical characteristics. In order to improve robustness, the embodiment of the present disclosure additionally introduces a random expulsion method to screen out a second token subset, so as to cooperate with the proxy token and improve the quality of expulsion. In the embodiment of the present disclosure, the second token subset is obtained by performing a random screening operation on the target token set. In order to screen out potential key tokens through the probability theory of random algorithms, improve the robustness of screening key tokens, and realize text reasoning acceleration.

[0095] In some possible implementations, the random algorithm can use a simple uniform distribution or Gaussian distribution algorithm for sampling. Alternatively, the algorithm can use the Attention Score as the sampling weight for sampling. The specific implementation steps are shown in Figure 5:

[0096] S501: Based on the attention scores of each token in the target token set obtained by performing a global query operation on the target token set using multiple proxy tokens, a random sampling weight for each token in a preset sampling set is constructed. The preset sampling set can be all tokens stored in the video memory, i.e., the target token set, or the difference between the target token set and the proxy token set.

[0097] Regardless of whether the preset sampling set is the target token set or the difference between the target token set and the first token subset, in the disclosed embodiment, all tokens in the video memory can be queried using the proxy token to obtain the attention score of each token in the target token set. The attention score is then converted into a random sampling weight for each token in the preset sampling set.

[0098] When the preset sampling set is the target token set, for example, the attention score can be simply used as the weight, or some mapping function can be used to map the attention score to the sampling weight. As long as the higher the attention score, the greater the probability of being randomly sampled, this is not specifically limited in the embodiments of the present disclosure.

[0099] If the preset sampling set is the difference between the target token set and the first token subset, the attention score of each token in the preset sampling set can be simply used as the random sampling weight, or the attention score can be mapped to the random sampling weight through a mapping function. Similarly, as long as the attention score is higher, the probability of being sampled is greater, and this is not specifically limited in the embodiments of the present disclosure.

[0100] Then, in S502 , based on the random sampling weight of each token, a second token subset is randomly selected from the preset sampling set.

[0101] In the disclosed embodiment, sampling weights are determined based on the attention score, allowing the sampling process to be implemented based on the attention score. This allows for a focus on sampling core tokens while ensuring robustness, thereby increasing the probability of sampling core tokens. Furthermore, in the case of multiple attention heads, each head also performs random sampling separately. Based on probabilistic knowledge, this effectively increases the probability of sampling core tokens in long text scenarios. This, combined with the proxy token-based eviction method, can effectively accelerate inference.

[0102] In the disclosed embodiment, the token eviction operation for the video memory is not performed once. To effectively accelerate inference, the token eviction operation is performed multiple times. In terms of the timing of eviction, the disclosed embodiment provides the following solution.

[0103] 3) Timing of eviction

[0104] As described above, the text generation process may include the encoding and decoding stages. To maximize the acceleration of inference, in the embodiment of the present disclosure, when any of the trigger conditions in the trigger condition set is met, the operation of filtering out the core token set to be retained from the target token set stored in the video memory is triggered.

[0105] The trigger condition set includes at least one of the following:

[0106] Trigger condition 1: During the encoding phase, any attention module completes reasoning;

[0107] During the encoding phase of a large language model, the inference acceleration method proposed in the embodiment of the present disclosure can be executed after any attention module completes inference. Thus, during the encoding phase, tokens can be expelled separately for each attention module to improve inference speed.

[0108] For example, a large language model composed of multiple neural network layers has each layer (i.e., each block) responsible for performing specific transformations or extracting features from the input data. During the encoding process, after a module (such as layer i) completes its computation and generates output, the large language model may store these results in a buffer or memory rather than sending them directly to the next layer. This allows non-critical tokens in the current module to be evicted after inference is complete, saving video memory and ensuring faster inference for subsequent layers.

[0109] Since large language models are mostly built based on transformers, in the disclosed embodiment, an eviction can be performed for each block during the encoding phase to better achieve inference acceleration.

[0110] Trigger condition 2: During the decoding phase, the amount of data in the video memory reaches a preset peak value;

[0111] During the decoding phase, when the amount of data in the video memory reaches a preset peak, the memory management strategy can be triggered to release video memory resources by evicting tokens, thereby achieving inference acceleration.

[0112] Trigger condition 3: During the decoding phase, an eviction cycle is reached, where each consecutive prediction of the target number of tokens is an eviction cycle, and tokens in different eviction cycles do not overlap.

[0113] For example, token1, token2, token3, token4, token5, and token6 are generated in sequence. Assuming that every 3 tokens generated is an eviction cycle, eviction is performed once when token3 is output, and once when token6 is output.

[0114] In this embodiment, the decoding stage outputs tokens one by one. When multiple newly generated tokens accumulate, eviction is performed once. Token eviction can be performed periodically to save computing resources and achieve inference acceleration.

[0115] The following takes the encoding stage and decoding stage as examples to illustrate the eviction method of each stage.

[0116] 1. Eviction during the encoding phase

[0117] As shown in Figure 6, assuming that the most recently generated m tokens are selected as proxy tokens, token eviction is performed for each attention head n of each blocki during the encoding phase, including the following steps:

[0118] S601, for the attention head n of blocki, its weight parameter is Based on these weight parameters, the proxy token is used to perform a global query on the target token set constructed by all current tokens to obtain the query result A. That is, the operation shown in expression (1) is performed to obtain the query result A.

[0119] S602: Define the difference between the target token set and the proxy token set constructed by all tokens currently in the video memory as R. Based on the submatrix constructed by the query results of the proxy tokens in the last m rows of the query result A, cumulatively sum each candidate token in R by column to obtain the attention score of each candidate token in R. The specific calculation method can be found in Expression (2).

[0120] S603: Select the top-k candidate tokens with the highest attention scores from R according to a preset ratio as key tokens, and determine the union of all key tokens and proxy token sets to obtain the first filtered token subset. In this way, the tokens to be retained are filtered out based on the proxy tokens.

[0121] S604: Randomly select a second subset of tokens from all current tokens in the video memory. For details, refer to the random eviction method in Section 2 above. This randomly selects the tokens that need to be retained.

[0122] S605: Merge the first token subset filtered out in S603 and the second token subset filtered out in S604 to obtain the core token set that needs to be retained this time corresponding to attention head n;

[0123] S606: Summarize the core token sets that need to be retained by all attention heads and retain them in the video memory, and evict the remaining tokens from the video memory.

[0124] It should be noted that, in the embodiment of the present disclosure, if a token needs to be retained in the video memory, the KV value corresponding to the token must be retained in the video memory. If a token is evicted, the KV value corresponding to the token is deleted from the video memory.

[0125] 2. Eviction during the decoding phase

[0126] As shown in Figure 7, in the decoding phase, tokens are output one by one. Assuming that every time m tokens are accumulated, an eviction is performed, the decoding phase includes the following steps:

[0127] For the attention head of each blocki in the decoding stage, perform operations S701-S707 respectively, and then execute S708.

[0128] S701: For the current token, use it as a proxy token to query the K value of the previously generated token to obtain a row of query results corresponding to the current proxy token.

[0129] The main calculation steps include:

[0130] Step B1: For the previous token of the current token, obtain the KV value of the previous token from the KV value corresponding to the attention head n of block i stored in the video memory;

[0131] The implementation process is shown in expression (3):

[0132] In expression (3), The K value of the nth attention head of the i-th block of the previous token stored in the video memory is assigned to The V value of the nth attention head of the i-th block of the previous token stored in the video memory is assigned to

[0133] In step B2, multiply the current token by the K value weight parameter corresponding to the attention n of blocki, and add the calculated result to the K value of the previous token to obtain the K value of the current token.

[0134] The implementation process is shown in expression (4):

[0135] In expression (4), Indicates the K value corresponding to the nth attention head in blocki of the current token; Indicates the current token in blocki; Represents the K value weight parameter of the nth attention head in blocki, which is a model parameter.

[0136] In step B3, multiply the current token by the V value weight parameter corresponding to blocki's attention n, and add the calculated result to the V value of the previous token obtained to obtain the V value of the current token.

[0137] The implementation process is shown in expression (5):

[0138] In expression (5), Represents the V value corresponding to the nth attention head in blocki of the current token; Indicates the current token in blocki; Represents the V value weight parameter of the nth attention head in blocki, which is a model parameter.

[0139] Among them, steps B2 and B3 calculate the KV value of the current token and also prepare for the inference acceleration of the next token.

[0140] In step B4, the query vector is obtained by multiplying the current token by the Q weight parameter corresponding to the attention head n of block i, and the K values ​​of all tokens before the current token are queried to obtain the query results for all tokens when the current token is used as a proxy token. This query result is equivalent to a row vector of the P matrix in an attention head in Figure 4.

[0141] Its implementation is shown in expression (6):

[0142] In expression (6), A represents the query result when the current token is used as a proxy token; Represents the Q value weight parameter of the nth attention head in blocki, which is a model parameter.

[0143] S702: accumulate the number of new tokens generated. When the accumulated number reaches m tokens, execute S703 to complete an eviction operation.

[0144] S703: Using the m tokens generated most recently as proxy tokens, determine the query results of each proxy token on all tokens stored in the video memory.

[0145] Since each token in S701 generates a row vector in the query result, i.e., a row vector in the P matrix of an attention head in Figure 4, the row vectors of the most recent m tokens are concatenated in order to obtain the query result. In this case, the concatenated query result can be considered as the aforementioned submatrix.

[0146] S704: Define the difference between the target token set and the proxy token set constructed by all tokens currently in the memory as R. For the query values ​​of multiple proxy tokens for the same candidate token in R, determine the attention score of the candidate token. For the specific calculation method, see Expression (2).

[0147] At step S705 , the top-k candidate tokens with the highest attention scores are selected from R according to a preset ratio as key tokens, and the union of all key tokens and proxy token sets is determined to obtain the first filtered token subset. Thus, the tokens to be retained are filtered out based on the proxy tokens.

[0148] S706: Randomly select a second token subset from all current tokens in the video memory. For details, refer to the random eviction method in Section 2 above. In this way, tokens to be retained are randomly selected.

[0149] S707: Merge the first token subset filtered out in S705 and the second token subset filtered out in S706 to obtain the core token set that needs to be retained this time corresponding to attention head n;

[0150] S708: Summarize the core token sets that need to be retained by all attention heads and retain them in the video memory, and evict the remaining tokens from the video memory.

[0151] Whether it's the accelerated inference during the encoding phase shown in Figure 6 or the accelerated inference during the decoding phase shown in Figure 7 , their implementation can be summarized and described using Figure 8 . Referring to Figure 8 , n attention heads are used as an example (only three are shown). In Figure 8 , each attention head can select tokens to be retained based on a random eviction method, and then determine the tokens to be evicted. For example, in Figure 8 , one attention head determines that three tokens need to be evicted, so a random eviction budget of 3 is marked in Figure 8 . Simultaneously, proxy tokens can also be used to select tokens to be retained. For example, in Figure 8 , P1-P4 are selected as proxy tokens, and based on the proxy tokens, two tokens are ultimately determined to be evicted, resulting in a proxy eviction budget of 2 in Figure 8 . The tokens to be retained, selected by both methods, are then merged, along with the tokens selected by all attention heads. The final eviction budget is 5 tokens, implementing the eviction of tokens from the video memory. In Figure 8 , since the decoding phase queries token by token, Figure 8 shows the situation at decoding phase t.

[0152] In the solution shown in FIG8 provided in the embodiment of the present disclosure, the calculation of the retained token includes two parts:

[0153] Key token (i.e., the first token subset): Q*K as shown in Expression (1) and Expression (6) ^T Calculate the Attention Score (i.e., the query result). The Attention Score corresponding to the proxy token is the last p rows. For the last p rows, add the sum column by column (as calculated in Expression (2)) to obtain a score with shape [1, S], where S is the total number of tokens in the video memory. Then, select m1 tokens from this using the Top-K algorithm.

[0154] Random token: Using the fraction of shape [1, S] calculated in the previous step as the random sampling weight, randomly select m2 tokens such that m1 + m2 = C. C is the cache budget, which is the budget value.

[0155] Based on the key methods described above, the reasoning acceleration method provided by the embodiments of this disclosure can be used in the encoding and decoding stages of the long text reasoning process:

[0156] During the encoding phase, for each layer (block), the inference acceleration method provided by this disclosure is first used to calculate the retained tokens, which are then evicted all at once to the given key-value buffer. With this layer-by-layer eviction framework, peak memory usage does not exceed the maximum memory usage.

[0157] Decoding stage: Token-by-token generation. When the current video memory reaches the maximum preset video memory or when periodic eviction is required, the eviction method of the embodiment of the present disclosure is used for each layer (block) to calculate the retained token and then evict it to the given key value buffer.

[0158] The disclosed embodiments, based on proxy tokens and random eviction, achieve globally optimal inference acceleration. Global optimization means that during the encoding phase, all prompt information is utilized during forward computation at each layer. This means that the proxy token query queries all preceding context, calculating the key token (i.e., the first token subset) in one go for eviction.

[0159] To understand the eviction method provided by this disclosure, we provide a specific example to evaluate its effectiveness. Define the encoding length as s, the decoding length as n, the key value cache buffer as C, and k = s / C. The details are as follows:

[0160] Set,training sample parameters,BatchSize=1;

[0161] Long text length (size), SequenceLength = 128*1024;

[0162] Each layer's HiddenSize = 40*128;

[0163] Data type, Dtype = BF16(2byte);

[0164] Default storage value, key value cache buffer = 32*1024;

[0165] The number of tokens generated is 1024, k = (128 * 1024) / (25.6 * 1024) = 5

[0166] In the case of non-eviction:

[0167] The shape of Key and Value is [1,128*1024], and the video memory is: 2*128*1024*40*128*2=2,684,354,560byte=2.5GB

[0168] Decoding stage calculation amount: S 2 +(S+1) 2 +(S+2) 2 +…+(S+1023) 2

[0169] When the proxy token-based random eviction method provided by the embodiment of the present disclosure is adopted:

[0170] The shape of Key and Value is [1,32*1024], and the video memory is: 2*32*1024*40*128*2=671,088,640byte=0.625GB

[0171] Decoding phase computational effort: (1 / 4*S*S)+(1 / 4*S*(S+1))+(1 / 4*S*(S+2))+…+(1 / 4*S*(S+1023))

[0172] It can be concluded that the video memory overhead is 1 / 4 of that without eviction, and the video memory saving is 3 times.

[0173] Comparing the computational complexity of the decoding phase, we can see that:

[0174] k*(2*n*n+3*n+6*s+6*s*s+6*s*n+1) / (2*s*(2*s+n))=7, theoretically 6 times faster.

[0175] Based on the same technical concept, the present disclosure also proposes a text reasoning acceleration device 900 for a large language model, as shown in FIG9 , including:

[0176] A screening module 901 is configured to screen a core word set to be retained from the target word set stored in the video memory; wherein the core word set includes at least a first word subset, the first word subset being determined based on attention scores obtained by performing a global query operation on the target word set using multiple proxy word sets, and each proxy word set being screened from the target word set;

[0177] The eviction module 902 is configured to evict the target word set in the video memory based on the core word set.

[0178] In some embodiments, the screening module comprises:

[0179] an acquiring unit, configured to acquire a target number of word-grams from the video memory to obtain a proxy word-gram set including the plurality of proxy word-grams;

[0180] a query unit, configured to perform a global query on the target word set using the query vector of each proxy word in the proxy word set, so as to obtain an attention score of each candidate word in the candidate word set in the target word set;

[0181] A screening unit is configured to screen out the first word-gram subset to be retained from the candidate word-gram set based on the attention score of each candidate word-gram.

[0182] In some embodiments, the screening unit is specifically configured to:

[0183] When the candidate word-gram set is a difference between the target word-gram set and the plurality of proxy word-grams, a plurality of keyword words are selected from the candidate word-gram set according to the size of the attention scores; wherein the larger the attention score, the greater the possibility of being selected as a keyword word;

[0184] A union of the plurality of keyword grammars and the proxy grammar set is determined to obtain the first grammar subset.

[0185] In some embodiments, the screening unit is specifically used to: when the candidate word set is the target word set, screen out multiple keyword words from the candidate word set according to the size of the attention score to obtain the first word subset; wherein, the larger the attention score, the greater the possibility of being selected as a keyword word.

[0186] In some embodiments, the core word set further includes a second word subset; the second word subset is obtained by performing a random screening operation on the target word set.

[0187] In some embodiments, the screening module comprises:

[0188] a weight determination unit, configured to construct a random sampling weight for each word-gram in a preset sampling set based on the attention score of each word-gram in the target word-gram set obtained after a global query operation is performed on the target word-gram set by the multiple proxy word-grams; the preset sampling set is the target word-gram set, or the preset sampling set is the difference between the target word-gram set and the first word-gram subset;

[0189] The random selection unit is configured to randomly select the second word-unit subset from the preset sampling set based on the random sampling weight of each word-unit.

[0190] In some embodiments, the screening module is specifically configured to:

[0191] In the case of including multiple attention heads, for each attention head, the core word set that needs to be retained is filtered out from the target word set stored in the video memory.

[0192] In some embodiments, for each attention head, the query unit is specifically configured to:

[0193] For each attention head, perform the following operations:

[0194] Based on the query matrix constructed by the query vector corresponding to the attention head of each word in the target word set, a key-value matrix constructed by the key vector of the attention head of each word in the target word set is queried to obtain a query result; wherein each word query is a word generated no later than the word;

[0195] In the submatrix corresponding to the proxy word in the query result, query values ​​of the same candidate word in the candidate word set are aggregated to obtain attention scores of the candidate word.

[0196] In some embodiments, the query unit is specifically configured to:

[0197] For the same candidate word-gram, obtaining at least one query value corresponding to the same candidate word-gram from the submatrix;

[0198] Mapping the at least one query value corresponding to the same candidate word to a target value interval to obtain a mapping value of each query value in the at least one query value;

[0199] Determine the cumulative sum of the mapping values ​​of the query values ​​corresponding to the same candidate word to obtain the attention score of the same candidate word.

[0200] In some embodiments, the eviction module is specifically configured to:

[0201] Summarize the core word sets corresponding to each attention head to obtain the total set of word sets that need to be retained;

[0202] The word units outside the total word unit set in the video memory are driven out of the video memory.

[0203] In some embodiments, the acquiring unit is specifically configured to:

[0204] The target number of word-grams are screened out from the target word-gram set in the video memory in descending order of generation time to obtain the plurality of proxy word-grams.

[0205] In some embodiments, the screening module is specifically configured to:

[0206] When any trigger condition in the trigger condition set is met, performing the operation of filtering out the core word set to be retained from the target word set stored in the video memory;

[0207] The trigger condition set includes at least one of the following:

[0208] In the encoding phase, any attention module completes reasoning;

[0209] During the decoding phase, the amount of data in the video memory reaches a preset peak value;

[0210] During the decoding phase, an expulsion cycle is reached, wherein each time the target number of words are continuously predicted is an expulsion cycle, and words in different expulsion cycles do not overlap.

[0211] For the description of specific functions and examples of each module, sub-module\unit of the device in the embodiment of the present disclosure, please refer to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0212] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0213] FIG10 shows a schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0214] As shown in FIG10 , the device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0215] Various components in device 1000 are connected to I / O interface 1005, including: an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0216] The computing unit 1001 can be a variety of general-purpose and / or specialized processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above, such as the text reasoning acceleration method applied to a large language model. For example, in some embodiments, the text reasoning acceleration method applied to a large language model can be implemented as a computer software program that is tangibly contained in a machine-readable medium, such as the storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the text reasoning acceleration method applied to a large language model described above can be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured in any other appropriate manner (for example, by means of firmware) to execute the text reasoning acceleration method applied to a large language model.

[0217] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0218] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0219] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0220] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0221] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0222] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0223] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0224] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for accelerating text reasoning for a large language model, comprising: Filtering a core word set to be retained from the target word set stored in the video memory; wherein the core word set includes at least a first word subset, the first word subset being determined based on attention scores obtained by performing a global query operation on the target word set using multiple proxy word sets, and each proxy word set being selected from the target word set; Based on the core word set, an eviction operation is performed on the target word set in the video memory.

2. The method according to claim 1, wherein Determining the first word-unit subset includes: Retrieving a target number of word units from the video memory to obtain a proxy word unit set including the plurality of proxy word units; Performing a global query on the target word set using the query vector of each proxy word in the proxy word set to obtain an attention score of each candidate word in the candidate word set in the target word set; Based on the attention score of each candidate word-gram, the first word-gram subset to be retained is screened out from the candidate word-gram set.

3. The method according to claim 2, wherein: The step of screening out the first word-gram subset to be retained from the candidate word-gram set based on the attention score of each candidate word-gram comprises: When the candidate word-gram set is a difference between the target word-gram set and the plurality of proxy word-grams, a plurality of keyword words are selected from the candidate word-gram set according to the size of the attention scores; wherein the larger the attention score, the greater the possibility of being selected as a keyword word; A union of the plurality of keyword grammars and the proxy grammar set is determined to obtain the first grammar subset.

4. The method according to claim 2, wherein: The step of screening out the first word-gram subset to be retained from the candidate word-gram set based on the attention score of each candidate word-gram comprises: When the candidate word-gram set is the target word-gram set, multiple keyword words are selected from the candidate word-gram set according to the size of the attention score to obtain the first word-gram subset; wherein, the larger the attention score, the greater the possibility of being selected as a keyword word.

5. The method according to any one of claims 1 to 4, wherein The core word set also includes a second word subset; the second word subset is obtained by performing a random screening operation on the target word set.

6. The method according to claim 5, wherein: Performing a random screening operation on the target word-gram set to obtain the second word-gram subset includes: constructing a random sampling weight for each word in a preset sampling set based on the attention score of each word in the target word set obtained after performing a global query operation on the target word set by the multiple proxy words; the preset sampling set is the target word set, or the preset sampling set is the difference between the target word set and the first word subset; Based on the random sampling weight of each word unit, the second word unit subset is randomly selected from the preset sampling set.

7. The method according to claim 2, wherein: The step of screening out the core word set to be retained from the target word set stored in the video memory includes: In the case of including multiple attention heads, for each attention head, the core word set that needs to be retained is filtered out from the target word set stored in the video memory.

8. The method according to claim 7, wherein: For each attention head, the query vector of each proxy word in the proxy word set is used to perform a global query on the target word set to obtain an attention score of each candidate word in the candidate word set in the target word set, including: For each attention head, perform the following operations: Based on the query matrix constructed by the query vector corresponding to the attention head of each word in the target word set, a key-value matrix constructed by the key vector of the attention head of each word in the target word set is queried to obtain a query result; wherein each word query is a word generated no later than the word; In the submatrix corresponding to the proxy word in the query result, query values ​​of the same candidate word in the candidate word set are aggregated to obtain attention scores of the candidate word.

9. The method according to claim 8, wherein In the submatrix corresponding to the proxy word in the query result, the query values ​​of the same candidate word in the candidate word set are aggregated to obtain the attention score of each candidate word, including: For the same candidate word-gram, obtaining at least one query value corresponding to the same candidate word-gram from the submatrix; Mapping the at least one query value corresponding to the same candidate word to a target value interval to obtain a mapping value of each query value in the at least one query value; Determine the cumulative sum of the mapping values ​​of the query values ​​corresponding to the same candidate word to obtain the attention score of the same candidate word.

10. The method according to claim 7, wherein: The step of performing an eviction operation on the target word set in the video memory based on the core word set includes: Summarize the core word sets corresponding to each attention head to obtain the total set of word sets that need to be retained; The word units outside the total word unit set in the video memory are driven out of the video memory.

11. The method according to any one of claims 2-4, 7-10, wherein The step of acquiring a target number of word units from the video memory to obtain a proxy word unit set including the plurality of proxy word units includes: The target number of word-grams are screened out from the target word-gram set in the video memory in descending order of generation time to obtain the plurality of proxy word-grams.

12. The method according to any one of claims 1 to 12, wherein The step of screening out the core word set to be retained from the target word set stored in the video memory includes: When any trigger condition in the trigger condition set is met, performing the operation of filtering out the core word set to be retained from the target word set stored in the video memory; The trigger condition set includes at least one of the following: In the encoding phase, any attention module completes reasoning; During the decoding phase, the amount of data in the video memory reaches a preset peak value; During the decoding phase, an expulsion cycle is reached, wherein each time the target number of words are continuously predicted is an expulsion cycle, and words in different expulsion cycles do not overlap.

13. A text reasoning acceleration device applied to a large language model, comprising: a screening module, configured to screen a core word set to be retained from the target word set stored in the video memory; wherein the core word set includes at least a first word subset, the first word subset being determined based on attention scores obtained by performing a global query operation on the target word set by multiple proxy word sets, and each proxy word set being screened from the target word set; An eviction module is configured to perform an eviction operation on the target word set in the video memory based on the core word set.

14. The device according to claim 13, wherein The screening module includes: an acquiring unit, configured to acquire a target number of word-grams from the video memory to obtain a proxy word-gram set including the plurality of proxy word-grams; a query unit, configured to perform a global query on the target word set using the query vector of each proxy word in the proxy word set, so as to obtain an attention score of each candidate word in the candidate word set in the target word set; A screening unit is configured to screen out the first word-gram subset to be retained from the candidate word-gram set based on the attention score of each candidate word-gram.

15. The device according to claim 14, wherein The screening unit is specifically used for: When the candidate word-gram set is a difference between the target word-gram set and the plurality of proxy word-grams, a plurality of keyword words are selected from the candidate word-gram set according to the size of the attention scores; wherein the larger the attention score, the greater the possibility of being selected as a keyword word; A union of the plurality of keyword grammars and the proxy grammar set is determined to obtain the first grammar subset.

16. The device according to claim 14, wherein The screening unit is specifically used to: when the candidate word set is the target word set, screen out multiple keyword words from the candidate word set according to the size of the attention score to obtain the first word subset; wherein, the larger the attention score, the greater the possibility of being selected as a keyword word.

17. The device according to any one of claims 13 to 16, wherein: The core word set also includes a second word subset; the second word subset is obtained by performing a random screening operation on the target word set.

18. The device according to claim 17, wherein The screening module comprises: a weight determination unit, configured to construct a random sampling weight for each word-gram in a preset sampling set based on the attention score of each word-gram in the target word-gram set obtained after a global query operation is performed on the target word-gram set by the multiple proxy word-grams; the preset sampling set is the target word-gram set, or the preset sampling set is the difference between the target word-gram set and the first word-gram subset; The random selection unit is configured to randomly select the second word-unit subset from the preset sampling set based on the random sampling weight of each word-unit.

19. The device according to claim 14, wherein The screening module is specifically used for: In the case of including multiple attention heads, for each attention head, the core word set that needs to be retained is filtered out from the target word set stored in the video memory.

20. The device according to claim 19, wherein For each attention head, the query unit is specifically used to: For each attention head, perform the following operations: Based on the query matrix constructed by the query vector corresponding to the attention head of each word in the target word set, a key-value matrix constructed by the key vector of the attention head of each word in the target word set is queried to obtain a query result; wherein each word query is a word generated no later than the word; In the submatrix corresponding to the proxy word in the query result, query values ​​of the same candidate word in the candidate word set are aggregated to obtain attention scores of the candidate word.

21. The device according to claim 20, wherein The query unit is specifically configured to: For the same candidate word-gram, obtaining at least one query value corresponding to the same candidate word-gram from the submatrix; Mapping the at least one query value corresponding to the same candidate word to a target value interval to obtain a mapping value of each query value in the at least one query value; Determine the cumulative sum of the mapping values ​​of the query values ​​corresponding to the same candidate word to obtain the attention score of the same candidate word.

22. The apparatus according to claim 19, wherein The eviction module is specifically used to: Summarize the core word sets corresponding to each attention head to obtain the total set of word sets that need to be retained; The word units outside the total word unit set in the video memory are driven out of the video memory.

23. The device according to any one of claims 14-16, 19-22, wherein The acquisition unit is specifically configured to: The target number of word-grams are screened out from the target word-gram set in the video memory in descending order of generation time to obtain the plurality of proxy word-grams.

24. The device according to any one of claims 13 to 23, wherein: The screening module is specifically used for: When any trigger condition in the trigger condition set is met, performing the operation of filtering out the core word set to be retained from the target word set stored in the video memory; The trigger condition set includes at least one of the following: In the encoding phase, any attention module completes reasoning; During the decoding phase, the amount of data in the video memory reaches a preset peak value; During the decoding phase, an expulsion cycle is reached, wherein each time the target number of words are continuously predicted is an expulsion cycle, and words in different expulsion cycles do not overlap.

25. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 12.

26. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-12.

27. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images