Text inference acceleration method and apparatus, electronic device, storage medium, and program applicable to large-scale language models

Abstract

This disclosure relates to a method for accelerating text inference applicable to large-scale language models. Bisou Place , electronic devices, storage media and programs The present invention provides a data processing technology, particularly in the fields of large-scale language models, deep learning, and long-text inference. Specifically, it includes screening a core token set that needs to be retained from a target token set stored in video memory, wherein the core token set includes at least a first token subset determined based on attention scores obtained by a plurality of proxy tokens performing a global query operation against the target token set, and each proxy token is screened from the target token set; and performing an eradication operation on the target token set in video memory based on the core token set. Embodiments of the present disclosure can screen a core token set that needs to be retained through a plurality of proxy tokens, reduce the processing load of unnecessary tokens, thereby effectively freeing up video memory space and increasing the inference speed of large-scale language models.

Description

[Technical Field]

[0001] This disclosure relates to data processing technologies, particularly to technologies such as large-scale language models, deep learning, long-text inference, and inference acceleration. [Background technology]

[0002] The "large scale" in large-scale language models refers to two aspects: the scale of the model parameters and the scale of the training data. In related technologies, large-scale language models are often built on transformer models. During the inference phase of large-scale language models, a key-value cache (KV cache) is typically used to accelerate the inference process.

[0003] However, as the sequence length increases, the demand for video memory to store the key verb (KV) increases linearly. Therefore, in accelerated inference scenarios for large language models, especially in accelerated long-text inference scenarios, video memory usage significantly impacts inference speed. [Overview of the project] [Problems that the invention aims to solve]

[0004] This disclosure provides a method and related apparatus for accelerating text inference applicable to large-scale language models. [Means for solving the problem]

[0005] One aspect of this disclosure provides a text inference acceleration method applicable to large-scale language models, the method being: The process involves screening a core token set that needs to be reserved from a target token set stored in video memory, wherein the core token set includes at least a first token subset determined based on attention scores obtained by a plurality of proxy tokens performing global query operations against the target token set, and each proxy token is screened from the target token set. This includes performing an elimination operation on the target token set in the video memory based on the core token set.

[0006] In another aspect of this disclosure, the disclosure provides a text inference device applicable to large-scale language models, the device, A screening module for screening a core token set that needs to be reserved from a target token set stored in video memory, wherein the core token set includes at least a first token subset determined based on attention scores obtained by a plurality of proxy tokens performing global query operations on the target token set, and each proxy token is screened from the target token set; The system includes an eradication module for performing an eradication operation on the target token set in the video memory based on the core token set.

[0007] In other aspects of this disclosure, the disclosure provides an electronic device, which is At least one processor, The system comprises at least one processor and memory that is communicated with, The memory stores instructions that can be executed by the at least one processor, and when these instructions are executed by the at least one processor, they cause the above method to be performed.

[0008] Another aspect of this disclosure provides a non-temporary computer-readable storage medium that stores computer instructions for causing a computer to perform the above method.

[0009] In another aspect of the present disclosure, a program product is provided which, when executed by a processor, includes a program for performing any of the methods in the embodiments of the present disclosure.

[0010] Examples of this disclosure allow for screening of core token sets that need to be held through multiple proxy tokens, reducing the processing load of unnecessary tokens, thereby effectively freeing up video memory space and increasing the inference speed of large language models. It should be understood that what is described herein is not intended to describe any key points or important features of the examples of this disclosure, nor is it intended to limit the scope of this disclosure. Further details of other features of this disclosure are provided in the specification below. [Brief explanation of the drawing]

[0011] The attached drawings are for the purpose of better understanding the solutions of this disclosure and do not constitute a limitation of this disclosure.

[0012] [Figure 1] This is a flowchart of a text inference acceleration method applied to a large-scale language model according to one embodiment of the present disclosure. [Figure 2] This is a flowchart for determining the first token subset according to one embodiment of the present disclosure. [Figure 3] This is a flowchart showing how to obtain the attention score for each candidate token for each attention head, according to one embodiment of the present disclosure. [Figure 4] This is a schematic diagram illustrating the process of each attention head query according to one embodiment of the present disclosure. [Figure 5] This is a flowchart of random algorithm sampling according to one embodiment of the present disclosure. [Figure 6] This is a flowchart of the elimination of the encoding stage according to one embodiment of the present disclosure. [Figure 7] This is a flowchart of the decryption process according to one embodiment of the present disclosure. [Figure 8] This is a schematic diagram illustrating a token erasure method combining proxy token erasure and random token erasure according to one embodiment of the present disclosure. [Figure 9]It is a schematic diagram showing the configuration of an accelerator for text inference applied to a large language model according to an embodiment of the present disclosure. [Figure 10] It is a block diagram of an electronic device for implementing a text inference acceleration method applied to a large language model according to an embodiment of the present disclosure.

Mode for Carrying Out the Invention

[0013] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the accompanying drawings. These drawings include various details of the embodiments of the present disclosure for the purpose of assisting understanding, and these should be considered to be merely exemplary. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described in this specification without departing from the scope of the present disclosure. Similarly, descriptions of well-known features and structures are omitted in the following description for clarity and conciseness.

[0014] Furthermore, the terms "first" and "second" are used only for the purpose of description and are not to be understood as indicating or implying relative importance, nor do they implicitly specify the number of the indicated technical features. Therefore, the features limited by the terms "first" and "second" may include one or more such features explicitly or implicitly. In the description of the present disclosure, "one or more" means two or more unless otherwise explicitly and specifically limited.

[0015] Furthermore, in order to better explain the present disclosure, a number of specific details are shown in the following specific embodiments. Those skilled in the art should understand that the present disclosure can be implemented similarly without specific specific details. In some examples, well-known methods, means, components, and circuits are not described in detail in order to emphasize the gist of the present disclosure.

[0016] The inference Generation of the large language model is divided into the following two stages.

[0017] This is the encoding stage. In this stage, prompt information is input and encoded, and the Key and Value calculated for each layer are cached to speed up the subsequent decoding stage. This can be considered a single forward run of the model, where the token is decoded. A token can be understood as a unit representing text, and may be a word, character, phrase, etc.

[0018] This is the Decoding stage. In this stage, the process enters a loop of the Generation process, and each token is decoded and generated one by one and output. The input is the token decoded in the previous process, and when the model performs forward calculations and performs Attention (Attention Mechanism operation), in order to provide acceleration for subsequent token calculations, the Attention operator completes the calculation using the previously cached Key and Value, and generates and caches a new Key and Value.

[0019] In long-text inference scenarios, in addition to model parameters, Keys and Values ​​are cached to accelerate inference. Typically, video memory increases linearly with the length of the token sequence. In the attention computation block, the complexity of the computation and the sequence length have a squared relationship.

[0020] With this in mind, embodiments of this disclosure propose a method for accelerating inference when caching KVs, which eliminates relatively secondary tokens. That is, it achieves inference acceleration by eliminating the KV values ​​of unimportant tokens in video memory. This method is suitable not only for large language models for short texts but especially for large language models for long texts. This method is applicable to large language models with a decoder-only structure or to large language models that predict tokens one by one.

[0021] In the embodiments of this disclosure, there is no fixed character limit for the definition of a long text, and the concept of a long text may vary depending on the model, application scenario, and needs. Generally, a long text refers to text that exceeds the default processing capacity of the model. For example, a typical NLP (Natural Language Processing) model has a maximum processing length of 2048 "tokens". Text that exceeds this limit can be considered a long text.

[0022] As shown in Figure 1, the text inference acceleration method applied to the large-scale language model provided in the embodiments of this disclosure includes the following:

[0023] In S101, the core token set that needs to be reserved is screened from the target token set stored in video memory, where the core token set includes at least a first token subset, the first token subset is determined based on the attention score obtained by multiple proxy tokens performing a global query operation against the target token set, and each proxy token is screened from the target token set.

[0024] In embodiments of this disclosure, it is possible to default that all current tokens in video memory are part of the target token set.

[0025] Since tokens stored in video memory are not equally important, eliminating relatively secondary tokens in video memory can reduce video memory usage and accelerate inference. Therefore, in the embodiments of this disclosure, core tokens can be understood and reserved as relatively important tokens in video memory. The core token set is a set of keywords for processing text. For example, if a large-scale language model generates a long novel based on a simple description, the large-scale language model determines the content and direction of the novel based on the core tokens in that description. These core tokens help the large-scale language model understand the task more accurately, and consequently provide more accurate and relevant results.

[0026] In the embodiments of this disclosure, the importance of a token can be indicated using an attention score. The attention score can describe the tightness of the relationships between tokens in the target token set from the perspective of a proxy token. This is useful for screening and reserving key tokens.

[0027] In S102, an elimination operation is performed on the target token set in video memory based on the core token set.

[0028] In the embodiments of this disclosure, by performing a global query on all tokens in video memory via multiple proxy tokens, the importance of tokens in the target token set can be measured multidimensionally from the perspective of multiple proxy tokens, thereby enabling screening of core token sets that need to be retained. By performing an extermination operation on tokens in video memory, important tokens can be retained as much as possible, reducing the processing load of unnecessary tokens and improving inference speed. At the same time, by exterminating a portion of the tokens, video memory space can be freed up, providing more video memory resources to subsequent inference processes. Thus, in the embodiments of this disclosure, by utilizing multiple proxy tokens, video memory space can be effectively freed up and the inference speed of large language models can be accelerated.

[0029] Furthermore, in the embodiments of this disclosure, we propose constructing a second token subset by screening tokens using a random algorithm to improve robustness. Therefore, the core token set in the embodiments of this disclosure may include not only the aforementioned first token subset but also the second token subset. This makes it possible to achieve token elimination in the embodiments of this disclosure based on a combination of proxy tokens and a random algorithm.

[0030] To better understand the text inference acceleration methods provided by the embodiments of this disclosure, the methods for accelerating long inferences will be described below in terms of proxy token-based elimination methods, random elimination methods, and elimination timing.

[0031] 1) Elimination by proxy token To improve the efficiency and quality of text processing using large-scale language models, the first token subset can be determined based on the following method, as shown in Figure 2.

[0032] In S201, the target number of tokens are obtained from video memory to obtain a set of proxy tokens containing multiple proxy tokens.

[0033] During implementation, a target number of tokens can be randomly screened from the target token set in video memory to obtain a set of proxy tokens.

[0034] In another possible embodiment, to improve the rate of extermination and to make the relationship between the output of the next token and recent tokens closer, embodiments of the present disclosure select several recently generated tokens as proxy tokens. This can be done as follows: From the target token set in the video memory, the target number of tokens can be screened from the slowest generation time to obtain the several proxy tokens. In embodiments of the present disclosure, several closest tokens are selected to be used as proxy tokens. That is, the values ​​of these proxy tokens are tokens corresponding to queries. The query is used to query the previous context and match the attention meaning. This can improve the quality of the screened core tokens, thereby increasing the timeliness and accuracy of text generation and effectively ensuring that the output content of the large model is better matched to the current context and demand. Furthermore, selecting the closest tokens makes implementation easier and increases inference speed.

[0035] In S202, a global query is performed on the target token set using the query vector of each proxy token in the proxy token set to obtain the attention score of each candidate token in the candidate token set within the target token set.

[0036] In embodiments of this disclosure, the candidate token set may be a subset of the target token set in video memory, for example, the difference between the target token set and the proxy token set. Alternatively, the candidate token set may be the target token set itself. In embodiments of this disclosure, it is possible to screen the candidate token set for a first token subset that needs to be reserved. That is, in S203, the first token subset that needs to be reserved is screened from the candidate token set based on the attention score of each candidate token.

[0037] In the embodiments of this disclosure, each candidate token in the candidate token set is evaluated from the dimensions of multiple proxy tokens to obtain an attention score for each candidate token, thereby facilitating screening of core tokens from the candidate token set, ultimately achieving token elimination and improving text inference speed.

[0038] In the embodiments of this disclosure, as described above, the candidate token set can include two cases, and screening the first token subset in the different cases will be described below.

[0039] In case (1), if the candidate token set is the target token set, multiple key tokens are screened from the candidate token set according to the magnitude of the attention score to obtain the first token subset, where the larger the attention score, the higher the probability of being selected as a key token.

[0040] Specifically, a first set of tokens can be constructed by screening multiple preset key tokens in order from those with the highest attention scores to those with the lowest.

[0041] Here, the number of presets can be determined based on the video memory capacity. For example, 20% of the total video memory capacity can be selected and reserved.

[0042] In embodiments of this disclosure, if the candidate token set is the target token set, the importance of each token in the target token set can be measured using attention scores by traversing the target token set. Subsequently, based on the magnitude of the attention scores, multiple key tokens can be screened from the candidate token set, and relatively important tokens can be screened and reserved from the entire target token set. This can improve inference speed by eliminating tokens.

[0043] In case (2), if the candidate token set is the difference between the target token set and the multiple proxy tokens, multiple key tokens are screened from the candidate token set according to the magnitude of the attention score, where the larger the attention score, the higher the probability of being selected as a key token. Then, the union of the multiple key tokens and the proxy token set is determined to obtain the first token subset.

[0044] For example, if the candidate token set is the difference between the target token set and multiple proxy tokens, then, depending on the magnitude of the attention score, multiple key tokens are screened from the candidate token set, and the union of the multiple key tokens and the proxy token set is determined to obtain the first token subset.

[0045] The sorting method is not limited in the embodiments of this disclosure. That is, it may be a sorting method from smallest to largest, or from largest to smallest. Based on the sorting of attention scores, multiple tokens with the highest attention scores can be selected as key tokens.

[0046] In the embodiments of this disclosure, the difference operation can be used to remove proxy tokens and avoid duplicate selection of proxy tokens as key tokens. This allows a first token subset to be jointly constructed using the screened tokens, complementarily with the proxy tokens, which helps to more effectively screen core tokens and ensures, to the greatest extent possible, that the first token subset helps to infer reasonable content.

[0047] Regardless of which of the above sets the candidate tokens belong to, in embodiments of this disclosure, for each proxy token, a query is performed against the target token set to obtain a single-row vector for each proxy token, thereby obtaining a two-dimensional matrix constructed of multi-row vectors. Here, for each proxy token, only tokens whose generation time is earlier than the proxy token are queried. Specifically, the query vector q of the proxy token can be used to query the K vector for each token in the target token set. For each proxy token, only tokens whose generation time is not later than its own are queried.

[0048] Large-scale language models often employ multiple head attentions, and in the embodiments of this disclosure, each attention head is screened for a set of core tokens that need to be reserved from a set of target tokens stored in video memory.

[0049] Each attention head is screened against a set of core tokens, allowing each attention head to screen a different set of core tokens based on its own situation. Particularly in long-text inference scenarios, the independent screening capabilities of different attention heads allow for the search of core tokens that truly need to be retained, accelerating inference while simultaneously ensuring better quality of inference.

[0050] Here, the core token set is screened for each attention head based on a proxy token and a random elimination method. The random elimination method will be explained later, but here we will first explain the elimination method using a proxy token.

[0051] Taking attention block i in the coding stage as an example, for the nth attention head, the query process can be as shown in equation (1).

number

[0052] In equation (1), X prompt This represents the target token set, which is the input information for attention block i. JPEG2026512557000003.jpg810 represents the model parameters indicated by the corresponding query weight of the nth attention head in the attention block i. JPEG2026512557000004.jpg910 represents the model parameters indicated by the corresponding value weights of the nth attention head in the attention block i. d is the size of the model parameters representing the attention head, i.e., All of JPEG2026512557000005.jpg823 are R d*d It belongs to [a specific category]. A represents the query result. As can be seen from equation (1), A is a two-dimensional matrix, and each element in A is the query result between two tokens.

[0053] For each attention head, it is necessary to perform a global query on the target token set using the query vector of each proxy token in the proxy token set to obtain the attention score of each candidate token in the candidate token set within the target token set. The specific implementation is as shown in Figure 3 and includes the following.

[0054] In S301, based on the query matrix constructed by the corresponding query vector of the attention head of each token in the target token set, a query is performed on the key value matrix constructed by the key vector of the attention head of each token in the target token set to obtain the query result. Here, the query generation time for each token is not slower than that of the token itself. That is, each token queries only itself and tokens generated before it.

[0055] As shown in Figure 4, the Q matrix represents the query matrix constructed by the corresponding query vectors of the attention heads for each token, the K matrix is ​​the key value matrix constructed by the key vectors of the attention heads for each token in the target token set, and the P matrix represents the query results obtained by the query operation. This is only the query result for one attention head; if there are multiple attention heads, each attention head completes the same operation individually, and the corresponding query result for each attention head is obtained. As shown in Figure 4, an overview of the obtained P matrix is ​​shown for the case of three attention heads. Also in Figure 4, the 2D matrices under one attention head are aggregated, with the same column showing the results after the same token has been queried on different tokens, and the same row showing the results after the same token has been queried on different tokens.

[0056] The proxy tokens selected for each attention head may be the same or different, and the embodiments of this disclosure are not limited thereto.

[0057] In S302, the corresponding submatrix of the proxy token in the query result is aggregated for the query values ​​of identical candidate tokens in the candidate token set to obtain the attention score for each candidate token.

[0058] In the embodiments of this disclosure, an attention score is obtained for each candidate token by aggregating the query values ​​of identical candidate tokens in the candidate token set. The attention score obtained through aggregation can indicate the degree to which the same token is attracting attention from different proxy tokens. This makes it possible to measure the importance of different tokens from the perspective of multiple proxy tokens. Furthermore, key tokens can be accurately screened based on the attention score, which helps to achieve faster inference speed.

[0059] Continuing with Figure 4 as an example, let's assume that in a single column, the tokens in the last three rows from top to bottom, where query operations are performed, are proxy tokens. The last three rows are the query results based on the proxy tokens, and the last three rows are the corresponding submatrices for the proxy tokens. In the submatrices, each value represents the query value. Assuming the length of the token sequence is S, each row in Figure 4 should have S elements. In the P matrix of Figure 4 (using only the P matrix with one attention head as an example), the same column corresponds to the same token and shows the query values ​​for that same token from different tokens. Therefore, the aggregation is performed on the query values ​​of all proxy tokens in the same column.

[0060] A specific aggregation method can be used: summing each column. That is, summing the query values ​​for all proxy tokens in the same column.

[0061] In some other possible embodiments, as shown in Figure 3, step S302 may include the following in order to better evaluate the importance of each candidate token:

[0062] In S3021, for the same candidate token, at least one corresponding query value for that candidate token is obtained from the submatrix.

[0063] Let's continue by assuming that the last three rows of the single attention head in Figure 4 represent the query results for the proxy token. Let's assume that the corresponding token in the first column of Figure 4 is the candidate token, and that the gray area in matrix P in Figure 4 represents the multiple query values ​​for that candidate token.

[0064] In S3022, the corresponding at least one query value of the same candidate token is mapped to the target value interval, and the mapping value for each query value in at least one query value is obtained.

[0065] In the embodiments of this disclosure, query values ​​can be mapped to a value interval of 0-1 using a nonlinear mapping method. This enables standardization of each query value. The significance of each query value can be better measured.

[0066] In S3023, the cumulative sum of the mapping values ​​for each corresponding query value of the same candidate token is determined to obtain the attention score for the same candidate token.

[0067] The specific implementation of aggregation is shown in equation (2),

number

[0068] In equation (2), F score This is the cumulative sum of the mapping values ​​for each corresponding query value of the same candidate token and the resulting attention score. Sofmax() represents mapping the corresponding query values ​​of the candidate token to the target value interval, where xa represents the candidate token, xb represents the proxy token, and xa represents the set of candidate tokens and xb represents the set of proxy tokens, F score This is a two-dimensional matrix containing the attention score for each candidate token.

[0069] In some embodiments, in addition to the direct summing method of equation (2), embodiments of the present disclosure can assign corresponding weights to multiple corresponding query values ​​of the same candidate token. After mapping the query values ​​to a target value interval, the attention score of the candidate token can be calculated using weighted summation.

[0070] In the embodiments of this disclosure, by mapping query values ​​for the same candidate token using different proxy tokens, it is possible to normalize query results from different proxy tokens and better measure the degree of attention given to different tokens. By aggregating query results for the same candidate token, it is possible to integrally determine the attention score of the candidate token based on the dimensions of different proxy tokens, which helps in accurately screening core tokens.

[0071] Furthermore, if there are multiple attention heads, the core token set is screened for each attention head. Then, based on the core token set, the elimination operation on the target token set in the video memory can be implemented by aggregating the corresponding core token sets of each attention head to obtain the set of tokens that need to be retained, and eliminating tokens other than the total token set from the video memory.

[0072] This allows for the screening of the core token set for each attention head in the case of multi-head attention, and by increasing the difference between the screened core tokens, key tokens that need to be reserved from different attention head dimensions can be effectively identified, which helps to accelerate inference.

[0073] Suppose that the tokens screened for attention head 1 that need to be retained are token1, token3, and token5, and the tokens screened for attention head 2 that may also need to be retained are token5, token8, and token10. If token5 and token10 are indeed relatively important, then based on this scheme, the core important tokens can be screened as much as possible and retained in video memory. In the case of long-text scenarios, the probability of retaining true core tokens can be probabilistically increased by individually screening tokens with different attention heads. Screening tokens that need to be retained using a random algorithm over time with different attention heads further increases robustness and the probability of screening key tokens. The random elimination method is described below.

[0074] 2) Random Elimination Method Proxy tokens can be used to screen for relatively important key tokens. However, in long-text scenarios, if there is a large amount of redundant non-key information, proxy tokens may fall into the trap of acting as local keys. To improve robustness, embodiments of this disclosure additionally introduce a random elimination scheme to screen a second token subset, which works in conjunction with the proxy tokens to improve the quality of elimination. In embodiments of this disclosure, the second token subset is obtained by randomly screening the target token set. The probability theory of the random algorithm allows for screening for potential key tokens, thereby increasing robustness and accelerating text inference.

[0075] In some possible embodiments, the random algorithm can be sampled using a simple uniform distribution or a Gaussian distribution algorithm. Sampling can also be performed using an attention score as the sampling weight. A specific implementation procedure is shown in Figure 5.

[0076] In S501, multiple proxy tokens perform global query operations on the target token set, and based on the attention score of each token in the target token set obtained, random sampling weights for each token in the preset sampling set are constructed. Here, the preset sampling set is all tokens stored in video memory, i.e., the target token set, or the difference between the target token set and the proxy token set.

[0077] Whether the preset sampling set is the target token set or the difference set between the target token set and the first token subset, in the embodiments of this disclosure, it is possible to query all tokens in video memory via a proxy token to obtain the attention score for each token in the target token set. Furthermore, the attention score is converted into a random sampling weight for each token in the preset sampling set.

[0078] If the preset sampling set is the target token set, then, for example, the sampling weights can be obtained by simply assigning attention scores to weights, or by mapping the attention scores using some mapping function. The higher the attention score, the more likely it is to be randomly sampled, and the embodiments of this disclosure are not limited to this.

[0079] If the preset sampling set is the difference between the target token set and the first token subset, the attention score of each token in the preset sampling set can simply be used as the random sampling weight, or the attention score can be mapped to the random sampling weight via a mapping function. Similarly, a higher attention score should indicate a higher probability of being sampled, and the embodiments of this disclosure are not limited thereto.

[0080] Subsequently, in S502, a second token subset is randomly screened from the preset sampling set based on the random sampling weight of each token.

[0081] In the embodiments of this disclosure, sampling weights are determined based on the attention score, The sampling process is implemented based on attention scores. To ensure robustness, weight is given to sampling core tokens, increasing the probability of obtaining a core token. In the case of multi-head attention, each attention head also performs random sampling, and according to probabilistic knowledge, this increases the probability of sampling a core token in long-text scenarios. By combining this with a proxy token elimination method, inference acceleration can be effectively implemented.

[0082] In the embodiments of this disclosure, the operation to eliminate tokens in video memory is not performed only once. To effectively accelerate inference, tokens in video memory may be eliminated multiple times. Therefore, the embodiments of this disclosure provide the following aspects for selecting the timing of elimination.

[0083] 3) Destroy timing As mentioned above, the text inference generation process may include encoding and decoding stages. To achieve inference acceleration as much as possible, embodiments of the present disclosure trigger and execute an operation to screen the target token set stored in video memory for a set of core tokens that need to be reserved, when any trigger condition in the trigger condition set is met.

[0084] The trigger condition set includes at least one of the following:

[0085] Trigger condition 1 is that inference for any attention block has been completed during the encoding stage.

[0086] During the encoding phase of a large-scale language model, once the inference of any attention block is completed, the inference acceleration method proposed in the embodiments of this disclosure can be executed. This allows for the elimination of tokens for each attention block during the encoding phase, thereby increasing the inference speed.

[0087] For example, in a large-scale language model consisting of multiple neural network layers, each layer (i.e., each block) performs specific transformations or feature extractions on the input data. In the encoding process, when a block (e.g., layer i) completes its computation and obtains an output, the large-scale language model may hold these results in a buffer or memory rather than sending them directly to the next layer. In this way, non-essential tokens of the current module are discarded after the inference of the current module is complete, thereby saving video memory and ensuring acceleration of subsequent inferences.

[0088] Since large-scale language models are often built on transformers, embodiments of this disclosure can perform one extermination for each block during the encoding stage to accelerate inference and improve performance.

[0089] Trigger condition 2 is that, during the decoding stage, the amount of data in video memory reaches a preset peak value.

[0090] During the decoding phase, if the amount of data in video memory reaches a preset peak value, a memory management strategy can be triggered, freeing up video memory resources by eliminating tokens and enabling accelerated inference.

[0091] Trigger condition 3 is that, in the decoding stage, one eradication period is reached, where one eradication period is defined as each time the target number of tokens can be predicted consecutively, and tokens do not overlap in different eradication periods.

[0092] For example, if token1, token2, token3, token4, token5, and token6 are generated in order, and assuming that there is one elimination cycle every three tokens generated, then elimination occurs once when token3 is output and once when token6 is output.

[0093] In the embodiment of the present invention, the decoding stage outputs one token at a time, and token erasure is performed once when multiple newly generated tokens have accumulated. This allows for periodic token erasure, saving computational resources and accelerating inference.

[0094] The following explains the elimination methods for each stage, using the encoding and decoding stages as examples.

[0095] 1. Elimination of the encoding stage As shown in Figure 6, assuming that the most recently generated m tokens are selected as proxy tokens and that token elimination is performed once for each attention head n in each blocki during the encoding stage, the following steps are involved.

[0096] In S601, for blocki's attention head n, its weight parameters are The filename is JPEG2026512557000007.jpg845, and based on these weight parameters, a global query is performed on the target token set constructed by all tokens using the proxy token to obtain query result A. That is, the operation shown in equation (1) is performed to obtain query result A.

[0097] In S602, R is defined as the set difference between the target token set and the proxy token set, which are constructed by all the current tokens in video memory. Based on the submatrix constructed by the proxy tokens in the last m row of query result A, the attention score for each candidate token in R is obtained by cumulatively summing each column. The specific calculation method can be found in equation (2).

[0098] In S603, the top-k candidate tokens with the highest attention scores from R are selected as key tokens according to the preset ratio, and the union of all key tokens and the proxy token set is determined to obtain a screened first token subset. This allows for screening of tokens that need to be reserved based on the proxy tokens.

[0099] In S604, a second set of tokens is randomly screened from all the current tokens in video memory. Specifically, the random elimination method described in part 2) above can be referenced. In this way, tokens that need to be reserved are screened using a random method.

[0100] In S605, the first token subset screened in S603 and the second token subset screened in S604 are merged to obtain the core token set that corresponds to attention head n and needs to be reserved for this time.

[0101] In S606, the core token set that needs to be held for all attention heads is aggregated and held in video memory, and the remaining tokens are removed from video memory.

[0102] In the embodiments of this disclosure, the token needs to be retained in video memory, that is, the KV value corresponding to that token needs to be retained in video memory. The token is removed, that is, the KV value corresponding to the token is deleted from video memory.

[0103] 2. Elimination at the con stage As shown in Figure 7, since one token is output at a time during the decoding stage, let's assume that one eradication is performed each time m tokens are generated cumulatively. This includes the following steps:

[0104] For each attention head of block i in the decoding stage, operations S701 to S707 are performed, and then S708 is executed.

[0105] In S701, the current token is used as a proxy token to query the K value of a previously generated token, and a single query result corresponding to the current proxy token is obtained.

[0106] The main calculation steps are as follows:

[0107] In step B1, for the token immediately preceding the current token, the KV value corresponding to the attention head n of block i stored in video memory is obtained, i.e., the KV value of the previous token.

[0108] The implementation process is shown in equation (3).

number

[0109] In equation (3), JPEG2026512557000009.jpg912 is the K value of the previous token stored in video memory, relating to the nth attention head of the i-th block. This indicates that the file will be assigned to JPEG2026512557000010.jpg89. JPEG2026512557000011.jpg811 is the V value for the nth attention head of the i-th block of the previous token stored in video memory. This indicates that the file will be assigned to JPEG2026512557000012.jpg89.

[0110] In step B2, the current token is multiplied by the K-value weight parameter corresponding to the attention head n of blocki, and the calculation result is concatenated with the K-value of the previous token obtained to obtain the K-value of the current token.

[0111] The implementation process is shown in equation (4).

number

[0112] In equation (4), JPEG2026512557000014.jpg79 currently represents the K value corresponding to the nth attention head in blocki of the token. JPEG2026512557000015.jpg76 represents the current token in blocki. JPEG2026512557000016.jpg711 is the K-value weight parameter of blocki's nth attention head, and is a model parameter.

[0113] In step B3, the current token is multiplied by the corresponding V-value weight parameter of blocki's attention n, and the calculation result is concatenated with the V-value of the previous token obtained to obtain the V-value of the current token.

[0114] The implementation process is shown in equation (5).

number

[0115] In equation (5), JPEG2026512557000018.jpg810 currently represents the V value corresponding to the nth attention head in block i of the token. JPEG2026512557000019.jpg77 represents the current token in blocki. JPEG2026512557000020.jpg710 represents the V-value weight parameter of blocki's nth attention head, and is a model parameter.

[0116] In steps B2 and B3, the KV value of the current token is calculated, and preparations are made for accelerating the inference of the next token.

[0117] In step B4, based on a query vector obtained by multiplying the current token by the corresponding Q weight parameter of blocki's attention mechanism header n, queries are performed on the K values ​​of all tokens prior to the current token. If the current token is used as a proxy token, query results for all tokens are obtained. These query results correspond to the first row vector of the P matrix in the attention head in Figure 4.

[0118] The implementation is shown in equation (6).

number

[0119] In equation (6), A represents the query result when the token is currently a proxy token. JPEG2026512557000022.jpg910 represents the Q-value weight parameter of the nth attention head in blocki, and is a model parameter.

[0120] In S702, the number of newly generated tokens is accumulated, and when the accumulation reaches m tokens, S703 is executed to complete one destroy operation.

[0121] In S703, the most recently generated m tokens are used as proxy tokens, and the query results for all tokens stored in video memory are determined by each proxy token.

[0122] Here, a single-row vector is generated for each token in S701, which is the single-row vector in the P matrix of one attention head in Figure 4. Therefore, the row vectors of the most recent m tokens are concatenated in order of priority, i.e., the query result is obtained. In this case, the concatenated query result can be considered as the aforementioned submatrix.

[0123] In S704, R is defined as the difference between the target token set and the proxy token set constructed by all current tokens in video memory. For query values ​​of the same candidate token in R from multiple proxy tokens, the attention score of that candidate token is determined. Refer to equation (2) for the specific calculation method.

[0124] In S705, the top-k candidate tokens with the highest attention scores from R are selected as key tokens according to the preset ratio, and the union of all key tokens and the proxy token set is determined to obtain a screened first token subset. This allows for screening of tokens that need to be reserved based on the proxy tokens.

[0125] In S706, a second set of tokens is randomly screened from all tokens currently in video memory. Specifically, the random elimination method described in part 2) above can be referenced. This randomly screens for tokens that need to be retained.

[0126] In S707, the first token subset screened in S705 and the second token subset screened in S706 are merged to obtain the core token set that corresponds to attention head n and needs to be reserved for this time.

[0127] In S708, the core token set that needs to be held for all attention heads is aggregated and held in video memory, and the remaining tokens are removed from video memory.

[0128] The implementation process for both the accelerated inference at the encoding stage shown in Figure 6 and the accelerated inference at the decoding stage shown in Figure 7 can be summarized using Figure 8. Referring to Figure 8, we will explain using n attention heads as an example (only three attention heads are shown in Figure 8). In Figure 8, based on a random elimination method, we can select tokens that need to be retained and then determine which tokens need to be eliminated. For example, in Figure 8, it is determined that three tokens need to be eliminated by one attention head, and therefore, Figure 8 marks a random elimination budget of 3. Alternatively, we can use proxy tokens to select tokens that need to be retained. For example, in Figure 8, P1-P4 are selected as proxy tokens, and based on the proxy tokens, it is ultimately determined that two tokens need to be eliminated. That is, a proxy elimination budget of 2 in Figure 8. Next, the tokens that need to be retained, which were screened in the two ways, are merged, and the tokens that need to be retained, which were screened by all attention heads, are merged. Finally, a budget of 5 tokens is eliminated, and the token elimination operation in video memory is achieved. In Figure 8, a query is performed for each token during the decryption stage, so Figure 8 shows the case with one decryption stage t.

[0129] In the embodiment shown in Figure 8 provided by the embodiments of this disclosure, the calculated retained token includes two parts.

[0130] The key token (i.e., the first token subset) is Q*K as shown in equations (1) and (6). ^T From this, the Attention Score (i.e., the query result) is calculated, and the corresponding Attention Score for the proxy token is in the last p row. For the last p row, the sum is accumulated column by column (calculation method shown in equation (2)) to obtain a score with shape [1, S], where S is the total amount of tokens in video memory. Then, based on the Top-K algorithm, m1 tokens are selected from this.

[0131] The random tokens are created by randomly selecting m2 tokens, using the scores with the shape [1,S] calculated in the above step as random sampling weights, such that m1 + m2 = C. C is the cash budget and is the budget value.

[0132] Based on the key method described above, the inference acceleration method provided by the embodiments of this disclosure can be used in the encoding and decoding stages of long-text inference.

[0133] During the encoding stage, for each layer (block), tokens to be retained by the inference acceleration method provided in the embodiments of this disclosure are first calculated and then collectively disposed of in a predetermined key-value buffer. Under such a layer-by-layer disposal framework, the peak video memory value does not exceed the maximum video memory.

[0134] During the decoding stage, token-by-token generation occurs when the current video memory reaches the maximum preset video memory, or when periodic eradication is required. For each block, the retained tokens are calculated using the eradication method of the embodiment of this disclosure and eradicated into a predetermined key-value buffer.

[0135] In embodiments of this disclosure, globally optimal inference acceleration can be achieved based on proxy tokens and a random extermination scheme. Globally optimal means that, during the encoding phase, when performing forward computations layer by layer, all prompt information is used, meaning the proxy token query queries all previous contexts, and the key token (i.e., the first token subset) is computed and exterminated all at once.

[0136] To understand the eradication methods provided by embodiments of this disclosure, this disclosure provides specific examples for evaluating their effectiveness.

[0137] By definition, the encoding length is s, the decoding length is n, the key-value cache buffer is C, and k = s / C. Specifically, it is as follows:

[0138] Set it as follows:

[0139] The training sample parameters are BatchSize=1. The length (size) of the long text is SequenceLength=128*1024. Each layer has a HiddenSize of 40*128. The data type is Dtype=BF 16 (2 bytes). The preset set of reserved values ​​is key value cache buffer = 32 * 1024. The number of tokens generated is 1024, and k = (128 * 1024) / (25.6 * 1024) = 5.

[0140] If they are not eliminated, the following will happen.

[0141] The shape of the Key and Value is [1,128*1024]. Video memory is 2 * 128 * 1024 * 40 * 128 * 2 = 268,435,4560 bytes = 2.5 GB. The computational complexity of the decoding stage is S 2+(S + 1) 2 +(S + 2) 2 +…+(S + 1023) 2 is as follows.

[0142] When proceeding using the combination of the proxy-based token method and the random eviction method provided by an embodiment of the present disclosure, the following occurs.

[0143] The shapes of Key and Value are [1, 32 * 1024], and the video memory is 2 * 32 * 1024 * 40 * 128 * 2 = 671088640 bytes = 0.625 GB. The computational complexity of the Decoding stage is (1 / 4 * S * S) + (1 / 4 * S * (S + 1) + (1 / 4 * S * (S + 2)) + … + (1 / 4 * S * (S + 1023)). As a result, the overhead of the video memory becomes 1 / 4 of that when not evicting, and the video memory can be saved three times.

[0144] Comparing the computational complexity of the Decoding stage, the following can be understood. k * (2 * n * n + 3 * n + 6 * s + 6 * s * s + 6 * s * n + 1) / (2 * s * (2 * s + n)) = 7, and theoretically, it accelerates six times.

[0145] Based on the same technical idea, in an embodiment of the present disclosure, a text inference acceleration device 900 applied to a large language model is provided. As shown in FIG. 9, the device includes

[0146] a screening module 901 for screening a set of core tokens that need to be retained from a set of target tokens stored in video memory. The set of core tokens includes at least a first token subset determined based on an attention score obtained by a plurality of proxy tokens performing a global query operation on the set of target tokens, and each proxy token is screened from the set of target tokens. The system includes an eradication module 902 for performing an eradication operation on the target token set in the video memory based on the core token set.

[0147] In some embodiments, the screening module is An acquisition unit for obtaining a target number of tokens from the video memory and obtaining a set of proxy tokens including the plurality of proxy tokens, A query unit for performing a global query on the target token set using the query vector of each proxy token in the proxy token set to obtain the attention score of each candidate token in the candidate token set within the target token set, The system includes a screening unit for screening the first set of tokens that need to be reserved from the set of candidate tokens based on the attention score of each of the candidate tokens.

[0148] In some examples, the screening module specifically, If the candidate token set is the difference set between the target token set and the plurality of proxy tokens, the process involves screening a plurality of key tokens from the candidate token set according to the magnitude of the attention score, where a larger attention score indicates a higher probability of being selected as a key token. This is used to determine the union of the plurality of key tokens and the set of proxy tokens to obtain the first token subset.

[0149] In some embodiments, the screening module specifically screens a plurality of key tokens from the candidate token set according to the magnitude of the attention score, when the candidate token set is the target token set, in order to obtain the first token subset, where a larger attention score indicates a higher probability of being selected as a key token.

[0150] In some embodiments, the core token set further includes a second token subset obtained by performing a random screening operation on the target token set.

[0151] In some embodiments, the screening module is A weight determination unit for constructing random sampling weights for each token in a preset sampling set based on the attention scores of each token in the target token set obtained by the plurality of proxy tokens performing a global query operation on the target token set, wherein the preset sampling set is the target token set, or the preset sampling set is the difference between the target token set and the first token subset, The system includes a random selection unit for randomly screening the second token subset from the preset sampling set based on the random sampling weight of each token.

[0152] In some examples, the screening module specifically, When multiple head attentions are involved, each attention head is used to screen for a core set of tokens that needs to be reserved from the target token set stored in video memory.

[0153] In some embodiments, the query unit specifically, For the same candidate token, obtain at least one corresponding query value for the same candidate token from the submatrix, The process involves mapping the corresponding at least one query value of the same candidate token to the target value interval, thereby obtaining the mapping value for each query value in the at least one query value. This is used to determine the cumulative sum of the mapping values ​​of the corresponding query values ​​for the same candidate token and to obtain the attention score of the same candidate token.

[0154] In some embodiments, the acquisition unit specifically, This is used to obtain the multiple proxy tokens by screening the target number of tokens from the target token set in the video memory, starting with those that have the slowest generation time.

[0155] In some examples, the screening module specifically, If any of the trigger conditions in the trigger condition set are met, it is used to perform an operation to screen the target token set stored in the video memory for the core token set that needs to be reserved. The aforementioned set of trigger conditions is: During the encoding stage, the inference of any attention block must be completed. During the decoding stage, the amount of data in the video memory reaches a preset peak value. The decryption stage includes at least one of the following: achieving one eradication cycle, where one eradication cycle is defined as each time the target number of tokens can be predicted consecutively, and no tokens are duplicated in different eradication cycles.

[0156] The specific functions and illustrative descriptions of each module and submodule of the apparatus according to the embodiments of this disclosure can be found in the relevant descriptions of the corresponding steps in the embodiments of the method described above, and will not be repeated here.

[0157] According to embodiments of the present disclosure, the present disclosure further provides electronic devices, non-temporary computer-readable storage media, and program products.

[0158] Figure 10 is a block diagram of an electronic device 1000 for implementing an embodiment of the present disclosure. The electronic device refers to various types of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, large computers, and other compatible computers. The electronic device further refers to various types of mobile devices, such as personal digital assistants, cellular phones, intelligent phones, wearable devices, and other similar computer devices. The components, their connections, and functions described in this disclosure are illustrative and do not limit the implementation of anything described or specified in this disclosure.

[0159] As shown in Figure 10, device 1000 includes a computing unit 1001 capable of performing various appropriate operations and processes based on computer program instructions stored in read-only memory (ROM) 1002, or computer program instructions loaded from storage unit 1008 into random access memory (RAM) 1003. RAM 1003 can further store various programs and data necessary for the operation of device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are connected to each other via bus 1004. An input / output (I / O) interface 1005 is also connected to bus 1004.

[0160] Multiple components in device 1000 are connected to an I / O interface 1005, which includes an input unit 1006 such as a keyboard and mouse, an output unit 1007 such as various displays and speakers, a storage unit 1008 such as a magnetic disk or optical disk, and a communication unit 1009 such as a network card, modem, or wireless communication transceiver. The communication unit 1009 allows device 1000 to exchange information / data with other devices via computer networks such as the Internet and / or various carrier networks.

[0161] The computing unit 1001 may be a variety of general-purpose and / or dedicated processing components having processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, a computing unit that executes various machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs each of the methods and processes described above, for example, a text inference acceleration method applied to a large-scale language model. For example, in some embodiments, a text inference acceleration method applied to a large-scale language model can be implemented as a computer software program tangibly contained in a machine-readable medium such as a storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed into device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the text inference acceleration method applied to the large-scale language model described above can be performed. In addition, in other embodiments, the computing unit 1001 may be configured to perform a text inference acceleration method applied to a large-scale language model by any other suitable method (e.g., firmware).

[0162] Various embodiments of the systems or technologies described in this disclosure can be implemented by digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standards (ASSPs), systems-on-a-chip (SOCs), complex-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. Each of these embodiments may be implemented by one or more computer programs that run and / or interpret on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0163] Program code for performing the methods of this disclosure can be written in any combination of one or more programming languages. These program codes are provided to a processor or controller of a general-purpose computer, a dedicated computer, or other programming data processing device, so that when the program code is executed by the processor or controller, it can perform the functions / operations defined in the flowcharts and / or block diagrams. The program code may run entirely in a mainscan, partially in a mainscan, partially as an independent soft encapsulation and partially in a remote mainscan, or entirely in a remote mainscan or server.

[0164] In this disclosure, machine-readable media may be tangible media containing or storing programs used by or in conjunction with instruction execution systems, devices, or equipment. Machine-readable media may be machine-readable signal media or machine-readable storage media. Machine-readable media may include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any suitable combination of the foregoing. Further specific examples of machine-readable storage media include electrical connections by one or more wires, portable computer disk cartridges, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any combination of the foregoing.

[0165] To provide user interaction, a computer may implement the systems and technologies described herein, which may include a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and pointing device for the user to provide input to the computer (e.g., a mouse or trackball). Other types of devices may also be used to provide user interaction; for example, the feedback provided to the user may be any form of sensor feedback (e.g., visual feedback, auditory feedback, or haptic feedback), and input from the user may be accepted in any form (e.g., acoustic input, voice input, haptic input).

[0166] The systems and technologies described herein can be implemented in computing systems that include background components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include front-end components (e.g., user computers having a graphics user interface or network browser, through which users can interact with embodiments of the systems and technologies described herein), or in any combination of such background components, middleware components, or front-end components. Components of the system can be connected to one another via digital data communication in any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0167] A computer system can include a client and a server. Typically, the client and server are geographically separated and interact via a communication network. The client-server relationship is created by a computer program that operates on the corresponding computer. The server may be a cloud server, a server in a distributed system, or a server incorporating blockchain technology, etc.

[0168] It should be understood that steps can be newly ranked, added, or deleted using the various forms of flows shown above. For example, each step described in this disclosure may be executed in parallel, sequentially, or in a different order. This disclosure is not limited to this, as long as the technical solutions disclosed herein can achieve the desired results.

[0169] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, subcombinations, and substitutions are possible due to design considerations and other factors. Any changes, equivalent substitutions, and improvements within the gist and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for accelerating text inference applied to large-scale language models, The process involves screening a core token set that needs to be reserved from a target token set stored in video memory, wherein the core token set includes at least a first token subset determined based on attention scores obtained by a plurality of proxy tokens performing global query operations against the target token set, and each proxy token is screened from the target token set. This includes performing an elimination operation on the target token set in the video memory based on the core token set, A method for accelerating text inference applied to large-scale language models.

2. Determining the aforementioned first token subset means Obtaining a target number of tokens from the video memory to obtain a set of proxy tokens containing the plurality of proxy tokens, Using the query vector of each proxy token in the proxy token set, a global query is performed on the target token set to obtain the attention score of each candidate token in the candidate token set within the target token set. The process includes screening the first set of tokens that need to be reserved from the set of candidate tokens based on the attention score of each of the candidate tokens, A method for accelerating text inference applied to a large-scale language model as described in claim 1.

3. Screening the first set of tokens that need to be reserved from the set of candidate tokens based on the attention score of each candidate token is: If the candidate token set is the difference set between the target token set and the plurality of proxy tokens, the process involves screening a plurality of key tokens from the candidate token set according to the magnitude of the attention score, where a larger attention score indicates a higher probability of being selected as a key token. This includes determining the union of the plurality of key tokens and the set of proxy tokens to obtain the first token subset, A method for accelerating text inference applied to a large-scale language model as described in claim 2.

4. Screening the first set of tokens that need to be reserved from the set of candidate tokens based on the attention score of each candidate token is: If the candidate token set is the target token set, the method involves screening multiple key tokens from the candidate token set according to the magnitude of the attention score to obtain the first token subset, wherein a larger attention score indicates a higher probability of being selected as a key token. A method for accelerating text inference applied to a large-scale language model as described in claim 2.

5. The core token set further includes a second token subset obtained by performing a random screening operation on the target token set. A method for accelerating text inference applied to a large-scale language model according to any one of claims 1 to 4.

6. Performing a random screening operation on the aforementioned target token set to obtain a second token subset is: The process involves constructing random sampling weights for each token in a preset sampling set based on the attention scores of each token in the target token set obtained by the plurality of proxy tokens performing a global query operation on the target token set, wherein the preset sampling set is the target token set, or the preset sampling set is the difference between the target token set and the first token subset. This includes randomly screening the second token subset from the preset sampling set based on the random sampling weight of each token, A method for accelerating text inference applied to a large-scale language model according to claim 5.

7. Screening the core token set that needs to be reserved from the target token set stored in the video memory is: If multiple head attentions are involved, each attention head will be screened for a set of core tokens that need to be held aside from a set of target tokens stored in video memory. A method for accelerating text inference applied to a large-scale language model as described in claim 2.

8. For each attention head, a global query is performed on the target token set using the query vector of each proxy token in the proxy token set to obtain the attention score of each candidate token in the candidate token set within the target token set. For each attention head, The method involves querying the key-value matrix constructed by the key-value matrix constructed by the key-value matrix constructed by the key-value matrix constructed by the key-value matrix constructed by the attention-value matrix of each token in the target token set, based on the query matrix constructed by the corresponding query vector of the attention-value matrix of each token in the target token set, and obtaining the query result, wherein the query generation time for each token is not slower than that of the token itself. The operation includes performing an operation in which, in the corresponding submatrix of the proxy tokens in the query result, aggregation is performed on the query values ​​of identical candidate tokens in the candidate token set to obtain the attention score of each candidate token, A method for accelerating text inference applicable to the large-scale language model described in claim 7.

9. In the corresponding submatrix of the proxy tokens in the query result, aggregating the query values ​​of identical candidate tokens in the candidate token set to obtain the attention score of each candidate token is: For the same candidate token, obtain at least one corresponding query value for the same candidate token from the submatrix, The process involves mapping the corresponding at least one query value of the same candidate token to the target value interval, thereby obtaining the mapping value for each query value in the at least one query value. This includes determining the cumulative sum of the mapping values ​​of the corresponding query values ​​for the same candidate token to obtain the attention score for the same candidate token, A method for accelerating text inference applied to a large-scale language model as described in claim 8.

10. Performing an elimination operation on the target token set in the video memory based on the core token set is: Each attention head aggregates its corresponding set of core tokens to obtain the total set of tokens that need to be held aside, This includes removing tokens other than the total set of tokens in the video memory from the video memory, A method for accelerating text inference applicable to the large-scale language model described in claim 7.

11. Obtaining a target number of tokens from the video memory to obtain a set of proxy tokens containing the plurality of proxy tokens is: This includes screening the target number of tokens from the target token set in the video memory, starting with those with the slowest generation time, to obtain the plurality of proxy tokens. A method for accelerating text inference applicable to a large-scale language model according to any one of claims 2 to 4 or 7 to 10.

12. Screening the core token set that needs to be reserved from the target token set stored in the video memory is: If any trigger condition in the trigger condition set is met, the operation includes screening the target token set stored in the video memory for a core token set that needs to be reserved, The aforementioned set of trigger conditions is: During the encoding stage, the inference of any attention block must be completed. During the decoding stage, the amount of data in the video memory reaches a preset peak value. The decryption stage includes achieving one eradication cycle, where one eradication cycle is defined as each time the target number of tokens can be predicted consecutively, and no tokens are duplicated in different eradication cycles. A method for accelerating text inference applied to a large-scale language model according to any one of claims 1 to 12.

13. A text inference accelerator applicable to large-scale language models, A screening module for screening a core token set that needs to be reserved from a target token set stored in video memory, wherein the core token set includes at least a first token subset determined based on attention scores obtained by a plurality of proxy tokens performing global query operations on the target token set, and each proxy token is screened from the target token set; The system comprises an eradication module for performing eradication operations on the target token set in the video memory based on the core token set, A text inference accelerator applied to large-scale language models.

14. The aforementioned screening module is An acquisition unit for obtaining a target number of tokens from the video memory and obtaining a set of proxy tokens including the plurality of proxy tokens, A query unit for performing a global query on the target token set using the query vector of each proxy token in the proxy token set to obtain the attention score of each candidate token in the candidate token set within the target token set, A screening unit for screening a first set of tokens that need to be reserved from the set of candidate tokens based on the attention score of each candidate token, A text inference accelerator applicable to the large-scale language model described in claim 13.

15. The aforementioned screening module specifically, If the candidate token set is the difference set between the target token set and the plurality of proxy tokens, the process involves screening a plurality of key tokens from the candidate token set according to the magnitude of the attention score, where a larger attention score indicates a higher probability of being selected as a key token. The union of the plurality of key tokens and the proxy token set is determined to obtain the first token subset, and is used for the following: A text inference accelerator applicable to the large-scale language model described in claim 14.

16. The aforementioned screening module specifically, When the candidate token set is the target token set, the method involves screening multiple key tokens from the candidate token set according to the magnitude of the attention score to obtain the first token subset, where a larger attention score indicates a higher probability of being selected as a key token. A text inference accelerator applicable to the large-scale language model described in claim 14.

17. The core token set further includes a second token subset obtained by performing a random screening operation on the target token set. A text inference accelerator applicable to a large-scale language model according to any one of claims 13 to 16.

18. The aforementioned screening module is A weight determination unit for constructing random sampling weights for each token in a preset sampling set based on the attention scores of each token in the target token set obtained by the plurality of proxy tokens performing a global query operation on the target token set, wherein the preset sampling set is the target token set, or the preset sampling set is the difference between the target token set and the first token subset, Includes a random selection unit for randomly screening the second token subset from the preset sampling set based on the random sampling weight of each token, A text inference accelerator applicable to the large-scale language model described in claim 17.

19. The aforementioned screening module specifically, When multiple head attentions are involved, each attention head is used to screen for a core set of tokens that needs to be reserved from the target token set stored in video memory. A text inference accelerator applicable to the large-scale language model described in claim 14.

20. For each attention head, the query unit: For each attention head, The method involves querying the key-value matrix constructed by the key-value matrix constructed by the key-value matrix constructed by the key-value matrix constructed by the key-value matrix constructed by the attention-value matrix of each token in the target token set, based on the query matrix constructed by the corresponding query vector of the attention-value matrix of each token in the target token set, and obtaining the query result, wherein the query generation time for each token is not slower than that of the token itself. This is used to perform an operation that includes, in the corresponding submatrix of the proxy tokens in the query result, aggregating the query values ​​of identical candidate tokens in the candidate token set to obtain the attention score of each candidate token. A text inference accelerator applicable to the large-scale language model described in claim 19.

21. The aforementioned query unit specifically, For the same candidate token, obtain at least one corresponding query value for the same candidate token from the submatrix, The process involves mapping the corresponding at least one query value of the same candidate token to the target value interval, thereby obtaining the mapping value for each query value in the at least one query value. The method used to determine the cumulative sum of the mapping values ​​of the corresponding query values ​​for the same candidate token and to obtain the attention score of the same candidate token, A text inference accelerator applicable to the large-scale language model described in claim 20.

22. The destroyer module, specifically, Each attention head aggregates its corresponding set of core tokens to obtain the total set of tokens that need to be held aside, Used to remove tokens other than the total set of tokens in the video memory from the video memory, A text inference accelerator applicable to the large-scale language model described in claim 19.

23. The acquisition unit specifically, This is used to obtain the multiple proxy tokens by screening the target number of tokens from the target token set in the video memory, starting with those that have the slowest generation time. A text inference accelerator applicable to a large-scale language model according to any one of claims 14 to 16 or 19 to 22.

24. The aforementioned screening module specifically, If any of the trigger conditions in the trigger condition set are met, it is used to perform an operation to screen the target token set stored in the video memory for the core token set that needs to be reserved. The aforementioned set of trigger conditions is: During the encoding stage, the inference of any attention block must be completed. During the decoding stage, the amount of data in the video memory reaches a preset peak value. The decryption stage includes achieving one eradication cycle, where one eradication cycle is defined as each time the target number of tokens can be predicted consecutively, and no tokens are duplicated in different eradication cycles. A text inference accelerator applicable to a large-scale language model according to any one of claims 13 to 23.

25. At least one processor, The system comprises at least one processor and a memory that is communicated with by it, The memory stores instructions that can be executed by the at least one processor, and when the instructions are executed by the at least one processor, they cause the at least one processor to perform the method according to any one of claims 1 to 12. Electronic devices.

26. A non-temporary computer-readable storage medium storing computer instructions that cause a computer to perform the method according to any one of claims 1 to 12.

27. A program product comprising a program that, when executed by a processor in a computer, implements the method according to any one of claims 1 to 12.

Images