Inference acceleration method and apparatus

Abstract

The application relates to a reasoning acceleration method and device. The reasoning acceleration method comprises: generating an original key matrix and an original value matrix of all word elements in an input sequence; processing the original key matrix to generate a processed key matrix, comprising: performing sparse processing on the original key matrix to generate a sparse key matrix; and performing low-precision quantization processing on the sparse key matrix to generate the processed key matrix; storing the processed key matrix, and storing the original key matrix and the original value matrix; calculating the similarity between a query vector for generating a word element to be generated and the processed key matrix; selecting a word element identifier based on the similarity, and acquiring the original key matrix and the original value matrix associated with the selected word element identifier; and calculating the word element to be generated based on the original key matrix and the original value matrix associated with the selected word element identifier.

Description

Technical Field

[0001] This application relates generally to the field of artificial intelligence, and more specifically, to a reasoning acceleration method and apparatus. Background Technology

[0002] The rapid development of Artificial Intelligence (AI) technology has led to the increasingly widespread application of Large Language Models (LLMs) in fields such as natural language processing, content generation, and intelligent interaction, resulting in a surge in demand for processing ultra-long texts. Typical scenarios for ultra-long text processing include extracting summaries from documents of tens of thousands of words, full analysis of code repositories, and generating multi-turn long dialogues. In such scenarios, the number of input tokens typically exceeds several thousand or even tens of thousands. The attention mechanism is a core component of LLMs, and its computation directly determines the accuracy and efficiency of LLM model inference. The computational complexity of the attention mechanism is proportional to the square of the input sentence length; therefore, the performance bottleneck of the attention mechanism in ultra-long text scenarios has become a key issue restricting the deployment of LLMs. Summary of the Invention

[0003] This application provides an inference acceleration method, apparatus, computer-readable storage medium, and computer program product for LLM, which can optimize attention calculation in LLM in ultra-long text scenarios, ensure inference accuracy, significantly improve attention calculation efficiency, and reduce hardware storage and computing costs.

[0004] According to one aspect of this application, an inference acceleration method is provided, comprising: generating an original key matrix and an original value matrix of all lexical units in an input sequence; processing the original key matrix to generate a processed key matrix, including: performing sparsification on the original key matrix to generate a sparse key matrix; and performing low-precision quantization on the sparse key matrix to generate the processed key matrix; storing the processed key matrix, and storing the original key matrix and the original value matrix; calculating the similarity between a query vector used to generate a lexical unit to be generated and the processed key matrix; selecting a lexical identifier based on the similarity, and obtaining the original key matrix and the original value matrix associated with the selected lexical identifier; and calculating the lexical unit to be generated based on the original key matrix and the original value matrix associated with the selected lexical identifier.

[0005] According to another aspect of this application, an inference acceleration apparatus is provided, comprising: one or more processors; and a memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the inference acceleration method described above.

[0006] According to another aspect of this application, a non-transitory computer-readable storage medium is provided, storing instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described inference acceleration method.

[0007] According to another aspect of this application, a computer program product is provided, including instructions that, when executed by one or more processors, cause the one or more processors to perform the above-described inference acceleration method. Attached Figure Description

[0008] Figure 1 A schematic flowchart of an inference acceleration method according to an embodiment of this application is shown.

[0009] Figure 2 A schematic flowchart of the pre-filling stage in the inference acceleration method according to an embodiment of this application is shown.

[0010] Figure 3 A schematic flowchart of the decoding stage in the inference acceleration method according to an embodiment of this application is shown.

[0011] Figure 4 A schematic diagram of an inference acceleration device according to an embodiment of this application is shown. Detailed Implementation

[0012] The features and exemplary embodiments of various aspects of this application will now be described in detail. Numerous specific details are provided in the following detailed description to provide a thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be implemented without some of these specific details. The following description of embodiments is merely intended to provide a better understanding of this application by illustrating examples. This application is by no means limited to any specific configurations and algorithms described below, but rather covers any modifications, substitutions, and improvements to elements, components, and algorithms without departing from the spirit of this application. Well-known structures and techniques are not shown in the accompanying drawings and the following description in order to avoid unnecessarily obscuring this application.

[0013] As mentioned earlier, the performance bottleneck of attention mechanisms in ultra-long text scenarios has become a key issue restricting the deployment of LLM. Currently, optimization schemes for attention computation in the LLM inference stage mainly fall into the following three categories.

[0014] The first type of solution is a global high-precision computation scheme, which adopts the native mechanism of the model training stage. It uses 16-bit or 32-bit dense high-precision storage and operation for both the key matrix and value matrix corresponding to all input words. It is usually used in scientific research scenarios with extremely high requirements for inference accuracy and small-scale text processing tasks. The disadvantage of this type of solution is that the computation and storage overhead increases exponentially. In ultra-long text scenarios, it not only requires a huge amount of hardware storage resources (e.g., GPU memory, RAM), but also leads to a significant increase in attention computation latency, making it difficult to meet the needs of real-time applications. Furthermore, it requires high-configuration hardware stacks (e.g., multi-GPU clusters) to run, resulting in extremely high hardware costs and making it unsuitable for large-scale commercial deployment.

[0015] The second type of solution is the global low-precision quantization scheme. This reduces storage overhead and computation by performing global low-precision quantization (e.g., 4-bit or 8-bit quantization) on the key / value matrix and model parameters. It typically involves quantization algorithms such as Generative Pre-trained Transformer Quantization (GPTQ) and Activation-aware Weight Quantization (AWQ), and is widely used in lightweight inference scenarios for edge devices. While this type of solution can significantly reduce hardware costs and improve inference speed, it suffers from unavoidable accuracy loss. This is because low-precision quantization leads to distortion of the semantic features of the key matrix, which in turn affects the accuracy of the correlation judgment in attention calculation. Ultimately, this results in logical breaks and semantic deviations in the generated content, making it unsuitable for scenarios with strict requirements for inference accuracy, such as professional document generation and core dialogue in intelligent customer service.

[0016] The third type of approach is the fixed-window pruning approach. This approach limits the scope of the word window used for attention computation, performing full computation only on words within the window. Typical techniques include the Longformer sliding window mechanism and the Reformer local attention mechanism. The drawback of this approach is that window pruning loses semantic association information of words outside the window, leading to a decrease in the model's global understanding of very long texts. This is especially problematic in scenarios requiring high textual logical coherence (e.g., novel sequels, legal document analysis), where contextual disconnects can easily occur. Furthermore, the window size setting lacks flexibility, making it difficult to balance inference efficiency and text understanding accuracy.

[0017] None of these solutions can solve the problem of low computational efficiency of LLM attention in ultra-long text scenarios without sacrificing inference accuracy or significantly increasing hardware costs.

[0018] This application provides an inference acceleration method that includes dual storage of the key matrix and selective recall of the key / value matrix, which can optimize attention computation during the inference phase of LLM. The inference acceleration method according to this application avoids the inherent logic of "global unified storage and computation" in the prior art. Addressing the phased requirements of attention computation, it combines the native acceleration capabilities of computing hardware for sparse matrix operations and low-precision multiplication operations through a dual storage strategy of the key matrix in the pre-filling stage and a selective recall mechanism of the key / value matrix in the decoding stage. This avoids the high computational overhead of globally dense computation and the loss of inference accuracy caused by low-precision quantization and window pruning. In ultra-long text scenarios, the inference acceleration method according to this application significantly improves attention computation efficiency and reduces hardware storage and computation costs while ensuring inference accuracy, achieving a triple balance of efficiency, accuracy, and cost.

[0019] The present application will now be described in further detail with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the application to these specific forms.

[0020] Figure 1 A schematic flowchart of an inference acceleration method 100 according to an embodiment of this application is shown. Figure 1 As shown, the inference acceleration method 100 may include the following steps S101-S106.

[0021] In step S101, the inference acceleration method 100 may include generating the original key matrix and the original value matrix of all words in the input sequence.

[0022] In some embodiments, the input sequence may include an extremely long input text sequence.

[0023] In step S102, the inference acceleration method 100 may include processing the original key matrix to generate a processed key matrix.

[0024] In some embodiments, step S102 may include: performing sparsification on the original bond matrix to generate a sparse bond matrix; and performing low-precision quantization on the sparse bond matrix to generate a processed bond matrix. In some embodiments, the sparsity of the processed bond matrix is ​​higher than that of the original bond matrix, and the precision of the processed bond matrix is ​​lower than that of the original bond matrix. In some embodiments, the sparsity of the processed bond matrix may be more than 8 times the sparsity of the original bond matrix.

[0025] In step S103, the inference acceleration method 100 may include storing the processed key matrix and storing the original key matrix and the original value matrix.

[0026] In some embodiments, storing the processed key matrix may include storing the processed key matrix in a hardware cache. In some embodiments, storing the original key matrix and the original value matrix may include storing the original key matrix and the original value matrix in a dedicated storage unit.

[0027] In step S104, the inference acceleration method 100 may include calculating the similarity between the query vector (Query) used to generate the tokens to be generated and the processed key matrix.

[0028] In some embodiments, the query vector used to generate the term to be generated can be obtained from calculations performed on the previous term. In some embodiments, similarity calculations may include, but are not limited to, sparse matrix operations and low-precision multiplication operations. In some embodiments, similarity calculations may be performed by hardware units in a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or a field-programmable gate array (FPGA).

[0029] In step S105, the inference acceleration method 100 may include selecting lexical identifiers based on similarity and obtaining the original key matrix and original value matrix associated with the selected lexical identifiers.

[0030] In some embodiments, step S105 may include: arranging the lexical identifiers of all previous lexical units in descending order of similarity; selecting multiple previous lexical units corresponding to the top-ranked lexical identifiers; and obtaining the original key matrix and the original value matrix of the selected multiple previous lexical units. All previous lexical units may include lexical units in the input sequence and newly calculated lexical units.

[0031] In step S106, the inference acceleration method 100 may include calculating the lexical to be generated based on the original key matrix and the original value matrix associated with the selected lexical identifier.

[0032] In some embodiments, step S106 may include: calculating the lexical unit to be generated based on the original key matrix and original value matrix of the selected plurality of previous lexical units. In some embodiments, calculating the lexical unit to be generated may include: performing attention calculation for the lexical unit to be generated.

[0033] Figure 2 A schematic flowchart of a pre-filling stage 200 in an inference acceleration method 100 according to an embodiment of this application is shown. It will be understood that the pre-filling stage 200 may correspond to... Figure 1 Steps S101-S103 in the inference acceleration method 100 shown.

[0034] The pre-filling stage 200 can be executed after receiving the complete input sequence. It generates the key / value matrix and a processed copy of the key matrix used in subsequent decoding stages. The original key / value matrix can be used for high-precision inference computation, and the processed copy of the key matrix can be used for efficient filtering of previous terms. Figure 2 As shown, the pre-filling stage 200 may include steps S201-S204.

[0035] In step S201, the pre-filling stage 200 may include generating the original dense high-precision key matrix / value matrix of the input sequence through an encoder.

[0036] For example, an input sequence containing very long text can be fed into an LLM encoder, which performs linear transformations, activation function calculations, and other operations on each word in the input sequence to generate a raw key vector and a raw value vector corresponding to each word. The raw key vectors of all words can be concatenated to form the raw dense high-precision key matrix of the input sequence, and the raw value vectors of all words can be concatenated to form the raw dense high-precision value matrix of the input sequence.

[0037] In this application, "dense" can refer to the fact that most elements in the matrix are non-zero, and "high precision" can refer to the fact that the data format of the matrix elements has the native precision used during model training, such as 16 bit or 32 bit. Generating the original dense high-precision key matrix / value matrix of the input sequence ensures that the semantic features extracted from the input sequence can be completely and accurately preserved in the matrix.

[0038] In step S202, the pre-filling stage 200 may include performing sparsification and low-precision quantization on the original dense high-precision key matrix to generate a high-sparseness low-precision key matrix for filtering. Step S202 can generate a copy of the key matrix with low storage overhead and easy hardware computation, which can be used for fast similarity calculation and filtering in the subsequent decoding stage.

[0039] In some embodiments, step S202 may include at least a first sub-step and a second sub-step.

[0040] In some embodiments, the first sub-step may include highly sparsified processing of the original bond matrix. For example, a sparsity algorithm may be applied to the original dense, high-precision bond matrix generated in step S201. For instance, if a target sparsity is set (e.g., 87.5%, i.e., 8 times the target sparsity), a sparsity algorithm can be used to set 87.5% of the elements in the original bond matrix to zero, thereby obtaining a highly sparsity bond matrix. Other target sparsities are also possible and are within the scope of this application, such as 4 times, 6 times, 8 times, or higher than the sparsity of the original bond matrix.

[0041] Furthermore, the sparse algorithm used in the first sub-step can include unstructured sparse algorithms or structured sparse algorithms; this application does not limit this, as long as it can meet the set target sparsity. A high-sparseness key matrix can significantly reduce the number of non-zero elements while preserving elements representing key semantic features as much as possible, thus enabling hardware sparse computing units to skip zero-value calculations.

[0042] In some embodiments, the second sub-step may include low-precision quantization of the high-sparse key matrix. For example, low-bit quantization may be performed on the remaining non-zero elements in the high-sparse key matrix obtained in the first sub-step to generate a high-sparse low-precision key matrix.

[0043] In some embodiments, for example, the second sub-step may employ symmetric uniform quantization: first, a scaling factor is determined by traversing the non-zero elements of the high-sparseness key matrix using mini-batch calibration data; then, each non-zero element is divided by the scaling factor and rounded to the nearest integer, representing each non-zero element as a 4-bit integer. Symmetric uniform quantization is merely an example of a low-precision quantization method that can be employed in this application; other low-precision quantization methods are also possible and are within the scope of this application.

[0044] The second sub-step can further compress the storage bits of each non-zero element in the key matrix from 16 / 32 bits to 4 bits, significantly reducing the overall storage overhead of the key matrix.

[0045] In step S203, the pre-filling stage 200 may include storing the high sparsity low precision key matrix generated in step S202 in a hardware cache, for example, in the secondary (L2) or tertiary (L3) cache of a GPU, or in the on-chip static random access memory (SRAM) of a dedicated artificial intelligence acceleration chip, for use in lexical filtering.

[0046] Storing the high-sparseness, low-precision key matrix in a hardware cache can facilitate frequent and low-latency access to the high-sparseness, low-precision key matrix during the lexical filtering operation in the subsequent decoding stage. The fast read and write characteristics of the hardware cache can effectively ensure the execution speed of the lexical filtering operation and avoid performance degradation caused by data access latency.

[0047] In step S204, the pre-filling stage 200 may include storing the original dense high-precision key matrix / value matrix generated in step S201 without any sparsification or quantization in a larger dedicated storage unit, such as the GPU's global memory or the system's dynamic random access memory (DRAM), for use in lexical computation.

[0048] Step S204 preserves the complete original dense high-precision key / value matrix, which can be recalled (read) in subsequent decoding stages for performing precise attention calculations, thereby providing original key / value data without precision loss for a small number of selected keyword elements.

[0049] In practical applications, such as Figure 2 The pre-filling stage 200 shown can be performed once or multiple times on the input sequence. For example, for an input sequence with a sentence length of 1024 words, the pre-filling stage 200 can be performed simultaneously on all 1024 words to obtain the key matrix / value matrix; alternatively, the pre-filling stage 200 can be performed first on 256 words, then on the subsequent 256 words, and so on, to obtain the complete key matrix / value matrix after performing the pre-filling stage 200 four times.

[0050] In addition, although Figure 2 The illustration shows the high-sparseness low-precision key matrix and the original dense high-precision key / value matrix stored in a hardware cache and a dedicated storage unit, respectively. However, in other embodiments, the high-sparseness low-precision key matrix and the original dense high-precision key / value matrix may also be stored in the same storage location, for example, both in a hardware cache. In some embodiments, steps S203 and S204 may be executed synchronously; in some other embodiments, steps S203 and S204 may be executed asynchronously.

[0051] Figure 3 A schematic flowchart of the decoding stage 300 in the inference acceleration method 100 according to an embodiment of this application is shown. It will be understood that the decoding stage 300 may correspond to... Figure 1 Steps S104-S106 in the inference acceleration method 100 shown.

[0052] Decoding phase 300 is an iterative computation process, where each iteration generates corresponding output terms until the inference ends. Decoding phase 300 may include low-cost, fast term filtering using the high-sparseness, low-precision key matrix generated in pre-filling phase 200 to locate a small number of key previous terms; and, based on this fast filtering, recalling only the original key / value matrices of these key previous terms for precise and computationally significantly reduced attention computation. Figure 3 As shown, the decoding stage 300 may include steps S301-S306.

[0053] In step S301, the decoding stage 300 may include obtaining a query vector for generating the current token to be generated in the current round of decoding.

[0054] For example, a query vector for generating the current word to be generated can be obtained based on the word segments generated in the previous decoding step and the word segment position of the word segment to be generated. This query vector indicates the contextual information required for the current decoding step.

[0055] In step S302, the decoding stage 300 may include calculating the similarity between the query vector used to generate the current term and the processed key matrix (i.e., the high-sparseness low-precision key matrix) through hardware-based sparse matrix operations and low-precision multiplication operations.

[0056] In some embodiments, for example, a high-sparseness low-precision key matrix can be read from a hardware cache (e.g., a GPU L2 / L3 cache), and sparse matrix operations and low-precision multiplication operations can be performed on the query vector used to generate the current token to be generated and the high-sparseness low-precision key matrix to calculate similarity.

[0057] In some embodiments, the similarity calculation between the query vector and the processed key matrix can be performed by hardware units in a GPU, ASIC, or FPGA. For example, the sparse matrix operations and low-precision multiplication operations described above can be performed by hardware computing units that support sparse operations, such as GPU tensor cores that support structured or unstructured sparse computing, or sparse multiply-accumulate trees in ASICs.

[0058] Step S302 utilizes the zero-element skipping mechanism of the sparse bond matrix to perform actual operations only on the non-zero elements in the sparse bond matrix. For example, for the target sparsity of 8 times mentioned above, invalid multiplication and addition operations associated with up to 87.5% of the zero elements can be skipped, significantly reducing computational overhead.

[0059] Furthermore, the non-zero elements in the high-sparseness low-precision bond matrix involved in the computation in step S302 have been processed by low-precision quantization (see step S202 of pre-filling stage 200), which can achieve higher parallel computation efficiency. Compared with the original dense high-precision bond matrix, the computational cost associated with the high-sparseness low-precision bond matrix can be reduced by at least one order of magnitude.

[0060] In this way, by utilizing hardware-accelerated computation on a high-sparse, low-precision key matrix, a similarity score vector can be obtained that is associated with the query vector used to generate the current term and the high-sparse, low-precision key matrix. The elements in this similarity score vector can indicate the relevance strength between the query vector used to generate the current term and the previous term.

[0061] In step S303, the decoding stage 300 may include filtering multiple keyword meta identifiers based on the similarity calculated in step S302, i.e., multiple keyword meta identifiers that need to be recalled for precise attention calculation.

[0062] In some embodiments, step S303 may include similarity score preprocessing and normalization. For example, to improve the stability of lexical filtering, the similarity score vector may be preprocessed, such as calculating the L2 norm of the similarity score vector and scaling it to control the sharpness of the similarity score distribution. For example, all elements in the similarity score vector may also be normalized to [0,1].

[0063] In some embodiments, step S303 may include employing a predetermined strategy to filter multiple keyword meta-identifiers based on similarity.

[0064] For example, the pre-defined strategy may include, but is not limited to: Top-K Select the one with the highest score. K The lexical identifiers corresponding to each similarity, where... K The strategy is simple to implement and has the highest determinism because it uses a preset positive integer that is less than the length of the input sequence. The dynamic threshold selection involves setting a dynamic threshold and selecting the lexical identifiers corresponding to similarity scores greater than the dynamic threshold. This strategy can take into account the overall matching situation between the query vector used to generate the current lexical and the previous lexical, as reflected by the similarity score vector, and adaptively adjust the number of lexical identifiers to be selected. Top-K Combining dynamic threshold selection with the two strategies mentioned above, for example, the highest score can be selected. K Calculate the similarity score and discard those below the dynamic threshold; etc.

[0065] In step S304, the decoding stage 300 may include obtaining the original dense high-precision key matrix / value matrix of multiple keyword elements corresponding to the selected multiple keyword element identifiers.

[0066] Based on the multiple keyword meta-identifiers selected in step S303, the original dense high-precision key-value matrices corresponding to the selected keyword meta-identifiers can be read from the hardware cache. In this way, it is possible to avoid loading the dense high-precision key-value matrices of all previous keywords into the computing core, significantly reducing data read bandwidth and on-chip storage pressure.

[0067] In step S305, the decoding stage 300 may include performing attention calculations based on the original dense high-precision key matrix / value matrix of multiple keyword elements corresponding to the selected multiple keyword element identifiers, to generate the current word element to be generated.

[0068] In some embodiments, the original dense high-precision key matrix / value matrix of multiple keyword elements corresponding to the selected multiple keyword element identifiers obtained in step S304 is used, combined with the query vector used to generate the current word element to be generated, to perform full-precision attention calculation, such as, but not limited to, similarity normalization, weight allocation, feature fusion, etc. The calculated attention output can be passed to subsequent layers of the model (e.g., a feedforward network) to finally generate the output word element of the current decoding step, i.e., the current word element to be generated.

[0069] The key / value data used in the attention calculation in step S305 are all original dense high-precision data. Therefore, the attention output generated in this step is consistent in accuracy with the result of dense high-precision calculation using all previous lexical units, thereby ensuring the accuracy of model inference.

[0070] Furthermore, in some embodiments, the key vector / value vector of the current word generated in step S305 will be stored (see step S204 above), and the key vector of the current word may also be sparsified and low-precision quantized (see step S202 above) and then filled into the processed key matrix.

[0071] In step S306, the decoding stage 300 may include determining whether all the tokens to be generated have been generated, or whether the current decoding step has reached the inference end identifier, in order to determine whether the inference end condition is met.

[0072] For example, if it is determined in step S306 that all words to be generated have been generated or the output sequence has reached the preset maximum length, the current decoding step can be ended, and the inference process ends. If the inference termination condition is not met, the decoding stage 300 returns to step S301 to execute the next decoding step to calculate the next word to be generated, until the inference termination condition is met.

[0073] The inference acceleration method according to embodiments of this application has one or more advantages in the following aspects: 1. Reduced hardware / software costs for LLM model deployment. The inference acceleration method according to embodiments of this application performs dense, high-precision calculations only on a small number of key-value pairs. The filtering operation utilizes high-sparseness, low-precision data and hardware acceleration, significantly reducing computational load and storage overhead, without requiring high-configuration hardware stacking. For example, in scenarios with sentence lengths exceeding 1000 words, hardware storage overhead can be reduced by at least 60%, and inference speed can be increased by 3-10 times, facilitating large-scale commercial deployment. 2. Avoidance of attention calculation accuracy loss. The inference acceleration method according to embodiments of this application uses low-precision, sparse data only in the filtering operation, while still using the original dense, high-precision key / value matrix in attention calculation. There is no quantization or sparsity loss, ensuring that the model inference accuracy is consistent with the global calculation, making it suitable for scenarios with strict accuracy requirements. 3. Avoidance of semantic loss. The reasoning acceleration method according to the embodiments of this application does not involve window pruning, but performs similarity calculation and filtering through a global high-sparseness low-precision value matrix. This can capture the semantic association between all input words and the currently generated words, avoid the loss of semantic information outside the window, and ensure the global logical coherence of ultra-long text processing.

[0074] The inference acceleration method according to the embodiments of this application does not require specific customized hardware and can be directly adapted to existing general-purpose computing hardware (e.g., GPU, ASIC, FPGA, etc.). It fully utilizes the hardware's native acceleration capabilities for sparse matrix operations and low-precision multiplication operations, thus resulting in low deployment costs. Furthermore, the inference acceleration method according to the embodiments of this application can be widely applied to various large-model inference deployment scenarios, including, for example, large-scale cloud inference services, lightweight inference on edge devices, and ultra-long text processing tasks (e.g., document summarization, code generation, multi-turn long dialogues). Without increasing hardware configuration or sacrificing inference accuracy, it significantly improves inference efficiency and reduces computational costs, promoting the deployment of LLM in more real-time, low-cost scenarios.

[0075] Figure 4 A schematic diagram of an inference acceleration apparatus according to an embodiment of this application is shown. The apparatus is shown as a computing device 400, which can be used to execute the above-described inference acceleration method. Figure 4 As shown, computing device 400 may include bus 402 or other communication mechanism for transmitting information, and one or more hardware processors 404 coupled to bus 402 for processing information. The one or more hardware processors 404 may include, for example, one or more general-purpose microprocessors.

[0076] like Figure 4As shown, in some embodiments, computing device 400 may further include main memory 406 coupled to bus 402. Main memory 406 is used to store information and instructions executed by one or more processors 404, such as random access memory (RAM), cache, and / or other dynamic storage devices. Main memory 406 may also be used to store temporary variables or other intermediate information during the execution of instructions executed by one or more processors 404. When these instructions are stored in storage media accessible to one or more processors 404, they can cause computing device 400 to become a dedicated machine customized to perform the operations specified in the instructions. Storage device 408 may include non-volatile and / or volatile storage media. Non-volatile storage media may include, for example, optical discs or magnetic disks. Volatile storage media may include dynamic memory. Common forms of storage media may include, for example, floppy disks, hard disks, solid-state drives, magnetic tape, or any other magnetic data storage media, CD-ROMs, any other optical data storage media, any physical media with a perforated pattern, RAM, DRAM, PROM, and EPROM, FLASH-EPROM, NVRAM, any other memory chip or cartridge, or networking versions thereof.

[0077] like Figure 4 As shown, in some embodiments, computing device 400 may further include one or more communication interfaces or network interfaces 410 coupled to bus 402. Network interface 410 may provide bidirectional data communication coupling to one or more network links connected to one or more networks. As another example, network interface 410 may be a local area network (LAN) card to provide data communication connectivity to a LAN-compatible (or WAN component communicating with a WAN) network. Wireless links may also be implemented.

[0078] The execution of certain operations can be distributed across processors rather than residing within a single machine, but rather deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0079] Each of the processes, methods, and algorithms described above may be embodied in code modules executed by one or more computer systems or computer processors including computer hardware, and may be fully or partially automated by these code modules. The processes and algorithms may be implemented partially or fully in dedicated circuit systems.

[0080] When the functions disclosed herein are implemented as software functional units and sold or used as stand-alone products, they may be stored in a processor-executable, non-volatile, computer-readable storage medium. Specific technical solutions (all or part) disclosed herein, or aspects contributing to the prior art, may be embodied in the form of a software product. The software product may be stored in a storage medium and includes several instructions that cause a computing device (which may be a personal computer, server, network device, etc.) to perform all or some steps of the methods of the embodiments of this application. The storage medium may include a flash drive, portable hard disk drive, ROM, RAM, magnetic disk, optical disk, other media operable to store program code, or any combination thereof.

[0081] Specific embodiments further provide an apparatus including a processor and a non-transitory computer-readable storage medium storing instructions executable by the processor to cause the apparatus to perform operations corresponding to steps in any method of the embodiments disclosed above. Specific embodiments further provide a non-transitory computer-readable storage medium storing instructions executable by one or more processors to cause the one or more processors to perform operations corresponding to steps in any method of the embodiments disclosed above.

[0082] The embodiments disclosed herein can be implemented via a cloud platform, server, or server cluster (also referred to as a “service system”) that interacts with a client. The client can be a terminal device or a client registered by a user at the platform, wherein the terminal device can be a mobile terminal, a personal computer (PC), or any device that can have the platform application installed.

[0083] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of this disclosure. Additionally, certain method or process blocks may be omitted in some embodiments. The methods and processes described herein are not limited to any particular order, and their associated blocks or states may be executed in other suitable orders. For example, described blocks or states may be executed in an order other than that specifically disclosed, or multiple blocks or states may be combined into a single block or state. Example blocks or states may be executed sequentially, in parallel, or in some other manner. Blocks or states may be added to or removed from the disclosed example embodiments. The exemplary systems and components described herein may be configured differently than described. For example, components may be added to, removed from, or rearranged compared to the disclosed example embodiments.

[0084] The various operations of the exemplary methods described herein can be performed at least in part by an algorithm. The algorithm may be included in program code or instructions stored in memory (e.g., the aforementioned non-transitory computer-readable storage medium). This algorithm may include a machine learning algorithm. In some embodiments, the machine learning algorithm may not explicitly refer to the computer as performing the function but may learn from training data to generate a predictive model of the function.

[0085] The various operations of the exemplary methods described herein can be performed, at least in part, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, these processors can constitute an engine of processor implementation that operates to perform one or more of the operations or functions described herein.

[0086] Similarly, the methods described herein may be implemented at least in part by a processor, wherein one or more specific processors are instances of hardware. For example, at least some operations of the methods may be performed by one or more processors or an engine implemented by a processor. Furthermore, one or more processors may also be operable to support the execution of relevant operations in a “cloud computing” environment or as the execution of relevant operations in a “Software as a Service” (SaaS) context. For example, at least some operations may be performed by a group of computers (as an example of a machine containing processors), wherein these operations are accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application programming interfaces (APIs)).

[0087] The execution of certain operations can be distributed across processors rather than residing within a single machine, and can be deployed across multiple machines. In some example embodiments, the processor or processor-implemented engine may reside in a single geographic location (e.g., in a home environment, office environment, or server farm). In other example embodiments, the processor or processor-implemented engine may be distributed across multiple geographic locations.

[0088] Throughout this specification, multiple instances may be implemented as components, operations, or structures described as a single instance. Although individual operations of one or more methods are illustrated and described as separate operations, one or more of these individual operations may be performed simultaneously, and not necessarily in the order illustrated. Structures and functions presented as separate components in the example configuration may be implemented as composite structures or components. Similarly, structures and functions presented as single components may be implemented as single components. These and other variations, modifications, additions, and improvements fall within the scope of this document.

[0089] As used herein, "or" is inclusive rather than exclusive unless explicitly indicated by the context. Therefore, in this document, "A, B, or C" means "A, B, A and B, A and C, B and C, or A, B, and C" unless explicitly indicated by the context. Furthermore, "and" is combined and separate unless explicitly indicated by the context. Therefore, in this document, "A and B" means "A and B, combined or separate" unless explicitly indicated by the context. Additionally, multiple instances of resources, operations, or structures described herein may be provided as a single instance. Furthermore, the boundaries between various resources, operations, engines, and data storage devices are somewhat arbitrary and specific operations are illustrated within the context of a particular illustrative configuration. Other functional assignments are foreseeable and fall within the scope of various embodiments of this disclosure. Generally, structures and functions presented as individual resources in example configurations may be implemented as combined structures or resources. Similarly, structures and functions presented as single resources may be implemented as single resources. These and other changes, modifications, additions, and improvements fall within the scope of the embodiments of this disclosure as expressed in the appended claims. Therefore, this specification and drawings should be considered illustrative rather than restrictive.

[0090] The terms “comprising” or “including” are used to indicate the presence of a subsequently claimed feature, but do not preclude the addition of other features. Unless otherwise specifically stated or otherwise understood in the context in which they are used, conditional language such as “may,” “can,” “may,” and “can” is generally intended to convey that certain embodiments include certain features, components, and / or steps that are not included in other embodiments. Therefore, this conditional language is generally not intended to imply that one or more embodiments require features, components, and / or steps in any way, or that one or more embodiments must include logic for determining whether such features, components, and / or steps are included in or performed in any particular embodiment, with or without user input or prompts.

[0091] Although the general outline of the subject matter has been described with reference to specific exemplary embodiments, various modifications and changes may be made to these embodiments without departing from the broad scope of embodiments of this disclosure. Where more than one embodiment is disclosed, these embodiments of the subject matter may be referred to individually or collectively herein as the term "invention," this is for convenience only and is not intended to automatically limit the scope of this application to any single disclosure or concept.

[0092] The embodiments illustrated herein are described in detail to enable those skilled in the art to practice the disclosed teachings. Other embodiments may be used and derived therefrom, such that structural and logical substitutions and changes may be made without departing from the scope of this disclosure. Therefore, “implementation” is not intended to be limiting, and the scope of the various embodiments is defined only by the appended claims and their full scope.

Claims

1. A method for accelerating inference, comprising: The pre-filling stage includes: Generate the original key matrix and original value matrix of all tokens in the input sequence, wherein the input sequence includes the input text sequence; Processing the original key matrix to generate a processed key matrix includes: Performing sparsification on the original bond matrix to generate a sparse bond matrix includes setting a subset of elements in the original bond matrix to zero using a sparsity algorithm based on a target sparsity, wherein the sparsity of the processed bond matrix is ​​at least eight times that of the original bond matrix; and Perform low-precision quantization on the sparse bond matrix to generate the processed bond matrix, including performing low-bit quantization on the remaining non-zero elements in the sparse bond matrix; The processed key matrix is ​​stored in a hardware cache for similarity calculation, and the original key matrix and the original value matrix are stored in a dedicated storage unit for attention calculation; and The decoding stage includes: The processed key matrix is ​​read from the hardware cache to calculate the similarity between the query vector used to generate the tokens to be generated and the processed key matrix, wherein calculating the similarity includes hardware-based sparse matrix operations and low-precision multiplication operations. Preprocessing and normalization are performed on the similarity; Based on the preprocessed and normalized similarity, lexical identifiers are selected, and the original key matrix and the original value matrix associated with the selected lexical identifiers are retrieved only from the dedicated storage unit; and The lexicon to be generated is calculated based on the original key matrix and the original value matrix associated with the selected lexicon identifier.

2. The method according to claim 1, wherein the precision of the processed key matrix is ​​lower than the precision of the original key matrix.

3. The method according to claim 1, wherein, The query vector used to generate the term to be generated is obtained from the calculation for the previous term.

4. The method according to claim 1, wherein, The calculation of the similarity is performed by a hardware unit in a graphics processing unit, an application-specific integrated circuit, or a field-programmable gate array.

5. The method according to claim 1, wherein, Selecting lexical identifiers based on the preprocessed and normalized similarity, and obtaining the original key matrix and the original value matrix associated with the selected lexical identifiers, includes: Arrange the lexical identifiers of all previous lexical units in descending order of similarity; Select multiple preceding lexical units corresponding to the top-ranked lexical identifiers; and Obtain the original key matrix and the original value matrix of the selected plurality of previous lexical units.

6. The method according to claim 1, wherein, Calculating the lexical to be generated includes performing attention calculations on the lexical to be generated.

7. A reasoning acceleration device, comprising: One or more processors; as well as A memory storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the method according to any one of claims 1-6.

8. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-6.

9. A computer program product comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 1-6.

Images