Inference acceleration method, device, equipment, medium and product of large language model

Abstract

The present disclosure provides a large language model inference acceleration method, device, equipment, medium and product. The large language model outputs key vectors and value vectors corresponding to a current word element and a previous word element of a to-be-processed task, adds the key vectors and the value vectors as cache key-value data to a cache data group, determines shared exponential bit data of the cache data group and compressed key-value data of each cache key-value data, and the large language model infers a next word element based on an updated word element sequence, first target compressed key-value data and target shared exponential bit data. The operation of inferring word elements of the to-be-processed task by the large language model is iteratively performed until all word elements are inferred. Thus, the cache key-value data occupies less memory, and the video memory cost is effectively saved without losing the inference accuracy of the large language model.

Description

Technical Field

[0001] This disclosure relates to model compression and model acceleration technologies, and in particular to a method, apparatus, device, medium, and product for accelerating reasoning of large language models. Background Technology

[0002] In the inference process of large language models (LLMs), a key-value cache (KV caching) mechanism is often used to improve the inference efficiency of LLMs. This involves caching the generated token key vectors and value vectors. When generating a new token, the LLM only needs to calculate the query vector of the current token, call the cached key and value to complete the attention calculation, and then store the current token's key and value back into the cache after the calculation is complete. However, since memory usage increases linearly with sequence length and batch size, as the amount of cached data (keys and values) increases, memory usage can rapidly rise, leading to out-of-memory errors. This is particularly problematic in long-context scenarios, severely limiting the LLM's context expansion and concurrent processing capabilities. To address the issue of KV Cache consuming excessive GPU memory, quantization is often used to reduce memory usage. This involves reducing the numerical precision of each Key and Value element (e.g., quantizing from FP16 to INT8 or INT4) to compress storage space. However, quantization inevitably introduces numerical errors. Due to the large dynamic range of activation values ​​(especially Keys) and the presence of outliers, low-precision quantization cannot accurately represent the original distribution, leading to deviations in attention weight calculations and reducing the quality of LLM-generated data. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this disclosure provide a method, apparatus, device, medium, and product for accelerating reasoning in large language models.

[0004] One aspect of this disclosure provides a method for accelerating inference using a large language model, comprising: responding to the output of the current lexical of a task to be processed by the large language model, adding the key vector and value vector corresponding to the previous lexical as cached key-value data to a cached data group; determining initial exponent data based on the exponent data of each cached key-value data in the cached data group; mapping the initial exponent data to a preset bit width to obtain shared exponent data; for each cached key-value data in the cached data group, compressing the mantissa and sign bits of the cached key-value data to obtain compressed key-value data, and caching the shared exponent data and each compressed key-value data; adding the current lexical to the lexical sequence of the task to be processed, wherein the large language model infers the next lexical of the task to be processed based on the updated lexical sequence, the first target compressed key-value data corresponding to the updated lexical sequence, and the target shared exponent data corresponding to the first target compressed key-value data, iteratively executing the operation of the large language model inferring the lexical of the task to be processed, until the large language model infers all the lexical of the task to be processed.

[0005] Another aspect of this disclosure provides an inference acceleration device for a large language model, comprising: a first storage module, configured to, in response to the output of the current lexical of the task to be processed by the large language model, add the key vector and value vector corresponding to the previous lexical as cached key-value data to a cached data group; an exponent determination module, configured to determine initial exponent data based on the exponent data of each cached key-value data in the cached data group; an exponent mapping module, configured to map the initial exponent data to a preset bit width to obtain shared exponent data; and a mantissa compression module, configured to, for each cached key-value data in the cached data group, compress the cached key-value data... The mantissa and sign bits of the data are compressed to obtain compressed key-value data, and the shared exponent bit data and each compressed key-value data are cached. The lexical reasoning module is used to add the current lexical to the lexical sequence of the task to be processed. The large language model infers the next lexical of the task to be processed based on the updated lexical sequence, the first target compressed key-value data corresponding to the updated lexical sequence, and the target shared exponent bit data corresponding to the first target compressed key-value data. The operation of the large language model inferring the lexical of the task to be processed is iteratively executed until the large language model infers all the lexical of the task to be processed.

[0006] In another aspect of this disclosure, an electronic device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, wherein when the computer program is executed, it implements the above-described method for accelerating inference using a large language model.

[0007] In another aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described method for accelerating inference using a large language model.

[0008] In another aspect, embodiments of this disclosure provide a computer program product including computer program instructions that, when executed by a processor, implement the reasoning acceleration method for any of the large language models described above.

[0009] Based on the embodiments of this disclosure, a layered optimization strategy of shared exponent bits and independent compression of mantissa bits is adopted for floating-point cached key-value data such as key vectors and value vectors generated by the self-attention layer. That is, the mantissa bits of each cached key-value data are compressed independently, and the shared exponent data (initial exponent bit data) of each cached key-value data is mapped to a preset bit width to construct standardized shared exponent bit data. This significantly reduces the memory occupation of cached key-value data without losing the inference accuracy of large language models, effectively saves GPU memory overhead, thereby improving the inference speed and context expansion capability of large language models, and thus taking into account both GPU memory optimization effect and inference stability.

[0010] The technical solutions of this disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Attached Figure Description

[0011] The accompanying drawings, which form part of this specification, illustrate embodiments of this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0012] This disclosure will become clearer with reference to the accompanying drawings and the following detailed description, wherein: Figure 1 This is a flowchart illustrating an exemplary embodiment of the present disclosure of a method for accelerating inference using a large language model.

[0013] Figure 2 This is a flowchart illustrating step S120 provided in an exemplary embodiment of this disclosure.

[0014] Figure 3 This is a flowchart illustrating step S130 provided in an exemplary embodiment of this disclosure.

[0015] Figure 4 This is a flowchart illustrating step S132 provided in an exemplary embodiment of this disclosure.

[0016] Figure 5 This is a flowchart illustrating step S130 provided in another exemplary embodiment of this disclosure.

[0017] Figure 6This is a flowchart illustrating step S140 provided in an exemplary embodiment of this disclosure.

[0018] Figure 7 This is a schematic diagram of large language model reasoning provided in an application example of this disclosure.

[0019] Figure 8 This is a structural block diagram of a large language model inference acceleration device provided in an exemplary embodiment of the present disclosure.

[0020] Figure 9 This is a schematic diagram of the structure of an application embodiment of the electronic device disclosed herein. Detailed Implementation

[0021] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0022] Those skilled in the art will understand that the terms "first," "second," etc., in the embodiments of this disclosure are only used to distinguish different steps, devices, or modules, and do not represent any specific technical meaning, nor do they indicate a necessary logical order between them.

[0023] It should also be understood that in the embodiments disclosed herein, "a plurality of" may refer to two or more, and "at least one" may refer to one, two or more.

[0024] It should also be understood that any component, data or structure mentioned in the embodiments of this disclosure can generally be understood as one or more unless expressly defined or given to the contrary in the context.

[0025] Furthermore, the term "and / or" in this disclosure is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this disclosure generally indicates that the preceding and following related objects have an "or" relationship.

[0026] It should also be understood that the description of the various embodiments in this disclosure emphasizes the differences between the various embodiments, and the similarities or similarities can be referred to each other. For the sake of brevity, they will not be described in detail.

[0027] At the same time, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn according to actual scale.

[0028] The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use.

[0029] Techniques, methods, and equipment known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and equipment should be considered part of the specification.

[0030] It should be noted that similar labels and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be discussed further in subsequent figures.

[0031] The embodiments disclosed herein can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate together with a wide range of other general-purpose or special-purpose computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, and servers include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments including any of the above systems, etc.

[0032] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. Typically, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in distributed cloud computing environments, where tasks are executed by remote processing devices linked through communication networks. In distributed cloud computing environments, program modules can reside on local or remote computing system storage media, including storage devices.

[0033] In Large Language Models (LLMs), the Transformer-based decoder is the core for tasks such as text generation, and its key mechanism is self-attention. This mechanism generates a query for each sequence position and interacts with the key and value of all positions, enabling the model to focus on all generated tokens and thus predict the next token. Because the inference process of LLMs has autoregressive characteristics—generating only one token at a time—without optimization strategies, when generating the nth token, the model needs to repeatedly calculate the key and value matrix for the previous n-1 tokens, leading to a large amount of redundant computation and severely slowing down inference. To eliminate this redundancy, existing technologies generally introduce a KV Cache mechanism to cache the key and value matrices of already generated tokens, avoiding repeated computation. However, while KV Cache improves speed, it also introduces a surge in GPU memory usage. For example, the Llama3 70B model, processing a single request with a 32K context, requires over 10GB of GPU memory for data storage based on the KV cache. During batch processing, this memory usage can exceed 80GB, severely limiting the expansion of the context length and the number of concurrent requests. Furthermore, the massive KV cache usage exacerbates GPU memory bandwidth pressure, limiting inference speed to memory read / write operations rather than computational power. Simultaneously, the allocation and release of caches for different requests can easily lead to GPU memory fragmentation, further reducing memory utilization. To address the GPU memory bottleneck of the KV cache, quantization technology has become the mainstream compression solution. Its core idea is to reduce storage overhead by lowering the numerical precision of key and value elements. Taking Post-Training Quantization (PTQ) as an example, this method predetermines the quantization parameters before inference, converting FP16 / BF16 precision key and value tensors into low-precision formats such as INT8 or INT4. This method is simple to implement and requires no modification to the inference framework. However, due to the large dynamic range and outliers of activation values ​​(especially keys), low-precision quantization struggles to accurately represent the original distribution, inevitably introducing a loss of precision. This loss directly manifests as a decline in the quality of generated text, such as increased content repetition, weakened semantic coherence, reduced factual consistency, and even logical inconsistencies. Research indicates that over-quantization can significantly increase model perplexity (PPL) and degrade performance in downstream tasks (such as BLEU scores in machine translation and accuracy in text classification). Therefore, effectively compressing the KV cache while maintaining model accuracy has become a significant challenge in current LLM inference optimization.

[0034] Figure 1This is a flowchart illustrating an exemplary embodiment of the present disclosure of a method for accelerating inference using a large language model. This embodiment can be applied to electronic devices, such as… Figure 1 As shown, the reasoning acceleration method for this large language model may include the following steps: In step S100, in response to the output of the current word of the task to be processed by the large language model, the key vector and value vector corresponding to the previous word are added to the cache data group as cached key-value data.

[0035] The task to be processed can be, for example, the generation of a target sentence, and the lexical units of the task to be processed are used to compose the target sentence. The key vector and value vector of the current lexical unit are generated by the self-attention layer of the large language model. The cached data group includes multiple cached key-value data, for example, the cached data group can include 32 or 64 cached key-value data. The cached key-value data can include key vectors and / or value vectors. The data type of the cached key-value data is floating point. For example, the data type of the cached key-value data can be half-precision floating-point (FP16), and the cached key-value data consists of a sign bit, an exponent bit, and a mantissa bit. The sign bit is 0 or 1, where 0 indicates that the cached key-value data is positive and 1 indicates that the cached key-value data is negative. The exponent bit represents the integer part of the cached key-value data, and the mantissa bit represents the fractional part of the cached key-value data.

[0036] In one implementation, the large language model can correspond to multiple cached data groups, each storing 32 or 64 cached key-value pairs generated by the large language model's self-attention layer. In this embodiment, the previous lexical is the lexical preceding the current lexical.

[0037] Step S110: Determine the initial exponent data based on the exponent data of each cached key-value data in the cached data group.

[0038] For example, the exponent data in each cached key-value data whose exponent data exceeds a preset threshold can be determined as the initial exponent data; if there are multiple cached key-value data whose exponent data exceeds the preset threshold, the exponent data of any one of the cached key-value data can be selected as the initial exponent data.

[0039] Step S120: Map the initial exponent data to a preset bit width to obtain shared exponent data.

[0040] The preset bit width represents the number of binary bits occupied by the shared exponent data. For example, the preset bit width can be 8 bits. The initial exponent data can be mapped to the preset bit width (e.g., 8 bits) through bitwise operations or linear scaling.

[0041] Step S130: For each cached key-value data in the cached data group, compress the mantissa and sign bits of the cached key-value data to obtain compressed key-value data, and cache the shared exponent bits and each compressed key-value data.

[0042] The compressed key-value data can include a sign bit and a compressed mantissa. The sign bit of the compressed key-value data is the same as the sign bit of the corresponding cached key-value data. The sign bit and mantissa of the cached key-value data can be compressed using data compression methods such as truncation, rounding, or fixed-ratio mapping. The data type of the compressed key-value data is Integer (INT), and its bit width can be set according to the actual application. In one example, the compressed key-value data uses the INT8 data type, with a bit width of 8 bits. The bit widths of the sign bit and the compressed mantissa in this compressed key-value data are 1 bit and 7 bits, respectively.

[0043] For example, Table 1 shows the bit composition of compressed key-value data. The data type of the compressed key-value data is INT8. The bit order within the compressed key-value data, from high to low, represents the sign bit data and the compressed mantissa, respectively. Specifically, the sign bit data of the compressed key-value data has a bit width of 1 bit, and the compressed mantissa has a bit width of 7 bits. In Table 1, mantissa 6 to mantissa 0 represent the data of the 6th to 0th bits of the compressed mantissa, respectively, while the sign represents the sign bit data of the compressed key-value data.

[0044] You can cache the shared exponent data and each compressed key value separately, or you can embed each bit of the shared exponent data into the least significant byte (LSB) of a compressed key value data, and then cache each compressed key value data.

[0045] Step S140: Add the current word to the word sequence of the task to be processed. Based on the updated word sequence, the first target compressed key value data corresponding to the updated word sequence, and the target shared index bit data corresponding to the first target compressed key value data, the big language model infers the next word of the task to be processed. Iteratively execute the operation of the big language model inferring the words of the task to be processed until the big language model infers all the words of the task to be processed.

[0046] In this embodiment, the compressed key value data corresponding to other words in the updated word sequence besides the current word can be called the first target compressed key value data, and the shared exponent bit data corresponding to the first target compressed key value data can be called the target exponent bit data.

[0047] The lexical sequence includes all lexical units generated by the large language model in the task to be processed. For example, if the task to be processed is to generate "I love artificial intelligence", the lexical sequence includes the already generated lexical units "I", "love", and "human", and the next lexical unit to be inferred is "intelligence". The compressed key-value data corresponding to the lexical units is obtained by compressing the cached key-value data corresponding to the lexical units in steps S110 to S130 above. The current lexical unit is added to the lexical sequence of the task to be processed to obtain the updated lexical sequence. The large language model uses the Transformer decoder as its core network structure, and the Transformer decoder has multiple self-attention layers running self-attention mechanisms. The large language model adopts an autoregressive generation method, that is, it generates one lexical unit at a time, and infers new lexical units based on the generated lexical units each time. When inferring the next lexical unit, the large language model uses self-attention calculation based on the first target compressed key-value data corresponding to the lexical unit sequence and the shared exponent data corresponding to each first target compressed key-value data to infer the next lexical unit.

[0048] In the disclosed embodiments, a layered optimization strategy of shared exponent bits and independent mantissa compression is adopted for floating-point cached key-value data such as key vectors and value vectors generated during the inference process. That is, the mantissa data of each cached key-value data is compressed independently, and the shared exponent data (initial exponent bit data) of each cached key-value data is mapped to a preset bit width to construct standardized shared exponent bit data. This significantly reduces the memory occupation of cached key-value data without losing the inference accuracy of large language models, effectively saves video memory overhead, thereby improving the inference speed and context expansion capability of large language models, and thus taking into account both video memory optimization effect and inference stability.

[0049] In some alternative implementations, in this embodiment of the disclosure, step S110 may include: obtaining the largest exponent bit data from each cached key-value data as the initial exponent bit data.

[0050] Specifically, the exponent data of each cached key-value data is extracted, and the maximum value among these extracted exponent data is selected as the initial exponent data.

[0051] In this embodiment of the disclosure, by selecting the maximum value of the exponent data in the cached data group as the initial exponent data, the alignment benchmark of multiple cached key-value data is determined. That is, all cached key-value data are quantized based on the maximum exponent data, ensuring that the maximum value in the cached key-value data will not overflow. At the same time, a unified exponent benchmark is provided for all cached key-value data, thereby preserving numerical accuracy to the maximum extent during the compression process.

[0052] Figure 2This is a flowchart illustrating step S120 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 2 As shown, step S120 may include the following steps: Step S121: Determine the effective bit length adjustment value based on the effective bit length of the preset compressed key value data and the preset calibration parameter value.

[0053] The preset calibration parameter value can be 1. The difference between the effective bit length of the preset compressed key value data and the preset calibration parameter value can be determined as the effective bit length adjustment value. For example, the effective bit length adjustment value = 7 (effective bit length of the preset compressed key value data) - 1 (preset calibration parameter value) = 6.

[0054] Step S122: Based on the exponent offset value, significant bit adjustment value, and preset offset value of the cached key-value data, offset calibration and bit width mapping are performed on the initial exponent bit data to obtain shared exponent bit data with a preset bit width.

[0055] The preset bias value can be 127. For cached key-value data of data type FP16, the exponent bias value is 15. The shared exponent data with a preset bit width can be obtained based on formula (1).

[0056] Shared exponent bits = Initial exponent data - Exponent offset of cached key-value data - Effective bits adjustment + Preset offset value Formula (1) In one example, when the data type of the cached key-value data is FP16, the preset bit width is 8 bits, the preset effective bit length of the compressed key-value data is 7 bits, and the preset bias value can be 127, then the shared exponent bit data (e8) = the initial exponent bit data - 15 - (7-1) + 127.

[0057] In this embodiment, the effective bit length adjustment value is determined by the preset effective bit length of the compressed key-value data and the preset calibration parameter value. The initial exponent bit data is offset calibrated and bit width mapped by combining the exponent offset value of the cached key-value data and the preset bias value. This achieves a precise conversion from the original floating-point exponent space to the shared exponent space. As a result, the shared exponent bit data is uniformly mapped to the preset bit width, ensuring the consistency of the exponent reference within the data group. Furthermore, by compensating for the precision loss caused by mantissa compression, the accurate restoration of the value during decoding is ensured, providing a reliable exponent reference for subsequent efficient inference.

[0058] Figure 3 This is a flowchart illustrating step S130 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 3 As shown, step S130 may include the following steps: Step S131: Concatenate the sign bit data and mantissa data of the cached key-value data to obtain the data to be compressed.

[0059] The data to be compressed includes a sign bit and a mantissa bit. When the data type of the cached key-value data is FP16, each cached key-value data can be parsed according to the IEEE 754 standard. Each cached key-value data is parsed into a sign bit, an exponent bit, and a mantissa. Since the mantissa part of FP16 implicitly contains an integer bit with a value of 1 (i.e., "1."), this bit is not explicitly stored, but together with the explicitly stored fractional part, it constitutes the complete mantissa bit data. Therefore, the hidden bit is restored for the mantissa of each cached key-value data to obtain the mantissa bit data of that cached key-value data. The mantissa bit data of the cached key-value data is then concatenated with the sign bit data to obtain the data to be compressed.

[0060] Step S132: Obtain the right shift bit corresponding to the cached key-value data.

[0061] Here, the right shift bit is the total number of bits used to right-shift the cached key-value data to be compressed. For example, the right shift bit can be preset.

[0062] Step S133: Based on the right shift bit, perform right shift and rounding operations on the cached key-value data to be compressed to obtain the compressed key-value data of the cached key-value data.

[0063] For ease of description, the last few bits of the cached key-value data are referred to as the last few bits. The data to be compressed is shifted right by these last few bits to obtain the initial compressed data. Based on the least significant 4 bits of the last few bits, rounding algorithms such as the Round to Nearest Even (RNE) algorithm are used to determine whether the last few bits need to be carried over. If a carry is needed, the initial compressed data is carried over by 1 to obtain the compressed key-value data. If no carry is needed, the initial compressed data is directly determined as the compressed key-value data.

[0064] For example, suppose the decimal values ​​of the three data to be compressed are +1650, +1654 and +1660, and the decimal values ​​of the corresponding mantissa data are 1650, 1654 and 1660, respectively. The binary representation of the halfpoint of these three mantissa data is 1000 (the corresponding decimal value is 8), and the corresponding right shift is 4.

[0065] For +1650, the binary value of the least significant 4 bits of its mantissa is 0010, corresponding to a decimal value of 2. Since 2 < 8, no carry is needed. Shifting +1650 right by 4 bits (1650 >> 4) yields a decimal value of 103, which is the decimal value of the compressed key-value data. For +1654, the binary value of the least significant 4 bits of its mantissa is 0110, corresponding to a decimal value of 6. Since 6 < 8, no carry is needed. Shifting +1654 right by 4 bits (1650 >> 4) yields a decimal value of 103, which is the decimal value of the compressed key-value data. For +1660, the binary value of the least significant 4 bits of its mantissa is 1100, corresponding to a decimal value of 12. Since 12 > 8, a carry is needed. Shifting +1660 right by 4 bits (1650 >> 4) yields a decimal value of 103, which is the decimal value of the compressed key-value data. 4) The decimal value after that is 103. Carry 103 by one bit to get 104. This 104 is the decimal value of the compressed key value data.

[0066] In this embodiment of the disclosure, by determining the corresponding right shift bit for each cached key-value data and performing right shift and rounding processing on the corresponding mantissa data based on the right shift bit, the mantissa data can be accurately compressed while maintaining exponent alignment within the cached data group. At the same time, the rounding mechanism effectively controls the loss of precision, thereby significantly reducing storage overhead while preserving the accuracy of the original value (cache key-value data) to the greatest extent, providing a reliable data foundation for subsequent efficient inference.

[0067] Figure 4 This is a flowchart illustrating step S132 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 4 As shown, step S132 may include the following steps: Step S1321: Determine the precision offset based on the effective bit length of the cached key-value data and the effective bit length of the preset compressed key-value data.

[0068] The preset effective bit length of the compressed key-value data refers to the length of the binary value occupied by the data bits excluding the sign bit. For example, the data type of the compressed key-value data can be INT8, which includes a 1-bit wide sign bit and a 7-bit wide compressed mantissa, meaning the preset effective bit length of the compressed key-value data is 7 bits. The precision offset (extraBits) is used to achieve format alignment between the data format of the cached key-value data (FP16) and the data format of the compressed key-value data (INT8).

[0069] In one embodiment, the effective bit length of the cached key-value data refers to the number of significant bits in the mantissa (including the hidden bit). For example, when the data type of the cached key-value data is FP16, the width of the mantissa is 10 bits and the hidden bit is 1 bit, then the effective bit length of the cached key-value data is 11 bits. The precision offset can be determined based on formula (2).

[0070] extraBits = Length of valid bits for cached key-value data - Length of valid bits for preset compressed key-value data (Formula 2) For example, when the data type of the cached key-value data is FP16 and the effective bit length of the preset compressed key-value data is 7 bits, based on formula (2), extraBits = 11 (effective bit length of the cached key-value data) - 7 (effective bit length of the preset compressed key-value data) = 4.

[0071] Step S1322: Determine the exponent alignment offset based on the initial exponent bit data and the exponent bit data of the cached key-value data.

[0072] The exponent alignment offset (expShift) is used to align the exponent bits (exp) of each cached key-value data in the cached data group to the initial exponent bits (maxExp). The exponent alignment offset (expShift) can be determined using formula (3) based on the initial exponent bits and the exponent bits of the cached key-value data.

[0073] expShift = maxExp - exp formula (3) For example, assuming the cached key-value data is 3.22265625, its exponent bit data has a binary value of 16, and the initial exponent bit data has a binary value of 17, then the exponent alignment offset is 17 - 16 = 1.

[0074] Step S1323: Determine the embedding correction amount based on the preset storage information and target storage information of the shared exponent bit data.

[0075] The target storage information refers to the storage information of the compressed key-value data corresponding to the cached key-value data. The storage information of the compressed key-value data includes its storage position in the storage space, where the storage position is the index of the data arranged in storage order (usually starting from 0 and increasing). The preset storage information includes the storage position of the compressed key-value data embedded for each bit of the shared exponent data, indicating which least significant bit of the compressed key-value data each bit of the shared exponent data should be embedded into. For example, the shared exponent data has a bit width of 8 bits, and the preset storage information includes a storage position set {16, 18, 20, 22, 24, 26, 28, 30}, which indicates that bits 0 to 7 of the shared exponent data are embedded into the least significant bit of the compressed mantissa of the compressed key-value data with storage positions 16, 18, 20, 22, 24, 26, 28, and 30, respectively, to achieve distributed storage of the shared exponent data across the compressed key-value data.

[0076] In one example, based on the preset storage information of the shared exponent bit data and the storage information (target storage information) of the compressed key-value data corresponding to the cached key-value data, it can be determined whether the compressed key-value data corresponding to the cached key-value data is used to store the shared exponent bit data. If the compressed key-value data needs to store the shared exponent bit data, the embedding correction amount (expPosShift) of the cached key-value data is determined to be 1; if it does not need to store the shared exponent bit data, the embedding correction amount of the cached key-value data is determined to be 0.

[0077] Step S1324: Determine the number of right shift bits based on the precision offset, exponent alignment offset, and embedding correction.

[0078] The number of right shifts can be calculated using formula (4) based on the precision offset, exponent alignment offset, and embedding correction.

[0079] Right shift bits = extraBits + expShift + expPosShift (Formula 4) In this embodiment of the disclosure, by coordinating the calculation of precision offset, exponent alignment offset and embedding correction, the right shift bits corresponding to each cached key-value data can be accurately determined. This achieves the goal of significantly compressing the data storage width of key-value data and reducing the memory usage, while maximizing the preservation of the numerical precision and dynamic range of the original cached key-value data. This provides accurate and reliable data support for the efficient and stable execution of autoregressive inference by the large language model.

[0080] Figure 5 This is a flowchart illustrating step S130 provided in another exemplary embodiment of this disclosure. In some alternative implementations, such as Figure 5As shown, step S130 may also include the following steps: Step S134: Based on preset storage information, determine the second target compressed key data from the compressed key data that has the same number of bit widths as the shared exponent bit data.

[0081] Specifically, based on the preset storage information of the shared exponent bit data and the storage bit order of each compressed key value data, the compressed key value data used to store the shared exponent bit data is determined from all the compressed key value data as the second target compressed key value data.

[0082] It should be noted that there is no specific order of execution between steps S133 and S134 in this embodiment.

[0083] Step S135: Store each bit of the shared exponent data into the least significant bit of the corresponding second target compressed key value data.

[0084] For example, the cached data group includes 32 cached key-value data, which are compressed into 32 compressed key-value data. Each compressed key-value data includes 1 sign bit and 7 compressed mantissa bits, meaning each compressed key-value data occupies 1 byte of storage. The bit width (preset bit width) of the shared exponent bit data is 8 bits. The preset storage information of this shared exponent bit data includes a storage bit sequence set {16, 18, 20, 22, 24, 26, 28, 30}. The compressed key-value data with storage bit sequences of 16, 18, 20, 22, 24, 26, 28, and 30 are determined as the second target compressed key-value data. Each bit of the shared exponent bit data is stored in the least significant bit of the corresponding second target compressed key-value data.

[0085] Table 2 shows the storage correspondence between the shared exponent data and the compressed key-value data. The byte number indicates the storage bit order. The 32 compressed key-value data occupies 32 consecutive bytes, with byte numbers 0 to 31 corresponding to the storage bit order of the 32 compressed key-value data. Bits 7 to 0 within each byte represent 8 binary bits from the most significant bit to the least significant bit, with bit 0 being the least significant bit. The shared exponent data consists of 8 bits, with each bit distributed across the least significant bit (bit 0) of the compressed key-value data with storage bit orders of 16, 18, 20, 22, 24, 26, 28, and 30. The specific embedding method is shown in Table 2, where "exp" represents one bit of the shared exponent data, "tail" represents one bit of the compressed mantissa, and "sign" represents the sign bit.

[0086] Table 2 0 symbol tail tail tail tail tail tail tail 1 symbol tail tail tail tail tail tail tail 2 symbol tail tail tail tail tail tail tail 3 symbol tail tail tail tail tail tail tail 4 symbol tail tail tail tail tail tail tail 5 symbol tail tail tail tail tail tail tail 6 symbol tail tail tail tail tail tail tail 7 symbol tail tail tail tail tail tail tail 8 symbol tail tail tail tail tail tail tail 9 symbol tail tail tail tail tail tail tail 10 symbol tail tail tail tail tail tail tail 11 symbol tail tail tail tail tail tail tail 12 symbol tail tail tail tail tail tail tail 13 symbol tail tail tail tail tail tail tail 14 symbol tail tail tail tail tail tail tail 15 symbol tail tail tail tail tail tail tail 16 symbol tail tail tail tail tail tail exp 17 symbol tail tail tail tail tail tail tail 18 symbol tail tail tail tail tail tail exp 19 symbol tail tail tail tail tail tail tail 20 symbol tail tail tail tail tail tail exp 21 symbol tail tail tail tail tail tail tail 22 symbol tail tail tail tail tail tail exp 23 symbol tail tail tail tail tail tail tail 24 symbol tail tail tail tail tail tail exp 25 symbol tail tail tail tail tail tail tail 26 symbol tail tail tail tail tail tail exp 27 symbol tail tail tail tail tail tail tail 28 symbol tail tail tail tail tail tail exp 29 symbol tail tail tail tail tail tail tail 30 symbol tail tail tail tail tail tail exp 31 symbol tail tail tail tail tail tail tail In this embodiment of the disclosure, each bit of the shared exponent data is embedded into the least significant bit of the compressed mantissa of the specified second target compressed key value data, thereby further reducing the storage space occupied; at the same time, since the embedding position is the least significant bit, the impact on the numerical precision of the compressed mantissa is minimal, thus balancing storage efficiency and data accuracy.

[0087] Figure 6 This is a flowchart illustrating step S140 provided in an exemplary embodiment of this disclosure. In some alternative embodiments, such as Figure 6 As shown, step S140 may include the following steps: Step S141: Obtain the first target compressed key value data and the target shared exponent bit data.

[0088] Specifically, the compressed key-value data corresponding to other words in the updated word sequence (excluding the current word) can be retrieved from the cache as the first target key-value data. The target shared exponent bit data can be obtained as follows: each first target compressed key-value data corresponds to its own preset storage information for shared exponent bit data. For each first target compressed key-value data, the preset storage information for the shared exponent bit data corresponding to that first target compressed key-value data is obtained; based on this preset storage information, each bit of the target shared exponent bit data is extracted from the least significant bit of the compressed key-value data storing the target shared exponent bit data; the extracted bits are then concatenated to obtain the target shared exponent bit data.

[0089] For example, taking the example in step S135, the preset storage information includes a storage bit sequence set {16, 18, 20, 22, 24, 26, 28, 30}. The compressed key-value data with storage bit sequences 16, 18, 20, 22, 24, 26, 28, and 30 contains shared exponent bit data. Each bit of the target shared exponent bit data is extracted from the least significant bit (bit0) of these compressed key-value data, and the bits are concatenated according to their bit order to obtain the shared exponent bit data. Step S142: Based on the target shared exponent data, decode the first target compressed key data to obtain decoded data of floating-point type.

[0090] The data type of the decoded data is FP16. The decoded data of the first target compressed key value data can be calculated based on the target shared exponent data using formula (5).

[0091] Decoded data = First target compressed key value data × 2 (共享指数位数据-127) Formula (5) In step S143, the large language model infers the next lexical unit based on the decoded data and the updated lexical unit sequence.

[0092] In one example, the updated word sequence includes t words (t is an integer ≥ 1), where the t-th word is the latest word added to the word sequence. First, the input vector of each word in the updated word sequence is calculated. Each input vector is formed by superimposing the word embedding (which maps discrete word IDs to dense vectors through a word embedding matrix) and its positional encoding (such as absolute positional encoding or RoPE). Then, step S141 is executed to obtain the decoded data corresponding to each first target compressed key value data. The updated word sequence is input into the large language model along with its input vectors and decoded data. The self-attention layer of the large language model utilizes its three internal weight matrices W. Q W K and W V A linear transformation is performed on the input vector of the t-th word to obtain the query vector, key vector, and value vector of the t-th word. Then, based on the decoded data and the query vector, key vector, and value vector of the t-th word, the (t+1)-th word (i.e., the next word) is inferred through self-attention calculation. The (t+1)-th word and the value vector and key vector of the t-th word are output. Next, the key vector and value vector of the t-th word are added to the cached data group as cached key-value data. Then, steps S110 to S130 are performed to compress and cache them. The above process is iteratively executed until all words of the task to be processed are generated. Among them, when inferring the first word of the task to be processed, the start symbol is... <s>As the initial lexical sequence input to the large language model, the large language model generates the key vector and value vector of the first lexical and the start symbol, and adds the key vector and value vector as cached key-value data to the cache data group.

[0093] In this embodiment of the disclosure, floating-point precision is restored by decoding compressed key-value data, which ensures the accuracy of large language model inference. At the same time, the newly generated cached key-value data is compressed, so that the entire inference process always maintains a compressed state with low video memory usage, thus achieving a closed-loop unity of efficient storage and high-precision inference.

[0094] In one application example, assume the data block (i.e., the cached data group) includes 32 cached key-value pairs, each in FP16 format. The first five cached key-value pairs in the storage bit order are 3.0, -1.5, 0.75, 0.25, and 3.222, respectively, while the remaining cached key-value pairs are all 0. The compressed key-value pairs are in INT8 format with an 8-bit width, consisting of a 1-bit sign bit and a 7-bit compressed mantissa. The default bit width for the shared exponent data is 8 bits, and the default effective bit length for the compressed key-value pairs is 7 bits. The default storage information specifies the storage bit order set for embedding the shared exponent data as {16, 18, 20, 22, 24, 26, 28, 30}.

[0095] Step 1: Convert all cached key-value data except 0 to FP16 format. See Table 3 for the results.

[0096] In FP16 representation, the storage exponent (i.e., the exponent bit data in this embodiment) is equal to the sum of the actual exponent and the bias value 15; the sign bit data (sign) indicates the positive or negative sign of the cached key-value data (0 for positive, 1 for negative); and the mantissa is the fractional part of the cached key-value data.

[0097] The second step is to compress the cached key-value data.

[0098] The results of compressing each cache key-value data are shown in Table 4. In Table 4, index indicates the storage bit order; "Restore Hidden Bit v" indicates the integer representation obtained by combining the mantissa of the FP16 floating-point number with the implicit integer bit "1", which is the mantissa data in this embodiment; "Signed vC" indicates the result after adding a sign bit (positive or negative) to the restored mantissa data, corresponding to the positive or negative sign of the original cache key-value data, which is the data to be compressed in this embodiment; the "int8" column gives the compressed key-value data.

[0099] In this application example, the maximum value of the exponent data (i.e., 16) in the data block is determined as the initial exponent data; simultaneously, the storage bit order of the cached key-value data is consistent with that of its corresponding compressed key-value data. The compression process is detailed below using the cached key-value data with a storage bit order (index) of 4 (value 3.222) as an example; the compression method for the remaining cached key-value data is the same. The FP16 representation of this cached key-value data is approximately 3.22265625, with its sign bit (sign) = 0, exponent bit (exp) = 16, and mantissa (mantissa) = 626. The compression parameters are calculated as follows: Precision offset extraBits = FP16 valid bit length 11 - default compressed key value data valid bit length 7 = 4; Exponential alignment offset expShift = maxExp - exp = 16 - 16 = 0; Since the storage bit order of this data is 4, according to the preset embedding position configuration (this storage bit order is not used to store shared exponent bit data), the embedding bit order correction expPosShift = 0; The number of bits to shift right is equal to the sum of extraBits, expShift, and expPosShift, which is 4.

[0100] The hidden bit of the mantissa of the cached key-value data is restored: the mantissa data v = 626 + 1024 = 1650 (mantissa data), its binary representation is 0b110_0111_0010 (11 bits), combined with the sign bit data (positive), the signed mantissa data vC = +1650 (data to be compressed) is obtained. The RNE algorithm is used for rounding check: the lowest 4 bits of 1650 are 0010 (decimal 2), the half point = 1000 (decimal 8), since 2 < 8, no carry, the rounding value rounding = 0; shift 1650 right by 4 bits: 1650 >> 4 = 103, so the output compressed key-value data output[4] = 103 (i.e. the compressed result of INT8 format).

[0101] The third step is to calculate the shared exponent data.

[0102] The initial exponent bit data is 16, the exponent offset value of the cached key-value data is 15 (the exponent offset value of FP16 format is 15), the effective bit length adjustment value is 6 (that is, the effective bit length of the preset compressed key-value data is 7 minus 1); the preset offset value is 127, based on which the shared exponent bit data e8 is calculated as 16 - 15 - 6 + 127 = 122.

[0103] Step 4: Embed shared exponent bit data.

[0104] The binary form of the shared exponent data 122 is 0b0111_1010 (8 bits). According to the preset storage bit sequence set {16, 18, 20, 22, 24, 26, 28, 30}, bits 0 (least significant bit) to 7 of this 8-bit data are sequentially written into the least significant bit (LSB) of the compressed key-value data for the corresponding storage bit sequence. Specifically, the LSB of the compressed mantissa of the compressed key-value data in storage bit sequence 16 stores bit 0 of the shared exponent, bit sequence 18 stores bit 1, and so on. See Table 5 for the embedded compressed key-value data (where "position" indicates the storage bit sequence and "value" is the final compressed key-value data).

[0105] The precision of the stored compressed key-value data can be verified. Specifically, the decoded data corresponding to each compressed key-value data is calculated based on formula (5). In this example, the shared exponent bit data is 122, therefore the decoded data = int8 (compressed key-value data) × 2 (122-127) = int8 (compressed key-value data) ÷ 32. Table 6 shows a comparison between the decoded data and the original cached key-value data, demonstrating almost no error between them.

[0106] In another application example, Figure 7 This is a schematic diagram of large language model inference provided in an application example of this disclosure. For example... Figure 7 As shown, it includes: 1. Output the t-th word of the task to be processed in the large language model, as well as the key vector and value vector of the (t-1)-th word. Add the value vector and the key vector to the cache data group. The data type of the cache key-value data is FP16. The t-th word is the current word, and the (t-1)-th word is the previous word. 2. Following the steps S110 to S130 above, the vector processor compresses each cache key-value data in the cache data group to obtain the corresponding compressed key-value data and shared exponent bit data. The data type of the compressed key-value data is INT8, and the bit width of the shared exponent bit data is 8 bits. 3. Based on the KV Cache mechanism, each compressed key-value data and the shared exponent data are stored in the cache; 4. Add the t-th word to the word sequence, and read the first target compressed key value data and the corresponding target shared index bit data corresponding to the first word to the (t-1)-th word in the word sequence from the cache; 5. The MAC Compute unit decodes the compressed key data of each first target by using the target shared exponent data to recover the decoded FP16 data; 6. Based on each decoded data and word sequence, the large language model infers and generates the (t+1)th word, outputs the (t+1)th word, and the key vector and value vector of the tth word. The key vector and value vector are added to the cached data group as cached key-value data. Steps 2 to 6 are executed iteratively until the inference ends.

[0107] Figure 8 This is a structural block diagram of a large language model inference acceleration device provided in an exemplary embodiment of the present disclosure. The large language model inference acceleration device utilizes electronic equipment and includes: The first storage module 200 is used to respond to the output of the current word of the task to be processed by the large language model, and add the key vector and value vector corresponding to the previous word as cache key-value data to the cache data group. The exponent determination module 210 is used to determine the initial exponent data based on the exponent data of each cache key value data in the cache data group; Exponent mapping module 220 is used to map the initial exponent bit data to a preset bit width to obtain shared exponent bit data; The mantissa compression module 230 is used to compress the mantissa bit data and sign bit data of each cache key value data in the cache data group to obtain compressed key value data, and cache the shared exponent bit data and each compressed key value data. The lexical reasoning module 240 is used to add the current lexical to the lexical sequence of the task to be processed. The large language model infers the next lexical of the task to be processed based on the updated lexical sequence, the first target compressed key value data corresponding to the updated lexical sequence, and the target shared index bit data corresponding to the first target compressed key value data. The operation of the large language model inferring the lexical of the task to be processed is iteratively executed until the large language model infers all the lexical of the task to be processed.

[0108] In some optional implementations, the exponent determination module 210 in this embodiment of the present disclosure is specifically used to obtain the exponent bit data with the largest value from each cached key-value data as the initial exponent bit data.

[0109] In some optional implementations, the exponent mapping module 220 in this embodiment of the present disclosure is specifically used to determine the effective bit length adjustment value based on the effective bit length of the preset compressed key value data and the preset calibration parameter value; and to perform offset calibration and bit width mapping on the initial exponent bit data based on the exponent offset value of the cached key value data, the effective bit length adjustment value and the preset bias value, so as to obtain shared exponent bit data with the preset bit width.

[0110] In some optional implementations, the mantissa compression module 230 in this embodiment of the present disclosure is specifically used to concatenate the sign bit data and the mantissa bit data of the cached key-value data to obtain the data to be compressed; obtain the right shift number corresponding to the cached key-value data; and perform a right shift operation and a rounding operation on the data to be compressed based on the right shift number to obtain the compressed key-value data.

[0111] In some optional implementations, the process of obtaining the right-shifted bits corresponding to the cached key-value data in this embodiment of the present disclosure is further used for: Based on the effective bit length of the cached key-value data and the effective bit length of the preset compressed key-value data, a precision offset is determined; based on the initial exponent bit data and the exponent bit data of the cached key-value data, an exponent alignment offset is determined; based on the preset storage information and target storage information of the shared exponent bit data, an embedding correction is determined, wherein the target storage information is the storage information of the compressed key-value data corresponding to the cached key-value data; based on the precision offset, the exponent alignment offset, and the embedding correction, the right shift bits are determined.

[0112] In some alternative implementations, the caching of the shared exponent bit data in this disclosure embodiment is further used for: According to the preset storage information, a second target compressed key value data with the same number of bits as the shared exponent data is determined from the compressed key value data; each bit of the shared exponent data is stored in the least significant bit of the corresponding second target compressed key value data.

[0113] In some alternative implementations, the lexical reasoning module 240 in this disclosure includes: The data acquisition submodule is used to acquire the first target compressed key value data and the target shared exponent bit data; The decoding submodule is used to decode the first target compressed key data based on the target shared exponent data to obtain decoded data of floating-point type; The large language model is used to infer the next lexical term based on the decoded data and the updated lexical sequence.

[0114] The inference acceleration device for the large language model in this disclosure corresponds to the inference acceleration method for the large language model described above. The relevant contents can be referred to each other, and will not be repeated here.

[0115] The beneficial technical effects of the exemplary embodiments of the reasoning acceleration device for large language models in this disclosure can be found in the corresponding beneficial technical effects of the exemplary methods and systems described above, and will not be repeated here.

[0116] In addition, this disclosure also provides an electronic device, including: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein, when the computer program is executed, it implements the reasoning acceleration method for a large language model as described in any of the above embodiments of the present disclosure.

[0117] Figure 9 This is a schematic diagram illustrating the structure of an application embodiment of the electronic device disclosed herein. Below, reference is made to… Figure 9 This describes an electronic device according to embodiments of the present disclosure. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0118] like Figure 9 As shown, the electronic device includes one or more processors and memory.

[0119] A processor can be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and can control other components in an electronic device to perform desired functions.

[0120] The memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor may execute the program instructions to implement the large language model inference acceleration methods of the various embodiments of this disclosure described above, and / or other desired functions.

[0121] In one example, the electronic device may also include input devices and output devices, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0122] In addition, the input device may include, for example, a keyboard, a mouse, etc.

[0123] This output device can output various information to the outside, including determined distance information, direction information, etc. The output device may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0124] Of course, for the sake of simplicity, Figure 9 Only some of the components of the electronic device relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device may include any other suitable components depending on the specific application.

[0125] In addition to the methods and apparatus described above, embodiments of this disclosure may also be computer program products comprising computer program instructions that, when executed by a processor, cause the processor to perform the steps of the reasoning acceleration method for large language models according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0126] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0127] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions that, when executed by a processor, cause the processor to perform the steps in the reasoning acceleration method for large language models according to various embodiments of this disclosure as described in the foregoing portion of this specification.

[0128] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0129] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as ROM, RAM, magnetic disk, or optical disk.

[0130] The basic principles of this disclosure have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0131] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For system embodiments, since they largely correspond to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0132] The block diagrams of devices, apparatuses, devices, and systems disclosed herein are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as "comprising," "including," "having," etc., are open-ended terms meaning "including but not limited to," and are used interchangeably with them. The terms "or" and "and" as used herein refer to the terms "and / or," and are used interchangeably with them unless the context clearly indicates otherwise. The term "such as" as used herein refers to the phrase "such as but not limited to," and is used interchangeably with it.

[0133] The methods and apparatus of this disclosure may be implemented in many ways. For example, they may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above-described order of steps for the methods is for illustrative purposes only, and the steps of the methods of this disclosure are not limited to the order specifically described above unless otherwise specifically stated. Furthermore, in some embodiments, this disclosure may also be implemented as a program recorded on a recording medium, the program including machine-readable instructions for implementing the methods according to this disclosure. Thus, this disclosure also covers recording media storing programs for performing the methods according to this disclosure.

[0134] It should also be noted that in the apparatus, devices, and methods of this disclosure, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions to this disclosure.

[0135] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this disclosure. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but rather to be carried out within the widest scope consistent with the principles and novel features disclosed herein.

[0136] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this disclosure to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.< / s>

Claims

1. A method for accelerating reasoning in a large language model, characterized in that, include: In response to the output of the large language model of the current word of the task to be processed, the key vector and value vector corresponding to the previous word are added to the cache data group as cached key-value data; Based on the exponent data of each cache key-value data in the cache data group, determine the initial exponent data; The initial exponent bit data is mapped to a preset bit width to obtain shared exponent bit data; For each cached key-value data in the cached data group, the mantissa and sign bits of the cached key-value data are compressed to obtain compressed key-value data, and the shared exponent bit data and each compressed key-value data are cached. The current lexical is added to the lexical sequence of the task to be processed. Based on the updated lexical sequence, the first target compressed key value data corresponding to the updated lexical sequence, and the target shared index bit data corresponding to the first target compressed key value data, the large language model infers the next lexical of the task to be processed. The operation of the large language model inferring the lexical of the task to be processed is iteratively executed until the large language model infers all the lexical of the task to be processed.

2. The method according to claim 1, characterized in that, The determination of the initial exponent data based on the exponent data of each cached key-value data in the cached data group includes: The initial exponent data is obtained by selecting the data with the largest exponent value from the cached key-value data.

3. The method according to claim 2, characterized in that, The step of mapping the initial exponent data to a preset bit width includes: Based on the effective bit length of the preset compressed key value data and the preset calibration parameter value, determine the effective bit adjustment value; Based on the exponent offset value of the cached key-value data, the effective bit adjustment value, and the preset offset value, the initial exponent bit data is offset-calibrated and bit-width-mapped to obtain shared exponent bit data with the preset bit width.

4. The method according to claim 1, characterized in that, The compression process for the mantissa and sign bits of the cached key-value data includes: The sign bit and mantissa bit of the cached key-value data are concatenated to obtain the data to be compressed; Obtain the right shift bits corresponding to the cached key-value data; Based on the number of right shifts, the data to be compressed is subjected to right shift and rounding operations to obtain the compressed key value data.

5. The method according to claim 4, characterized in that, The step of obtaining the right-shifted bits corresponding to the cached key-value data includes: The precision offset is determined based on the effective bit length of the cached key-value data and the effective bit length of the preset compressed key-value data. Based on the initial exponent bit data and the exponent bit data of the cached key-value data, determine the exponent alignment offset; Based on the preset storage information and target storage information of the shared exponent data, the embedding correction amount is determined, wherein the target storage information is the storage information of the compressed key-value data corresponding to the cached key-value data; The number of right shifts is determined based on the precision offset, the exponent alignment offset, and the embedding correction.

6. The method according to claim 1, characterized in that, Cache the shared exponent bit data, including: Based on preset storage information, a second target compressed key value data with the same number of bit widths as the shared exponent bit data is determined from the compressed key value data; Each bit of the shared exponent data is stored in the least significant bit of the corresponding second target compressed key value data.

7. The method according to claim 1, characterized in that, The large language model infers the next lexical term of the task to be processed based on the updated lexical sequence, the first target compressed key value data corresponding to the updated lexical sequence, and the target shared index bit data corresponding to the first target compressed key value data. Obtain the first target compressed key value data and the target shared exponent bit data; Based on the target shared exponent data, the first target compressed key data is decoded to obtain decoded data of floating-point type; The large language model infers the next lexical term based on the decoded data and the updated lexical sequence.

8. A reasoning acceleration device for a large language model, characterized in that, include: The first storage module is used to respond to the current word of the task to be processed output by the large language model, and add the key vector and value vector corresponding to the previous word as cache key-value data to the cache data group. The exponent determination module is used to determine the initial exponent data based on the exponent data of each cache key-value data in the cache data group; The exponent mapping module is used to map the initial exponent bit data to a preset bit width to obtain shared exponent bit data; The mantissa compression module is used to compress the mantissa bit and sign bit of each cache key value data in the cache data group to obtain compressed key value data, and cache the shared exponent bit data and each compressed key value data. The lexical reasoning module is used to add the current lexical to the lexical sequence of the task to be processed. The large language model infers the next lexical of the task to be processed based on the updated lexical sequence, the first target compressed key value data corresponding to the updated lexical sequence, and the target shared index bit data corresponding to the first target compressed key value data. The operation of the large language model inferring the lexical of the task to be processed is iteratively executed until the large language model infers all the lexical of the task to be processed.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method described in any one of claims 1-7.

11. A computer program product comprising computer program instructions, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1-7.

Images