Memory optimization method, system and electronic device for real-time quantization unit
By acquiring the variance of grouped data and calculating the quantization coefficients based on the variance, the K and V caches in the KV cache are dynamically and adaptively quantized, which solves the problem of limited memory bandwidth in Large Language Model (LLM) inference and improves the efficiency and computational performance of model inference.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHANGHAI JIAOTONG UNIV
- Filing Date
- 2024-12-30
- Publication Date
- 2026-04-24
AI Technical Summary
Existing technologies suffer from limited memory bandwidth in Large Language Model (LLM) inference, and traditional quantization methods are computationally complex and cannot adapt to dynamically changing data streams in real time, especially in key-value caches, resulting in low efficiency.
By acquiring the variance of the grouped data, the grouped quantization coefficient is calculated based on the variance. The K and V caches in the KV cache are dynamically and adaptively quantized. A two-stage quantization scheme is adopted to optimize the real-time quantization process of the V cache, and pipelined calculation is combined to improve efficiency.
Dynamic adaptive data type quantization of KV cache was implemented, the memory of real-time quantization unit was optimized, the efficiency and computational performance of model inference were improved, and the latency was reduced.
Smart Images
Figure CN119902707B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of memory control technology, and in particular to the field of memory optimization technology for real-time quantization units. Background Technology
[0002] Currently, Large Language Model (LLM) inference exhibits memory-bound characteristics. Therefore, optimizing memory usage is crucial for accelerating LLM inference, with model weights and key-value (KV) caches being the primary sources of memory overhead. Quantization techniques are a major method for reducing memory overhead. Traditional LLM quantization methods typically perform mapping at the tensor or channel level, referred to as tensor-level quantization or channel-level quantization, respectively. In this case, outliers significantly impact quantization performance because they substantially increase the rounding error of standard values across the entire tensor or channel. To address the challenge posed by outliers, many studies have proposed group quantization methods. Group quantization uses smaller groups (e.g., 64 consecutive elements within a channel) as the basic unit of quantization. Although the mapping parameters introduced by each group incur some overhead, fine-grained group quantization can limit the impact of outliers to a smaller region, thereby improving overall quantization performance.
[0003] The diversity of data distribution at the group level is significantly higher than at the tensor level. In short, while different tensors may exhibit similar distributions, the distributions of different groups within a tensor can differ significantly. This observation underscores the importance of achieving full adaptivity in group quantization, an emerging paradigm for accelerating large language models, where significant group-level diversity requires high adaptability to fully realize its potential. Therefore, finding the correspondence between data distribution and specific data types is crucial, as it significantly improves the quantization performance of weights and key-value caches. Furthermore, for key-value caches, dynamically determining the scaling factor and data type for each group presents a challenge to the quantization process.
[0004] This invention proposes a method for determining the data type based on the statistical characteristics of data distribution, and designs a corresponding architecture to support dynamic adaptive data type quantization of KV cache. Summary of the Invention
[0005] In view of the shortcomings of the prior art described above, the purpose of this application is to provide a memory optimization method, system and electronic device for real-time quantization units, which optimizes the memory of real-time quantization units by using dynamic adaptive data type quantization of KV cache, so as to improve the efficiency of model inference.
[0006] To achieve the above and other related objectives, this application provides a memory optimization method for a real-time quantization unit. The method includes: during the KV cache quantization process, obtaining the variance of the grouped data and obtaining the grouped quantization coefficients based on the variance; in the pre-filling stage, quantizing the K cache and V cache in the KV cache into corresponding data types based on the grouped quantization coefficients; in the decoding stage, obtaining the data of the entire group in each iteration of the K cache, thereby achieving real-time quantization; configuring a processing window for the V cache, and performing real-time quantization on the V cache based on the processing window.
[0007] In one embodiment of this application, obtaining the variance of the grouped data and obtaining the grouped quantization coefficient based on the variance includes: performing streaming processing on the grouped data to calculate the variance of the grouped data; and obtaining the grouped quantization coefficient of the current group based on a pre-configured mapping relationship between the variance and the data type.
[0008] In one embodiment of this application, the variance is Gaussian distributed.
[0009] In one embodiment of this application, during the pre-filling stage, the input data is sequence data, and the K-cache and the V-cache are matrix data; the K-cache and the V-cache are quantized into 4-bit data types.
[0010] In one embodiment of this application, during the decoding stage, the input data is vector data, and the K buffer and the V buffer are vector data.
[0011] In one embodiment of this application, configuring a processing window for the V cache and performing real-time quantization on the V cache based on the processing window includes: defining each G iteration in the decoding stage as a processing window for the V cache; where G is the group size; performing quantization processing on the V cache until the processing window is filled; when the processing window is filled, calculating the quantization coefficient of the current data, and performing real-time quantization on the V cache based on the quantization coefficient of the current data.
[0012] In one embodiment of this application, the V cache is quantized until the processing window is filled, and the V cache is quantized into an 8-bit data type; when the processing window is filled, the quantization coefficient of the current data is calculated, and the V cache is quantized in real time based on the quantization coefficient of the current data, and the 8-bit data type of the V cache is quantized into a 4-bit data type.
[0013] To achieve the above and other related objectives, this application also provides a memory optimization system for a real-time quantization unit, used to execute the memory optimization method for the real-time quantization unit described above. The memory optimization system for the real-time quantization unit includes: a plurality of real-time quantization units, each of which includes a comparator and two accumulators.
[0014] In one embodiment of this application, the number of real-time quantization units is matched with the size of the systolic array, and is used to calculate the maximum value of the K-buffer and V-buffer and the variance in a pipelined manner.
[0015] To achieve the above and other related objectives, this application also provides an electronic device including a memory optimization system for a real-time quantization unit as described above.
[0016] As described above, the memory optimization method, system, and electronic device for the real-time quantization unit of this application have the following beneficial effects:
[0017] This application can support dynamic adaptive data type quantization of KV cache and optimize the memory of real-time quantization unit to improve the efficiency of model inference. Attached Figure Description
[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This illustrates the calculation process of the K-cache and V-cache in the decoding stage of the memory optimization method for the real-time quantization unit in one embodiment of this application.
[0020] Figure 2 The diagram shows the overall flow of a memory optimization method for a real-time quantization unit in one embodiment of this application.
[0021] Figure 3 The diagram shows a flowchart of the process for obtaining grouped quantization coefficients in a memory optimization method for a real-time quantization unit according to an embodiment of this application.
[0022] Figure 4 The diagram shown is a schematic representation of the pre-filling stage in the memory optimization method for a real-time quantization unit according to an embodiment of this application.
[0023] Figure 5 The flowchart shown is a process of quantizing the V cache in the memory optimization method of the real-time quantization unit in one embodiment of this application.
[0024] Figure 6 The diagram shown is a schematic diagram of the memory optimization system of the real-time quantization unit in one embodiment of this application.
[0025] Figure 7 The diagram shown is a hardware architecture diagram of a memory optimization system for a real-time quantization unit in one embodiment of this application.
[0026] Figure 8 The diagram shown is an overall implementation principle diagram of the memory optimization method and system for a real-time quantization unit in one embodiment of this application.
[0027] Figure 9 The diagram shown is a schematic block diagram of an electronic device according to an embodiment of this application. Detailed Implementation
[0028] The following specific examples illustrate the implementation of this embodiment. Those skilled in the art can easily understand other advantages and effects of this embodiment from the content disclosed in this specification. This embodiment can also be implemented or applied through other different specific implementation methods, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this embodiment. It should be noted that, in the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.
[0029] As cached data increases, the pressure on GPU memory also gradually increases. To alleviate this problem, strategies such as weighted quantization, cache strategy optimization, and model parallelism and data parallelism can be adopted. Currently, most quantization methods for cached data in real-time quantization units use INT or FP data types. Adaptive data type methods are mainly divided into two categories: data type-based methods and clustering-based methods. Data type-based adaptive methods select appropriate data types from a discrete set of data types based on the tensor's data distribution. ANT (MICRO'22) is a typical data type-based method. ANT provides a variety of different data types to choose from, including INT for uniform distribution, PoT (Power of Two) for Laplace distribution, and flint for Gaussian distribution. Clustering-based adaptive methods use clustering algorithms to generate cluster centers aligned with the data distribution, thus providing strong adaptability. Mokey (MICRO'22) and GOBO (MICRO'20) are typical representatives of this method, although they are mainly used for tensor-level or channel-level quantization. MANT proposes a method based on mathematical expressions to determine data types by adjusting a coefficient 'a' to fit the expression of different data types. The expression for MANT is as follows:
[0030] Value grid =±(a×|INT|+2)|INT| )
[0031] In the above formula, Value grid represents the quantization grid, 'a' is a grouping constant, and INT is an INT4 data type with sign-magnitude representation, used for symmetric quantization, covering the range [-7, 7]. This embodiment uses INT4 as an example to explain the design because INT4 calculations are easy to support and aligned in memory.
[0032] Current data type-based adaptive methods primarily rely on calibration datasets, selecting the optimal data type for tensors, channels, or groups by minimizing the mean squared error (MSE) after quantization. However, this approach suffers from significant technical limitations in practical applications. First, the calibration process requires multiple rounds of quantization and dequantization operations on the calibration dataset to evaluate each possible data type. This search process is not only computationally complex but also extremely time- and computationally resource-intensive, especially when dealing with high-dimensional tensors or large-scale neural networks. This makes it difficult for traditional methods to meet the efficiency requirements of practical applications.
[0033] Furthermore, for data streams with dynamically changing characteristics (such as KV cache), the calibration dataset may not adequately represent their distribution characteristics, thus reducing the effectiveness of data type selection. Especially in the context of KV cache, this method requires searching for the optimal data type for each group or channel individually. As the data size increases, the computational overhead grows exponentially, becoming almost unacceptable. This approach lacks real-time adaptability and cannot quickly respond to changes in data distribution, further limiting its application in high-efficiency, low-latency scenarios. In summary, existing methods have significant limitations in terms of computational efficiency and real-time performance, urgently requiring a more efficient and real-time-responsive adaptive data type selection method.
[0034] Existing cached data quantization methods require establishing a mapping between data distribution and data type to support dynamic data type selection for different groups, enabling real-time data type determination during quantization. Current methods lack this mapping. Secondly, key-value (KV) caching faces challenges in real-time group quantization because it is dynamically generated during large language model inference. To achieve low-precision computation in group quantization, K and V need to be quantized along the inner dimension. This requirement stems from the support of matrix inner product operations in most GPUs and TPUs. In these operations, the scaling factor for the groups can be extracted from the multiply-accumulate calculation. Figure 1The calculation process of K and V during the decoding stage is illustrated. The dimension used for matrix inner product operations is called the inner dimension. The inner dimensions of the K cache and V cache are different; the K cache requires a transpose operation, while the V cache does not, which complicates matters further.
[0035] This application proposes a memory optimization method for real-time quantization units, which optimizes the memory of real-time quantization units by using dynamic adaptive data type quantization through KV cache, thereby improving the efficiency of model inference.
[0036] In addressing the challenge of KV cache group quantization, this application dynamically applies adaptive data types. This application presents a method and its hardware architecture for dynamically calculating the number and type of each group in the KV cache based on data flow, and performing adaptive data type group quantization according to the characteristics of K and V respectively.
[0037] The following will describe in detail the principles and implementation methods of the memory optimization method, system, and electronic device of the real-time quantization unit in this embodiment, so that those skilled in the art can understand the memory optimization method, system, and electronic device of the real-time quantization unit in this embodiment without creative effort.
[0038] This embodiment improves a memory optimization method for a real-time quantization unit (RQU), which is a hardware unit used to accelerate real-time inference in deep learning models. It reduces computation and memory usage, thus improving inference speed, by converting the weights and activation functions in the model from floating-point numbers to low-precision integers. An RQU typically contains multiple sub-modules, such as quantizers, dequantizers, and multipliers, to support different quantization strategies and algorithms. During inference, the RQU converts the input data and model parameters from floating-point format to low-precision integer format (such as INT8 or INT4), performs integer operations, and finally converts the result back to floating-point format for output.
[0039] Figure 2 This is a schematic diagram illustrating the overall flow of a memory optimization method for a real-time quantization unit in one embodiment of this application; as shown below. Figure 2 As shown, the memory optimization method for the real-time quantization unit includes the following steps S110 to S130.
[0040] Step S110: During the KV cache quantization process, the variance of the grouped data is obtained and the grouped quantization coefficients are obtained based on the variance;
[0041] Step S120: In the pre-filling stage, the K cache and V cache in the KV cache are quantized into corresponding data types based on the group quantization coefficients.
[0042] In step S130, during the decoding stage, the K cache acquires the data of the entire group in each iteration, thereby achieving real-time quantization; a processing window is configured for the V cache, and real-time quantization is performed on the V cache based on the processing window.
[0043] This application can efficiently utilize cluster resources to support higher load throughput and achieve the lowest end-to-end latency under various request characteristics of dynamic workflows.
[0044] The following provides a detailed description of steps S110 and S130 in the memory optimization method of the real-time quantization unit in this embodiment.
[0045] Step S110: During the KV cache quantization process, the variance of the grouped data is obtained and the group quantization coefficients are obtained based on the variance.
[0046] Key-value (KV) caching enables fast data reading and writing by storing the mapping between keys and values. In LLM inference, keys typically represent features of the model input or intermediate states, while values store computational results or model parameters related to those features. KV caching includes both key-value (K) and value-value (V) caches to store the intermediate representations needed for each token in the generation sequence, avoiding the need to recompile the key-value tensors of these tokens in each generation step. By reducing redundant computation, KV caching significantly improves the speed of LLM inference. This optimization is particularly pronounced when dealing with long sequences or scenarios requiring multiple iterations.
[0047] Real-time data type selection is a crucial and challenging task in KV cache quantization. While mean squared error (MSE)-based search methods can effectively reduce accuracy loss, they require performing quantization operations on each candidate data type individually for error searching. This high computational overhead makes it virtually infeasible in real-time applications. Therefore, this embodiment proposes a dynamic mapping mechanism based on data features (such as variance), which can efficiently determine the optimal data type using streaming computation, thus avoiding the computational bottleneck of traditional methods.
[0048] Variance, as an important statistical feature, can accurately reflect the distribution characteristics of data groups. For different grouped data, there is a corresponding relationship between the distribution characteristics of specific data types and their variances. Therefore, this embodiment uses variance as a criterion to dynamically determine the most suitable data type. The core idea of this method is to infer the most suitable quantization coefficient by quickly calculating the variance of the grouped data, thereby achieving a low-latency quantization process.
[0049] Figure 3The diagram shown is a flowchart illustrating the process of obtaining grouped quantization coefficients in a memory optimization method for a real-time quantization unit according to an embodiment of this application; as follows: Figure 3 As shown, in a specific implementation of this embodiment, obtaining the variance of the grouped data and obtaining the grouped quantization coefficients based on the variance includes:
[0050] Step S111: Perform streaming processing on the grouped data and calculate the variance of the grouped data;
[0051] Step S112: Based on the pre-configured mapping relationship between variance and data type, obtain the grouping quantization coefficient of the current group.
[0052] Specifically, the quantization process consists of two key steps. First, this embodiment quickly calculates the variance of the grouped data through streaming processing. In this way, this embodiment can obtain the distribution characteristics of the grouped data in real time without additional complex calculations. Second, based on the predefined mapping relationship between variance and data type, a quantization coefficient suitable for the current group is directly selected. The specific formula is as follows:
[0053]
[0054] Where, x i It represents the numerical value of each element, with the first term being x. i The mean of the sum of squares, the second term is x. i The variance of the data can be calculated from the square of the mean and these two terms. In a specific implementation of this embodiment, the variance follows a Gaussian distribution. The variance of a Gaussian distribution is σ. 2 That is, the square of the standard deviation. The mean μ determines the central location of the distribution, while the variance σ... 2 This describes the dispersion of data points around the mean. Specifically, the variance σ 2 Variance is the average of the squares of the differences between each data point and its mean, reflecting the range and dispersion of the data. Variance is closely related to standard deviation (σ), which is the arithmetic square root of variance. Although both describe the dispersion of data, variance, being a squared term, may be more intuitive and convenient to calculate and interpret.
[0055] In data distribution, a larger variance indicates that the data is close to a uniform distribution, while the variance of a Gaussian distribution is relatively small. This embodiment first samples the K and V tensors by calibrating the dataset to select the data type for each group with the goal of reducing MSE, thereby minimizing quantization error. Subsequently, this embodiment calculates the variance of each group under different data types and determines the corresponding range. To further optimize efficiency, this embodiment integrates the variance calculation process with matrix multiplication operations, thereby effectively hiding latency and improving overall performance.
[0056] Step S120: In the pre-filling stage, the K cache and V cache in the KV cache are quantized into corresponding data types based on the group quantization coefficients.
[0057] During the pre-filling phase, the K-cache and V-cache can compute the scaling factor for each group. However, during the decoding phase, the newly generated K-vectors are concatenated along the inner dimensions of the K-cache, enabling immediate quantization. But the newly generated V-vectors are associated with different groups, with only one element generated for each group in each iteration. This process makes it impossible to obtain the scaling factor for the entire group in a single iteration, posing a significant challenge to real-time quantization of the V-cache.
[0058] In one specific implementation of this embodiment, during the pre-filling stage, the input data is sequence data, and the K-cache and the V-cache are matrix data; the K-cache and the V-cache are quantized into 4-bit data types.
[0059] Figure 4 The diagram shown is a schematic representation of the pre-filling stage in the memory optimization method for a real-time quantization unit according to an embodiment of this application; as follows: Figure 4 As shown, in the pre-filling stage, the input is a sequence, so both K and V are matrices, and the sequence length is typically larger than the group size. Therefore, both the K-buffer and V-buffer can obtain the data needed to calculate the variance. By selecting an appropriate coefficient 'a' based on the variance, the K-buffer and V-buffer can be quantized into a 4-bit data type.
[0060] In step S130, during the decoding stage, the K cache acquires the data of the entire group in each iteration, thereby achieving real-time quantization; a processing window is configured for the V cache, and real-time quantization is performed on the V cache based on the processing window.
[0061] In one specific implementation of this embodiment, during the decoding stage, the input data is vector data, and the K buffer and the V buffer are vector data.
[0062] Decoding Phase: In the decoding phase, since the input is a vector, the newly generated K and V are also vectors. Therefore, the K buffer can retrieve the data for the entire group in one iteration, thus achieving real-time quantization, similar to activation quantization in the decoding phase. The difference is that x also needs to be calculated for each group. i and x i 2 The partial sum and maximum value are determined because the K cache needs to be quantized as a 4-bit MANT data type. However, a new challenge arises for the V cache, as each iteration only generates one element from a group.
[0063] To address this issue, this embodiment proposes a two-stage quantization scheme for V-caching. Figure 5The flowchart shown is a process for quantizing the V cache in a memory optimization method for a real-time quantization unit according to an embodiment of this application; as follows: Figure 5 As shown, in a specific implementation of this embodiment, configuring a processing window for the V cache and performing real-time quantization for the V cache based on the processing window includes:
[0064] Step S131: Define each G iteration in the decoding stage as a processing window of the V buffer; where G is the block size;
[0065] Step S132: Quantize the V cache until the processing window is filled;
[0066] Step S133: When the processing window is full, calculate the quantization coefficient of the current data, and perform real-time quantization on the V cache based on the quantization coefficient of the current data.
[0067] In one specific implementation of this embodiment, the V cache is quantized until the processing window is filled, and the V cache is quantized into an 8-bit data type; when the processing window is filled, the quantization coefficient of the current data is calculated, and the V cache is quantized in real time based on the quantization coefficient of the current data, and the 8-bit data type of the V cache is quantized into a 4-bit data type.
[0068] In this embodiment, each G iteration in the decoding stage is defined as a processing window of the V buffer, where G is the block size. In the first stage, the newly generated V vector is quantized to INT8, and its channel-level scaling factor comes from the pre-padding stage, denoted as "scales". Simultaneously, this embodiment updates the v... i and v i 2 Find the maximum value and partial sum, and denote the parameters in the group as v. i This operation continues until the processing window is full.
[0069] The second stage quantizes the 8-bit V buffer into a 4-bit MANT data type. In this embodiment, when the processing window is full, v is used. i and v i 2 The variance is calculated. Then, similar to the pre-filling stage, this embodiment determines the coefficient 'a' based on the variance and quantizes the stacked INT8 V cache to a 4-bit data type. This two-stage quantization scheme can efficiently quantize all vectors in the processing window except for the most recently generated V vector to 4 bits. The overhead of INT8 operations on the V cache in the processing window is small and acceptable. Furthermore, this scheme can improve the quality of newly generated tokens, as some studies have shown that the most recently generated tokens are more important.
[0070] This embodiment also provides a memory optimization system for a real-time quantization unit, used to execute the memory optimization method for the real-time quantization unit described above. Figure 6 The diagram shown is a schematic representation of the memory optimization system for a real-time quantization unit in one embodiment of this application. Figure 6 As shown, the memory optimization system of the real-time quantization unit includes: multiple real-time quantization units, wherein each real-time quantization unit includes a comparator and two accumulators. Figure 7 The diagram shown is a hardware architecture diagram of a memory optimization system for a real-time quantization unit according to an embodiment of this application.
[0071] In one specific implementation of this embodiment, the number of real-time quantization units is matched with the size of the systolic array, and is used to calculate the maximum value of the K-buffer and V-buffer in a pipelined manner and to calculate the variance.
[0072] like Figure 7 As shown, the Real-Time Quantization Unit (RQU) in this embodiment comprises two components: an FP16 comparator and two FP16 accumulators. In this invention, this embodiment uses 32 RQUs, matched to the size of the systolic array, to determine the maximum values of the K-buffer and V-buffer and to calculate the variance parameter. Furthermore, it can also be used for maximum value extraction in INT-activated quantization. The RQU supports both spatial and temporal data stream modes. Figure 7 Both modes are demonstrated, and an example of using RQU to calculate the maximum value is given. Once the systolic array has completed the final calculation of the accumulated dimensions, the complete result of the output tile is captured and quantized to a low-bit format.
[0073] like Figure 7 As shown, at t=0, the first RQU (RQU 0) obtains the value of C_{0,0} and calculates the square of C_{0,0}, passing it to RQU 1. At t=1, RQU 1 adds C_{0,1} to C_{0,0} and also adds their squares, passing the result to RQU 2. By t=32, RQU 31 outputs the final sum. From this point onward, the RQUs operate in a fully pipelined manner, with RQU 31 outputting the sum in each cycle.
[0074] Figure 8 The diagram shown illustrates the overall implementation principle of a memory optimization method and system for a real-time quantization unit in one embodiment of this application; as follows: Figure 8As shown, the memory optimization method of the real-time quantization unit in this embodiment is applied to the memory optimization system of the real-time quantization unit. The memory optimization system of the real-time quantization unit forms a hardware architecture of KV cache quantization. The purpose of this embodiment is to accelerate LLM inference and reduce memory overhead. The key to KV cache real-time quantization is to determine the data type based on variance and then perform KV cache quantization in stages.
[0075] The scope of protection for the memory optimization method of the real-time quantization unit described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.
[0076] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the memory optimization method for the real-time quantization unit provided in any embodiment of this application.
[0077] In the embodiments of this application, any combination of one or more storage media can be used. The storage medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0078] This application also provides an electronic device. The electronic device includes a memory optimization system for a real-time quantization unit as described above. Figure 9The diagram shown is a structural schematic of the electronic device 100 provided in an embodiment of this application. In some embodiments, the electronic device may be a mobile phone, tablet computer, wearable device, in-vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), or other terminal device. Furthermore, the memory optimization method for the real-time quantization unit provided in this application can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. This application does not limit the specific application scenarios of the memory optimization method for the real-time quantization unit.
[0079] like Figure 9 As shown, the electronic device 100 provided in this application embodiment includes a memory 101 and a processor 102.
[0080] The memory 101 is used to store computer programs; preferably, the memory 101 includes various media that can store program code, such as ROM, RAM, magnetic disk, USB flash drive, memory card or optical disk.
[0081] Specifically, memory 101 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) and / or cache memory. Electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. Memory 101 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of the embodiments of this application.
[0082] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 so that the electronic device 100 executes the memory optimization method of the real-time quantization unit provided in any embodiment of this application.
[0083] Optionally, the processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0084] Optionally, in this embodiment, the electronic device 100 may further include a display 103. The display 103 is communicatively connected to the memory 101 and the processor 102, and is used to display the relevant GUI interactive interface of the memory optimization method of the real-time quantization unit.
[0085] In summary, this application supports dynamic adaptive data type quantization of KV cache and optimizes the memory of real-time quantization units to improve model inference efficiency. Therefore, this embodiment effectively overcomes the various shortcomings of the prior art and has high industrial application value.
[0086] The above embodiments are merely illustrative of the principles and effects of this embodiment and are not intended to limit this embodiment. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this embodiment. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this embodiment should still be covered by the claims of this embodiment.
Claims
1. A memory optimization method for a real-time quantization unit, characterized in that: The method includes: During the KV cache quantization process, the variance of the grouped data is obtained and the group quantization coefficients are obtained based on the variance; During the pre-filling stage, the K cache and V cache in the KV cache are quantized into their corresponding data types based on the group quantization coefficients. During the decoding phase, the K-cache acquires the data of the entire group in each iteration, thereby achieving real-time quantization; a processing window is configured for the V-cache, and real-time quantization is performed on the V-cache based on the processing window; The process of configuring a processing window for the V cache and performing real-time quantization for the V cache based on the processing window includes: Each G iteration in the decoding phase is defined as a processing window of the V buffer; where G is the block size; The V cache is quantized until the processing window is full; When the processing window is full, the quantization coefficient of the current data is calculated, and the V cache is quantized in real time based on the quantization coefficient of the current data.
2. The memory optimization method for the real-time quantization unit according to claim 1, characterized in that: The step of obtaining the variance of the grouped data and obtaining the grouped quantization coefficient based on the variance includes: The grouped data is stream-processed to calculate the variance of the grouped data; Based on the pre-configured mapping relationship between variance and data type, the group quantization coefficient of the current group is obtained.
3. The memory optimization method for the real-time quantization unit according to claim 2, characterized in that: The variance is Gaussian distributed.
4. The memory optimization method for the real-time quantization unit according to claim 1, characterized in that: In the pre-filling stage, the input data is sequence data, and the K-cache and the V-cache are matrix data; the K-cache and the V-cache are quantized into 4-bit data types.
5. The memory optimization method for the real-time quantization unit according to claim 1 or 4, characterized in that: During the decoding stage, the input data is vector data, and the K buffer and the V buffer are vector data.
6. The memory optimization method for the real-time quantization unit according to claim 1, characterized in that: The V-cache is quantized until the processing window is full, during which the V-cache is quantized into an 8-bit data type. When the processing window is full, the quantization coefficient of the current data is calculated, and the V-cache is quantized in real time based on the quantization coefficient of the current data, during which the 8-bit data type V-cache is quantized into a 4-bit data type.
7. A memory optimization system for a real-time quantization unit, used to execute the memory optimization method for a real-time quantization unit as described in any one of claims 1 to 6, characterized in that: The memory optimization system of the real-time quantization unit includes: multiple real-time quantization units, each of which includes a comparator and two accumulators.
8. The memory optimization system for the real-time quantization unit according to claim 7, characterized in that: The number of real-time quantization units is matched to the size of the systolic array, and is used to pipeline the calculation of the maximum values of the K-buffer and V-buffer and to calculate the variance.
9. An electronic device, characterized in that, A memory optimization system including the real-time quantization unit as described in claim 7 or claim 8.