Memory optimization method and system of real-time quantization unit and electronic equipment

Abstract

The invention provides a memory optimization method and system for a real-time quantization unit and electronic equipment, and the method comprises the steps: obtaining a packet data variance in a KV cache quantization process, and obtaining a packet quantization coefficient based on the variance; in a pre-filling stage, respectively quantizing the K cache and the V cache in the KV cache into corresponding data types based on the grouping quantization coefficient; in the decoding stage, the K cache obtains the data of the whole group in each iteration, so that real-time quantization is realized; and configuring a processing window for the V cache, and carrying out real-time quantification on the V cache based on the processing window. According to the method, dynamic self-adaptive data type quantization of KV cache can be supported, and the memory of the real-time quantization unit is optimized, so that the efficiency of model reasoning is improved.

Description

Technical Field

[0001] The present application relates to the field of memory control technology, and in particular to the field of memory optimization technology of a real-time quantization unit. Background Art

[0002] At present, the reasoning of large language models (LLMs) presents the characteristics of memory-bound bandwidth. Therefore, optimizing memory usage becomes the key to accelerate LLM reasoning, among which model weights and KV cache are the main sources of memory overhead. Quantization technology is the main method to reduce memory overhead. Traditional large language model quantization methods usually map at the tensor or channel level, which are called tensor-level quantization or channel-level quantization, respectively. In this case, outliers will significantly affect the quantization performance because they will greatly increase the rounding error of the standard value in the entire tensor or channel. In order to address the challenges brought by outliers, many studies have proposed group quantization methods. Group quantization uses smaller groups (such as 64 consecutive elements in a channel) as the basic unit of quantization. Although the mapping parameters introduced by each group bring some overhead, fine-grained group quantization can limit the impact of outliers to a smaller area, thereby improving the overall quantization performance.

[0003] The diversity of data distribution at the group level is significantly higher than that at the tensor level. In short, although different tensors show similar distributions, the distributions of different groups in a tensor may be significantly different. This observation emphasizes the importance of achieving full adaptability in group quantization, which is an emerging paradigm for accelerating large language models, and the significant group-level diversity requires a high degree of adaptability to fully unleash its potential. Therefore, it is very important to find the correspondence between data distribution and specific data types, which will significantly improve the quantization effect of weights and KV caches. In addition, for KV caches, it is also necessary to dynamically determine the scaling factor and data type of each group, which brings certain challenges to the quantization process.

[0004] The present invention mainly proposes a method for determining the data type based on the statistical characteristics of data distribution, and designs corresponding architecture support to support dynamic adaptive data type quantization of KV cache. Summary of the invention

[0005] In view of the shortcomings of the prior art described above, the purpose of the present application is to provide a memory optimization method, system and electronic device for a real-time quantization unit, which is used to optimize the memory of the real-time quantization unit through dynamic adaptive data type quantization of the KV cache to improve the efficiency of model reasoning.

[0006] To achieve the above-mentioned purpose and other related purposes, the present application provides a memory optimization method for a real-time quantization unit, the method comprising: in a KV cache quantization process, obtaining a variance of grouped data and obtaining a grouped quantization coefficient based on the variance; in a pre-filling stage, quantizing the K cache and the V cache in the KV cache into corresponding data types based on the grouped quantization coefficient; in a decoding stage, obtaining data of the entire group in each iteration of the K cache, thereby realizing real-time quantization; configuring a processing window for the V cache, and performing real-time quantization for the V cache based on the processing window.

[0007] In one embodiment of the present application, obtaining the variance of the grouped data and obtaining the grouped quantization coefficient based on the variance includes: streaming the grouped data to calculate the variance of the grouped data; and obtaining the grouped quantization coefficient of the current group based on a pre-configured mapping relationship between the variance and the data type.

[0008] In one embodiment of the present application, the variance is Gaussian distributed.

[0009] In one embodiment of the present application, in the pre-filling stage, the input data is sequence data, the K buffer and the V buffer are matrix data; the K buffer and the V buffer are quantized into 4-bit data types.

[0010] In one embodiment of the present application, in the decoding stage, the input data is vector data, and the K cache and the V cache are vector data.

[0011] In one embodiment of the present application, configuring a processing window for the V cache and performing real-time quantization for the V cache based on the processing window includes: defining each G iterations in the decoding stage as a processing window of the V cache; wherein G is the packet size; performing quantization processing on the V cache until the processing window is filled; when the processing window is filled, calculating the quantization coefficient of the current data, and performing real-time quantization for the V cache based on the quantization coefficient of the current data.

[0012] In one embodiment of the present application, the V cache is quantized until the processing window is filled, and the V cache is quantized into an 8-bit data type; when the processing window is filled, the quantization coefficient of the current data is calculated, and the V cache is quantized in real time based on the quantization coefficient of the current data, and the 8-bit data type of the V cache is quantized into a 4-bit data type.

[0013] To achieve the above-mentioned purpose and other related purposes, the present application also provides a memory optimization system for a real-time quantization unit, which is used to execute the memory optimization method for the real-time quantization unit as described above, and the memory optimization system for the real-time quantization unit includes: multiple real-time quantization units, each of which includes a comparator and two accumulators.

[0014] In one embodiment of the present application, the number of the real-time quantization units matches the size of the systolic array, and is used to calculate the maximum value of the K cache and the V cache and calculate the variance in a pipeline manner.

[0015] To achieve the above-mentioned purpose and other related purposes, the present application also provides an electronic device, including a memory optimization system of the real-time quantization unit as described above.

[0016] As described above, the memory optimization method, system and electronic device of the real-time quantization unit of the present application have the following beneficial effects:

[0017] This application can support dynamic adaptive data type quantization of KV cache and optimize the memory of real-time quantization unit to improve the efficiency of model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 Shown is the calculation process of K cache and V cache in the decoding stage in the memory optimization method of the real-time quantization unit in one embodiment of the present application;

[0020] Figure 2 Shown is a schematic diagram of the overall process of a memory optimization method for a real-time quantization unit in an embodiment of the present application;

[0021] Figure 3 Shown is a schematic diagram of a flow chart of obtaining grouped quantization coefficients in a memory optimization method of a real-time quantization unit in an embodiment of the present application;

[0022] Figure 4 It is a schematic diagram showing the principle of the pre-filling stage in the memory optimization method of the real-time quantization unit in one embodiment of the present application;

[0023] Figure 5 Shown is a flow chart of a quantization process of a V cache in a memory optimization method of a real-time quantization unit in an embodiment of the present application;

[0024] Figure 6 Shown is a schematic diagram of the principle structure of a memory optimization system of a real-time quantization unit in an embodiment of the present application;

[0025] Figure 7 Shown is a hardware architecture diagram of a memory optimization system of a real-time quantization unit in an embodiment of the present application;

[0026] Figure 8 Shown is an overall implementation principle diagram of a memory optimization method and system for a real-time quantization unit in an embodiment of the present application;

[0027] Fig. 9 It is a principle block diagram of an electronic device in one embodiment of the present application. DETAILED DESCRIPTION

[0028] The following describes the implementation of this embodiment through specific examples, and those skilled in the art can easily understand other advantages and effects of this embodiment from the content disclosed in this specification. This embodiment can also be implemented or applied through other different specific implementations, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of this embodiment. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0029] As the cache data increases, the pressure on the GPU memory will gradually increase. To alleviate this problem, strategies such as weight quantization, cache strategy optimization, model parallelism and data parallelism can be adopted. At present, most of the quantization methods for cache data in real-time quantization units use INT or FP data types. Adaptive data type methods are mainly divided into two categories: data type-based methods and clustering-based methods. Data type-based adaptive methods select appropriate data types from a discrete data type set according to the data distribution of the tensor. ANT (MICRO'22) is a typical data type-based method. ANT provides a variety of different data types to choose from, including INT for uniform distribution, PoT (Power of Two) for Laplace distribution, and flint for Gaussian distribution. Clustering-based adaptive methods use clustering algorithms to generate cluster centers aligned with data distribution, thereby providing strong adaptability. Mokey (MICRO'22) and GOBO (MICRO'20) are typical representatives of this method, although they are mainly used for tensor-level or channel-level quantization. MANT proposes a method to determine the data type based on a mathematical expression, by adjusting a coefficient a to fit the expression of different data types. The expression of MANT is as follows:

[0030] Value grid =±(a×|INT|+2|INT| )

[0031] In the above formula, Value grid represents a quantization grid, a is a grouping constant, and INT is a sign-magnitude INT4 data type for symmetric quantization, which covers a range of [-7,7]. This embodiment takes INT4 as an example to explain the design because the calculation of INT4 is easy to support and aligned in memory.

[0032] Current data type-based adaptive methods mainly rely on calibration datasets to select the optimal data type for tensors, channels, or groups by minimizing the mean square error (MSE) after quantization. However, this approach has significant technical limitations in practical applications. First, the calibration process requires multiple rounds of quantization and dequantization operations on the calibration dataset to evaluate each possible data type. This search process is not only computationally complex, but also extremely time-consuming and computationally expensive, especially when dealing with high-dimensional tensors or large-scale neural networks. This makes it difficult for traditional methods to meet the efficiency requirements in practical applications.

[0033] In addition, for data streams with dynamically changing characteristics (such as KV cache), the calibration data set may not fully represent its distribution characteristics, thereby reducing the effectiveness of data type selection. Especially in the context of KV cache, this method requires searching for the best data type for each group or channel one by one. As the data scale increases, the computational overhead grows exponentially and is almost unacceptable. This method lacks real-time adaptability and cannot quickly respond to changes in data distribution, further limiting its application in high-efficiency, low-latency scenarios. In summary, the existing methods have significant limitations in computational efficiency and real-time performance, and a more efficient and real-time adaptive data type selection method is urgently needed.

[0034] In existing cache data quantization methods, in order to support dynamic selection of data types for different groups, it is necessary to establish a mapping relationship between data distribution and data type so that the data type can be determined in real time during quantization. Existing methods lack such a mapping relationship. Secondly, in the real-time group quantization process, KV cache faces certain challenges because KV cache is dynamically generated during the inference process of large language models. In order to achieve low-precision calculations in group quantization, K and V need to be quantized along the inner dimension. This requirement stems from the support of matrix inner product operations by most GPUs and TPUs. In these operations, the scaling factors of the groups can be extracted from the multiplication and addition calculation process. Figure 1The calculation of K and V during the decoding phase is shown. The dimension used for the inner product operation is called the inner dimension. The inner dimensions of the K and V caches are different; the K cache needs to perform a transpose operation, while the V cache does not, which makes the situation more complicated.

[0035] The present application proposes a memory optimization method for a real-time quantization unit, which is used to optimize the memory of the real-time quantization unit through dynamic adaptive data type quantization of a KV cache to improve the efficiency of model reasoning.

[0036] In solving the challenge of KV cache group quantization, the present application can dynamically apply adaptive data types. The present application is a method and hardware architecture for dynamically calculating the number type of each group of KV cache based on data flow, and performing adaptive data type group quantization according to the characteristics of K and V respectively.

[0037] The following will describe in detail the principles and implementation methods of the memory optimization method, system and electronic device of the real-time quantization unit of the present embodiment, so that those skilled in the art can understand the memory optimization method, system and electronic device of the real-time quantization unit of the present embodiment without creative work.

[0038] This embodiment improves a memory optimization method for a real-time quantization unit, which is applied to the real-time quantization unit. The real-time quantization unit (Real-Time Quantization Unit, RQU for short) is a hardware unit for realizing real-time reasoning acceleration in deep learning models. It reduces the amount of calculation and memory usage and improves the reasoning speed by converting the weights and activation functions in the model from floating point numbers to low-precision integers. RQU usually includes multiple sub-modules, such as quantizers, dequantizers, multipliers, etc., to support different quantization strategies and algorithms. During the reasoning process, RQU converts the input data and model parameters from floating point format to low-precision integer format (such as INT8 or INT4), then performs integer operations, and finally converts the results back to floating point format for output.

[0039] Figure 2 It is a schematic diagram showing the overall flow of the memory optimization method of the real-time quantization unit in one embodiment of the present application; Figure 2 As shown, the memory optimization method of the real-time quantization unit includes the following steps: step S110 to step S130.

[0040] Step S110, in the KV cache quantization process, obtaining the variance of the grouped data and obtaining the grouped quantization coefficient based on the variance;

[0041] Step S120, in the pre-filling stage, quantizing the K cache and the V cache in the KV cache into corresponding data types based on the grouping quantization coefficients;

[0042] Step S130, in the decoding stage, the K cache obtains data of the entire group in each iteration, thereby realizing real-time quantization; a processing window is configured for the V cache, and real-time quantization is performed for the V cache based on the processing window.

[0043] The present application can efficiently utilize cluster resources to support higher load throughput and achieve the lowest end-to-end latency under various request characteristics of dynamic workflows.

[0044] The above-mentioned step S110 and step S130 in the memory optimization method of the real-time quantization unit of this embodiment are described in detail below.

[0045] Step S110 : in the KV cache quantization process, obtaining the variance of the grouped data and obtaining the grouped quantization coefficient based on the variance.

[0046] The KV cache achieves fast reading and writing of data by storing the corresponding relationship between keys and values. In the LLM reasoning process, the key usually represents certain features of the model input or intermediate state, while the value stores the calculation results or model parameters related to these features. The KV cache includes K cache and K cache, which are used to store the intermediate representations required to generate each token in the sequence to avoid recalculating the key and value tensors of these tokens in each generation step. Due to the reduction of repeated calculations, the KV cache can significantly improve the speed of LLM reasoning. This optimization effect is especially obvious in scenarios where long sequences are processed or multiple iterations are required.

[0047] In the KV cache quantization process, achieving real-time data type selection is an important and challenging task. Although the search method based on mean square error (MSE) can effectively reduce the loss of precision, this method requires that quantization operations be performed on each candidate data type one by one for error search. This high computational overhead makes it almost infeasible in real-time application scenarios. Therefore, this embodiment proposes a dynamic mapping mechanism based on data features (such as variance), which can efficiently determine the optimal data type in a streaming computing manner, thereby avoiding the computational bottleneck of traditional methods.

[0048] Variance, as an important statistical feature, can accurately reflect the distribution characteristics of data groups. For different grouped data, there is a corresponding relationship between the distribution characteristics of a specific data type and the variance. Therefore, this embodiment uses variance as a basis for judgment to dynamically determine the most suitable data type. The core idea of ​​this method is to infer the most suitable quantization coefficient by quickly calculating the variance of the grouped data, thereby achieving a low-latency quantization process.

[0049] Figure 3It is a schematic diagram showing the flow of obtaining grouped quantization coefficients in the memory optimization method of the real-time quantization unit in one embodiment of the present application; Figure 3 As shown, in a specific implementation of this embodiment, obtaining the variance of the grouped data and obtaining the grouped quantization coefficient based on the variance includes:

[0050] Step S111, performing stream processing on the grouped data to calculate the variance of the grouped data;

[0051] Step S112: acquiring the group quantization coefficient of the current group based on the pre-configured mapping relationship between the variance and the data type.

[0052] Specifically, the quantization process is divided into two key steps. First, this embodiment quickly calculates the variance of the grouped data by streaming it. In this way, the distribution characteristics of the grouped data can be obtained in real time without additional complex calculations. Secondly, according to the predefined mapping relationship between the variance and the data type, the quantization coefficient suitable for the current group is directly selected. The specific formula is as follows:

[0053]

[0054] Among them, x i is the value of each element, the first item is x i The second term is the mean of the sum of squares of i The square of the mean, through these two terms, the variance of the data can be calculated. In a specific implementation of this embodiment, the variance is Gaussian distribution. The variance of Gaussian distribution is σ 2 , which is the square of the standard deviation. The mean μ determines the center of the distribution, while the variance σ 2 describes the dispersion of the data points around the mean. Specifically, the variance σ 2 It is the average of the squares of the differences between each data and its mean, which reflects the fluctuation range and dispersion of the data. Variance is closely related to standard deviation σ, which is the arithmetic square root of variance. Although both can describe the dispersion of data, variance may be more intuitive and convenient in calculation and interpretation because it is a square term.

[0055] In data distribution, the larger the variance, the closer the data is to uniform distribution, while the variance of Gaussian distribution is relatively small. This embodiment first samples the K and V tensors through the calibration data set, and selects the data type of each group with the goal of reducing MSE to minimize the quantization error. Subsequently, this embodiment calculates the variance of each group under different data types and determines the corresponding range. In order to further optimize efficiency, this embodiment merges the variance calculation process with the matrix multiplication operation, thereby effectively hiding the delay and improving the overall performance.

[0056] Step S120: In the pre-filling stage, the K cache and the V cache in the KV cache are quantized into corresponding data types based on the grouping quantization coefficients.

[0057] In the pre-filling phase, the K buffer and the V buffer can calculate the scaling factor for each group. However, in the decoding phase, the newly generated K vectors are concatenated along the inner dimension of the K buffer so that they can be quantized immediately. But the newly generated V vectors are associated with different groups, and only one element is generated for each group in each iteration. This process makes it impossible to obtain the scaling factor of the entire group in a single iteration, which poses a significant challenge to the real-time quantization of the V buffer.

[0058] In a specific implementation of this embodiment, in the pre-filling stage, the input data is sequence data, the K buffer and the V buffer are matrix data; the K buffer and the V buffer are quantized into 4-bit data types.

[0059] Figure 4 It is a schematic diagram showing the principle of the pre-filling stage in the memory optimization method of the real-time quantization unit in one embodiment of the present application; Figure 4 As shown in the figure, in the pre-filling stage, the input is a sequence, so K and V are both matrices, and the sequence length is usually larger than the packet size. Therefore, both the K cache and the V cache can obtain the data required to calculate the variance. By selecting an appropriate coefficient a according to the variance, the K cache and the V cache can be quantized to a 4-bit data type.

[0060] Step S130, in the decoding stage, the K cache obtains data of the entire group in each iteration, thereby realizing real-time quantization; a processing window is configured for the V cache, and real-time quantization is performed for the V cache based on the processing window.

[0061] In a specific implementation of this embodiment, in the decoding stage, the input data is vector data, and the K cache and the V cache are vector data.

[0062] Decoding stage: In the decoding stage, since the input is a vector, the newly generated K and V are also vectors. Therefore, the K cache can obtain the data of the entire group in one iteration, thereby achieving real-time quantization, which is similar to the activation quantization in the decoding stage. The difference is that each group also needs to calculate x i and x i 2 The K buffer needs to be quantized into a 4-bit MANT data type. However, for the V buffer, new challenges arise because each iteration only generates one element in a group.

[0063] To solve this problem, this embodiment proposes a dual-stage quantization solution for V cache. Figure 5It is a flow chart showing the quantization process of the V cache in the memory optimization method of the real-time quantization unit in one embodiment of the present application; Figure 5 As shown, in a specific implementation of this embodiment, configuring a processing window for the V cache and performing real-time quantization for the V cache based on the processing window includes:

[0064] Step S131, defining each G iterations in the decoding phase as a processing window of the V cache; wherein G is the packet size;

[0065] Step S132, performing quantization processing on the V buffer until the processing window is filled;

[0066] Step S133: when the processing window is filled, the quantization coefficient of the current data is calculated, and real-time quantization is performed for the V buffer based on the quantization coefficient of the current data.

[0067] In a specific implementation of this embodiment, the V cache is quantized until the processing window is filled, and the V cache is quantized into an 8-bit data type; when the processing window is filled, the quantization coefficient of the current data is calculated, and the V cache is quantized in real time based on the quantization coefficient of the current data, and the 8-bit data type of the V cache is quantized into a 4-bit data type.

[0068] This embodiment defines each G iteration in the decoding phase as a processing window of the V cache, where G is the packet size. In the first stage, the newly generated V vector is quantized to INT8, and its channel-level scaling factor comes from the pre-filling stage, marked as "scales". At the same time, this embodiment updates v i and v i 2 The maximum value and partial sum of the group are recorded as v i This operation continues until the processing window is filled.

[0069] The second stage quantizes the 8-bit V buffer into a 4-bit MANT data type. i and v i 2 The variance is calculated by the partial sum. Then, similar to the pre-filling stage, this embodiment determines the coefficient a based on the variance and quantizes the stacked INT8 V cache to a 4-bit data type. This two-stage quantization scheme can effectively quantize all vectors except the most recently generated V vector in the processing window to 4 bits. The INT8 operation overhead of the V cache in the processing window is small and acceptable. In addition, this scheme can also improve the quality of newly generated word units, because some studies have shown that the most recently generated word units are more important.

[0070] This embodiment further provides a memory optimization system for a real-time quantization unit, which is used to execute the memory optimization method for the real-time quantization unit as described above. Figure 6 The schematic diagram of the principle structure of the memory optimization system of the real-time quantization unit in one embodiment of the present application is shown; Figure 6 As shown, the memory optimization system of the real-time quantization unit includes: a plurality of real-time quantization units, wherein each of the real-time quantization units includes a comparator and two accumulators. Figure 7 Shown is a hardware architecture diagram of a memory optimization system for a real-time quantization unit in an embodiment of the present application.

[0071] In a specific implementation of this embodiment, the number of the real-time quantization units matches the size of the systolic array, and is used to calculate the maximum value of the K cache and the V cache and calculate the variance in a pipeline manner.

[0072] like Figure 7 As shown, the real-time quantization unit (RQU) of this embodiment includes two components: one FP16 comparator and two FP16 accumulators. In the present invention, this embodiment uses 32 RQUs, which matches the size of the systolic array, to determine the maximum values ​​of the K cache and the V cache and calculate the variance parameters. In addition, it can also be used for maximum value extraction in INT activation quantization. RQU supports both spatial and temporal data flow modes. Figure 7 Both modes are demonstrated, and an example of RQU being used to compute the maximum value is given. Once the systolic array completes the final calculation of the accumulation dimension, the full result of the output tile is captured and quantized to the low-bit format.

[0073] like Figure 7 As shown in the figure, at t = 0, the first RQU (RQU 0) obtains the value of C_{0,0} and calculates the square of C_{0,0}, passing it to RQU 1. At t = 1, RQU 1 adds C_{0,1} and C_{0,0}, also adds their squares, and passes the result of the addition to RQU 2. At t = 32, RQU 31 outputs the final sum. From this point on, the RQU works in a fully pipelined manner, and RQU 31 can output the summed result every cycle.

[0074] Figure 8 The overall implementation principle diagram of the memory optimization method and system of the real-time quantization unit in one embodiment of the present application is shown; Figure 8As shown, the memory optimization method of the real-time quantization unit of this embodiment is applied to the memory optimization system of the real-time quantization unit. The memory optimization system of the real-time quantization unit forms a hardware architecture of KV cache quantization. The purpose of this embodiment is to accelerate LLM reasoning and reduce memory overhead. The key to KV cache real-time quantization is to determine the data type based on variance, and then perform KV cache quantization in stages.

[0075] The protection scope of the memory optimization method of the real-time quantization unit described in the embodiment of the present application is not limited to the execution order of the steps listed in this embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0076] An embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the memory optimization method of the real-time quantization unit provided in any embodiment of the present application is implemented.

[0077] In the embodiment of the present application, any combination of one or more storage media can be used. The storage medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, - but not limited to - an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0078] The embodiment of the present application also provides an electronic device. The electronic device includes a memory optimization system of the real-time quantization unit as described above, Fig. 9Shown is a structural diagram of an electronic device 100 provided in an embodiment of the present application. In some embodiments, the electronic device may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA) and other terminal devices. In addition, the memory optimization method of the real-time quantization unit provided in the present application can also be applied to databases, servers, and service response systems based on terminal artificial intelligence. The present application embodiment does not impose any restrictions on the specific application scenarios of the memory optimization method of the real-time quantization unit.

[0079] like Fig. 9 As shown, the electronic device 100 provided in the embodiment of the present application includes a memory 101 and a processor 102 .

[0080] The memory 101 is used to store computer programs; preferably, the memory 101 includes: ROM, RAM, disk, USB flash drive, memory card or CD-ROM and other media that can store program codes.

[0081] Specifically, the memory 101 may include a computer system readable medium in the form of a volatile memory, such as a random access memory (RAM) and / or a cache memory. The electronic device 100 may further include other removable / non-removable, volatile / non-volatile computer system storage media. The memory 101 may include at least one program product having a set (e.g., at least one) of program modules, which are configured to perform the functions of the various embodiments of the present application.

[0082] The processor 102 is connected to the memory 101 and is used to execute the computer program stored in the memory 101 so that the electronic device 100 executes the memory optimization method of the real-time quantization unit provided in any embodiment of the present application.

[0083] Optionally, the processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0084] Optionally, the electronic device 100 in this embodiment may further include a display 103. The display 103 is communicatively connected with the memory 101 and the processor 102, and is used to display a GUI interaction interface related to the memory optimization method of the real-time quantization unit.

[0085] In summary, this application can support dynamic adaptive data type quantization of KV cache, optimize the memory of real-time quantization unit, and improve the efficiency of model reasoning. Therefore, this embodiment effectively overcomes various shortcomings in the prior art and has high industrial utilization value.

[0086] The above embodiments are merely illustrative of the principles and effects of the present embodiment, and are not intended to limit the present embodiment. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present embodiment. Therefore, all equivalent modifications or changes made by those of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present embodiment should still be covered by the claims of the present embodiment.

Claims

1. A memory optimization method for a real-time quantization unit, characterized in that: The method comprises: In the KV cache quantization process, obtaining a variance of grouped data and obtaining a grouped quantization coefficient based on the variance; In the pre-filling stage, the K cache and the V cache in the KV cache are quantized into corresponding data types based on the grouping quantization coefficients; In the decoding stage, the K cache obtains data of the entire group in each iteration, thereby realizing real-time quantization; a processing window is configured for the V cache, and real-time quantization is performed for the V cache based on the processing window.

2. The memory optimization method of the real-time quantization unit according to claim 1, characterized in that: The obtaining of the variance of the grouped data and obtaining the grouped quantization coefficient based on the variance comprises: Performing stream processing on the grouped data to calculate the variance of the grouped data; Based on the pre-configured mapping relationship between the variance and the data type, a group quantization coefficient of the current group is obtained.

3. The memory optimization method of the real-time quantization unit according to claim 2, characterized in that: The variance is Gaussian distributed.

4. The memory optimization method of the real-time quantization unit according to claim 1, characterized in that: In the pre-filling stage, the input data is sequence data, the K buffer and the V buffer are matrix data; the K buffer and the V buffer are quantized into 4-bit data types.

5. The memory optimization method of the real-time quantization unit according to claim 1 or 4, characterized in that: In the decoding stage, the input data is vector data, and the K cache and the V cache are vector data.

6. The memory optimization method of the real-time quantization unit according to claim 1, characterized in that: The configuring a processing window for the V buffer and performing real-time quantization for the V buffer based on the processing window includes: Define each G iteration in the decoding phase as a processing window of the V buffer; where G is the packet size; Quantizing the V buffer until the processing window is filled; When the processing window is filled, the quantization coefficients of the current data are calculated, and real-time quantization is performed for the V buffer based on the quantization coefficients of the current data.

7. The memory optimization method of the real-time quantization unit according to claim 6, characterized in that: The V cache is quantized until the processing window is filled, and the V cache is quantized into an 8-bit data type; when the processing window is filled, the quantization coefficient of the current data is calculated, and the V cache is quantized in real time based on the quantization coefficient of the current data, and the 8-bit data type of the V cache is quantized into a 4-bit data type.

8. A memory optimization system for a real-time quantization unit, used to execute the memory optimization method for a real-time quantization unit according to any one of claims 1 to 7, characterized in that: The memory optimization system of the real-time quantization unit comprises: a plurality of real-time quantization units, each of which comprises a comparator and two accumulators.

9. The memory optimization system of the real-time quantization unit according to claim 8, characterized in that: The number of the real-time quantization units matches the size of the systolic array, and is used to calculate the maximum value of the K cache and the V cache and calculate the variance in a pipeline manner.

10. An electronic device, characterized in that: A memory optimization system comprising a real-time quantization unit as claimed in claim 8 and claim 9.

Images