Data compression method and electronic equipment

Abstract

The embodiment of the invention provides a data compression method and electronic equipment. The method is applied to the electronic equipment and comprises the steps that a first matrix is decomposed, a second matrix and a third matrix are obtained, the size of the first matrix is [a first value and a second value], the size of the second matrix is [a first value and a first value], and the size of the third matrix is [a first value and a second value]; the bit width of the first matrix, the second matrix and the third matrix is a first bit width; based on a second bit width, the third matrix is quantized, a first quantization result is generated, and the second bit width is lower than the first bit width. According to the data compression method provided by the embodiment of the invention, data compression with a high compression rate can be realized on the data matrix on the premise of ensuring high precision, and the compression effect of data compression is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a data compression method and an electronic device. Background Technology

[0002] In data application scenarios such as data transmission and data storage, data compression is necessary to reduce data transmission bandwidth usage and data storage space consumption. Therefore, a data compression method is needed. Summary of the Invention

[0003] To address the issue of how to achieve data compression, this application provides a data compression method and an electronic device, and also provides a computer-readable storage medium.

[0004] The embodiments of this application adopt the following technical solutions:

[0005] In a first aspect, this application provides a data compression method, the method being applied to an electronic device, the method comprising:

[0006] Decompose the first matrix to obtain the second and third matrices, wherein the size of the first matrix is ​​[first value, second value], the size of the second matrix is ​​[first value, first value], the size of the third matrix is ​​[first value, second value], and the bit width of the first matrix, the second matrix, and the third matrix is ​​the first bit width;

[0007] Based on the second bit width, the third matrix is ​​quantized to generate a first quantization result, wherein the second bit width is lower than the first bit width.

[0008] The data compression method according to the embodiments of this application can achieve high compression ratio of data matrix while ensuring high accuracy.

[0009] The data compression method according to the embodiments of this application can improve the compression effect of data compression.

[0010] In one implementation of the first aspect, the method further includes:

[0011] Save the second matrix and the first quantization result.

[0012] Based on the above implementation method, only the larger of the two matrices is compressed, ensuring the accuracy of the decompressed data while achieving data compression.

[0013] In one implementation of the first aspect, the method further includes:

[0014] Based on the third bit width, the second matrix is ​​quantized to generate a second quantization result, wherein the third bit width is lower than the first bit width and higher than the second bit width;

[0015] Save the first quantization result and the second quantization result.

[0016] Based on the above implementation method, the two matrices are compressed, thereby improving the compression ratio.

[0017] In one implementation of the first aspect, the first quantization result is a fourth matrix, the size of the fourth matrix is ​​[first value, second value], and the bit width of the fourth matrix is ​​the second bit width.

[0018] In one implementation of the first aspect, the first value is a preset hyperparameter, and before the first matrix is ​​decomposed, the method further includes:

[0019] Based on the first value, the fifth matrix is ​​transformed to generate the first matrix, wherein the bit width of the fifth matrix is ​​the first bit width, and the size of the fifth matrix is ​​[the third value, the fourth value].

[0020] In one implementation of the first aspect, the first value is much smaller than the product of the third value and the fourth value.

[0021] In one implementation of the first aspect, the fifth matrix is ​​the key value that needs to be saved to the key value cache, the third value is the sequence input length of the key value, and the fourth value is the hidden layer dimension.

[0022] In one implementation of the first aspect, the first value is 32.

[0023] According to the data compression method of this application embodiment, data compression can be performed on key values ​​as objects, so that key values ​​occupy less storage overhead while ensuring that the accuracy degradation is not obvious, thereby reducing the overall inference latency.

[0024] Secondly, this application provides a decompression method, which is applied to an electronic device, and the method includes:

[0025] Obtain the third quantization result and the sixth matrix, wherein: the size of the sixth matrix is ​​[fifth value, fifth value], and the bit width of the sixth matrix is ​​the fourth bit width; the third quantization result is the quantization result generated by quantizing the seventh matrix based on the fifth bit width; the size of the seventh matrix is ​​[fifth value, sixth value], and the bit width of the seventh matrix is ​​the fourth bit width; the fifth bit width is smaller than the fourth bit width;

[0026] Based on the fourth bit width, the third quantization result is dequantized to generate the seventh matrix;

[0027] Based on the sixth matrix and the seventh matrix, a matrix reconstruction is performed to generate an eighth matrix, wherein the bit width of the eighth matrix is ​​the same as the fourth bit width.

[0028] In one implementation of the second aspect, the third quantization result is a ninth matrix, the size of the ninth matrix is ​​[the fifth value, the sixth value], and the bit width of the ninth matrix is ​​the fifth bit width.

[0029] In one implementation of the second aspect, obtaining the third quantization result and the sixth matrix includes:

[0030] Obtain the fourth quantization result, wherein: the fourth quantization result is the quantization result generated by quantizing the sixth matrix based on the sixth bit width; the sixth bit width is smaller than the fourth bit width, and the sixth bit width is larger than the fifth bit width;

[0031] Based on the fourth bit width, the fourth quantization result is dequantized to generate the sixth matrix.

[0032] In one implementation of the second aspect, the eighth matrix is ​​the key value required for key value calculation, the size of the eighth matrix is ​​[seventh value, eighth value], the seventh value is the sequence input length of the key value, and the eighth value is the hidden layer dimension.

[0033] In one implementation of the second aspect, the fifth value is much smaller than the product of the seventh value and the eighth value.

[0034] In one implementation of the second aspect, the fifth value is 32.

[0035] In one implementation of the second aspect, the method includes:

[0036] Read the third quantization result and the sixth matrix from the memory;

[0037] Based on the first fusion operator, and according to the third quantization result and the sixth matrix, the eighth matrix is ​​generated, wherein:

[0038] The first fusion operator is a fusion of the inverse quantization operator and the matrix reconstruction operator;

[0039] The dequantization operator is used to dequantize the third quantization result based on the fourth bit width to generate the seventh matrix;

[0040] The matrix reconstruction operator is used to perform matrix reconstruction based on the sixth matrix and the seventh matrix to generate the eighth matrix.

[0041] In one implementation of the second aspect, the dequantization operator is manifested as multiplying the dequantization matrix with the input matrix; the matrix reconstruction operator is manifested as multiplying two input matrices; and the first fusion operator is manifested as multiplying one input matrix with the dequantization matrix and then multiplying it by another input matrix.

[0042] According to the method in the embodiments of this application, the dequantization operator and the matrix reconstruction operator are mathematically equivalent and merged into a large operator, thereby reducing the number of memory read and write operations and improving the operator execution efficiency.

[0043] In one implementation of the second aspect, the method further includes:

[0044] Based on the eighth matrix, key values ​​are calculated, and key value calculation results are generated.

[0045] In one implementation of the second aspect, the method includes:

[0046] Read the third quantization result and the sixth matrix from the memory;

[0047] Based on the second fusion operator, and according to the third quantization result and the sixth matrix, the key value calculation result is generated, wherein:

[0048] The second fusion operator is a fusion of the dequantization operator, the matrix reconstruction operator, and the key value calculation operator;

[0049] The dequantization operator is used to dequantize the third quantization result based on the fourth bit width to generate the seventh matrix;

[0050] The matrix reconstruction operator is used to perform matrix reconstruction based on the sixth matrix and the seventh matrix to generate the eighth matrix;

[0051] The key value calculation operator is used to perform key value calculation based on the input key value and generate key value calculation results.

[0052] Thirdly, this application provides an electronic device, which includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the steps of the method described in the first or second aspect.

[0053] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method described in the first or second aspect. Attached Figure Description

[0054] Figure 1 The diagram shows the GPT inference process using KV cache technology.

[0055] Figure 2 The diagram shown is a basic flowchart of the conversion calculation performed by a graphics processor / neural network processor.

[0056] Figure 3 The diagram shown is a flowchart of a data compression method according to an embodiment of this application;

[0057] Figure 4 The diagram shown is a data flow diagram of a data compression method according to an embodiment of this application;

[0058] Figure 5 The diagram shown is a data flow diagram of a data compression method according to an embodiment of this application;

[0059] Figure 6 The diagram shown is a flowchart of a data compression method according to an embodiment of this application;

[0060] Figure 7 The diagram shown is a flowchart of a decompression method according to an embodiment of this application;

[0061] Figure 8 The diagram shown is a flowchart of a decompression method according to an embodiment of this application;

[0062] Figure 9 The diagram shown is a flowchart of a decompression method according to an embodiment of this application;

[0063] Figure 10 The diagram shown is an operator fusion schematic diagram according to an embodiment of this application;

[0064] Figure 11 The diagram shown is a flowchart of a decompression method according to an embodiment of this application;

[0065] Figure 12 The figure shown is a statistical chart of the decomposition results of KV matrix decomposition according to an embodiment of the data compression method of this application;

[0066] Figure 13 The diagram shown is a flowchart of KV calculation according to an embodiment of this application;

[0067] Figure 14 The diagram shown is a flowchart of KV calculation according to an embodiment of this application;

[0068] Figure 15 The diagram shown is a flowchart of KV calculation according to an embodiment of this application;

[0069] Figure 16The figure shows the memory consumption and input / output latency statistics for data compression and decompression of KV.

[0070] Figure 17 The diagram shown is a flowchart of KV calculation according to an embodiment of this application;

[0071] Figure 18 This is a schematic diagram of an electronic device structure according to an embodiment of this application. Detailed Implementation

[0072] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0073] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0074] To address the issue of how to achieve data compression, this application provides a data compression method.

[0075] This application does not specifically limit the object of data compression. Those skilled in the art can apply the data compression method provided in this application to various types of data according to actual needs. For example, it can be used to compress communication data; or, for example, to compress streaming media data.

[0076] Large-scale language models have sparked a new wave in the field of Artificial Intelligence (AI). Generative Pre-trained Transformers (GPT) models, represented by generative models, have a unique inference process. Given a sequence of input tokens of length S, the model outputs a sequence of output tokens of length N. This process involves N inference steps. Specifically, a GPT model outputs only one token per inference step, which is then concatenated with previous input tokens (S+1, S+2, ..., S+N-1) as input for the next inference step, continuing until a terminator or the maximum output length is encountered.

[0077] This approach introduces a problem: as the number of iterations increases, the computational cost of each inference increases accordingly; furthermore, since each iteration includes the input from the previous iteration, there is a significant amount of redundant computation. Therefore, engineers use key-value caching (KV caching) technology to keep the computational cost of each inference round after the first iteration relatively constant, thereby saving computing power and reducing inference latency.

[0078] Figure 1 The diagram shows the GPT inference process using KV cache technology.

[0079] A schematic diagram of a solution using KV cache technology is shown below. Figure 1 The process is divided into a first-round context filling (Prefill) phase and a decoding (Decode) phase. The Prefill phase is executed once, and the Decode phase is executed N times (N is an integer greater than 1).

[0080] In the Prefill phase, the input data passes through the embedding module, the TransformerBlockxL module, and the Logit GEMM module.

[0081] In the Prefill phase, the input data dimension is [B, S], where B represents the batch size, a common feature in machine learning. The intermediate layer data processing and transmission dimension is [B, S, H], where H represents the hidden layer dimension. The output data dimension is [B, 1], representing the number of tokens in a batch; if there is only one batch, there is only one token. Inference is performed only once. During inference, the key-value (KV) information of the intermediate layer transformation block (TransformerBlock xL) is cached in the key-value cache for later use in the Decode phase.

[0082] During the Decode phase, the input data passes through the embedding module, the TransformerBlockxL module, and the Logit GEMM module.

[0083] In the Decode phase, the input data dimension is [B, 1]. The data processing and transmission dimension in the intermediate layers of the model is [B, 1, H]. The output data dimension is [B, 1]. The Decode phase will be executed N times until a termination condition is encountered. Similarly, the inference process first retrieves the key-value information stored in the previous round from the key-value cache, and simultaneously updates the newly generated key-value information by caching it in the key-value cache.

[0084] During the Decode phase, memory access latency is the main component of the total inference latency (computation latency + memory access latency). Memory access includes access to model weights and the KVCache. The size of the KVCache satisfies the following formula:

[0085] KVCache_Size = 2B * S * N bits * H * L. (Formula 1)

[0086] In Equation 1, B is the batch size, S is the length of the input sequence, Nbits is the storage size occupied by the KVCache information corresponding to each Token (e.g., 16 / 8 / 4 bits are 2 / 1 / 0.5 bytes respectively), H and L are the hidden layer dimension and the number of network layers corresponding to the model.

[0087] KVCache grows linearly with the increase of sequence length and batch size. Tables 1 and 2 below show the relationship between the KVCache size and batch size and sequence length (Seqlen) of a certain model.

[0088] Table 1

[0089]

[0090] Table 2

[0091]

[0092] Figure 2 The diagram shows the basic flowchart of the transformation computation performed by a graphics processing unit (GPU) / neural network processing unit (NPU).

[0093] like Figure 2 As shown, the transformer consists of multiple identical blocks. The KV cache-related computations in the i-th block can be simply abstracted into three parts: KV storage (i-th KV storage), KV reading (i-th KV reading), and KV computation (i-th KV computation). The latency related to KV memory access (i.e., KV storage and KV reading latency) far exceeds the KV computation latency, which is the memory-bound bottleneck.

[0094] Therefore, during the transformer decode inference phase, memory access latency constitutes the majority of the overall inference latency, and it is strongly correlated with model weights and the capacity of the KV cache. Since the KV cache grows linearly with the sequence length and batch size, this leads to a continuous increase in memory access latency.

[0095] Therefore, in order to control memory access latency, it is necessary to control the storage capacity of the KV Cache.

[0096] Therefore, in one application scenario, the data compression method provided in this application compresses the KV data that needs to be saved to the KVCache. By compressing the KV data, the storage capacity of the KVCache is controlled, thereby controlling the memory access latency.

[0097] One feasible approach to data compression using key-value pairs (KV) is to compress the KV sequence to a given length. Specifically, the KV sets a given size, and each time a KV is updated, the KV with the lowest cumulative attention score is removed. This step is repeated to compress the KV sequence to the given length by relying on the attention score and TopK calculation.

[0098] While the above scheme can control the length of the key-value (KV) sequence to a given length, thus achieving KV compression, the KV attention score only represents the dependency relationship between the KV and the current token. Discarded KV values ​​are not visible in subsequent token calculations and may have a high attention score. In other words, discarded KV values ​​may be data with high importance in subsequent calculations. Since discarded KV values ​​cannot be recovered, the act of discarding KV values ​​may affect the calculation accuracy.

[0099] Another feasible approach to data compression using key-value pairs is to perform low-bit quantization on the key-value information. For example, the original 16-bit calculation can be performed and stored using 8 bits, thereby achieving a compression of 1.5x-2x and an acceleration of 1.2-1.5x.

[0100] However, since key-value pairs belong to the activation category, they are more difficult to quantize than weights, and generally require validation using a dataset. Furthermore, due to limitations in quantization precision, compression ratios are typically restricted; pure 8-bit quantization achieves 2x compression, and 4-bit quantization achieves 4x compression. A mixture of 16-bit and 8-bit quantization is commonly used, but achieving 2x compression is difficult.

[0101] Another feasible approach to data compression using key-value pairs (KV) is to decompose the KV information into a low-rank form: compressing it using the lower projection matrix KV to obtain a one-dimensional vector (thus significantly reducing the size of the KV), and then restoring it using the upper projection matrix during inference, thereby achieving significant compression of KV storage.

[0102] However, the above compression scheme relies on model training and cannot perform low-rank decomposition compression on the trained model. Furthermore, the up-projection matrix is ​​very large (even exceeding the original KV size), requiring fusion with the model weights.

[0103] To address the problems existing in the aforementioned KV compression schemes, this application proposes a data compression method in one embodiment. Specifically, in the data compression method of this embodiment, the data matrix of the data to be compressed is decomposed into two matrices, a large matrix and a small matrix, wherein the large matrix has the same matrix size as the data matrix of the data to be compressed. Data quantization is performed on the large matrix to reduce the data volume, thereby achieving the effect of data compression.

[0104] The data compression method according to the embodiments of this application can achieve high compression ratios for data matrices while ensuring high precision. The data compression method according to the embodiments of this application can improve the compression effect of data compression.

[0105] Figure 3 The diagram shown is a flowchart of a data compression method according to an embodiment of this application.

[0106] In one embodiment, the data to be compressed is matrix T1 (the fifth matrix), and the size of matrix T1 can be represented as [the third value, the fourth value]. The bit width of matrix T1 is the first bit width, for example, 16 bits.

[0107] S is the third value, and D is the fourth value.

[0108] Electronic devices perform Figure 3 The following process is shown to achieve data compression of matrix T1.

[0109] S300, obtain the matrix T1 to be compressed.

[0110] Specifically, in one embodiment, the third value is set to S, the fourth value is set to D, and the size of matrix T1 can be represented as [S, D].

[0111] S301, based on the given first value, perform shape transformation on matrix T1 to obtain matrix T2 (first matrix), the size of matrix T2 is [first value, second value].

[0112] Specifically, in one embodiment, S301 is implemented based on a shape transformation (reshape) operation. In S301, only the element arrangement in matrix T1 is changed, without affecting the precision of each element. Therefore, the bit width of matrix T2 is the same as that of matrix T1, which is the first bit width, for example, 16 bits.

[0113] Specifically, in one embodiment, the first value is a given hyperparameter, which is much smaller than the product of the third and fourth values. For example, in one embodiment, S = D = 1024, K = 32.

[0114] Specifically, in one embodiment, a first value is set to K, where K is much smaller than S*D (K < 0.05). <S*D)。

[0115] Based on the dimensions of matrix T1 [S, D], we can obtain the dimensions of matrix T2 as [K, S*D / K]. S*D / K is the second value.

[0116] S302, perform matrix decomposition on matrix T2 to obtain matrix TS1 (second matrix) and matrix TL1 (third matrix). The bit width of matrix TS1 and matrix TL1 is the same as the first bit width, for example, 16 bits.

[0117] Specifically, in S302, the size of matrix TS1 is [first value, first value]([K, K]).

[0118] In S302, the first value (K) determines the numerical values ​​of the two matrices TS1 and TL1. The larger the first value (K), the smaller the numerical range of matrix TL1.

[0119] In S302, the size of matrix TL1 is the same as that of matrix T2, which is [first value, second value]([K, S*D / K]). That is:

[0120] T2 [K,SD / K] =TS1 [K,K] *TL1 [K,SD / K] . (Formula 2)

[0121] This application does not impose specific restrictions on the matrix decomposition method used in S302 for matrix T1. Those skilled in the art can adopt a suitable matrix decomposition method according to actual application requirements.

[0122] For example, in one embodiment, Singular Value Decomposition (SVD) is used to perform matrix decomposition.

[0123] Following S302, taking S=D=1024 and K=32 as an example, the values ​​of matrix T1, matrix TS1 and matrix TL1 are statistically analyzed, and their distribution is shown in Table 3 below.

[0124] Table 3

[0125] Percentage (Ratio) Maximum (Max) Min Matrix T1 100％ 5.9055 -6.339 TS1 0.10％ 6.0625 -5.9671 TL1 99.9％ 0.864 -0.864

[0126] As can be seen from Table 3, after matrix decomposition, the small matrix TS1, which occupies 0.1% of the values, has a data distribution comparable to the original matrix T1, while the large matrix TL1, which occupies 99.9% of the parameters, has its value distribution greatly compressed to a very small range (such as -0.864 to 0.864 in the table). This allows the large matrix TL1 to be quantized and compressed with lower bit depth, thereby greatly reducing the data volume of the matrix TL.

[0127] Therefore, in order to reduce the amount of data in matrix T1 and compress matrix T1, S310 is executed after S302.

[0128] S310, based on the second bit width, quantize matrix TL1 and obtain the quantization result (first quantization result) of matrix TL1, wherein the second bit width (e.g., 4 bits) is lower than the first bit width.

[0129] Specifically, the first quantization result can be any data format. For example, the first quantization result is matrix TL2 (the fourth matrix), where the size of matrix TL2 is the same as that of matrix TL1, the bit width of matrix TL2 is smaller than that of matrix TL1, and the amount of data in matrix TL2 is smaller than that in matrix TL1.

[0130] Specifically, the larger the first value (K), the smaller the numerical range of matrix TL1, which is more beneficial for quantization in S310.

[0131] This application does not impose specific limitations on the quantization algorithm of the quantization matrix TL1 in S310. Those skilled in the art can select a suitable quantization algorithm according to actual needs (e.g., according to compression accuracy requirements) to reduce the data volume of matrix TL.

[0132] Specifically, in one embodiment, in S310, matrix TL1 is quantized to low bit levels (e.g., 8 bits / 4 bits / 2 bits). Furthermore, in application scenarios where matrix T1 is KV, a KV-specific quantization method can be used to quantize matrix TL1 to low bit levels.

[0133] In one embodiment, after S302, matrix TS1 is not processed, and after S310, matrix TL2 and matrix TS1 are used as the data compression result of matrix T1.

[0134] Figure 4 The diagram shown is a data flow diagram of a data compression method according to an embodiment of this application.

[0135] like Figure 4 As shown, the matrix T1[S, D] to be compressed has a width of D and a height of S. The bit width of matrix T1 is 16 bits.

[0136] After matrix shape transformation, matrix T1[S, D] becomes matrix T2[K, S*D / K]. Matrix T2[K, S*D / K] has a width of S*D / K and a height of K. The bit width of matrix T2 is 16 bits.

[0137] Perform matrix decomposition on matrix T2[K,S*D / K] to obtain matrices TS1[K,K] and TL1[K,S*D / K]. Matrix TS1 has a width of / K and a height of K. The bit width of matrix TS1 is 16 bits. Matrix TL1 has a width of S*D / K and a height of K. The bit width of matrix TL1 is 16 bits.

[0138] Quantize matrix TL1 using N bits (N is less than 16, for example, N is 4) to obtain matrix TL2[K, S*D / K]. Matrix TL2 has a width of S*D / K and a height of K. The bit width of matrix TL2 is N bits.

[0139] Matrix TS1[K,K](16bits) and matrix TL2[K,S*D / K](Nbits) are the data compression results of matrix T1[S,D](16bits).

[0140] According to one embodiment of the method in this application, only the larger of the two matrices is compressed, thereby ensuring the accuracy of the decompressed data while achieving data compression.

[0141] In another embodiment, after S302, the following is also performed:

[0142] The quantization result (second quantization result) of matrix TS1 is obtained by quantizing matrix TS1 with a third bit width (e.g., 12 bits) that is lower than the first bit width (the bit width of matrix TL1) and higher than the second bit width (the quantization result of matrix TL1 (the first quantization result)).

[0143] Specifically, the second quantization result can be any data format. For example, the second quantization result is matrix TS2, where the size of matrix TS2 is the same as that of matrix TS1, the bit width of matrix TS2 is smaller than that of matrix TS1, and the amount of data in matrix TS2 is smaller than that in matrix TS1.

[0144] Matrix TS2, where the data size of matrix TS2 is smaller than that of matrix TS1. Matrix TL2 and matrix TS2 are used as the data compression result of matrix T1.

[0145] This application does not impose specific limitations on the quantization algorithm of the quantization matrix TS. Those skilled in the art can select a suitable quantization algorithm according to actual needs (e.g., according to compression accuracy requirements) to reduce the data volume of the matrix TS.

[0146] Specifically, in one embodiment, the matrix TS is quantized to a high bit level (e.g., 12 bits / 8 bits). Furthermore, in application scenarios where matrix T1 is KV, a quantization method for KV can be used to quantize the matrix TS to a high bit level.

[0147] Figure 5 The diagram shown is a data flow diagram of a data compression method according to an embodiment of this application.

[0148] like Figure 5 As shown, the matrix T1[S, D] to be compressed has a width of D and a height of S. The bit width of matrix T1 is 16 bits.

[0149] After matrix shape transformation, matrix T1[S, D] becomes matrix T2[K, S*D / K]. Matrix T2[K, S*D / K] has a width of S*D / K and a height of K. The bit width of matrix T2 is 16 bits.

[0150] Perform matrix decomposition on matrix T2[K,S*D / K] to obtain matrices TS1[K,K] and TL1[K,S*D / K]. Matrix TS1 has a width of / K and a height of K. The bit width of matrix TS1 is 16 bits. Matrix TL1 has a width of S*D / K and a height of K. The bit width of matrix TL1 is 16 bits.

[0151] Quantize matrix TL1 using N bits (N is less than 16, for example, N is 4) to obtain matrix TL2[K, S*D / K]. Matrix TL2 has a width of S*D / K and a height of K. The bit width of matrix TL2 is N bits.

[0152] Quantize matrix TS1 using Mbits (M is less than 16 but greater than N, for example, M is 12) to obtain matrix TS2[K,K]. Matrix TS2 has a width of K and a height of K. The bit width of matrix TS2 is Mbits.

[0153] Matrix TS2[K,K](Mbits) and matrix TL2[K,S*D / K](Nbits) are the data compression results of matrix T1[S,D](16bits).

[0154] According to a method of one embodiment of this application, two matrices are compressed, thereby increasing the compression ratio.

[0155] In another application scenario, given a first value (K), if the matrix T4 to be compressed satisfies the size [first value, second value]([K, S*D / K]), then S301 can be skipped. T4 is treated as T2 in S301, and the steps following S301 are executed to compress T4.

[0156] Furthermore, in one embodiment, instead of setting a K value, an F value is set based on the size of the matrix to be compressed, with the numerical value F serving as K, thus skipping step S301. For example, if the matrix to be compressed is T5, and the size of T5 is [A, B], then F = A. Using T5 as T2 in step S301, and the numerical value F as K, the steps following step S301 are executed to compress T5.

[0157] Figure 6 The diagram shown is a flowchart of a data compression method according to an embodiment of this application.

[0158] In one embodiment, the data to be compressed is matrix T5, ​​and the size of matrix T5 can be represented as [A, B].

[0159] Electronic devices perform Figure 6 The following process is shown to achieve data compression of matrix T5.

[0160] S600, obtain the matrix T5 to be compressed. The size of matrix T5 can be represented as [A, B].

[0161] S601, set K = A.

[0162] S602, perform matrix decomposition on matrix T5 to obtain matrix TS5 and matrix TL5. The size of matrix TS5 is [K, K], and the size of matrix TL5 is the same as that of matrix T5. The size of matrix TL5 is [K, AB / K], where AB / K = B.

[0163] S610 quantizes matrix TL5 based on a bit width lower than that of matrix TL5, and obtains the quantization result of matrix TL5.

[0164] In one embodiment, after S602, matrix TS1 is not processed, and after S610, the quantization result of matrix TL5 and matrix TS5 are used as the data compression result of matrix T5.

[0165] In another embodiment, after S602, the following steps are also performed: quantizing matrix TS5 and obtaining the quantization result of matrix TS5. The quantization result of matrix TS5 and the quantization result of matrix TL5 are used as the data compression result of matrix T5.

[0166] Furthermore, in some application scenarios, compressed data needs to be decompressed before use. Therefore, in addition to the data compression method proposed in the embodiments of this application, an embodiment of this application also proposes a decompression method.

[0167] In one embodiment, the compressed data includes a third quantization result.

[0168] The third quantization result is the quantization result generated based on the fifth bit width and the quantization matrix TL4 (the seventh matrix).

[0169] The third quantization result can be matrix TL3 (the ninth matrix), with dimensions [the fifth value, the sixth value]. The bit width of matrix TL4 is the fifth bit width.

[0170] The third quantification result can be referenced. Figure 3 The first quantization result of the illustrated embodiment. Matrix TL3 can be referenced. Figure 3 The matrix TL2 shown in the example.

[0171] The dimensions of matrix TL4 are [the fifth value, the sixth value]. The bit width of matrix TL4 is the fourth bit width.

[0172] Matrix TL4 can be referenced Figure 3 The matrix TL1 shown in the example.

[0173] The fifth value can be referenced. Figure 3 The first and sixth values ​​shown in the embodiment can be referred to. Figure 3 The second value shown in the embodiment, the size of matrix TL4 and matrix TL3 can be expressed as [K, S*D / K].

[0174] The fifth bit width is smaller than the fourth bit width. The fifth bit width can be referenced... Figure 3 The second bit width shown in the embodiment is, for example, 4 bits. The fourth bit width can be referenced... Figure 3 The first bit width shown in the embodiment is, for example, 16 bits.

[0175] Optionally, in one embodiment, the compressed data further includes matrix TS3 (sixth matrix), which can be referenced... Figure 3 , 4 The example shows matrix TS1. The size of matrix TS3 is [fifth value, fifth value]([K,K]). The bit width of matrix TS3 is the fourth bit width, for example, 16 bits.

[0176] Figure 7 The diagram shown is a flowchart of a decompression method according to an embodiment of this application.

[0177] Electronic devices perform Figure 7 The following process is shown to decompress compressed data and obtain the original matrix data.

[0178] S700, obtain matrix TS3 and the third quantization result.

[0179] S710, dequantize the third quantization result to obtain matrix TL4.

[0180] S720: Based on matrix TS3 and matrix TL4, perform matrix reconstruction to generate matrix T3 (the eighth matrix).

[0181] Specifically, in S720, matrix TS3 is multiplied by matrix TL4 to achieve matrix reconstruction.

[0182] Matrix T3 is the matrix data obtained after decompression. Matrix T3 can be referenced... Figure 3 The example shows matrix T1 or matrix T2. Matrix T3 has a bit width of the fourth bit width and a size of [third value, fourth value] ([S, D]) or [fifth value, sixth value] ([K, S*D / K]).

[0183] Furthermore, in some application scenarios, computing devices support the direct use of low-bit N-bit matrices. For this application scenario, in one embodiment, S710 is skipped, the third quantization result is not dequantized, and the matrix TS3 and the third quantization result are directly used for data calculation after S720.

[0184] Alternatively, in another embodiment, the compressed data may also include a fourth quantization result.

[0185] The fourth quantization result is the quantization result generated based on the sixth bit width and the quantization matrix TS3 (sixth matrix).

[0186] The fourth quantization result can be matrix TS4. The size of matrix TS4 is [fifth value, fifth value]. The bit width of matrix TS4 is the sixth bit width.

[0187] The fourth quantification result can be referenced. Figure 3 The second quantization result of the illustrated embodiment. Matrix TS4 can be referenced. Figure 3 The matrix TS2 shown in the example.

[0188] The sixth bit width can be referenced from the third bit width, for example, 12 bits.

[0189] Figure 8 The diagram shown is a flowchart of a decompression method according to an embodiment of this application.

[0190] Electronic devices perform Figure 8 The following process is shown to decompress compressed data and obtain the original matrix data.

[0191] S800, obtain the fourth quantization result and the third quantization result.

[0192] S811, dequantize the third quantization result to obtain matrix TL4.

[0193] S812, dequantize the fourth quantization result to obtain TS3.

[0194] S820: Based on matrices TS3 and TL4, perform matrix reconstruction to generate matrix T3 (the eighth matrix). (Refer to S720)

[0195] Furthermore, in some application scenarios, computing devices support the direct use of low-bit N-bit or M-bit matrices. For this application scenario, in one embodiment, S811 and S812 are skipped, the third quantization result and the fourth quantization result are not dequantized, and the third quantization result and the fourth quantization result are directly used for data calculation after S820.

[0196] Furthermore, refer to Figure 7 The decompression process shown in this application includes a dequantization process (S710) and a matrix reconstruction process (S720) in a data compression method of one embodiment of the application. In implementing the dequantization process (S710) and the matrix reconstruction process (S720), data saving and data reading operations are inevitably required.

[0197] Figure 9 The diagram shown is a flowchart of a decompression method according to an embodiment of this application.

[0198] Reference Figure 7 The decompression process shown, combined with data storage and data retrieval, is executed by the electronic device. Figure 9 The following process is shown to decompress compressed data and obtain the original matrix data.

[0199] S901, read matrix TL3 from memory 900.

[0200] S902, based on the dequantization operator, dequantizes matrix TL3 to obtain matrix TL4.

[0201] In the embodiments described in this application, an operator can be understood as an executable computation function. The operator performs calculations based on the input and outputs the calculation results. Different operators correspond to different computational logic.

[0202] For example, in S902, the dequantization operator can be represented by multiplying the input matrix with a preset dequantization matrix to obtain the output matrix. That is, multiplying matrix TL3 with the preset dequantization matrix to obtain matrix TL4.

[0203] S903, save matrix TL4 to memory 900.

[0204] S904, read matrix TL4 from memory 900.

[0205] S905, read matrix TS3 from memory 900.

[0206] S906, based on the matrix reconstruction operator, generates matrix T3 from matrix TL4 and matrix TS3.

[0207] The matrix reconstruction operator can be represented as multiplying two input matrices to obtain an output matrix. That is, multiplying matrix TL4 and matrix TS3 yields matrix T3.

[0208] exist Figure 9 In the process of the illustrated embodiment, there are memory read and write operations between S902 and S906. That is, the calculation result of S902 is first written to memory 900, and the calculation result of S902 needs to be read from memory 900 before S906 is calculated.

[0209] In order to reduce data storage and data retrieval steps, in one embodiment of this application, the dequantization operator used in S902 and the reconstruction operator used in S906 are merged into a large operator, that is, the dequantization (S902) and reconstruction (S906) of matrix TL3 are realized in one calculation.

[0210] Specifically, the dequantization operator and the matrix reconstruction operator undergo mathematically equivalent transformations and are merged into a single large operator.

[0211] Figure 10 The diagram shown is an operator fusion schematic diagram according to an embodiment of this application.

[0212] like Figure 10 As shown, the input to the dequantization operator (S902) is TL3 read from the memory area, and the output is TL4. The output TL4 of the dequantization operator (S902) is saved to memory. The input to the reconstruction operator is TL4 and TS3 read from memory.

[0213] exist Figure 10 The process shown requires saving and reading TL4 once.

[0214] The core of both the dequantization operator (S902) and the matrix reconstruction operator (S906) is matrix multiplication. Specifically, the dequantization operator is the multiplication of the dequantization matrix and the input matrix (TL3); the matrix reconstruction operator is the multiplication of two input matrices (TL4 and TS3).

[0215] A new fusion operator can be generated by pre-memorizing and fusing these two multiplication matrices to generate a new multiplication matrix. Specifically, the fusion operator is manifested as multiplying the first input matrix (TL3) by the inverse quantization matrix and then multiplying it by the second input matrix (TS3).

[0216] The input to the fusion operator is TL3 and TS3 read from the storage area. Based on the fusion operator, it is not necessary to save or read TL4.

[0217] Figure 11 The diagram shown is a flowchart of a decompression method according to an embodiment of this application.

[0218] Reference Figure 7 The decompression process shown, combined with data storage and data retrieval, is executed by the electronic device. Figure 11 The following process is shown to decompress compressed data and obtain the original matrix data.

[0219] S1101, Read matrix TL3 from memory 1100.

[0220] S1102, Read matrix TS3 from memory 1100.

[0221] S1103, based on the fusion operator, generates matrix T3 from matrix TL3 and matrix TS3. The fusion operator is a fusion of the inverse quantization operator and the reconstruction operator.

[0222] Specifically, the fusion operator can be represented as multiplying the first input matrix by a preset inverse quantization matrix, and then multiplying by the second input matrix to obtain the output matrix. That is, multiplying matrix TL3 by the preset inverse quantization matrix, and then multiplying by matrix TS3, yields matrix T3.

[0223] According to the method in the embodiments of this application, the dequantization operator and the matrix reconstruction operator are mathematically equivalent and merged into a large operator, thereby reducing the number of memory read and write operations and improving the operator execution efficiency.

[0224] Furthermore, the data compression and decompression methods proposed in this application embodiment can be applied to large-scale transformer model training scenarios to compress and decompress key-value pairs.

[0225] Specifically, in application scenarios where the data to be compressed is KV, refer to Figure 3 In the illustrated embodiment, in S302, for the trained large transformer model, matrix T1 represents the key-value pairs (KV) within a certain block. In the dimensions [S, D] of matrix T1, S is the sequence input length, and D is the hidden layer dimension.

[0226] In one embodiment, the data format (bit width) of the KV storage is 16 bits.

[0227] Figure 12 The figure shown is a statistical chart of the decomposition results of KV matrix decomposition according to an embodiment of the data compression method of this application.

[0228] For the application scenarios of large Transformer models, using KV as matrix T1 to execute S302, the distribution of the K values (Keys in Layer16) in the layer16 network layer is statistically analyzed, and the statistical results are as Figure 12 shown. The large matrix TL has a strong density (i.e., a large number of parameters), but the range of values is relatively narrow; on the contrary, the small matrix TS has a weak density (i.e., a small number of parameters), but the range of values is relatively wide.

[0229] Therefore, for KV, after executing S302, S310 can be executed to achieve data compression.

[0230] According to the data compression method of the embodiments of the present application, data compression can be performed on KV as an object. On the premise that the accuracy degradation is not obvious, KV occupies a small storage overhead, thereby reducing the overall inference latency.

[0231] Specifically, Figure 13 shown is the KV calculation flowchart according to an embodiment of the present application.

[0232] The electronic device executes the following process as Figure 13 shown to implement KV calculation.

[0233] S1300, receive the KV matrix with the original [S, D] size and the original bit width of 16 bits (refer to matrix T1) as input.

[0234] S1301, KV shape transformation (refer to S301): Given the hyperparameter K (K << S * D), perform shape transformation on the KV matrix to obtain a KV matrix with the size of [K, S * D / K] (refer to matrix T2), and the bit width of the KV matrix after shape transformation is the original 16 bits.

[0235] S1310, KV matrix decomposition (refer to S302): Using matrix decomposition methods such as SVD, decompose the KV matrix with the size of [K, S * D / K] after shape transformation to obtain a small matrix SKV (size of [K, K]) (refer to matrix TS1) and a large matrix LKV (size of [K, S * D / K]) (refer to matrix TL1), that is:

[0236] KV [K,SD / K] = SKV [K,K] * LKV [K,SD / K] . (Formula 3)

[0237] The bit widths of the small matrix SKV and the large matrix LKV are the original 16 bits.

[0238] S1320, Small Matrix SKV Storage: Store the decomposed small matrix SKV[K,K] in the video memory according to the original 16 bits or high bit width (such as 12 bits / 8 bits).

[0239] S1321, Small matrix SKV read: Before subsequent KV calculation, read the stored small matrix SKV[K,K] (refer to matrix TS1 or TS3) from the video memory.

[0240] S1330, Low-bit quantization of large matrix LKV (refer to S310): The large matrix LKV is quantized to low bit (Nbits) using the KV quantization method (the quantization result is the first quantization result or the third quantization result, matrix TL2 or TL3). For example, a suitable quantization algorithm is selected according to the actual accuracy requirements to perform low-bit quantization such as Nbits (8bits / 4bits / 2bits).

[0241] S1331, Large Matrix LKV Storage: Store the quantized and compressed N-bit (8-bit / 4-bit / 2-bit, etc.) large matrix LKV (refer to the first or third quantization result, matrix TL2 or TL3) into the video memory. Since the bit width is reduced to N / 16 of the original, the corresponding write (storage) latency is also reduced to N / 16 of the original.

[0242] S1332, Large Matrix LKV Read: Reads the compressed N-bit large matrix LKV (refer to the first or third quantization result, matrix TL2 or TL3) from the video memory for subsequent KV operations. Similar to the large matrix LKV storage in S1331, this part of the latency is also reduced to N / 16 of the original.

[0243] S1340, Large Matrix LKV Dequantization (Refer to S710): Based on the dequantization operator (Refer to S902), the compressed N-bit large matrix LKV read from S1332 (refer to the first or third quantization result, matrix TL2 or TL3) is dequantized to generate the original fp16 (16-bit) large matrix LKV (refer to TL1 or TL4) for subsequent KV calculations.

[0244] Furthermore, in some application scenarios, the computing device supports direct KV calculation using low-bit N bits of KV. For this application scenario, in one embodiment, S1340 is skipped.

[0245] When the computing device does not support direct KV calculation using low-bit N-bit KV values—for example, pure 8-bit calculation has limited precision, or 4-bit / 2-bit calculation lacks corresponding hardware support—it is necessary to follow the... Figure 13The process shown will execute S1340.

[0246] S1341, KV Reconstruction (Refer to S720): Based on the matrix reconstruction operator (Refer to S906), it receives the small matrix SKV (16 bits) of size [K,K] read in S1321 (refer to matrix TS1 or TS3), and the large matrix LKV (16 bits) of size [K,S*D / K] generated after dequantization in S1340 (refer to TL1 or TL4) as input. Through matrix multiplication, it is restored to the KV matrix used for KV calculation (size [S,D] or [K,S*D / K], bit width 16 bits) (refer to T3, T1, or T2), that is:

[0247]

[0248] from Figure 13 As can be seen from the large matrix branch on the right, the latency of KV storage (S1331) and KV reading (S1332) is reduced to N / 16 of the original, while the latency of KV dequantization (S1340) and KV reconstruction (S1341) is increased. However, the latency of S1331 and S1332 is significantly reduced overall, while S1340 and S1341 are slightly increased, resulting in a decrease in the total latency.

[0249] S1350, KV Calculation: Perform the corresponding KV calculation on the reconstructed KV matrix (refer to T3, T1, or T2) to generate the KV calculation result.

[0250] Specifically, in S1350, the reconstructed KV matrix is ​​calculated based on the key value calculation operator. The key value calculation operator is used to calculate key values ​​based on the input key values ​​and generate key value calculation results.

[0251] Table 4 shows the compression accuracy and compression rate statistics for KV data compression.

[0252] Table 4

[0253]

[0254]

[0255] As shown in Table 4, the original 16-bit (FP16) data size is 46.7MB.

[0256] Under the activations-only setting, according to the method (DecoQuant) of this application embodiment, compression based on 4-bit quantization (16-4) (4-bit quantization of the large matrix LKV(TL), no quantization of TS, using the original fp16 precision) can achieve a data volume of 11.7MB, thus achieving a maximum compression ratio of 4x. Furthermore, after 4-bit quantization (16-4) compression, according to the method of this application embodiment, compression precisions of LLaMA-7B (88.1), LLaMA-13B (88.9), OPT-1.3B (73.6), OPT-6.7B (80.9), and an average of (82.9) can be achieved. It is higher than the compression accuracy achieved by RTN-based compression, which is LLaMA-7B (86.0), LLaMA-13B (88.1), OPT-1.3B (71.7), OPT-6.7B (80.6), and average (81.6).

[0257] Under the weights and activations settings, the method (DecoQuant) according to the embodiments of this application, based on 4-bit quantization (4-4) compression (4-bit quantization of matrix TL, no quantization of TS, using the original fp16 precision), can achieve a data volume of 11.7MB, thus achieving a maximum compression ratio of 4x. Furthermore, after 4-bit quantization (4-4) compression, the method according to the embodiments of this application can achieve compression accuracies of LLaMA-7B (88.4), LLaMA-13B (88.5), OPT-1.3B (70.8), OPT-6.7B (79.1), and an average of (81.7). This is higher than the compression accuracies achieved by compression based on RTN and SmoothQuant.

[0258] The data compression method according to the embodiments of this application has the maximum compression ratio for KV while maintaining the accuracy of KV compression.

[0259] Furthermore, regarding S1340 and S1341, in one embodiment, after S1340 is executed, the execution result of S1340 is saved to memory (e.g., video memory). Before executing S1341, the execution result of S1340 is read from memory (e.g., video memory).

[0260] Figure 14 The diagram shown is a flowchart of KV calculation according to an embodiment of this application.

[0261] Electronic devices perform such Figure 14 The following process is shown to perform KV calculation.

[0262] S1400 receives a KV matrix with the original [S, D] size and a bit width of 16 bits (reference matrix T1) as input.

[0263] S1401, KV shape transformation (reference S1301): Given a hyperparameter K (K << S * D), perform a shape transformation on the KV matrix to obtain a KV matrix with a size of [K, S * D / K] (reference matrix T2). The bit width of the KV matrix after the shape transformation is 16 bits.

[0264] S1410, KV matrix decomposition (reference S1310): Decompose the KV matrix with a size of [K, S * D / K] after the shape transformation to obtain a small matrix SKV (size [K, K]) (reference matrix TS1) and a large matrix LKV (size [K, S * D / K]) (reference matrix TL1). The bit width of the small matrix SKV and the large matrix LKV is 16 bits.

[0265] S1420, storage of the small matrix SKV (reference S1320): Store the decomposed small matrix SKV [K, K] (reference matrix TS1) in the video memory according to the original 16 bits or a high - bit quantization bit width (such as 12 bits / 8 bits).

[0266] S1421, reading of the small matrix SKV (reference S1321): Before subsequent KV calculations, read the stored small matrix SKV [K, K] (reference matrix TS1 or TS3) from the video memory.

[0267] S1430, low - bit quantization of the large matrix LKV (reference S1330): Use the KV quantization method to perform low - bit (N bits) quantization on the large matrix LKV (the quantization result is the first quantization result or the third quantization result, matrix TL2 or TL3).

[0268] S1431, storage of the large matrix LKV (reference S1331): Store the quantized and compressed N - bit large matrix LKV (reference the first quantization result or the third quantization result, matrix TL2 or TL3) in the video memory.

[0269] S1432, reading of the large matrix LKV (reference S1332): Read the compressed N - bit large matrix LKV (reference the first quantization result or the third quantization result, matrix TL2 or TL3) from the video memory to participate in subsequent KV operations.

[0270] S1440, Large Matrix LKV Dequantization (Refer to S1340): Perform dequantization on the compressed N-bit large matrix LKV read from S1432 (refer to the first or third quantization result, matrix TL2 or TL3) to generate the original fp16 (16-bit) large matrix LKV (refer to TL1 or TL4).

[0271] S1441 saves the dequantization result of S1440 (a large 16-bit matrix LKV) (refer to TL1 or TL4) to the video memory.

[0272] S1442 reads the dequantization result (a large 16-bit matrix LKV) stored in the video memory (refer to TL1 or TL4).

[0273] S1443, KV Reconstruction (Refer to S1341): Receive the small matrix SKV (16 bits) of size [K,K] read from S1421 (refer to matrix TS1 or TS3), and the dequantized result read from S1442 (large matrix LKV (16 bits) of size [K,S*D / K]) (refer to TL1 or TL4) as input. Through matrix multiplication, reconstruct the KV matrix (size [S,D] or [K,S*D / K], bit width 16 bits) for KV calculation (refer to T3, T1, or T2).

[0274] S1450, KV Calculation (Refer to S1350): Perform the corresponding KV calculation on the reconstructed KV matrix (Refer to T3, T1, or T2) to generate the KV calculation results.

[0275] like Figure 14 As shown, after the KV dequantization operation (S1440) and before the KV reconstruction operation (S1443), there needs to be a write (S1441) and read (S1442) video memory operation. The size of the read and write video memory is the original KV capacity (i.e., S*D size) and the KV value bit width (i.e., 16 bits). This part of the delay greatly exceeds the KV calculation delay.

[0276] To reduce KV calculation latency, in one embodiment, refer to... Figure 10 As shown, the dequantization operator of S1340 (S1440) and the rectangle reconstruction operator of S1341 (S1443) are merged into a single fusion operator (the first fusion operator), realizing S1340 (S1440) and S1341 (S1443) in a single calculation. This eliminates the need to write the KV dequantization result back to video memory, thus avoiding the latency of reading and writing video memory in a single operation.

[0277] Figure 15 The diagram shown is a flowchart of KV calculation according to an embodiment of this application.

[0278] The electronic device executes the following process as shown Figure 15 below to implement KV calculation.

[0279] S1500, receive the KV matrix with the original [S, D] size and a bit width of the original 16 bits (reference matrix T1) as input.

[0280] S1501, KV shape transformation (reference S1301): Given the hyperparameter K (K << S * D), perform a shape transformation on the KV matrix to obtain a KV matrix with a size of [K, S * D / K] (reference matrix T2). The bit width of the KV matrix after shape transformation is the original 16 bits.

[0281] S1510, KV matrix decomposition (reference S1310): Decompose the KV matrix with a size of [K, S * D / K] after shape transformation to obtain a small matrix SKV (size [K, K]) (reference matrix TS1) and a large matrix LKV (size [K, S * D / K]) (reference matrix TL1). The bit width of the small matrix SKV and the large matrix LKV is the original 16 bits.

[0282] S1520, small matrix SKV storage (reference S1320): Store the decomposed small matrix SKV [K, K] (reference matrix TS1) in the video memory according to the original 16 bits or a high - bits quantization bit width (such as 12 bits / 8 bits).

[0283] S1521, small matrix SKV reading (reference S1321): Before subsequent KV calculation, read the stored small matrix SKV [K, K] (reference matrix TS1 or TS3) from the video memory.

[0284] S1530, low - bit quantization of the large matrix LKV (reference S1330): Use the KV quantization method to perform low - bit (N bits) quantization on the large matrix LKV (the quantization result is the first quantization result or the third quantization result, matrix TL2 or TL3).

[0285] S1531, large matrix LKV storage (reference S1331): Store the quantized and compressed N - bit large matrix LKV (reference the first quantization result or the third quantization result, matrix TL2 or TL3) in the video memory.

[0286] S1532, large matrix LKV reading (reference S1332): Read the compressed N - bit large matrix LKV (reference the first quantization result or the third quantization result, matrix TL2 or TL3) from the video memory to participate in subsequent KV operations.

[0287] S1540, Large Matrix LKV Dequantization + KV Reconstruction: Based on the first fusion operator, according to the compressed N-bit large matrix LKV read in S1532 (refer to the first quantization result or the third quantization result, matrix TL2 or TL3) and the small matrix SKV[K,K] read in S1521 (refer to matrix TS1 or TS3), a KV matrix (size [S,D] or [K,S*D / K], bit width 16 bits) for KV calculation is generated (refer to T3 or T1 or T2).

[0288] The first fusion operator is a fusion of the inverse quantization operator (refer to S902) and the matrix reconstruction operator (refer to S906).

[0289] The dequantization operator is used to dequantize the compressed N-bit large matrix LKV read by S1532 (refer to the first or third quantization result, matrix TL2 or TL3) based on the fourth bit width (16 bits) to generate the original fp16 (16 bits) large matrix LKV (refer to TL1 or TL4).

[0290] The matrix reconstruction operator is used to reconstruct the matrix based on the original fp16 (16-bit) large matrix LKV (refer to TL1 or TL4) and the small matrix SKV[K,K] read by S1521 (refer to matrix TS1 or TS3), generating a KV matrix (size [S,D] or [K,S*D / K], bit width 16 bits) for KV calculation (refer to T3, T1 or T2).

[0291] S1550, KV Calculation (refer to S1350): Perform the corresponding KV calculation on the reconstructed KV matrix (refer to T3, T1, or T2) to generate the KV calculation results.

[0292] Figure 16 The figure shows the memory consumption and input / output latency statistics for data compression and decompression of KV.

[0293] like Figure 16 As shown, compared with the scheme (KV) without KV matrix decomposition and compression, the KV compression method (Compress-KV) according to the embodiments of this application can achieve a significant reduction in memory cost.

[0294] Furthermore, compared to the input-output (IO) of a scheme that does not perform KV matrix decomposition and compression, the input-output (IO) of the KV compression method according to the embodiments of this application can achieve a significant reduction in input-output latency.

[0295] Further, in another embodiment, the KV dequantization operator of S1340 (S1440), the KV reconstruction operator of S1341 (S1443), and the key value calculation operator of S1350 (S1450) are fused into a fused operator (the second fused operator) to implement S1340 (S1440), S1341 (S1443), and S1350 (S1450) in one calculation.

[0296] Figure 17 The figure shows a KV calculation flowchart according to an embodiment of the present application.

[0297] The electronic device executes the following process as Figure 17 shown to implement KV calculation.

[0298] S1700, receive a KV matrix (reference matrix T1) with the original [S, D] size and the original bit width of 16 bits as input.

[0299] S1701, KV shape transformation (refer to S1301): Given a hyperparameter K (K << S * D), perform a shape transformation on the KV matrix to obtain a KV matrix with the size of [K, S * D / K] (reference matrix T2). The bit width of the shape-transformed KV matrix is the original 16 bits.

[0300] S1710, KV matrix decomposition (refer to S1310): Decompose the KV matrix with the size of [K, S * D / K] after shape transformation to obtain a small matrix SKV (size [K, K]) (reference matrix TS1) and a large matrix LKV (size [K, S * D / K]) (reference matrix TL1). The bit widths of the small matrix SKV and the large matrix LKV are the original 16 bits.

[0301] S1720, small matrix SKV storage (refer to S1320): Store the decomposed small matrix SKV [K, K] (reference matrix TS1) in the video memory according to the original 16 bits or a high-bit quantization bit width (such as 12 bits / 8 bits).

[0302] S1721, small matrix SKV reading (refer to S1321): Before subsequent KV calculations, read the stored small matrix SKV [K, K] (reference matrix TS1 or TS3) from the video memory.

[0303] S1730, low-bit quantization of the large matrix LKV (refer to S1330): Use the KV quantization method to perform low-bit (N bits) quantization on the large matrix LKV (the quantization result is the first quantization result or the third quantization result, matrix TL2 or TL3).

[0304] S1731, Large Matrix LKV Storage (Refer to S1331): Store the quantized and compressed N-bit large matrix LKV (refer to the first or third quantization result, matrix TL2 or TL3) into the video memory.

[0305] S1732, Large Matrix LKV Read (refer to S1332): Read the compressed N-bit large matrix LKV (refer to the first or third quantization result, matrix TL2 or TL3) from the video memory for use in subsequent KV operations.

[0306] S1740, Large matrix LKV dequantization + KV reconstruction + KV calculation: Based on the second fusion operator, the KV calculation result is generated according to the compressed Nbits large matrix LKV read in S1532 (refer to the first quantization result or the third quantization result, matrix TL2 or TL3) and the small matrix SKV[K,K] read in S1521 (refer to matrix TS1 or TS3). (The KV calculation result refers to the KV calculation result in S1350.)

[0307] The second fusion operator is a fusion of the inverse quantization operator (refer to S902), the matrix reconstruction operator (refer to S906), and the key value calculation operator (refer to S1350).

[0308] The dequantization operator is used to dequantize the compressed N-bit large matrix LKV read by S1732 (refer to the first or third quantization result, matrix TL2 or TL3) based on the fourth bit width (16 bits) to generate the original fp16 (16 bits) large matrix LKV (refer to TL1 or TL4).

[0309] The matrix reconstruction operator is used to reconstruct the matrix based on the original fp16 (16-bit) large matrix LKV (refer to TL1 or TL4) and the small matrix SKV[K,K] read by S1721 (refer to matrix TS1 or TS3), generating a KV matrix (size [S,D] or [K,S*D / K], bit width 16 bits) for KV calculation (refer to T3, T1 or T2).

[0310] The key value calculation operator is used to calculate key values ​​based on the input key values ​​(KV matrix (size is [S, D] or [K, S*D / K], bit width 16 bits) (refer to T3 or T1 or T2)) and generate key value calculation results.

[0311] In the description of the embodiments of this application, for the sake of convenience, the device is described by dividing it into various modules according to its functions. The division of each module is only a logical functional division. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0312] Specifically, the apparatus proposed in this application can be fully or partially integrated onto a single physical entity (e.g., a GPU or other type of processor), or it can be physically separated. These modules can be implemented entirely in software via processing element calls; entirely in hardware; or some modules can be implemented in software via processing element calls, while others are implemented in hardware. For example, the detection module can be a separate processing element or integrated into a chip in an electronic device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together or implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0313] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0314] An embodiment of this application also proposes an electronic device. This electronic device is used to execute the method flow or part of the method flow described in the embodiments of this application.

[0315] Figure 18 This is a schematic diagram of an electronic device structure according to an embodiment of this application.

[0316] like Figure 18 As shown, the electronic device 2500 includes a memory 2502 for storing computer program instructions and a processor 2501 for executing the program instructions. When the computer program instructions are executed by the processor 2501, the electronic device 2500 is triggered to perform the steps of the method described in the embodiments of this application.

[0317] Specifically, in one embodiment of this application, the aforementioned one or more computer programs are stored in the aforementioned memory 2502. The aforementioned one or more computer programs include instructions that, when executed by the aforementioned electronic device 2500, cause the aforementioned electronic device 2500 to perform the method steps described in the embodiments of this application.

[0318] It is understood that the structural description of the electronic device 2500 in this application does not constitute a specific limitation on the electronic device 2500. In other embodiments of this application, the electronic device 2500 may include other components besides the processor 2501 and the memory 2502.

[0319] The processor 2501 may be an on-chip device (SOC) that may include a central processing unit (CPU) and may further include other types of processors.

[0320] The processor 2501 may include, for example, a CPU, DSP, microcontroller, or digital signal processor, and may also include a GPU, embedded neural network processing units (NPUs), and image signal processors (ISPs). The processor may also include necessary hardware accelerators or logic processing hardware circuitry, such as an ASIC, or one or more integrated circuits for controlling the execution of the program in this application. Furthermore, the processor may have the function of operating one or more software programs, which may be stored in a storage medium.

[0321] Processor 2501 may include one or more processing units. For example, a processor may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, electronic device 2500 may also include one or more processors 2501. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0322] In some embodiments, the processor 2501 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. The USB interface is a USB standard-compliant interface, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface can be used to connect a charger to charge the electronic device, and can also be used for data transfer between the electronic device and peripheral devices.

[0323] Electronic device 2500 may also include an external memory interface for connecting an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device. The external memory card communicates with processor 2501 through the external memory interface to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0324] The memory 2502 may include a code storage area and a data storage area. The code storage area may store the operating system. The data storage area may store data created during the use of the electronic device 2500. Furthermore, the memory 2502 may include high-speed random access memory, and may also include non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (UFS), etc.

[0325] The memory 2502 may be a read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), or other types of dynamic storage devices capable of storing information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices. Alternatively, it may be any computer-readable medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer.

[0326] Processor 2501 and memory 2502 can be combined into a single processing device, but more commonly they are separate components.

[0327] An embodiment of this application also provides an electronic chip. This electronic chip is used to execute the method flow or part of the method flow described in the embodiments of this application.

[0328] Specifically, the electronic chip includes a processor for executing program instructions. When the computer program instructions are executed by the processor, the electronic chip is triggered to perform the steps described in the embodiments of this application. The processor of the electronic chip can refer to the processor of the above-described electronic device.

[0329] Optionally, the devices, apparatuses, and modules described in the embodiments of this application may be implemented by computer chips or physical entities, or by products with certain functions.

[0330] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0331] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0332] Specifically, one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute the method provided in the embodiment of this application.

[0333] An embodiment of this application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the method provided in the embodiment of this application.

[0334] The embodiments described in this application are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0335] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0336] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0337] It should also be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of singular or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0338] In this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0339] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0340] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0341] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments of this application can be implemented using electronic hardware, computer software, or a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0342] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0343] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A data compression method, characterized in that, The method is applied to an electronic device, and the method includes: Decompose the first matrix to obtain the second and third matrices, wherein the size of the first matrix is ​​[first value, second value], the size of the second matrix is ​​[first value, first value], the size of the third matrix is ​​[first value, second value], and the bit width of the first matrix, the second matrix, and the third matrix is ​​the first bit width; Based on the second bit width, the third matrix is ​​quantized to generate a first quantization result, wherein the second bit width is lower than the first bit width.

2. The method according to claim 1, characterized in that, The method further includes: Save the second matrix and the first quantization result.

3. The method according to claim 1, characterized in that, The method further includes: Based on the third bit width, the second matrix is ​​quantized to generate a second quantization result, wherein the third bit width is lower than the first bit width and higher than the second bit width; Save the first quantization result and the second quantization result.

4. The method according to claim 1, characterized in that, The first quantization result is a fourth matrix, the size of which is [first value, second value], and the bit width of which is the second bit width.

5. The method according to any one of claims 1-4, characterized in that, The first value is a preset hyperparameter. Before decomposing the first matrix, the method further includes: Based on the first value, the fifth matrix is ​​transformed to generate the first matrix, wherein the bit width of the fifth matrix is ​​the first bit width, and the size of the fifth matrix is ​​[the third value, the fourth value].

6. The method according to claim 5, characterized in that, The first value is much smaller than the product of the third value and the fourth value.

7. The method according to claim 5 or 6, characterized in that, The fifth matrix represents the key values ​​that need to be saved to the key value cache, the third value represents the sequence input length of the key values, and the fourth value represents the hidden layer dimension.

8. The method according to claim 7, characterized in that, The first value is 32.

9. A decompression method, characterized in that, The method is applied to an electronic device, and the method includes: Obtain the third quantization result and the sixth matrix, wherein: the size of the sixth matrix is ​​[fifth value, fifth value], and the bit width of the sixth matrix is ​​the fourth bit width; the third quantization result is the quantization result generated by quantizing the seventh matrix based on the fifth bit width; the size of the seventh matrix is ​​[fifth value, sixth value], and the bit width of the seventh matrix is ​​the fourth bit width; the fifth bit width is smaller than the fourth bit width; Based on the fourth bit width, the third quantization result is dequantized to generate the seventh matrix; Based on the sixth matrix and the seventh matrix, a matrix reconstruction is performed to generate an eighth matrix, wherein the bit width of the eighth matrix is ​​the same as the fourth bit width.

10. The method according to claim 9, characterized in that, The third quantization result is the ninth matrix, the size of the ninth matrix is ​​[the fifth value, the sixth value], and the bit width of the ninth matrix is ​​the fifth bit width.

11. The method according to claim 9 or 10, characterized in that, The acquisition of the third quantization result and the sixth matrix includes: Obtain the fourth quantization result, wherein: the fourth quantization result is the quantization result generated by quantizing the sixth matrix based on the sixth bit width; the sixth bit width is smaller than the fourth bit width, and the sixth bit width is larger than the fifth bit width; Based on the fourth bit width, the fourth quantization result is dequantized to generate the sixth matrix.

12. The method according to any one of claims 9-11, characterized in that, The eighth matrix contains the key values ​​required for key value calculation. The size of the eighth matrix is ​​[seventh value, eighth value], where the seventh value is the sequence input length of the key values ​​and the eighth value is the hidden layer dimension.

13. The method according to claim 12, characterized in that, The fifth value is much smaller than the product of the seventh value and the eighth value.

14. The method according to claim 12, characterized in that, The fifth value is 32.

15. The method according to any one of claims 12-14, characterized in that, The method includes: Read the third quantization result and the sixth matrix from the memory; Based on the first fusion operator, and according to the third quantization result and the sixth matrix, the eighth matrix is ​​generated, wherein: The first fusion operator is a fusion of the inverse quantization operator and the matrix reconstruction operator; The dequantization operator is used to dequantize the third quantization result based on the fourth bit width to generate the seventh matrix; The matrix reconstruction operator is used to perform matrix reconstruction based on the sixth matrix and the seventh matrix to generate the eighth matrix.

16. The method according to claim 15, characterized in that, The inverse quantization operator is expressed as multiplying the inverse quantization matrix by the input matrix; the matrix reconstruction operator is expressed as multiplying two input matrices; the first fusion operator is expressed as multiplying one input matrix by the inverse quantization matrix and then multiplying by another input matrix.

17. The method according to any one of claims 12-14, characterized in that, The method further includes: Based on the eighth matrix, key values ​​are calculated, and key value calculation results are generated.

18. The method according to claim 17, characterized in that, The method includes: Read the third quantization result and the sixth matrix from the memory; Based on the second fusion operator, and according to the third quantization result and the sixth matrix, the key value calculation result is generated, wherein: The second fusion operator is a fusion of the dequantization operator, the matrix reconstruction operator, and the key value calculation operator; The dequantization operator is used to dequantize the third quantization result based on the fourth bit width to generate the seventh matrix; The matrix reconstruction operator is used to perform matrix reconstruction based on the sixth matrix and the seventh matrix to generate the eighth matrix; The key value calculation operator is used to perform key value calculation based on the input key value and generate key value calculation results.

19. An electronic device, characterized in that, The electronic device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method steps as described in any one of claims 1-8 or 9-18.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-8 or 9-18.

Images