A tensor data compression method, system and device for artificial intelligence processing

Abstract

The application belongs to the technical field of computer data processing, and particularly relates to a tensor data compression method, system and device for artificial intelligence processing. Random orthogonal transformation is applied to a high-dimensional vector to be compressed to obtain a transformed vector and convert the transformed vector into a polar coordinate form to obtain a module length component and a unit direction vector. The module length component is subjected to non-uniform scalar quantization to obtain a module length quantization index. Each dimension component of the unit direction vector is subjected to non-uniform scalar quantization to obtain a direction quantization index sequence. The module length quantization index and the direction quantization index sequence are used to reconstruct a first-stage approximate vector, and a residual vector of the transformed vector and the first-stage approximate vector is calculated. Each dimension element of the residual vector is subjected to sign quantization to generate a one-bit sign vector. A compressed data stream is output, and the compressed data stream comprises the module length quantization index, the direction quantization index sequence and the one-bit sign vector. The application greatly reduces storage and bandwidth overhead while ensuring inference accuracy.

Description

Technical Field

[0001] This invention belongs to the field of computer data processing technology, and specifically relates to a tensor data compression method, system and device for artificial intelligence processing. Background Technology

[0002] With the rapid increase in the scale of deep learning models, especially the widespread application of large language models and visual Transformers, the storage capacity and bandwidth requirements of intermediate tensor data during inference, such as the key and value caches in attention mechanisms, have become a bottleneck. For example, in autoregressive generation tasks, the KV cache generated by long sequences often exceeds the GPU memory capacity, severely limiting batch size and sequence length. Meanwhile, in edge computing scenarios, if multimedia data such as images and videos undergoes traditional compression encoding, such as JPEG's DCT transform or HEVC / H.265 transmission, AI applications must first fully decode and restore it to pixel-domain or tensor form before performing inference calculations. This "decoding-re-encoding" process introduces redundant transcoding latency and energy consumption, making it impossible to achieve "compress once, reuse multiple times."

[0003] Existing technologies such as DCT and H.265, traditional image and video compression algorithms, offer high compression rates and are mature systems. However, their drawbacks are also significant: these methods are not designed for tensor inner product operations, and the compressed bitstream cannot directly support the core operations required by AI models, such as vector inner product and matrix multiplication. If you want to directly use the compressed bitstream for inference, you must perform transcoding, which leads to low end-to-end efficiency.

[0004] In recent years, compression methods for AI tensors have emerged, such as scalar quantization and vector quantization to compress key-value caches. However, simple quantization introduces systematic biases related to inner product calculations, significantly impairing the accuracy of attention weights. Another type of method utilizes feature transformation and dimensionality reduction, but none have been able to simultaneously preserve the approximate unbiasedness of inner product operations and achieve data-independent online compression at extremely low bit counts. Therefore, there is a need for a tensor data compression method that can unify traditional multimedia and AI feature encoding, directly support inner product operations in the compression domain, and is hardware-friendly. To this end, this invention provides a tensor data compression method, system, and apparatus for artificial intelligence processing. Summary of the Invention

[0005] The purpose of this invention is to provide a tensor data compression method, system, and apparatus for artificial intelligence processing. The compressed bitstream can be directly used for high-precision inner product operations without complete decompression, thereby achieving "compression once, reuse many times" of compressed data, significantly reducing storage and bandwidth overhead, while ensuring inference accuracy.

[0006] The specific technical solution adopted by this invention is as follows: This invention relates to the field of computer technology, specifically to the fields of data compression, artificial intelligence acceleration and high-performance computing, and in particular to an online lossless compression method, dedicated processor and system for tensor data, which is suitable for neural network inference and especially for key-value caching compression in attention mechanisms; A tensor data compression method for artificial intelligence processing includes the following steps: Apply a random orthogonal transformation to the high-dimensional vector to be compressed to obtain the transformed vector; The transformation vector is converted into polar coordinates to obtain the magnitude component and the unit direction vector. The modulus component is non-uniformly scalar quantized based on a predetermined first quantization codebook to obtain a modulus quantization index; the dimensional components of the unit direction vector are non-uniformly scalar quantized based on a predetermined second quantization codebook to obtain a direction quantization index sequence. The first-stage approximate vector is reconstructed using the modulus quantization index and the direction quantization index sequence, and the residual vector between the transformed vector and the first-stage approximate vector is calculated. Each element of the residual vector is sign-quantized to generate a one-bit sign vector; Output a compressed data stream, the compressed data stream containing the modulus quantization index, the direction quantization index sequence, and the one-bit symbol vector.

[0007] Preferably, it further includes an inner product calculation step: Retrieve the query vector; The first-stage approximate vector is reconstructed based on the magnitude quantization index and the direction quantization index sequence; Calculate the projection vector of the query vector after the random orthogonal transformation; Calculate the inner product of the projection vector and the first-stage approximation vector as the preliminary inner product; A correction term is generated using the inner product of the projection vector and the one-bit symbol vector, along with a preset correction coefficient. The preliminary inner product is added to the correction term to obtain the corrected inner product estimate.

[0008] Preferably, the random orthogonal transformation is implemented by multiplying the fast Hadamard transformation with a random diagonal sign matrix, wherein the diagonal elements of the random diagonal sign matrix are independently and randomly taken from {+1, -1}.

[0009] Preferably, the first quantization codebook is pre-calculated based on the probability model of the kinematic distribution that the modulus component follows; the second quantization codebook is pre-calculated based on the probability distribution model of the high-dimensional spherical edge that each component of the unit direction vector follows, and the codebook is independent of the data to be compressed.

[0010] Preferably, the preset correction coefficient is set as the statistical expectation of the absolute value of each element of the residual vector, or determined through a calibration process that minimizes the inner product estimation error.

[0011] A tensor data compression system for artificial intelligence processing, used to execute the method, wherein the tensor data compression system acts as a tensor data compression processor; comprising: The random projection unit is used to perform random orthogonal transformations on the high-dimensional vector to be compressed, and obtain the transformed vector. A coordinate separation unit, connected to the random projection unit, is used to calculate the magnitude and unit direction vector of the transformation vector; The first-level quantization unit includes a magnitude quantizer and a direction quantizer, which are used to perform non-uniform scalar quantization on each component of the magnitude and the unit direction vector, respectively, output the magnitude quantization index and the direction quantization index sequence, and reconstruct the first-stage approximate vector; the magnitude quantizer and the direction quantizer are connected to a pre-stored codebook memory. The residual calculation unit is used to calculate the residual vector between the transformed vector and the first-stage approximation vector; A second-level quantization unit is used to design each element of the residual vector to generate a one-bit sign vector; The bitstream assembly unit is used to assemble the modulus quantization index, the direction quantization index sequence, and the one-bit symbol vector into a compressed bitstream.

[0012] Preferably, it further includes a decoding and inner product calculation unit, which is used to receive the query vector and the compressed bitstream, reconstruct the first-stage approximate vector according to the index, calculate the projection vector of the query vector after the random orthogonal transformation, and output the corrected inner product value using the projection vector, the one-bit symbol vector and the preset correction coefficient.

[0013] Preferably, the random projection unit includes a fast Hadamard transform circuit and a sign-flip register, and the codebook memory is a read-only memory that stores a quantized codebook pre-calculated based on a theoretical probability distribution.

[0014] A tensor data compression device for artificial intelligence processing includes a host processor, a tensor data compression system, and video memory; During neural network inference, the tensor data compression processor compresses the generated intermediate tensor data online and stores the resulting compressed bitstream in the video memory. When it is necessary to calculate the inner product related to the compressed tensor, the decoding and inner product calculation unit in the tensor data compression processor or the corresponding calculation unit in the system directly performs the inner product operation for deviation correction based on the compressed bitstream and the query vector.

[0015] Preferably, the intermediate tensor data is a key cache and a value cache generated by the attention mechanism, and the inner product operation is used to calculate the attention weights.

[0016] The technical effects achieved by this invention are as follows: This invention achieves efficient compression of tensor vectors through a combination of random orthogonal transformation, polar coordinate separation, and two-stage hybrid quantization. In particular, the sign information of the second-stage residual is preserved and used for unbiased correction in inner product calculations, ensuring that even at extremely low bit rates, inner product operations such as attention weight calculations remain nearly unbiased, minimizing the impact on model inference accuracy. This scheme is data distribution-independent, and the codebook can be pre-trained completely offline, making it suitable for ASIC or FPGA implementations, achieving high throughput and low power consumption. More importantly, this encoding method is applicable to both traditional multimedia data such as image patches and AI feature tensors, unifying the encoding format. This allows the same compressed bitstream to be used for efficient storage and transmission of images / videos, and also directly consumed by AI models, truly achieving "compression once, reuse many times," eliminating transcoding steps and significantly saving computational resources and energy. Attached Figure Description

[0017] Figure 1 This is a flowchart of a tensor data compression method for artificial intelligence processing according to the present invention; Figure 2 This is a system block diagram of a tensor data compression system for artificial intelligence processing according to the present invention. Detailed Implementation

[0018] To make the objectives and advantages of this invention clearer, the invention will be specifically described below with reference to embodiments. It should be understood that the following text is merely used to describe one or more specific embodiments of the invention and does not strictly limit the scope of protection specifically claimed by the invention.

[0019] Example 1: like Figures 1-2 As shown, a tensor data compression method for artificial intelligence processing includes: Step 1: First-stage quantization. Apply a random orthogonal transformation to the original high-dimensional vector to convert its components into approximately independent and identically distributed components; represent the transformed vector in polar coordinates and separate its magnitude and direction components; based on a predetermined, data-independent quantization codebook, perform non-uniform scalar quantization on the magnitude and direction components respectively to obtain the first-stage quantization result and the first-level residual.

[0020] Step 2: Second-stage quantization. Project and sign quantization are performed on the first-level residual to generate a one-bit sign vector with the same dimension as the vector, which serves as the inner product bias correction information.

[0021] Step 3: Output compressed data stream. The compressed data stream includes the index of the first-stage quantization result and a one-bit symbol vector.

[0022] Furthermore, when obtaining the query vector, the method also includes decompression and inner product calculation steps: based on the first-stage approximation of the index-reconstructed vector, the inner product result calculated from the first-stage approximation is unbiasedly corrected using the query vector and a one-bit symbol vector to obtain the final inner product estimate.

[0023] Preferably, the stochastic orthogonal transformation employs a combination of fast Hadamard transform and random sign flipping to enhance the independence of each dimension with extremely low computational complexity. The quantization codebook is pre-calculated based on the theoretical probability distribution of the vector magnitude and direction components after the stochastic orthogonal transformation, ensuring that the quantizer is independent of the data distribution and achieving zero-latency online compression.

[0024] In actual operation, this invention is used for example: This embodiment provides a tensor data compression method for artificial intelligence processing, specifically including the following steps: S101: Apply a random orthogonal transformation to the original high-dimensional vector x∈R^d to obtain the transformed vector y.

[0025] In this embodiment, the random orthogonal matrix P is a combination of the Fast Hadamard Transform (FHT) and a random sign-flipping matrix, i.e., P = H·D, where H is the normalized Hadamard matrix and D is a diagonal matrix with diagonal elements independently and randomly drawn from {+1, -1}. The Hadamard Transform can be completed in O(d log d) complexity using a butterfly algorithm. For vectors whose shape is not a power of 2, zeros can be padded to powers of 2 first. The random sign flipping ensures that different data streams are statistically independent. This transformation makes the components of the transformed vector y approach an independent and identically distributed zero-mean Gaussian distribution (assuming that the components of the original vector are independent and have finite variance).

[0026] S102: Convert the transformation vector y to polar coordinates to obtain the magnitude r = ||y||2 and the unit direction vector u = y / r.

[0027] S103: Non-uniform scalar quantization based on a pre-generated quantization codebook.

[0028] Modular codebook C r Includes K r The centroid is designed to be derived by minimizing the mean square error on a theoretical distribution of modulus r (a chiral distribution with d degrees of freedom) using the Lloyd-Max algorithm. The modulus r is then quantized to obtain the index i. r And remodel the length r q =C r (i r ).

[0029] The direction vector u is a d-dimensional unit vector, and the marginal probability density functions of its components can be derived based on a high-dimensional spherical distribution: For a vector uniformly distributed on a d-dimensional unit sphere, any one of its components u k The density is proportional to (1-u k 2 ) (d-3) / 2 Based on this distribution, we also pre-calculate the distribution containing K. u scalar quantization codebook C with centroid u Each component of u is independently scalar quantized to obtain the index sequence i. u，1 , ..., i u，d And reconstruct the values ​​of each component u q，k =C u (i u，k ).

[0030] Calculate the first-stage approximation vector using the reconstructed magnitude and direction: , where u q =[u q，1 , ..., u q，d ] T .

[0031] Calculate the first-order residual vector: .

[0032] S104: Second-stage quantization. Each element e_k of the residual vector e is sign-quantized to generate a 1-bit sign vector s∈{+1, -1}. d , where s k =sign(e k This symbol information will be crucial for inner product bias correction.

[0033] S105: Output compressed data stream. The stream contains: modulus index i r (log2K) r (bits), d directional indices (d·log2K) u The total number of bits is log₂K (1 bit for +1, 0 bit for -1). r +d·log2K u +d.

[0034] When you need to perform inner product calculations using a compressed bitstream, perform the following steps: S106: Given a query vector q, this invention needs to calculate the inner product q of the original vector x and q. T x. Since the orthogonal matrix P satisfies P T P=I, and in this embodiment the Hadamard matrix is ​​symmetric, so x=P Ty, therefore the inner product is equivalent to (Pq). T First, calculate the projection q'=Pq of the query vector.

[0035] S107: Reconstruct the first-stage approximate vector based on the index .

[0036] S108: Calculate the preliminary inner product: .

[0037] S109: Calculate the correction term: Δ = α·(q') T s), where α is a preset correction coefficient.

[0038] S110: Obtain the corrected inner product estimate: v = v0 + Δ.

[0039] The principle for setting the correction coefficient α is to achieve unbiased estimation. Because... And s is known k =sign(e k Assuming that the residual components are independent and the mean of the first-stage quantization error is zero, then e k Given s k The conditional expectation under the following condition can be approximated as E[e k |s k ]≈β·s k , where β=E[|e k In practice, E[|e] can be statistically analyzed on a large number of samples during the calibration phase. k |] is used as the α value, or the optimal α is searched by minimizing the mean square error of the inner product using a small amount of calibration data for a specific network layer. This correction mechanism greatly eliminates the need to use only the α value. The systematic bias in calculating the inner product has almost no loss of weight accuracy in attention calculation.

[0040] like Figures 1-2 As shown, Example 2: A tensor data compression system for artificial intelligence processing, used to execute tensor data compression methods, the tensor data compression system as a tensor data compression processor, includes: The random projection unit employs a pipelined fast Hadamard transform kernel and a random symbol register to perform random orthogonal transformations.

[0041] The coordinate separation unit includes a square accumulator, a square root circuit, and a divider, which calculates the modulus and direction.

[0042] The first-level quantization unit consists of a modulus quantizer and a direction quantizer, both of which store the offline pre-trained codebook C through read-only memory (ROM). r and C u The lookup table quickly outputs the quantized index and reconstructed values.

[0043] The residual calculation unit is a set of subtractors that calculates the residual between the transformed vector and the reconstructed value in the first stage.

[0044] The second-level quantization unit is a set of comparators that extract the sign bit of each element of the residual and generate a bit vector.

[0045] The bit stream assembly unit concatenates the index bits and sign bits into an output bit stream.

[0046] In addition, the processor can be optionally equipped with a decoding and inner product calculation unit. This unit receives an external query vector and a compressed bitstream, and internally includes vector reconstruction logic, a correction coefficient register α, and a multiply-accumulator array for directly outputting the corrected inner product, bypassing the decompression of the complete tensor.

[0047] like Figures 1-2 As shown, Example 3: This embodiment of a tensor data compression device for artificial intelligence processing includes a host processor, a tensor data compression system, and off-chip video memory. When running a large language model, the new Key and Value tensors generated by each Transformer layer are immediately sent to the compression processor, and the compressed compact bitstream is stored in the KV buffer in the video memory. When performing self-attention calculation, the query vector, output from the matrix multiplication unit, is directly sent to the decoding and inner product calculation unit of the compression processor. This unit reads the compressed Key vector bitstream from the video memory and pipelines the corrected attention inner integral, without the attention calculation unit needing to access the decompressed complete tensor. This significantly reduces video memory usage and bandwidth requirements. The same bitstream can also be directly transmitted to a display or multimedia decoding module. If the multimedia module also integrates the same decompression inner product unit, it can directly perform AI tasks such as image retrieval based on feature similarity, achieving unified encoding reuse.

[0048] Practical application examples: Case 1: Long Text Generation from a Large Language Model. When generating a 32k context, a standard FP16-based KV cache requires approximately 4GB of GPU memory. Using the method of this invention, d=128, K... r =256、K u =16, compression ratio of about 6 times, and memory usage reduced to about 680MB. At the same time, after proper calibration of the correction coefficient α, the perplexity of the model decreased by no more than 0.2%, achieving a lossless inference experience.

[0049] Case Study 2: Intelligent Surveillance Video Analysis. The camera compresses the depth feature map of each frame block by block using this method, obtaining a bitstream that is then sent to the central server. The server directly calculates the inner product similarity between features in the compressed domain, completing target re-identification without the need for decoding and reconstructing pixels or full-precision feature maps. End-to-end latency is reduced by 45%, and bandwidth consumption is only 1 / 10 of traditional video transmission.

[0050] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

[0051] The above description is merely a preferred embodiment of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention. Structures, devices, and operating methods not specifically described or explained in this invention are implemented according to conventional methods in the art unless otherwise specified or limited.

Claims

1. A tensor data compression method for artificial intelligence processing, characterized in that: Includes the following steps: Apply a random orthogonal transformation to the high-dimensional vector to be compressed to obtain the transformed vector; The transformation vector is converted into polar coordinates to obtain the magnitude component and the unit direction vector. Based on a predetermined first quantization codebook, the modulus component is subjected to non-uniform scalar quantization to obtain the modulus quantization index. Based on a predetermined second quantization codebook, non-uniform scalar quantization is performed on each dimension component of the unit direction vector to obtain a direction quantization index sequence. The first-stage approximate vector is reconstructed using the modulus quantization index and the direction quantization index sequence, and the residual vector between the transformed vector and the first-stage approximate vector is calculated. Each element of the residual vector is sign-quantized to generate a one-bit sign vector; Output a compressed data stream, the compressed data stream containing the modulus quantization index, the direction quantization index sequence, and the one-bit symbol vector.

2. The method according to claim 1, characterized in that: It also includes the inner product calculation steps: Retrieve the query vector; The first-stage approximate vector is reconstructed based on the magnitude quantization index and the direction quantization index sequence; Calculate the projection vector of the query vector after the random orthogonal transformation; Calculate the inner product of the projection vector and the first-stage approximation vector as the preliminary inner product; A correction term is generated using the inner product of the projection vector and the one-bit symbol vector, along with a preset correction coefficient. The preliminary inner product is added to the correction term to obtain the corrected inner product estimate.

3. The method according to claim 1 or 2, characterized in that: The random orthogonal transformation is implemented by multiplying the fast Hadamard transformation with a random diagonal sign matrix, wherein the diagonal elements of the random diagonal sign matrix are independently and randomly selected from {+1, -1}.

4. The method according to claim 1, characterized in that: The first quantization codebook is pre-calculated based on the probability model of the kinematic distribution that the modulus component follows; the second quantization codebook is pre-calculated based on the probability distribution model of the high-dimensional spherical edge that each component of the unit direction vector follows, and the codebook is independent of the data to be compressed.

5. The method according to claim 2, characterized in that: The preset correction coefficient is set as the statistical expectation of the absolute value of each element of the residual vector, or determined through a calibration process that minimizes the inner product estimation error.

6. A tensor data compression system for artificial intelligence processing, characterized in that: For performing the method according to any one of claims 1-5, the tensor data compression system serves as a tensor data compression processor; comprising: The random projection unit is used to perform random orthogonal transformations on the high-dimensional vector to be compressed, and obtain the transformed vector. A coordinate separation unit, connected to the random projection unit, is used to calculate the magnitude and unit direction vector of the transformation vector; The first-level quantization unit includes a magnitude quantizer and a direction quantizer, which are used to perform non-uniform scalar quantization on each component of the magnitude and the unit direction vector, respectively, output the magnitude quantization index and the direction quantization index sequence, and reconstruct the first-stage approximate vector; the magnitude quantizer and the direction quantizer are connected to a pre-stored codebook memory. The residual calculation unit is used to calculate the residual vector between the transformed vector and the first-stage approximation vector; A second-level quantization unit is used to design each element of the residual vector to generate a one-bit sign vector; The bitstream assembly unit is used to assemble the modulus quantization index, the direction quantization index sequence, and the one-bit symbol vector into a compressed bitstream.

7. The system according to claim 6, characterized in that: It also includes a decoding and inner product calculation unit, which is used to receive the query vector and the compressed bitstream, reconstruct the first-stage approximate vector according to the index, calculate the projection vector of the query vector after the random orthogonal transformation, and output the corrected inner product value using the projection vector, the one-bit symbol vector and the preset correction coefficient.

8. The system according to claim 6, characterized in that: The random projection unit includes a fast Hadamard transform circuit and a sign flip register. The codebook memory is a read-only memory that stores a quantized codebook pre-calculated based on a theoretical probability distribution.

9. A tensor data compression device for artificial intelligence processing, characterized in that: Includes a host processor, a tensor data compression system as described in any one of claims 6 to 8, and video memory; During neural network inference, the tensor data compression processor compresses the generated intermediate tensor data online and stores the resulting compressed bitstream in the video memory. When it is necessary to calculate the inner product related to the compressed tensor, the decoding and inner product calculation unit in the tensor data compression processor or the corresponding calculation unit in the system directly performs the inner product operation for deviation correction based on the compressed bitstream and the query vector.

10. The apparatus according to claim 9, characterized in that: The intermediate tensor data consists of key and value caches generated by the attention mechanism, and the inner product operation is used to calculate the attention weights.

Images