Sliding barrier product quantification method and apparatus

Abstract

The application belongs to the technical field of information, and particularly relates to a sliding fence product quantization method and device. The method comprises the following steps: input N D-dimensional vectors are regarded as an N*D matrix and initial segmentation is performed to obtain a small value matrix, a large value matrix and an indication matrix; then, recursive segmentation is performed, the small value matrix and the large value matrix obtained after the recursive segmentation are combined into a new matrix, and a sliding window is used to divide a subspace; K-means clustering is performed in each subspace, and cluster centers are used as codebooks to encode all vectors in the subspace; a Count-min operation is used to obtain the final encoding of each element, and the quantization result is obtained after combination. The application realizes improvement and optimization of the product quantization algorithm, can effectively reduce memory overhead under the premise of ensuring accuracy, and can be widely applied to fields such as large language model weight quantization, vector database management, KV cache optimization, graph compression, image compression and other fields requiring efficient matrix quantization.

Description

Technical Field

[0001] This invention belongs to the field of information technology, and specifically relates to a sliding fence product quantization method and apparatus. Background Technology

[0002] In the development and application of Large Language Models (LLMs) such as ChatGPT, the storage and computational efficiency of matrices is a critical issue. These models have a huge number of parameters, and storing and operating on large matrices imposes a significant computational burden. Matrix quantization techniques record the numerical information of matrices in a more compact form, reducing storage requirements, and the original matrix can be recovered through dequantization during use. This process plays an important role in applications such as LLM weight quantization, key-value (KV) cache quantization, vector database management, graph compression, and image compression.

[0003] RTN quantization is a traditional matrix element quantization technique. Typically, given a matrix with N elements, the elements are represented as A1, A2, A3...A... N The RTN algorithm rounds each element A using a simple rounding operation. k This maps the matrix elements to the set of its nearest discrete values. The core operation in this process is rounding the matrix elements to a predefined quantization range, typically using formula A. k ≈Round(A k The matrix RTN is represented by the Round(·) function, which maps element values ​​to the nearest quantization interval. RTN is simple and efficient to compute, but the size of the quantization interval often depends on the range of values ​​in the matrix. When outliers exist, the quantization error can be large.

[0004] Vector quantization is a traditional vector compression technique. Typically, given N D-dimensional real vectors V1, V2, ..., V... N Vector quantization algorithm generates a quantizer, which converts V1, V2, ..., V... N A vector V k Mapping to codebook A codeword c(i(V)) where i takes values ​​in a finite set of indicators. k The function i(·) is called the encoder, and the function c(·) is called the decoder. Q(V) is also commonly used. k ) as c(i(V) k )) is short for V k An approximate estimate. Quantization distortion (or compression error) is defined as MSE(Q) = ∑ 1≤k≤N |V k -Q(V k )|2 The lower the quantization distortion, the higher the quantization accuracy of the quantization algorithm. Excellent quantization algorithms often have low quantization distortion.

[0005] K-means is a mainstream clustering algorithm. It divides N vectors into K clusters, assigning each vector to the nearest cluster center. By repeatedly performing the process of "taking the mean of the vectors within a cluster as the new cluster center - re-dividing the clusters," the total distance of each vector to the cluster center is gradually reduced. K-means can be viewed as a vector quantization algorithm, with a codebook of K vectors representing cluster centers, V1, V2, ..., V... N Each vector in the array is encoded into an integer between 1 and K. Figure 1 This is a schematic diagram illustrating the performance of the K-means algorithm.

[0006] Product quantization is also a type of vector quantization algorithm, and its process is as follows: Figure 2 As shown. For N D-dimensional real vectors V1, V2, ..., V... N The product quantization algorithm first divides each vector into M sub-vectors, each with a dimension of D / M. Therefore, for N D-dimensional real vectors V1, V2, ..., V... N The N subvectors corresponding to each dimension form a subspace of dimension D / M. After partitioning the subspace, the product quantization algorithm uses the K-means algorithm to cluster within each subspace to obtain the codebook and encoding. The codebook and encoding of each subspace are then merged to obtain the quantization results of these N D-dimensional real vectors. Assume V... k =(V k1 V k2 , ..., V kM The vector V is divided into M subvectors. k1 V k2 , ..., V kM (Each is D / M dimension), V k1 V k2 , ..., V kM The approximate estimates generated by the K-means algorithm are Q1(V) k1 ), Q2(V k2 ), ..., Q M (V kM ), then V k The approximate estimate under the product quantization algorithm is Q(V) k )=(Q1(V k1 ), Q2(V k2 ), ..., Q M (V kM )).

[0007] The main problems with existing product quantization algorithms are excessively long encoding and decoding times and large compression errors. Summary of the Invention

[0008] This invention mainly addresses the improvement and optimization of the product quantization algorithm, specifically by minimizing compression error while keeping the encoding and decoding time relatively short.

[0009] The technical solution adopted in this invention is as follows:

[0010] A sliding fence product quantization method includes the following steps:

[0011] Obtain the N D-dimensional input vectors;

[0012] Consider N D-dimensional vectors as an N×D matrix, and perform an initial segmentation on the N×D matrix; the initial segmentation includes: treating two adjacent elements in the matrix as a pair of elements, placing the element with the smaller value in each pair into a small-value matrix, placing the element with the larger value in a large-value matrix, and using an indicator variable to record the relative size relationship of each pair of elements, and storing the indicator variable in a first indicator matrix;

[0013] The initial small and large value matrices are recursively divided by continuing to compare and divide element pairs to generate new small, large, and first indicator matrices.

[0014] The small and large value matrices obtained after recursive partitioning are merged into a new N×D matrix, and the merged matrix is ​​divided into subspaces by a sliding window method.

[0015] K-means clustering is performed within each of the divided subspaces, and the cluster centers obtained from the clustering are used as codebooks to encode all vectors within the subspaces.

[0016] The final code of each element in the merged matrix is ​​obtained by the count-min operation. The final codes of each element are then merged to obtain the quantization result.

[0017] Furthermore, the initial segmentation also includes: if the position of any element in the original matrix is ​​(a, b), then its corresponding position in the small value matrix or the large value matrix is ​​(a, [b / 2]), and the indicator variable is stored in (a, [b / 2]) of the first indicator matrix, where [·] represents the floor function; traverse the element pairs of the entire matrix in order to generate the small value matrix, the large value matrix, and the first indicator matrix of size N×D / 2.

[0018] Furthermore, by specifying the number of recursive divisions, a tree-like division structure is formed. If the number of recursive divisions is l, then the total size of the first indicator matrix is ​​l×N×D / 2.

[0019] Furthermore, the method of dividing the merged matrix into subspaces by sliding windows includes: setting the window size and the distance of each slide, treating the vector within each sliding window as a subspace, mapping each element of the merged matrix to a different subspace to form a copy of that element, and having multiple copies of each element of the merged matrix.

[0020] Furthermore, the Count-min operation includes: for each element of the merged matrix, calculating the distance between the element and the cluster center of the subspace where each of its replicas is located, recording the subspace number of the nearest replica to the element at the corresponding position in the second exponential matrix, and using the code of the nearest replica to the element as the final code of the element.

[0021] An image compression method based on sliding fence product quantization includes the following steps:

[0022] The original image to be compressed is initially segmented; the initial segmentation includes: treating the image as an N×D pixel matrix, treating each pair of adjacent pixels as an element pair, storing the smaller value in the element pair into a small value matrix and the larger value into a large value matrix, and generating a first indicator matrix to record the relative size relationship of each pair of pixels.

[0023] Recursively partition the small-value matrix and the large-value matrix to form a new small-value matrix, a new large-value matrix, and a corresponding first indicator matrix;

[0024] The small and large value matrices obtained after recursive partitioning are merged into a new N×D matrix. The merged matrix is ​​then divided into subspaces using a sliding window approach, with the pixel vectors within each sliding window serving as a subspace. Multiple copies of each pixel are generated.

[0025] K-means clustering is performed within each of the divided subspaces, and the cluster centers obtained from the clustering are used as codebooks to encode all vectors within the subspaces.

[0026] The final code of each element in the merged matrix is ​​obtained by the count-min operation. The final codes of each element are merged to obtain the quantization result. The original image is recovered by dequantizing the quantization result, thereby achieving image compression.

[0027] A method for storing weight quantization of large language models based on sliding fence product quantization includes the following steps:

[0028] The initial segmentation of the weight matrix of the large language model includes: treating the weight matrix of the large language model as an N×D matrix, treating each pair of adjacent weights as an element pair, placing the smaller value in the element pair into the small value matrix and the larger value into the large value matrix, and generating a first indicator matrix to record the size relationship of each pair of weights.

[0029] Recursively partition the small-value matrix and the large-value matrix to generate new small-value matrices and large-value matrices, and generate the corresponding first indicator matrix;

[0030] The small and large value matrices obtained after recursive partitioning are merged into a new N×D matrix. The merged weight matrix is ​​divided into subspaces by sliding window method, with the weight vector in each sliding window as a subspace, and multiple copies of each weight are generated.

[0031] K-means clustering is performed within each of the divided subspaces, and the cluster centers obtained from the clustering are used as codebooks to encode all vectors within the subspaces.

[0032] The final encoding of each element in the merged matrix is ​​obtained by the count-min operation. The final encodings of each element are then merged to obtain the quantization result of the entire weight matrix, thereby realizing the weight quantization and storage of a large language model.

[0033] A sliding fence product quantization device, comprising:

[0034] The input module is used to obtain N D-dimensional vectors as input;

[0035] The initial segmentation module is used to treat N D-dimensional vectors as N×D matrices and perform initial segmentation on the N×D matrices. The initial segmentation includes: treating two adjacent elements in the matrix as a pair of elements, placing the element with the smaller value in each pair into a small-value matrix and the element with the larger value into a large-value matrix, and using an indicator variable to record the relative size relationship of each pair of elements, and storing the indicator variable in a first indicator matrix.

[0036] The recursive partitioning module is used to recursively partition the initially partitioned small and large value matrices. By continuing to compare and partition element pairs, new small value matrices, large value matrices, and a first indicator matrix are generated.

[0037] The subspace partitioning module is used to merge the small-value matrix and large-value matrix obtained after recursive partitioning into a new N×D matrix, and to partition the merged matrix into subspaces by sliding window.

[0038] The clustering module is used to perform K-means clustering in each of the partitioned subspaces, and uses the cluster centers obtained by clustering as a codebook to encode all vectors in the subspace;

[0039] The Count-min operation and merging module is used to obtain the final code of each element in the merged matrix through the Count-min operation, and then merge the final codes of each element to obtain the quantization result.

[0040] The beneficial effects of this invention are as follows:

[0041] This invention improves and optimizes the product quantization algorithm, minimizing compression errors while maintaining reasonable encoding and decoding times, and effectively reducing memory overhead while ensuring accuracy. This invention can be widely applied in fields requiring efficient matrix quantization, such as Large Language Model (LLM) weight quantization, vector database management, key-value cache optimization, graph compression, and image compression. Attached Figure Description

[0042] Figure 1 This is a schematic diagram illustrating the performance of the K-means algorithm.

[0043] Figure 2 This is a schematic diagram of the product quantization algorithm.

[0044] Figure 3 This is a schematic diagram of the sliding fence product quantization algorithm of the present invention. Detailed Implementation

[0045] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to specific embodiments and accompanying drawings.

[0046] This invention proposes an improved product quantization algorithm—the sliding fence product quantization algorithm. This algorithm first adjusts the local order of matrix elements, then divides the matrix into subspaces using a sliding window, performs K-means clustering in each subspace, and finally selects the most accurate quantized copy of each subvector as the final quantization result. In this invention, the "sliding fence" refers to isolating and grouping vector elements in the original matrix according to their relative size, i.e., the initial segmentation and recursive segmentation steps in the algorithm flow; "sliding" refers to dividing the subspace using a sliding window.

[0047] The flow of the sliding fence product quantization algorithm of the present invention is as follows: Figure 3 As shown. Assuming the input consists of N D-dimensional vectors, the core method of this invention follows this process:

[0048] (1) Initial Segmentation: Treat the N D-dimensional vectors as an N×D matrix. Consider any two adjacent elements of the matrix as a pair. Place the element with the smaller value into the small part and the element with the larger value into the big part. Use a 1-bit indicator variable to record the relative size relationship of this pair of elements. Let the position of any element in the original matrix be (a, b), then its corresponding position in the small or big part matrix is ​​(a, [b / 2]). The indicator variable is stored in (a, [b / 2]) of indicator matrix 1, where [·] represents the floor function. Iterate through the element pairs of the entire matrix in order to generate the small or big part matrix and indicator matrix 1 (Indicator Map) of size N×D / 2.

[0049] In this context, two adjacent elements of a matrix refer to two elements that are adjacent horizontally. Specifically, a matrix element is denoted as a. ij (i≤N, j≤D, i and j are positive integers), let a i，2k-1 and Consider them as a pair, and compare a. i，2k-1 and a i，2k The relative size.

[0050] (2) Recursive partitioning: For the initially partitioned small or large matrix, the element-pair comparison and partitioning in step (1) above can be repeated to generate new small, large, and indicator matrices. The number of recursive partitions can be specified to form... Figure 3 The tree-like segmentation structure is shown. The indicator matrix 1 of each segmentation level can be combined into an indicator matrix 1 of size N×D / 2. Assuming the number of recursive segmentations is l, the total size of the indicator matrix 1 is l×N×D / 2.

[0051] (3) Subspace partitioning using a sliding window: Merge the partitioned matrices into a single matrix, restoring the matrix size to N×D. Set the window size and the distance of each slide, and partition the merged matrix into subspaces using a sliding window method, treating each vector within a sliding window as a subspace. Thus, each element of the merged matrix is ​​mapped to a different subspace, and elements in different subspaces are called copies of that element; each element has multiple copies. For example... Figure 3 The window shown is 4 in size, and the sliding distance is 2.

[0052] (4) K-Means clustering: Perform K-means clustering in each subspace divided in the previous step, use the cluster center as the codebook, encode all vectors in the subspace, and obtain the code.

[0053] (5) Count-min operation and merging: For each element of the merged matrix in step (3), calculate the distance between the element and the cluster center of the subspace where each of its replicas resides. Record the subspace number of the nearest replica to the element at the corresponding position in the exponent matrix 2. Use the code of the nearest replica to the element as the final code of the element. This step is called the "Count-min" operation. Merge the codes of each element to obtain the final quantization result.

[0054] When the quantization result obtained by the method of the present invention needs to be recovered from the original matrix through dequantization in practical applications, for each position of the matrix, the corresponding subspace is first found according to the indicator matrix 2, and then the codebook vector of the subspace is queried according to its encoding to recover the value at that position, thus obtaining a preliminary recovered matrix. For the preliminary recovered matrix, the positional relationship of the matrix elements is recursively recovered according to the value of the indicator matrix 1.

[0055] This invention can be widely applied to fields requiring efficient matrix quantization, such as large language model (LLM) weight quantization, vector database management, key-value cache optimization, graph compression, and image compression.

[0056] For example, one embodiment of the present invention provides an image compression method based on sliding fence product quantization, comprising the following steps:

[0057] 1) Initial segmentation: The original image to be compressed is processed, treating the image as an N×D pixel matrix. Each pair of adjacent pixels is compared as a pair, the smaller value is stored in the small value matrix, and the larger value is stored in the large value matrix. At the same time, an indicator matrix 1 is generated to record the relative size relationship of each pair of pixels.

[0058] 2) Recursive Segmentation: The initial segmentation operation described above is repeated on the small and large value matrices to form new small and large value matrices and their corresponding indicator matrices. This process can be repeated recursively multiple times, thus forming a tree-like segmentation structure that gradually refines the information in different parts of the image. Finally, the segmentation results of each layer are merged into a total indicator matrix of size l×N×D / 2.

[0059] 3) Subspace partitioning using a sliding window: The recursively partitioned matrix is ​​restored to an N×D matrix. The size of the sliding window and the step size for each slide are set. The matrix is ​​divided into multiple subspaces using a sliding window operation. The pixel vector within each window forms a subspace for subsequent quantization. In this way, multiple copies will be generated at each pixel location for subsequent selection of the optimal encoding.

[0060] 4) K-Means clustering: Perform K-means clustering in each subspace divided in the previous step, using the cluster center as the codebook, and encode all vectors in the subspace to obtain the code.

[0061] 5) Count-min operation and merging: For each element of the merged matrix in step (3), calculate the distance between the element and the cluster center of the subspace containing each of its replicas. Record the subspace number of the nearest replica in the corresponding position in the exponent matrix 2. Use the code of the nearest replica as the final code of the element. This step is called the "Count-min" operation. Merge the codes of each element to obtain the final quantization result. By dequantizing the coded data, an approximate original image can be recovered, thereby achieving image compression.

[0062] For example, another embodiment of the present invention provides a method for storing the weight quantization of a large language model (LLM) based on sliding fence product quantization, comprising the following steps:

[0063] 1) Initial segmentation: Treat the weight matrix of the large-scale language model as an N×D matrix, treat each pair of adjacent weights as an element pair, and put the smaller value into the small value matrix and the larger value into the large value matrix by comparison, and generate indicator matrix 1 to record the size relationship.

[0064] 2) Recursive Segmentation: The small-value matrix and the large-value matrix are recursively segmented to generate multiple layers of small-value and large-value matrices, and corresponding indicator matrix 1 is generated. The number of recursive segmentation layers is specified by the user, forming a progressively refined tree-like weighted representation.

[0065] 3) Divide the subspace by sliding window: merge the recursively divided matrices back to N×D size, and divide the weight matrix by sliding window by setting the sliding window size and sliding step, to obtain multiple subspaces. The weight vector in each sliding window is a subspace, thereby quantizing the weight matrix with finer granularity.

[0066] 4) K-Means clustering: Perform K-means clustering in each subspace divided in the previous step, using the cluster center as the codebook, and encode all vectors in the subspace to obtain the code.

[0067] 5) Count-min operation and merging: For each element of the merged matrix in step (3), calculate the distance between the element and the cluster center of the subspace where each of its replicas resides. Record the subspace number of the nearest replica to the element at the corresponding position in the exponent matrix 2. Use the code of the nearest replica to the element as the final code of the element. This step is called the "Count-min" operation. Merge the codes of each element to obtain the final quantization result of the entire weight matrix, thereby realizing the weight quantization storage of a large-scale language model.

[0068] Another embodiment of the present invention provides a sliding fence product quantization device, comprising:

[0069] The input module is used to obtain N D-dimensional vectors as input;

[0070] The initial segmentation module is used to treat N D-dimensional vectors as N×D matrices and perform initial segmentation on the N×D matrices. The initial segmentation includes: treating two adjacent elements in the matrix as a pair of elements, placing the element with the smaller value in each pair into a small-value matrix and the element with the larger value into a large-value matrix, and using an indicator variable to record the relative size relationship of each pair of elements, and storing the indicator variable in a first indicator matrix.

[0071] The recursive partitioning module is used to recursively partition the initially partitioned small and large value matrices. By continuing to compare and partition element pairs, new small value matrices, large value matrices, and a first indicator matrix are generated.

[0072] The subspace partitioning module is used to merge the small-value matrix and large-value matrix obtained after recursive partitioning into a new N×D matrix, and to partition the merged matrix into subspaces by sliding window.

[0073] The clustering module is used to perform K-means clustering in each of the partitioned subspaces, and uses the cluster centers obtained by clustering as a codebook to encode all vectors in the subspace;

[0074] The Count-min operation and merging module is used to obtain the final code of each element in the merged matrix through the Count-min operation, and then merge the final codes of each element to obtain the quantization result.

[0075] The above division of modules is merely illustrative. In practical applications, the functions described above can be assigned to different functional modules as needed to complete all or part of the functions described in the aforementioned method. The specific working process of each module can be found in the corresponding process in the aforementioned method embodiments, and will not be repeated here.

[0076] Another embodiment of the present invention provides a computer device (computer, server, smartphone, etc.) including a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the method of the present invention.

[0077] Another embodiment of the present invention provides a computer-readable storage medium (such as ROM / RAM, disk, optical disk) storing a computer program that, when executed by a computer, implements the various steps of the method of the present invention.

[0078] The specific embodiments of the present invention disclosed above are intended to help understand the content of the present invention and to implement it accordingly. Those skilled in the art will understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention. The present invention should not be limited to the content disclosed in the embodiments of this specification; the scope of protection of the present invention is defined by the claims.

Claims

1. An image compression method based on sliding fence product quantization, characterized in that, Includes the following steps: The original image to be compressed is initially segmented; the initial segmentation includes: treating the image as an N×D pixel matrix, treating each pair of adjacent pixels as an element pair, storing the smaller value in the element pair into a small value matrix and the larger value into a large value matrix, and generating a first indicator matrix to record the relative size relationship of each pair of pixels. Recursively partition the small value matrix and the large value matrix to form a new small value matrix, a large value matrix and the corresponding first indicator matrix; The small and large value matrices obtained after recursive partitioning are merged into a new N×D matrix. The merged matrix is ​​then divided into subspaces using a sliding window approach, with the pixel vectors within each sliding window serving as a subspace. Multiple copies of each pixel are generated. K-means clustering is performed within each of the divided subspaces, and the cluster centers obtained from the clustering are used as codebooks to encode all vectors within the subspaces. The final code of each element in the merged matrix is ​​obtained by the count-min operation. The final codes of each element are merged to obtain the quantization result. The original image is recovered by dequantizing the quantization result, thereby achieving image compression.

2. The method according to claim 1, characterized in that, The initial segmentation further includes: if the position of any element in the original matrix is ​​(a, b), then its corresponding position in the small value matrix or the large value matrix is ​​(a, [b / 2]), and the indicator variable is stored in (a, [b / 2]) of the first indicator matrix, where [·] represents the floor function; traverse the element pairs of the entire matrix in order to generate the small value matrix, the large value matrix, and the first indicator matrix of size N×D / 2.

3. The method according to claim 1, characterized in that, By specifying the number of recursive partitions, a tree-like partitioning structure is formed. If the number of recursive partitions is l, then the total size of the first indicator matrix is ​​l×N×D / 2.

4. The method according to claim 1, characterized in that, The method of dividing the merged matrix into subspaces by sliding windows includes: setting the window size and the distance of each slide, treating the vector within each sliding window as a subspace, mapping each element of the merged matrix to different subspaces to form a copy of that element, and having multiple copies of each element of the merged matrix.

5. The method according to claim 4, characterized in that, The Count-min operation includes: for each element of the merged matrix, calculating the distance between the element and the cluster center of the subspace where each of its replicas is located, recording the subspace number of the nearest replica to the element at the corresponding position in the second exponential matrix, and using the code of the nearest replica to the element as the final code of the element.

6. A computer device, characterized in that, It includes a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the method of any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer, implements the method according to any one of claims 1 to 5.

Images