A high-precision quantization method for non-uniformly distributed data

Abstract

This invention discloses a high-precision quantization method for non-uniformly distributed data, aiming to solve the problems of significantly increased reconstruction error and wasted codebook resources faced by traditional uniform quantization algorithms when dealing with complex data distributions. This scheme first uses a dynamic programming algorithm to locally homogenize the data, achieving optimal segmentation by minimizing the sum of squared weighted ranges. Then, a particle swarm optimization algorithm is introduced to iteratively optimize the endpoints of the quantization interval, constructing an efficient codebook with the goal of minimizing the mean square error (MSE). Experimental results show that this method reduces the quantization error by 93.52% and 98.05% on 128-dimensional and 420-dimensional datasets, respectively, compared to traditional methods, while maintaining hardware compatibility. It is suitable for scenarios with stringent requirements for compression efficiency and real-time performance, such as edge computing nodes and AI chips.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence and machine learning technology, specifically relating to the theory and technology of quantization compression of non-uniformly distributed data. Background Technology

[0002] With the rapid development of edge computing, the Internet of Things, and deep learning, the demand for efficient storage and transmission of high-dimensional non-uniform data (such as gradient features of images, word vectors in natural language processing, and spatiotemporal sequences collected by sensors) is becoming increasingly urgent. Traditional uniform quantization methods cannot adapt to the non-uniform distribution characteristics of data, and therefore cannot meet the real-time processing requirements of edge devices (such as smart cameras and wearable sensors) and AI chips (such as NPUs and TPUs) in terms of compression efficiency and accuracy.

[0003] Traditional uniform quantization assumes that the data follows a uniform distribution and uses equally spaced quantization intervals. However, real-world data often exhibits complex distribution characteristics. When the data has a multimodal distribution, the histogram of image features typically has multiple peaks. Uniform quantization can lead to excessively large quantization intervals in peak regions, significantly increasing reconstruction errors. Furthermore, in skewed data distributions, the distribution of high-frequency and low-frequency words in text word vectors is highly unbalanced. Uniform quantization can allocate too many codebooks to low-frequency word regions, resulting in wasted resources. Experiments show that in non-uniform distribution scenarios, uniform quantization has higher reconstruction errors, and approximately 65% ​​of the codebooks correspond to less than 5% of the sample size. Summary of the Invention

[0004] To address the aforementioned issues, this invention proposes a high-precision quantization method for non-uniformly distributed data. Through a two-layer optimization mechanism of "dynamic programming segmentation + swarm intelligence optimization," a balance between quantization error and codebook efficiency is achieved.

[0005] A high-precision quantization method for non-uniformly distributed data includes the following steps:

[0006] S1. Data Preprocessing and Segmentation: For high-dimensional non-uniform data, the feature values ​​of each dimension are first sorted in ascending order and deduplicated to generate an ordered sequence of unique values. At the same time, the frequency of occurrence of each value is counted. Based on the dynamic programming algorithm, the sequence of unique values ​​is divided into... Mutual exclusive sub-intervals The objective of partitioning is to minimize the weighted sum of squares over the range, i.e.:

[0007] ,

[0008] in, Subinterval The number of samples;

[0009] The state transition equation of the dynamic programming algorithm is:

[0010] ,

[0011] in, Indicates the preceding Each unique value is divided into The minimum weighted sum of squares over each interval; For the sorted number A unique value; For the front The cumulative frequency of each value;

[0012] S2. Quantization parameter initialization: For each sub-interval The endpoints of the quantization interval are initialized using the Min-Max normalization method. ,in , Quantization step size is defined as:

[0013] ,

[0014] In the formula, This is the initial quantization step size. The number of quantization levels for each sub-interval is determined by the quantization bit width.

[0015] S3. Particle Swarm Optimization for Intelligent Parameter Optimization: With minimizing the mean square error (MSE) as the optimization objective, the particle swarm optimization algorithm is used to optimize the parameters at the endpoints of the quantization interval. Perform a global search; the position parameter of the particle swarm is defined as the center point of the interval. and width To decouple the optimization of interval position and range, where This is the lower bound of the quantization interval. The upper bound of the quantization interval is defined; the search space expands around the initial solution, specifically as follows:

[0016] ,

[0017] in, The initial interval width, The expansion factor is used to balance the search range and accuracy; the initial particles are generated through grid sampling, with the grid density set to [value missing]. To ensure a uniform distribution of the initial solution, The mesh particle width is given; the fitness function is defined as the reciprocal of MSE, guiding particles to iterate towards the region of minimum error; the particle velocity update formula is:

[0018] ,

[0019] in, Let be the velocity of the particle at time t. Let the mass at time t be the inertia weight. It decays linearly with the number of iterations; As a learning factor, A random number in the interval [0,1]. and These represent the optimal position for the individual particle and the global optimal position, respectively.

[0020] S4. Data Quantization: Based on the optimized parameters ,in To quantize the step size, the data within the sub-interval is... Mapped to the discrete codebook space, the specific quantization formula is as follows:

[0021] ,

[0022] in, For data Corresponding quantization code, This is for rounding operations.

[0023] Furthermore, the parameter search space for particle swarm optimization is expanded by an expansion factor. Dynamic adjustment; when When the value is small, the search focuses on the vicinity of the initial solution, performing local fine-tuning; when... When the size is large, the search range is expanded for global exploration; a grid sampling strategy using initial particles is used to ensure uniform coverage of the solution space and avoid getting trapped in local optima.

[0024] Furthermore, the particle swarm velocity update formula adopts an inertial weight decay strategy. In the early stage, a larger inertial weight is used to encourage particles to search globally, and in the later stage, a smaller weight is used to guide particles to focus on local optima.

[0025] The beneficial effects of this invention include:

[0026] 1) Improved accuracy: On the Scale Invariant Feature Transform dataset (SIFT dataset), the mean squared error of this invention is significantly reduced, showing a significant error reduction of 93.52% compared to uniform quantization; on the audio processing dataset (MSong dataset), the mean squared error is significantly reduced, and the quantization error is reduced by 98.05%;

[0027] 2) Efficiency optimization: Particle swarm optimization supports GPU parallel acceleration to meet real-time quantization requirements;

[0028] 3) Hardware compatibility: The quantization process only relies on addition, subtraction, multiplication, division and integer operations, and can run on microcontrollers without floating-point units, reducing power consumption by more than 70% compared to floating-point operations. Attached Figure Description

[0029] Figure 1 This is a comparison chart of the quantization error results of the present invention and existing methods. Detailed Implementation

[0030] The technical solution of the present invention will be described in detail below with reference to specific embodiments.

[0031] A high-precision quantization method for non-uniformly distributed data includes the following steps:

[0032] S1. Data Preprocessing and Segmentation: For high-dimensional non-uniform data, the feature values ​​of each dimension are first sorted in ascending order and deduplicated to generate an ordered sequence of unique values. At the same time, the frequency of occurrence of each value is counted. Based on the dynamic programming algorithm, the sequence of unique values ​​is divided into... Mutual exclusive sub-intervals The objective of partitioning is to minimize the weighted sum of squares over the range, i.e.:

[0033] ,

[0034] in, Subinterval The number of samples;

[0035] The state transition equation of the dynamic programming algorithm is:

[0036] ,

[0037] in, Indicates the preceding Each unique value is divided into The minimum weighted sum of squares over each interval; For the sorted number A unique value; For the front The cumulative frequency of each value;

[0038] S2. Quantization parameter initialization: For each sub-interval The endpoints of the quantization interval are initialized using the Min-Max normalization method. ,in , Quantization step size is defined as:

[0039] ,

[0040] In the formula, This is the initial quantization step size. The number of quantization levels for each sub-interval is determined by the quantization bit width.

[0041] S3. Particle Swarm Optimization for Intelligent Parameter Optimization: With minimizing the mean square error (MSE) as the optimization objective, the particle swarm optimization algorithm is used to optimize the parameters at the endpoints of the quantization interval. Perform a global search; the position parameter of the particle swarm is defined as the center point of the interval. and width To decouple the optimization of interval position and range, where This is the lower bound of the quantization interval. The upper bound of the quantization interval is defined; the search space expands around the initial solution, specifically as follows:

[0042] ,

[0043] in, The initial interval width, The expansion factor is used to balance the search range and accuracy; the initial particles are generated through grid sampling, with the grid density set to [value missing]. To ensure a uniform distribution of the initial solution, The mesh particle width is given; the fitness function is defined as the reciprocal of MSE, guiding particles to iterate towards the region of minimum error; the particle velocity update formula is:

[0044] ,

[0045] in, Let be the velocity of the particle at time t. Let the mass at time t be the inertia weight. It decays linearly with the number of iterations; As a learning factor, A random number in the interval [0,1]. and These represent the optimal position for the individual particle and the global optimal position, respectively.

[0046] S4. Data Quantization: Based on the optimized parameters ,in To quantize the step size, the data within the sub-interval is... Mapped to the discrete codebook space, the specific quantization formula is as follows:

[0047] ,

[0048] in, For data Corresponding quantization code, This is for rounding operations.

[0049] Furthermore, the parameter search space for particle swarm optimization is expanded by an expansion factor. Dynamic adjustment; when When the value is small, the search focuses on the vicinity of the initial solution, performing local fine-tuning; when... When the size is large, the search range is expanded for global exploration; a grid sampling strategy using initial particles is used to ensure uniform coverage of the solution space and avoid getting trapped in local optima.

[0050] Furthermore, the particle swarm velocity update formula adopts an inertial weight decay strategy. In the early stage, a larger inertial weight is used to encourage particles to search globally, and in the later stage, a smaller weight is used to guide particles to focus on local optima.

[0051] Example 1

[0052] Taking the Global Vector Dataset (GloVe dataset) as an example, the image feature quantization process is as follows:

[0053] Step 1. Input data: 1 million 300-dimensional image feature vectors, with feature values ​​ranging from [0, 1000].

[0054] Step 2. Dynamic Programming Segmentation:

[0055] 1) Set the number of segments M=4 to divide the data into 4 sub-intervals with similar frequencies;

[0056] 2) Calculate the optimal partition point through the state transition equation and obtain the optimal partition point [32, 128, 200], and divide the data into [0,32], (32,128], (128,200], (200,255];

[0057] 3) The dynamic ranges of each sub-interval after segmentation are 32, 96, 72, and 55, respectively. The average dynamic range is compressed to 31.6% compared with the traditional uniform segmentation, reducing the pressure of local quantization.

[0058] Step 3. Particle Swarm Optimization: Initialize parameters: (Expanded range is ±20% of the initial interval width), G=16, generating 256 initial particles, T=128 iterations; inertia weight linearly decays from 0.9 to 0.4, learning factor Optimization objective: For each sub-interval... Optimization was performed, such as optimizing the initial interval [0,32] to [2,30], compressing the width to 28, and adjusting the step size from 4 (when k=8) to 3.5, so that the quantization interval in the high-frequency center region is more dense.

[0059] Step 4. Quantization Output: Generate 4 independent codebooks, each containing 256 quantization levels; the data compression ratio reaches 8:1, compressing from 2400 bytes to 300 bytes, and the cosine similarity of the decompressed features is 0.987 with the original data, meeting the requirements of image retrieval tasks.

[0060] Example 2

[0061] Complexity analysis:

[0062] Step 1. Dynamic Programming Segmentation: After optimization using the Knuth-Yao algorithm, the search range for decision points is reduced from... Shrink to ,in and This represents the optimal decision point from the previous step, reducing the time complexity to O(NM). For N=10... 4 In the scenario where M=10, the computation time is approximately 0.1 seconds. The space complexity is O(N), requiring only the storage of the cumulative frequency array. and dynamic programming table The current layer and the previous layer, suitable for memory-constrained devices.

[0063] Step 2. Particle Swarm Optimization: The initialization complexity is O(G²). When G=16, 256 particles need to be generated for each sub-interval. The iteration complexity is O(nTG²), where n is the number of samples in the sub-interval. For n=10... 5 In a scenario where T=100, a single GPU takes approximately 50 milliseconds to perform parallel computation.

[0064] Through a two-layer optimization mechanism of "dynamic programming segmentation + swarm intelligence optimization", this invention achieves a balance between quantization error and codebook efficiency. Figure 1 As shown, the optimized uniform quantization is a method that discards the segmentation after the segmentation of the piecewise optimized uniform quantization method. Experimental results show that the quantization error of the method of the present invention is reduced by 93.52% and 98.05% respectively on the scale-invariant feature transform dataset (128-dimensional) and the audio processing dataset (420-dimensional) compared with the traditional method.

[0065] Although the present invention has been described herein with reference to illustrative embodiments, the above embodiments are merely preferred embodiments of the present invention, and the implementation of the present invention is not limited to the above embodiments.

[0066] The scope of protection of this invention includes, but is not limited to, the algorithm flow of the aforementioned high-precision quantization method, the combination of dynamic programming and particle swarm optimization, and the hardware implementation based on this framework, such as application-specific integrated circuit (ASIC) IP cores, FPGA modules, and application scenarios such as data compression for edge devices and model quantization for AI chips. Those skilled in the art can design many other modifications and implementations, which will fall within the principles and spirit disclosed in this application.

Claims

1. A high-precision quantization method for non-uniformly distributed data, characterized in that, The method comprises the following steps: S1. Data preprocessing and segmentation: For high-dimensional non-uniform data, first sort the characteristic value of each dimension in ascending order and remove duplicates to generate an ordered unique value sequence , while counting the frequency of each value ; Based on dynamic programming algorithm, the unique value sequence is divided into mutually exclusive sub-intervals , and the division target is to minimize the weighted range square sum, that is: ， wherein represents the number of samples in the sub-interval . The state transition equation of the dynamic programming algorithm is as follows: ， wherein, represents the first unique value is divided into minimum weighted range sum of squares of intervals; is the first unique value after sorting; is the cumulative frequency of the first values; S2. Quantization parameter initialization: for each sub-interval , the Min-Max normalization method is used to initialize the quantization interval endpoints where , ; the quantization step is defined as: ， In the formula, is an initial quantization step, is a quantization bit width, which determines the number of quantization levels for each sub-interval; S3. Particle swarm intelligence optimization parameter: with the minimum mean square error (MSE) as the optimization objective, the particle swarm optimization algorithm is used to search the global optimization solution of the quantization interval endpoints ; the position parameter of the particle swarm is defined as the interval center point and width , so as to decouple the optimization of the interval position and range, wherein is the lower bound of the quantization interval, is the upper bound of the quantization interval; the search space is expanded around the initialized solution, specifically: ， wherein, is the initial interval width, is the expansion factor, used to balance the search range and accuracy; the initial particles are generated by grid sampling, and the grid density is set to to ensure the uniform distribution of the initial solution, is the grid particle width; the fitness function is defined as the inverse of the MSE, guiding the particles to iterate to the area with the minimum error; the particle velocity update formula is: ， wherein, is the velocity of the particle at time t, is the mass at time t, the inertia weight linearly decays with the iteration number; is the learning factor, is a random number in the interval [0, 1], and is the individual best position of the particle and is the global best position, respectively. S4. Data quantization: according to the optimized parameters wherein is the quantization step, the data within the sub-interval is mapped to the discrete codebook space, and the specific quantization formula is: ， wherein is data corresponding quantization code, is a rounding operation.

2. The method of claim 1, wherein, Parameter search space of particle swarm optimization by expansion factor Dynamic adjustment; when When the value is small, the search is concentrated around the initial solution, and the local fine tuning is carried out; when When the value is large, the search range is expanded for global exploration; the grid sampling strategy of initial particles is used to ensure uniform coverage of the solution space and avoid falling into local optimum.

3. The method of claim 1, wherein, The inertia weight attenuation strategy is adopted in the particle swarm velocity updating formula, a larger inertia weight is used in the early stage to encourage global search of particles, and a smaller weight is used in the later stage to guide the particles to focus on local optimal solution.

Images