Method and processing element for compressing and decompressing neural network weights

Abstract

Methods and apparatus for compressing and decompressing weight values associated with a neural network. The input weight values are divided into a plurality of groups, wherein each group is processed using a scaling factor. For each set of scaled input weight values, a codebook is identified from a plurality of codebooks, where each codebook represents a discrete set of centroid values. The input weight values within each group are encoded using centroid values from the identified codebook, thereby producing encoded weight values including a codebook index and a centroid index. During decompression, the encoded weight values are processed using corresponding codebooks and scaling factors to reconstruct output weight values. A codebook is generated by identifying similar distributions of scaled input weight values across different groups and clustering these values to determine centroid values. The processing element performs a decompression operation to reconstruct the weight values for neural network operations.

Description

[0001] Cross-references to related applications

[0002] This application claims the benefit of U.S. Provisional Application No. 63 / 709,189, filed October 18, 2024, pursuant to 35 USC § 119(a). The entire contents of the patent application mentioned above are incorporated herein by reference. Background Technology Technical Field

[0003] This disclosure relates to data compression and decompression techniques, and more specifically to methods and systems for compressing and decompressing weight values ​​associated with neural networks. More specifically, this disclosure relates to quantization techniques for neural network weight values ​​using codebooks and centroid values.

[0004] Related technical descriptions

[0005] Neural networks process data through interconnected nodes, which are characterized by parameters that define the relationships between the nodes. These parameters form the basis of neural network computation and determine how input data propagates through network layers. Modern neural network architectures can contain millions of parameters, requiring significant memory resources to store them during both training and inference operations. The number of parameters tends to be proportional to the depth and width of the neural network architecture.

[0006] Storing and transmitting neural network parameters presents technical challenges in practical applications. As the size and complexity of neural networks increase, the associated memory requirements increase proportionally. Data transfer bandwidth limitations can impact the deployment of neural networks across different computing platforms and environments. Processing large parameter sets requires computational resources during both the training phase (when iteratively updating parameters) and the inference phase (when using parameters to generate predictions or outputs from input data).

[0007] Various techniques can be applied to reduce the memory and bandwidth requirements associated with neural network parameters. Traditional data compression methods may not adequately preserve the numerical accuracy required for neural network computations. The specific implementation of data simplification schemes involves considerations of efficiency ratios and computational overhead. Processed parameter data requires recovery operations to retrieve the values ​​used for neural network computations, which introduces additional processing steps into the computational pipeline.

[0008] Parameter reduction methods provide ways to reduce the memory usage of neural networks by representing parameters with modified numerical precision. These techniques involve transforming floating-point parameter values ​​into alternative representations that can be stored with fewer bits. Various implementations utilize different mechanisms to map the original parameter values ​​to their modified representations. Mathematical operations can be incorporated into parameter processing to maintain numerical relationships while reducing the memory footprint of the neural network. Summary of the Invention

[0009] Based on the example, a method is provided that processes a set of input weight values ​​associated with a neural network to produce encoded weight values. The input weight values ​​are organized into multiple groups. For each group, a scaling factor is applied to these weight values. Codebooks are identified from multiple available codebooks, where each codebook represents a discrete set of centroid values. The method processes each scaled input weight value within a group by selecting a centroid value from the discrete set of centroid values ​​of the identified codebooks to represent that scaled input weight value.

[0010] For each group, the method encodes a codebook index representing the codebook identified from the available codebooks. The method also encodes centroid indices representing the centroid values ​​selected to represent the input weight values ​​in that group. The output includes encoded weight values, which for each group include the codebook index and the corresponding centroid index of the input weight value in that group.

[0011] Different scaling factors and codebooks can be used to process input weight values ​​for different groups. For example, the first group can use a first scaling factor and a first codebook, while the second group can use a second scaling factor and a second codebook. This method can apply different scaling factors across groups when selecting a codebook from the same set of available codebooks. The number of available codebooks can range from four to sixteen, and in some cases can consist of exactly four codebooks. The number of input weight values ​​in each group can correspond to an integer multiple of the width of the Single Instruction Multiple Data (SIMD) vector.

[0012] Two bits can be used to encode the codebook index used to represent the identified codebook. The centroid index representing the selected centroid value can be encoded using 2 or 3 bits, and in some specific implementations, 2-bit encoding is specifically used. The number of codebooks can be selected so that all codebooks can fit into the processor's local register file.

[0013] The input weight values ​​can be derived from the weight matrix of a large language model (LLM).

[0014] Input weight values ​​can be FP16 (16-bit floating-point) or FP32 (16-bit floating-point), where the scaling factor is used to reduce the bit width of the centroid values. The centroid values ​​can be signed 8-bit integers. For example, if the required bit width of the centroid values ​​in the codebook is a signed 8-bit integer, the group scaling factor will scale the group weights to the range of -128 to 127.

[0015] The codebook used in the compression method may contain centroids that are not uniformly distributed across its value range.

[0016] The decompression method processes the encoded weight values ​​to reconstruct the output weight values ​​associated with the neural network. The encoded weight values ​​are organized into multiple groups. For each group of encoded weight values, the method obtains a codebook index and multiple centroid indices. The codebook index is used to identify a specific codebook from multiple available codebooks, where each codebook represents a discrete set of centroid values. Using the identified codebooks and centroid indices, the method identifies the centroid values ​​of that group of encoded weight values. A scaling factor is then applied to these values.

[0017] The decompression method produces output weight values ​​corresponding to the encoded weight values. For each group, the output includes weight values ​​corresponding to the centroid values ​​identified after applying the appropriate scaling factor to that group.

[0018] The decompression process can use different parameters to process different groups. For example, the first group can use a first codebook (identified by the first codebook index) and a first scaling factor to process its centroid index, while the second group can use a second codebook and a second scaling factor. This method can apply different scaling factors across groups when selecting codebooks from the same set of available codebooks. The number of available codebooks can range from four to sixteen. The number of input weight values ​​in each group can correspond to an integer multiple of the width of the Single Instruction Multiple Data (SIMD) vector.

[0019] The set of codebooks used in the decompression method can consist of exactly four codebooks. Two bits can be used to encode the codebook index used to identify a specific codebook. Two or three bits can be used to encode the centroid index specifying a particular centroid value from the identified codebook, and in some implementations, two-bit encoding is specifically used. The number of codebooks can be selected so that all codebooks can fit into the processor's local register file.

[0020] The weight values ​​processed by the decompression method can be derived from the weight matrix of a large language model (LLM). The codebook used in the decompression process may contain centroids that are not uniformly distributed across its value range.

[0021] A method for generating codebooks for quantizing weight values ​​processes input weight values ​​associated with one or more neural networks. The method applies a grouping structure to organize the input weight values ​​and applies scaling factors to the corresponding groups to generate scaled input weight values. For each codebook being generated, the method identifies similar distributions of scaled input weight values ​​across different groups. The scaled input weight values ​​from these similar distributions are clustered into discrete clusters to identify a predetermined number of centroid values, which are then used to generate the codebook.

[0022] The processing element can be configured to perform decompression of the encoded weight values. The processing element obtains the weight values ​​of the encoded data organized by groups, processes the codebook index and centroid index of each group, identifies the appropriate codebook and centroid values, applies a scaling factor, and outputs the reconstructed weight values.

[0023] The computing device may implement these methods using memory, at least one processor, and a network interface for receiving data such as input data and weight data. The device includes a user interface for interaction and command input. These components are interconnected via a system bus for data transfer between components. The memory contains computer program instructions that, when executed by the processor, implement the compression method, decompression method, or codebook generation method. Attached Figure Description

[0024] Examples of this disclosure will now be described with reference to the accompanying drawings.

[0025] Figure 1 This is a decomposition diagram showing the three-dimensional weight tensor;

[0026] Figure 2 This is a schematic diagram showing the internal components of a computing device;

[0027] Figure 3 A is a schematic diagram illustrating the user interface of a computing device;

[0028] Figure 3 B is a diagram showing the hardware of the mobile device;

[0029] Figure 4 This is a flowchart illustrating a method for compressing weight values ​​in a neural network;

[0030] Figure 5 This is a flowchart illustrating a method for group quantization of weight values;

[0031] Figure 6 This is a flowchart illustrating a method for processing multiple sets of input weight values, based on an example;

[0032] Figure 7 This is a flowchart illustrating a method for processing multiple sets of input weight values, based on a further example;

[0033] Figure 8 This is a flowchart illustrating a method for determining weight values ​​used in decompression encoding; and

[0034] Figure 9 This is a flowchart illustrating a method for generating multiple codebooks used to quantize weight values. Detailed Implementation

[0035] Generative large language models (LLMs) have demonstrated remarkable results for a wide range of tasks. Given context or a question, a language model can predict the next word. LLMs are trained on massive amounts of data to learn language patterns. They can perform tasks ranging from summarizing and translating text to responding in chatbot conversations. Therefore, promoting their efficient execution on CPUs extends their reach to billions of devices. LLMs are typically memory bandwidth and memory capacity limited, where memory access is dominated by weights, thus allowing CPUs a chance to achieve competitive performance and outperform other processors and accelerators in terms of overall inference / cost. Furthermore, CPUs are ubiquitous, offering portability and flexibility, so new LLM compression schemes can work seamlessly on CPUs with minimal effort.

[0036] The example demonstrates the operation of an LLM deployed on a CPU in data centers, smartphones, and edge devices. Deploying these LLMs for inference has become a significant challenge due to their unprecedented size and resource requirements. One of the main performance bottlenecks in LLM inference for generation tasks is memory bandwidth. Quantization is an effective method for converting high-precision (16- or 32-bit) model weights to lower-precision values ​​without significantly affecting accuracy. It reduces the model's memory and computational requirements, making it more suitable for deployment on resource-constrained devices.

[0037] While 8-bit quantization halves the storage requirements of LLMs (from 16-bit precision), the large size of LLMs necessitates quantizing them to even lower precisions (e.g., 4 bits or even lower bit widths). The expensive decoding schemes and large model footprints of existing quantization methods have a significant impact on runtime performance. This has spurred the development of faster inference kernels and runtime-friendly, fast, and accurate quantization methods. Existing quantization methods, such as 2-bit quantization, perform poorly and cause significant quality degradation at very high compression ratios.

[0038] The examples described herein present a novel post-training quantization method based on non-uniform codebooks, which achieves ultra-low-precision quantization of LLMs while outperforming existing techniques in terms of text generation quality and runtime performance. This is achieved by applying non-uniform codebook-based quantization to the weight matrix of the grouped LLM structure, combined with the identification and use of a reduced number of codebooks (preferably as few as required) to capture the diverse non-uniform weight distributions across all LLM layers. Preferably, all these codebooks can be stored in a CPU register file (e.g., a single 128-bit vector register). This, combined with a CPU-optimized codebook-based grouped quantization matrix multiplication kernel, significantly improves the runtime performance of the model in the LLM domain. It should be noted that while CPU-based generative inference is described, the techniques described herein are general and should be scalable to other settings.

[0039] The example provides a group codebook-based quantization method for ultra-low precision quantization in LLMs to better match non-uniform patterns in their weight distributions. This allows large-scale LLMs to fit on smaller devices and exhibits better throughput during token generation, while ensuring better quality than existing techniques. The example provides a Pareto-optimal solution for model quality and runtime performance with 2-bit quantization and outperforms existing 2-bit quantization techniques in LLM quality, requiring fewer bits per weight and eliminating the need for additional training or fine-tuning.

[0040] Weighted Tensor Structure

[0041] To enable the technology described in this article, Figure 1 An example decomposition plot of a 2×2×4 three-dimensional weight tensor 100 is shown. The three-dimensional weight tensor 100 can be defined using three dimensions: height (h), width (w), and channels (c), which correspond to the x, y, and z dimensions, respectively. These dimensions are labeled in the lower right of the plot. Although a three-dimensional weight tensor 100 is shown here for clarity, the tensor can also be implemented in two or four dimensions, including two or more stacked three-dimensional kernels.

[0042] Figure 1 Four 2×2 tensors are illustrated, labeled 110, 120, 130, and 140. Each 2×2 tensor contains four weight values. The four 2×2 tensors 110 through 140 together form a 2×2×4 three-dimensional weight tensor 100, which contains 16 weight values. Each weight value is represented as a numerical value. Weight values ​​can be stored in various data formats based on model requirements, hardware capabilities, and trade-offs between precision, memory, and computation. High-precision weight values ​​are typically stored in 32-bit floating-point (float32) or 16-bit floating-point (float16) formats. However, these high-precision formats require significant memory resources; for example, a float32 value requires 4 bytes per weight value, while a float16 value requires 2 bytes per weight value.

[0043] Due to the enormous size of some neural networks (such as LLMs), post-training quantization (PTQ) can be used to accelerate and efficiently run them during inference. By reducing the accuracy of pre-trained LLMs, PTQ saves memory and speeds up LLM inference, while preserving most of the model quality at scale compared to the performance and computational requirements of other compression techniques such as pruning and quantization-aware training (QAT). While aggressive weight quantization is used in LLMs to reduce inference costs, activation quantization may not be an issue due to their smaller memory footprint, allowing activations to be quantized to a higher bit width than the weights, such as 8 bits.

[0044] The example uses grouped PTQ to provide an improved kernel for group-quantized LLMs, demonstrating significant performance improvements across various low bit widths on the CPU.

[0045] Instead of uniform quantization, post-training non-uniform quantization is used in the example to better match the non-uniform patterns typically found in LLM weight distributions. The high overhead of accessing the codebook, combined with a complex decompression path, can lead to poor runtime performance. However, in the example, block-based codebook quantization not only provides faster throughput but also better quality in very high compression scenarios (e.g., 2-bit quantization).

[0046] The LLM is constructed using transformer layers. Given an input prompt, a new token is generated in each round of the LLM network, and this new token is fed into the LLM for token generation in the next round. For the next round, the LLM may need the initial prompt and the answer generated up to the input context to generate the next token. However, since all tokens except the last one generated remain the same as in the previous round, to save redundant computation, the LLM stores their embeddings in a key-value cache when they are first generated. Therefore, in the next round, the LLM simply retrieves the history, state, or embedding of the previous tokens and combines the previous embeddings with the last generated token to generate the next token. The LLM updates the history with the last token and repeats the process until a complete answer is generated.

[0047] For typical operators in LLMs, the weight matrix is ​​significantly larger than the activation matrix. Therefore, compressing the weight matrix is ​​crucial for reducing memory and bandwidth consumption; hence, in this example, they are quantized to 4 bits or fewer. Group quantization offers a finer granularity than standard demultiplexing or dechannel quantization, allowing it to naturally reduce quantization noise while maintaining near-full-precision (floating-point) quality of the underlying model. Group quantization quantizes in groups, dividing the weights into 32, 64, or 256 groups. Each group is then quantized individually to mitigate outliers and improve accuracy. This weight quantization is chosen to optimize space and bandwidth, and to make the decompression process easier during inference, while the activation quantization format is chosen to facilitate subsequent integer dot product calculations utilizing the group-quantized weights.

[0048] Uniform quantization divides the range of weight values ​​into equal intervals and assigns a quantization level to each interval. It distributes quantization values ​​evenly and at equal intervals. Therefore, although it is often used in conjunction with grouped quantization in LLM, it is not very flexible in matching non-uniform patterns typically found in the LLM weight distribution, resulting in suboptimal accuracy, especially for low-precision LLM quantization.

[0049] Non-uniform quantization adjusts the quantization interval based on the probability distribution of the input signal, with smaller intervals in regions of higher probability, resulting in lower average distortion and quantization noise. Given a weight distribution, non-uniform codebook-based quantization identifies k centroids that best represent the weight values ​​and maps the weights to them. For example, when quantizing the weight distribution to 4 bits, prior art codebook-based quantization techniques aim to determine the 16 centroid values ​​that best represent these values. Each high-precision weight can then be represented by a 4-bit index of the centroid in the codebook instead of its original bit width. Additionally, non-uniform codebook-based quantization requires storing the codebook itself and incurs association overhead.

[0050] The examples described in this paper address the limitations of existing non-uniform quantization techniques and attempt to fill the gaps by ensuring not only faster throughput but also better quality in very high compression scenarios of LLM (e.g., 2-bit quantization).

[0051] Computer device components

[0052] The methods described herein can be implemented on any suitable computing device. Figure 2 This is a schematic diagram illustrating the internal components of a computing device 200 used in the methods described herein. The computing device 200 includes a storage device 202, a network interface 204, a network 206, an interface 208, a user interface 210, a system bus 212, and a processing element 214.

[0053] Storage device 202 may include one or more volatile memories (such as random access memory (RAM)) and non-volatile memories (such as read-only memory (ROM) or solid-state drives (SSDs) such as flash memory). Storage device 202 may include magnetic or optical storage devices or other storage media. Storage device 202 may be removable, such as SD or USB type drives, or may not be removable from computing device 200. Storage device 202 may store data elements including input data, weighted bits, and output data. Storage device 202 includes computer program instructions that, when processed by processing element 214, implement methods for processing data. The computer program instructions and data may be stored in an accessible non-transitory computer-readable medium and loaded into memory.

[0054] Network interface 204 can communicatively connect to network 206 to receive data, such as input data and weight data. Network interface 204 can connect to and receive data via any known wired or wireless data transmission technology. Network 206 can be the Internet, a local intranet, a local or remote data server, or a link to another local or remote computing device.

[0055] Interface 208 and user interface 210 can be configured to combine to provide a user with the ability to interact with computing device 200 to input commands, data, and / or other information. Computing device 200 can accumulate data from network 206 into a dataset in storage device 202.

[0056] Processing element 214 is communicatively coupled to storage device 202 and can be used to implement neural networks. These processing element types include neural processing units (NPUs) and other custom processors specifically suited for neural network computations, as well as more generalized processors suitable for performing neural network computations, including central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), etc., with neural network capabilities. Generally, these types of processors may have “on-chip” data storage devices, such as one or more static random access memory (SRAM) cells, dynamic random access memory (DRAM) cells, data buffers, or any other type of memory.

[0057] Processing element 214 may also include a microprocessor, a general-purpose processor, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA) or other programmable logic device, discrete hardware components, discrete gate or transistor logic, or any suitable combination thereof capable of performing decompression functions. Processing element 214 may also be implemented as a combination of computing devices, such as multiple processors, a combination of DSP and microprocessor, one or more microprocessors combined with a DSP core, or other configurations.

[0058] The components of computing device 200 are interconnected using system bus 212, thereby allowing data to be transferred between the various components of computing device 200.

[0059] Overview of computing devices

[0060] Figure 3 A shows an example computing device in the form of a mobile device 300. As described above, the ultra-low precision quantization of neural networks as described herein allows large-scale LLMs to be adapted to smaller devices, such as mobile devices 300. Although described in relation to mobile devices, these techniques can be applied to any type of computing device for retrieving weight values ​​associated with neural networks, including tablets, laptops, personal computers (PCs), servers, and other computing devices.

[0061] Mobile device 300 includes a user interface 302 for displaying information to a user. User interface 302 may include a touchscreen display, physical buttons, a keyboard, a trackpad, or other input / output mechanisms typically found on mobile devices. User interface 302 allows users to interact with applications, view content, and control device functions.

[0062] Figure 3 B is a diagram illustrating the hardware of the mobile device. A processing element 304, in the form of a CPU, handles general computing tasks. The computing device 300 also includes a neural processing unit (NPU) 306, which serves as a dedicated processor for performing computations related to artificial intelligence, specifically computations related to neural networks. The NPU 306 can efficiently handle matrix multiplication, convolution, and other operations commonly used in neural network inference and training. The NPU 306 includes dedicated hardware optimized for parallel processing of neural network operations.

[0063] Random access memory (RAM) 308 provides storage for computing device 300. Although not illustrated, additional non-volatile storage devices are also provided. RAM 308 stores program instructions, data, neural network parameters, and other information required during operation of computing device 300.

[0064] Communication system 310 enables computing device 300 to connect to and transmit data through various data networks. These communication systems 310 can utilize technologies such as Wi-Fi. ™ And cellular network technologies. Supported additional data network technologies include Bluetooth, NFC, Ethernet, and other wired or wireless communication protocols.

[0065] The components of computing device 300 are interconnected via a system bus that facilitates data transfer between the various components. This allows processing element 304 and NPU 306 to access data stored in RAM 308, receive input from user interface 302, and communicate via communication system 310.

[0066] Quantization methods based on block codebooks

[0067] For a typical group-quantized LLM weight matrix, the shape of the Gaussian distribution of values ​​between groups may vary. However, after scaling by a group scaling factor, the Gaussian distributions of the individual groups with different shapes should be clustered into a small set of shapes, each of which can be represented by its own codebook.

[0068] In the example described in this article, a grouping structure is applied to divide the high-bit-width floating-point weights into groups, and each group is first scaled individually using its own scaling factor. The scaling factor is chosen such that the range of scaled values ​​can be represented by the bit width of the codebook. For example, if the required bit width for the centroid values ​​in the codebook is a signed 8-bit integer, the scaling factor for the group scales the weights of the group to the range of -128 to 127. Then, a clustering algorithm (e.g., a constraint-guided clustering algorithm) can be used to cluster the scaled weight values ​​of a group into a small number of codebooks, each with the same discrete number of centroid values, which can be a small number, such as four or eight. Clustering similar groups into the same codebook helps the group locate the closest codebook that best represents its values, while the small number of centroid values ​​in each codebook ensures that high-precision centroid values ​​(such as four 8-bit signed centroids in the codebook) can be encoded using a lower bit-width index (here, a 2-bit index) in the codebook.

[0069] For very high compression, the quantization bit width and the number of associated distinct values ​​or quantization bins (e.g., 16 distinct values ​​for 4-bit quantization) have the greatest impact on LLM accuracy. On the other hand, variations in the bit precision of the scaling factor have a smaller impact on LLM accuracy. The variety of codebooks in group-based quantization inherently helps to adapt the group to select a subset of relatively high-precision data types while using low-bit-width indices (here, the four most important quantization bins out of the 16 quantization bins in the group using 2-bit indices). This can result in a distribution of quantization bins that closely follows the higher-precision data type and bridges the accuracy gap with it.

[0070] Furthermore, codebook-based quantization examples keep the decompression path of converting low-bit codebook indices to high-precision centroid values ​​relatively simple, resulting in improved throughput.

[0071] 2-bit quantization based on block codebook (Q2_NL)

[0072] The example described in this article applies a codebook-based quantization technique to compress LLM to slightly more than 2 bits per weight. In the example, the 2-bit codebook-based quantization scheme has four codebooks. These are found by first applying a grouping structure and scaling factor to the LLM weight matrix, then partitioning and placing similar groups (groups with similar distributions) into four clusters. The four clusters are then processed separately using a clustering algorithm to cluster the values ​​into four codebooks, each with four centroids. Later, during post-training quantization of the grouped structure LLM, each weight value group selects one of the four codebooks that is considered to be the best match for its distribution according to the cost function, such as having the lowest reconstruction MSE (mean squared error). In the example, each weight value group requires two bits to encode the index of the selected codebook, and two bits for each of its elements to encode one of the four centroids of the codebook.

[0073] In the example, the 2-bit codebook-based quantization technique has four codebooks, and each codebook can have four centroids, for example, four signed 8-bit integer centroids. In the example, a group size of 16 can be used, with 2-bit quantization elements and its own scaling factor. Each group also has a 2-bit codebook index, which can be used to index into one of the four codebooks and extract the centroid value corresponding to the 2-bit quantization index element. In the example, the FP16 (16-bit floating-point) scaling factor can be used for 16-wide groups.

[0074] Note that codebook-based quantization schemes can be extended to other bit widths, such as 3-bit and 4-bit quantization. This can be achieved, for example, by dividing similar groups into a small number of clusters (e.g., one or two), and then encoding each cluster group with an 8-entry (for 3-bit index) or 16-entry (for 4-bit index) codebook.

[0075] Algorithm 1

[0076] Q2_NL quantization based on non-uniform codebook:

[0077] Input: High-precision model weights W, divided into groups of size g.

[0078] Step 1: Scale each set of high-precision weights to the range of -128 to 127 using the FP16 scaling factor.

[0079] Step 2: Cluster the similar weight reorganizations (groups of scaled weight values ​​with a similar Gaussian distribution) created in Step 1 into C clusters by performing the following steps:

[0080] - Convert each set of scaled weight values ​​into a probability distribution. One approach is to scale each set of values ​​to a smaller range (e.g., the range [-m, m], where m=8), then find a histogram of the scaled values ​​and normalize it, thus converting the discrete distribution of intensity into a discrete distribution of probability.

[0081] - Apply cluster analysis (e.g., k-means clustering) to these probability distributions, which now represent distinct groups, to cluster similar groups.

[0082] Step 3: Now apply cluster analysis to the value groups of each cluster (created in step 2) to identify several centroid values ​​that best represent the probability distribution of the weight values ​​within each value group. Repeat step 3 for the value groups of C clusters to create C codebooks to find the distinct non-uniform weight distributions present in the high-precision weights W.

[0083] Step 4: Using the C codebooks created in Step 3, perform post-training quantization of high-precision input weight values ​​by executing the following steps:

[0084] - Scaled input weight value group ← High-precision input weight value group / FP16 scaling factor

[0085] - Codebook index, group codebook index ← For a scaled set of input weight values, select the codebook (out of C codebooks) that best matches the distribution with the lowest reconstruction MSE (mean squared error), and map the weights to the centroid values ​​in the selected codebook. The output represents the encoded data of the selected codebook and the mapped centroid values.

[0086] Repeat the high-precision input weight values ​​for each group.

[0087] Output: C codebooks for each set of high-precision input weights, FP16 scaling factor, codebook index, and corresponding centroid index.

[0088] Algorithm 2

[0089] Q2_NL Decompression and Inference

[0090] Input: C codebooks from each quantized weight group in Algorithm 1, an FP16 scaling factor, a codebook index, and a corresponding centroid index.

[0091] For each set of quantized weight values:

[0092] Centroid value ← Codebook [Codebook Index][Centroid Index]

[0093] Decompressed value ← FP16 scaling factor * centroid value

[0094] Output: The decompressed model weights W, divided into groups of size g.

[0095] Weighted compression method

[0096] Figure 4 A general method 400 for compressing a set of input weight values ​​to produce a set of encoded weight values ​​is illustrated according to an example. Method 400 can be performed by any suitable computing device, such as... Figure 2 and Figure 3 The computing device shown.

[0097] At box 402, obtain a set of input weight values ​​associated with the neural network. The neural network can be of any type, including large language models (LLM), convolutional neural networks (CNN), recurrent neural networks (RNN), or transformer networks. The input weight values ​​may include high-precision weight values.

[0098] At box 404, select multiple sets of input weight values ​​from a set of input weight values. The minimum number of input weight values ​​in a set can be indicated by the Single Instruction Multiple Data (SIMD) vector width. This allows the resulting matrix multiplication kernel to benefit from vectorization and leverage efficient vectorized integer computation and matrix multiplication operations. Each set can include integer multiples of 16 input weight values, such as 16, 32, 64, or 128 values.

[0099] At box 406, a scaling factor is applied to the selected group. The scaling factor can be selected such that the range of scaled values ​​can be represented by the bit width of the codebook. For example, if the required bit width for the centroid values ​​in the codebook is a signed 8-bit integer, the group scaling factor scales the input weight values ​​of the group to the range of -128 to 127. As previously mentioned, for some neural network weight matrices, such as those of LLMs, which are typically group-quantized, the shape of the Gaussian distribution of values ​​between groups can vary. However, it has been observed that after scaling with a group scaling factor, the Gaussian distributions of the individual groups with different shapes can be clustered into a small set of shapes, each represented by its own codebook.

[0100] At box 408, a codebook is identified from multiple codebooks. Clustering algorithms (such as constraint-guided clustering) can be used to cluster a scaled set of weight values ​​into a small number of codebooks. A codebook represents a discrete set of centroid values ​​that can be used to represent the scaled input weight values. Codebooks may have non-uniformly distributed centroids. In some examples, each centroid may be a signed 8-bit integer. The codebook found to best match the distribution of the current group can be selected from multiple codebooks based on a cost function (such as minimum reconstruction MSE (mean squared error)). Figure 9 The method shown generates multiple codebooks.

[0101] At box 410, for each of the multiple scaled input weight values ​​in the selected group, an input weight value is selected, and a centroid value is identified from the discrete set of centroid values ​​represented by the identified codebook. Clustering similar groups into the same codebook helps the group locate the closest codebook that best represents its scaled input weight values. The small number of centroid values ​​in each codebook ensures that high or medium precision centroid values ​​(such as four 8-bit signed centroids in the codebook) can be encoded using index numbers with lower bit widths (such as 2-bit index numbers). Centroid values ​​can have more than 8 bits. In some examples, 8-bit signed centroid values ​​can be used, making it possible to perform 8-bit integer arithmetic on the centroid values.

[0102] At box 412, the codebook index representing the identified codebook is encoded. The codebook index is one of multiple codebook indices representing a codebook. The codebook index can include a 2-bit codebook index, thus allowing for four codebooks. Other examples include a 3-bit codebook index for eight codebooks or a 4-bit codebook index for sixteen codebooks. Specific utilities have been found in applications using 2-bit or 3-bit codebook indices.

[0103] At box 414, the centroid index representing the identified centroid value is encoded. Each centroid can be indexed, and the centroid index can be a 2-bit, 3-bit, or 4-bit centroid index. The size of the centroid index is related to the number of centroids in the codebook.

[0104] At box 416, output a set of encoded weight values ​​corresponding to a set of input weight values. For each set of input weight values, the encoded weight values ​​include the codebook index of the set and multiple centroid indices corresponding to the multiple input weight values ​​in the set.

[0105] Encoding weight values ​​using codebooks and centroid indices reduces storage requirements compared to storing full-precision weights. Codebook and centroid indices can represent weight values ​​using fewer bits than the original weight values. This compact representation allows neural networks to be stored and deployed with lower memory overhead. This allows large-scale neural networks to be fitted to smaller devices.

[0106] The combination of scaling factors and multiple codebooks provides adaptability to different weight distributions across groups while maintaining compression efficiency. Multiple codebooks enable the encoding of different weight groups using codebook distributions that match their characteristics.

[0107] Group weight quantization process

[0108] Figure 5 Examples Figure 4 Additional aspects of the weight compression method described herein. Method 500 illustrates the group quantization process.

[0109] In this example, the weight values ​​are divided into groups, each containing V elements and having an associated scaling factor. A group size of 32 V is used, although other group sizes can be chosen (as described above). For each group of 32 floating-point weights, the values ​​are quantized to 4-bit integer values ​​(4-bit centroid indexes for codebook-based quantization) using a local scaling factor specific to that group. For non-uniform codebook-based quantization, each group also has a codebook index field (not shown in the figure) to index into a codebook within the codebook.

[0110] The process continues sequentially through the weight tensor, where each subsequent group of 32 consecutive weights is quantized to 4 bits using a different scaling factor calculated for that particular group. This grouping method continues until the entire weight tensor has been processed. FP16 (16-bit floating-point) precision is used to store the scaling factors to maintain sufficient numerical precision while keeping storage requirements low.

[0111] Multi-group weight processing method

[0112] Figure 6 Method 600 is illustrated, which provides a method for... Figure 4 The method shown handles additional details regarding multiple sets of input weight values. Boxes 602 to 608 and boxes 610 to 616 can be similar to those described above. Figure 4 The procedure described in boxes 406 to 412 shall be followed.

[0113] For the input weight values ​​of the first selected group, at box 602, a first scaling factor is applied to the first selected group. At box 604, a first codebook is identified from multiple codebooks based on the distribution of the scaled input weight values. At box 606, for each of the multiple input weight values ​​in the first selected group, an input weight value is selected, and a centroid value is identified from a discrete set of centroid values ​​represented by the first codebook to represent the selected input weight value. At box 608, a first codebook index representing the first codebook and a first centroid index representing the centroid value are encoded; these centroid values ​​are identified as representing the multiple input weight values ​​in the first selected group.

[0114] For the input weight values ​​of the second selected group, at box 610, a second scaling factor is applied to the second selected group. At box 612, a second codebook is identified from a plurality of codebooks. At box 614, for each of the plurality of input weight values ​​in the second selected group, an input weight value is selected, and a centroid value is identified from a discrete set of centroid values ​​represented by the second codebook to represent the selected input weight value. At box 616, a second codebook index representing the second codebook and a second centroid index representing the centroid value, which are identified as representing the plurality of input weight values ​​in the second selected group, are encoded.

[0115] Processing multiple groups allows different scaling factors and codebooks to be applied to different parts of the input weight values. This enables compression techniques to adapt to variations in weight distribution across different parts of the neural network. Each group can be processed with a scaling factor and codebook suitable for the characteristics of the weight values ​​within that particular group.

[0116] Using separate scaling factors and codebooks to independently process input weight values ​​for different groups provides flexibility in compression. Weight distributions can vary significantly across different parts of a neural network. Using group-specific parameters allows compression to better match the statistical properties of the weight values ​​for each group.

[0117] Group codebook selection process

[0118] Figure 7 Method 700 is illustrated, which provides a method based on Figure 4 and Figure 6 The method shown handles additional details of multiple sets of input weight values, where a codebook from a set of codebooks is identified for each of the multiple groups after applying different scaling factors to each group. As mentioned, there may be some variation in the shape of the Gaussian distribution of the values ​​between groups. However, after scaling by the grouping scaling factor, the Gaussian distributions of the individual groups with different shapes can be clustered into a small set of shapes, each of which can be represented by its own codebook.

[0119] At box 702, multiple sets of input weight values ​​are received. At box 704, the first or next set of input weight values ​​is selected for processing. At box 706, a scaling factor specific to the current set is applied. The scaling factor can be selected to map the weight values ​​in the current set to an appropriate range for the codebook representation.

[0120] At box 708, a codebook is identified from a set of codebooks used for the current group. The codebook found to best match the distribution of its current group is selected from this set of codebooks based on a cost function (such as the lowest reconstruction MSE (mean squared error)). The identified codebook represents a discrete set of centroid values ​​that can be used to represent the input weight values ​​in the current group. Input weight values ​​for different groups can be encoded using different codebooks or the same codebook. The same codebook can be used for multiple scaled sets of values ​​to suit its distribution. The selection of codebooks across different groups allows for consistent encoding, while group-specific scaling factors enable adaptation to different weight value distributions.

[0121] At decision box 710, a check is performed to determine if there are any more groups to process. If condition 712 indicates that an additional group exists (Yes), the method returns to box 704 to select and process the next group. If condition 714 indicates that there are no more groups to process (No), the method proceeds to box 716, where all groups are processed.

[0122] Steps 702-708 can be similar to those regarding... Figure 4 Steps 402-408 are performed as described. Iterative processing of groups with different scaling factors and identification of the codebook from the set achieve efficient compression while maintaining adaptability to changes in weight distribution across different parts of the neural network.

[0123] Using a common codebook across different groups while applying different scaling factors provides a simplified implementation. Since the codebook can be stored once and referenced by multiple groups, memory requirements are reduced. Processing overhead is minimized by reusing the established codebook structure instead of maintaining a separate codebook for each group.

[0124] Weighted decompression method

[0125] Figure 8 Examples of supplements are provided. Figure 4 The compression method shown is followed by decompression method 800. Method 800 decompresses a set of output weight values ​​from a set of encoded weight values, wherein the set of output weight values ​​may have been previously used. Figure 4 The method involved compression.

[0126] At box 802, a set of encoded weight values ​​corresponding to a set of output weight values ​​is obtained. This set of output weight values ​​includes weight values ​​associated with a neural network, which can be of any type, including large language models, convolutional neural networks, recurrent neural networks, or transformer networks.

[0127] At step 804, weight values ​​for the multiple sets of codes corresponding to the weight values ​​for the multiple sets of outputs are selected from the weight values ​​of the group of codes. The size of the group can be set according to the width of the Single Instruction Multiple Data (SIMD) vector, such as a multiple of 16 for a 128-bit vector register divided into 16 8-bit channels.

[0128] At box 806, for the weight value of the code of the selected group, the codebook index and multiple centroid indices are obtained from the weight value of the code of the selected group.

[0129] At box 808, the obtained codebook index is used to identify one of the multiple codebooks. Each codebook represents a discrete set of centroid values. The codebook may contain non-uniformly distributed centroids, where each centroid can be a signed 8-bit integer.

[0130] At box 810, multiple centroid values ​​for the weight values ​​of the selected group of codes are identified from multiple centroid indices using the identified codebook. The centroid values ​​are derived from a discrete set of centroid values ​​represented by the identified codebook. A table lookup of the codebook register can be performed to obtain the codebook and centroid values. For example, for a 128-bit vector register comprising 16 8-bit channels, each group of weights can be associated with a corresponding codebook ID (or codebook index). The codebook index of a group of size 16 can be copied across all 16 channels of the vector register, shifted by a specified amount to create space, and combined with the centroid ID of each channel using an OR operation to generate a table lookup vector for the codebook register.

[0131] At box 812, apply the scaling factor to the selected group. The scaling factor maps the centroid values ​​back to their original value range.

[0132] At decision box 814, the method determines whether there are more groups to process. If yes, at 816, the method returns to box 806 to process the next group. If no, at 818, the method proceeds to box 820.

[0133] At box 820, the output weight values ​​for that group, corresponding to the weight values ​​encoded in that group, are output. These output weight values ​​include weight values ​​corresponding to the identified centroid values ​​for each group, scaled by the appropriate scaling factor for that group.

[0134] The structured approach to decompressing weight values ​​enables rapid reconstruction of the original neural network weights. By utilizing a codebook and centroid indexing, the decompression process can be performed through an efficient lookup operation. Organizing weight values ​​into groups allows for the parallel processing of multiple values ​​simultaneously.

[0135] The hierarchical coding structure reduces the memory requirements during the storage and transmission of neural network weights. The codebook stores a compact set of representative weight values, while indices reference these values. The combination of codebook and indices provides an efficient representation that maintains the ability to accurately reconstruct the original weight values.

[0136] Applying scaling factors on a per-group basis supports precise reconstruction of weight sizes. Different regions of a neural network can contain weights spanning different numerical ranges. Using multiple codebooks in conjunction with group-specific scaling allows mapping each group of weights back to appropriate values ​​for its corresponding network region.

[0137] Multi-codebook generation method

[0138] Figure 9 A method 1100 for generating multiple codebooks is illustrated, which can be used for... Figure 4 , Figure 6 , Figure 7 and Figure 8 In the aforementioned compression and decompression techniques, the codebook generated for the first neural network using this method can be used for a second neural network with the same architecture, thereby allowing the generation of a single set of codebooks for a given category of neural networks.

[0139] At box 902, a set of input weight values ​​associated with one or more neural networks is obtained. These weight values ​​can come from any type of neural network, including large language models, convolutional neural networks, recurrent neural networks, or transformer networks.

[0140] At box 904, a grouping structure is applied to a set of input weight values. This structure organizes the weight values ​​into multiple groups, where each group can contain weight values ​​from different parts of the neural network. The size of the groups can be determined based on requirements such as the width of a Single Instruction Multiple Data (SIMD) vector.

[0141] At box 906, a scaling factor is applied to the corresponding group of the grouping structure to generate scaled input weight values ​​for each group. The scaling factor maps the weight values ​​in each group to a range suitable for the codebook representation while preserving the relative weight magnitudes within the group.

[0142] At box 908, similar distributions of scaled input weight values ​​are identified across different groups in the grouping structure. This identification process examines the statistical properties and patterns in the scaled weight values ​​to find groups that share a comparable distribution of values.

[0143] At box 910, the scaled input weight values ​​identified from similar distributions in different groups are clustered into discrete clusters. Input weight values ​​from similar distributions can be clustered into a small, predetermined number of clusters. The clustering process groups scaled weight values ​​that exhibit similar properties together, thereby allowing the identification of common representation patterns.

[0144] At box 912, for each cluster of scaled input weight values, a predetermined number of centroid values ​​are identified. These centroid values ​​serve as representative values ​​that can efficiently encode the weight values ​​in the currently selected cluster. The number of centroid values ​​can be chosen based on factors such as desired compression ratio and accuracy requirements.

[0145] At box 914, a codebook is generated from the identified centroid values. The codebook contains a discrete set of centroid values ​​that can represent the input weight values ​​in the currently selected cluster. This process is repeated for each scaled weight value cluster for each codebook to be generated, thereby creating multiple codebooks that can capture the different patterns and distributions present in the neural network weights.

[0146] The generated codebook provides a compact representation that can be used across multiple neural networks with similar architectures. By identifying and utilizing the distribution of a small number of weight values ​​across different groups, the codebook achieves efficient compression while maintaining the ability to accurately represent the original weight values.

[0147] Codebooks are generated by identifying similar distributions across different groups, enabling the sharing of weight pattern representations. This method reduces redundancy by allowing the encoding of weight values ​​with comparable statistical properties using common centroid values. The resulting codebook provides compact and representative encodings that preserve the characteristics of the original weight distribution.

[0148] Applying scaling factors to each group preserves the relative relationships between weight values ​​while promoting efficient compression. The scaling factor maps weight values ​​to a range suitable for the codebook representation without losing important magnitude information. This group-specific scaling method adapts to varying weight distributions across different parts of the neural network.

[0149] Clustering based on distribution-similarity-based weight values ​​produces centroid values ​​highly suitable for compression. By considering statistical patterns across groups, the clustering process identifies representative values ​​that capture common features. This distribution-aware approach generates a codebook that efficiently represents weight values ​​across multiple groups.

[0150] Other variations

[0151] In some examples, a set of codebooks may include four to sixteen codebooks. A particular implementation may utilize exactly four codebooks. The codebook index can be implemented as a 2-bit index, thereby allowing for efficient storage and processing of codebook selection.

[0152] Centroid indexes can be implemented as 2-bit or 3-bit indexes, with 2-bit centroid indexes used in some examples. This choice of bit width for centroid indexes achieves a balance between compression efficiency and representation accuracy.

[0153] The codebook can be implemented using a non-uniformly distributed centroid. This non-uniform distribution may be advantageous in capturing the statistical properties of the compressed weight values, potentially leading to improved compression quality compared to a uniformly distributed centroid.

[0154] These variations can be combined in various ways. For example, the system can use four codebooks with 2-bit codebook indices and 2-bit centroid indices to compress the input weights from the LLM weight matrix, while taking advantage of the non-uniformly distributed centroids in the codebooks.

[0155] Using four to sixteen codebooks allows for efficient data compression while maintaining representational accuracy. This range provides sufficient flexibility to capture meaningful patterns in the data. The number of codebooks can be chosen within this range based on specific requirements for compression ratio and accuracy. Using four to sixteen codebooks allows for practical implementation while maintaining reconstruction quality. This range provides sufficient granularity for the weight values ​​used for encoding without requiring excessive computational resources. The upper limit helps control memory usage and processing overhead, while the lower limit ensures sufficient representational power. Limiting the number of codebooks to a small number of values ​​to load them into the processor's local register file allows for efficient vector lookups, such as table lookup instructions, which can be used to access multiple centroid values ​​from them simultaneously.

[0156] Using four codebooks provides a practical balance between data compression and reconstruction quality. The four-codebook configuration allows for efficient implementation while maintaining a suitable compression ratio. This approach helps manage computational resource requirements during compression and decompression operations.

[0157] The use of 2-bit codebook indexes provides a compact representation for storing weight values. This format efficiently compresses weight data while maintaining sufficient precision for practical applications. The 2-bit structure of the centroid index in the codebook allows for four distinct index values, thus supporting an efficient mapping between weight values ​​and their corresponding codebook entries. The use of 2-bit codebook indexes allows for a compact representation of codebook selection. Each index requires only two bits of storage space, resulting in efficient memory utilization during compression and decompression operations. This method supports high compression ratios while maintaining sufficient precision to accurately identify and select the appropriate codebook for weight quantization.

[0158] Using 2-bit or 3-bit centroid indexes allows for flexibility in precision selection. A 2-bit implementation prioritizes memory efficiency and processing speed, while a 3-bit implementation offers enhanced representation accuracy. This adaptability allows the system to be configured based on specific model requirements and available computational resources. The use of 2-bit and 3-bit centroid indexes provides implementation flexibility. 2-bit representation allows for higher compression ratios, while 3-bit representation achieves higher reconstruction accuracy. This adaptable precision allows compression schemes to be tailored to specific application requirements and resource constraints.

[0159] In the preceding description, numerous specific details of certain examples were set forth for illustrative purposes. References to “example” or similar language in this specification mean that a particular feature, structure, or characteristic described in connection with that example is included in at least one example, but not necessarily in other examples.

[0160] The examples above are to be understood as illustrative examples of this disclosure. Further examples of this disclosure are contemplated. It should be understood that any feature described with respect to any example may be used alone or in combination with other features described, and may also be used in combination with one or more features of any other example or with any combination of any other example. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of this disclosure as defined in the appended claims.

[0161] This disclosure implements the following additional embodiments.

[0162] Example 1: A method for compressing a set of input weight values ​​to generate a set of encoded weight values, the method comprising:

[0163] Obtain a set of input weight values, the set of input weight values ​​including multiple weight values ​​associated with the neural network;

[0164] Select multiple sets of input weight values ​​from the set of input weight values;

[0165] For a selected group among the multiple sets of input weight values:

[0166] A scaling factor is applied to the selected group to produce scaled input weight values ​​from the input weight values ​​in the group;

[0167] Identify one of a plurality of codebooks, each codebook representing a discrete set of centroid values;

[0168] For each of the multiple scaled input weight values ​​in the selected group, the scaled input weight value is selected, and a centroid value is identified from the discrete set of centroid values ​​represented by the identified codebook to represent the selected scaled input weight value.

[0169] Encode the codebook index representing the identified codebook, wherein the codebook index is one of a plurality of codebook indices representing the codebook among the plurality of codebooks; and

[0170] Encode multiple centroid indices representing the centroid values, the centroid values ​​being identified as input weight values ​​representing the multiple scaling operations; and

[0171] Output a set of encoded weight values ​​corresponding to the set of input weight values. For each set of input weight values, the set of encoded weight values ​​includes data representing the codebook index of the code in the set and data representing the centroid indices corresponding to the multiple input weight values ​​in the set.

[0172] Example 2: The method according to Example 1, wherein the method includes:

[0173] For the first selected group among the multiple sets of input weight values:

[0174] A first scaling factor is applied to the first selected group to produce scaled input weight values ​​from the input weight values ​​in the first selected group;

[0175] Identify the first codebook among the plurality of codebooks;

[0176] For each of the multiple scaled input weight values ​​in the first selected group, the scaled input weight value is selected, and the centroid value is identified from the discrete set of centroid values ​​represented by the first codebook to represent the selected scaled input weight value.

[0177] Encode the first codebook index representing the first codebook; and

[0178] Encode a first plurality of centroid indices representing the centroid values, the centroid values ​​being identified as input weight values ​​representing the plurality of scaling values ​​in the first selected group; and

[0179] For the second selected group among the multiple sets of input weight values:

[0180] The second scaling factor is applied to the second selected group to produce scaled weight values ​​from the input weight values ​​in the second selected group;

[0181] Identify the second codebook among the plurality of codebooks;

[0182] For each of the multiple scaled input weight values ​​in the second selected group, the scaled input weight value is selected, and the centroid value is identified from the discrete set of centroid values ​​represented by the second codebook to represent the selected scaled input weight value.

[0183] Encode the second codebook index representing the second codebook; and

[0184] The second plurality of centroid indices representing the centroid values ​​are encoded, and the centroid values ​​are identified as input weight values ​​representing the plurality of scaling in the second selected group.

[0185] Example 3: According to the method described in Example 1, the method includes: applying different scaling factors, and identifying codebooks from the same group for each of the multiple groups of input weight values.

[0186] Example 4: According to the method described in Example 3, a set of codebooks consists of between four and sixteen codebooks.

[0187] Example 5: According to the method described in Example 1, the number of input weight values ​​in each group is equal to an integer multiple of the width of the Single Instruction Multiple Data (SIMD) vector.

[0188] Example 6: According to the method described in Example 1, the codebook index includes a 2-bit codebook index.

[0189] Example 7: According to the method described in Example 1, the centroid index includes a 2-bit centroid index or a 3-bit centroid index.

[0190] Example 8: According to the method described in Example 1, the number of the plurality of codebooks is such that the plurality of codebooks are suitable for the processor's local register file.

[0191] Example 9: According to the method of Example 1, the codebook includes a codebook with a non-uniformly distributed centroid.

[0192] Example 10: A method for decompressing a set of output weight values ​​from a set of encoded weight values, the method comprising:

[0193] Obtain a set of encoded weight values ​​corresponding to a set of output weight values, wherein the set of output weight values ​​includes multiple weight values ​​associated with the neural network;

[0194] Select weight values ​​from the set of output weight values ​​that correspond to the weight values ​​of the multiple sets of output weight values;

[0195] For the weight values ​​of the selected group of codes from the multiple groups of codes:

[0196] The codebook index and multiple centroid indices are obtained from the weight values ​​of the encoding of the selected group;

[0197] Identify one of a plurality of codebooks from the obtained codebook index for weight values ​​used in encoding the selected group, each codebook representing a discrete set of centroid values;

[0198] Using the identified codebook, multiple centroid values ​​for weight values ​​used in the encoding of the selected group are identified from the multiple centroid indices. These centroid values ​​are derived from multiple centroid values ​​derived from a discrete set of centroid values ​​represented by the identified codebook.

[0199] Apply the scaling factor to the selected group; and

[0200] The set of output weight values ​​corresponds to the weight values ​​of the set of codes. For each set of weight values ​​in the plurality of sets of codes, the set of output weight values ​​includes weight values ​​corresponding to a plurality of centroid values ​​of the corresponding identification scaled by a corresponding scaling factor.

[0201] Example 11: The method according to Example 10, wherein the method includes:

[0202] For the weight values ​​of the first selected group of codes among the multiple groups of codes:

[0203] The first codebook index and the first plurality of centroid indices are obtained from the weight values ​​of the encoding of the first selected group;

[0204] Identify the first codebook from the first codebook index;

[0205] Using the first codebook, identify first plurality of centroid values ​​from the first plurality of centroid indices the weight values ​​for the encoding of the first selected group; and

[0206] Apply the first scaling factor to the selected group; and

[0207] For the weight values ​​of the second selected group of codes among the multiple groups of codes:

[0208] The second codebook index and the second and a plurality of centroid indices are obtained from the weight values ​​of the encoding of the second selected group;

[0209] Identify the second codebook from the second codebook index;

[0210] Using the second codebook, identify a second plurality of centroid values ​​from the second plurality of centroid indices to determine the weight values ​​for the encoding of the second selected group; and

[0211] Apply the second scaling factor to the selected group.

[0212] Example 12: The method according to Example 10, wherein the method includes: applying different scaling factors and identifying codebooks from the same group for each of the plurality of input weight values.

[0213] Example 13: According to the method described in Example 12, a set of codebooks consists of between four and sixteen codebooks.

[0214] Example 14: The method described in Example 10, wherein the number of input weight values ​​in each group is an integer multiple of the width of the Single Instruction Multiple Data (SIMD) vector.

[0215] Example 15: According to the method of Example 10, the number of the plurality of codebooks is such that the plurality of codebooks are suitable for the processor's local register file.

[0216] Example 16: According to the method described in Example 10, the centroid index includes a 2-bit centroid index or a 3-bit centroid index.

[0217] Example 17: The method according to Example 10, wherein the input weight values ​​include an LLM weight matrix.

[0218] Example 18: The method according to Example 10, wherein the codebook includes a codebook having a non-uniformly distributed centroid.

[0219] Example 19: A method for generating multiple codebooks, wherein the multiple codebooks are used to quantize multiple sets of input weight values ​​to generate multiple sets of encoded weight values, the method comprising:

[0220] Obtain a set of input weight values, the set of output weight values ​​including multiple weight values ​​associated with one or more neural networks;

[0221] Apply the grouping structure to the set of input weight values;

[0222] The scaling factor is applied to the grouping structure of the corresponding groups to generate the input weight value for scaling of each corresponding group; and

[0223] For each of the multiple codebooks to be generated:

[0224] Identify multiple similar distributions of the scaled input weight values ​​in different groups within the grouping structure of the corresponding groups;

[0225] Clustering is performed on the scaled input weight values ​​from the multiple similar distributions of different groups in the corresponding discrete clusters to identify a predetermined number of centroid values ​​from the clustered scaled input weight values; and

[0226] Generate a codebook from the identified centroid values.

[0227] Example 20: A processing element adapted to decompress a set of output weight values ​​from a set of encoded weight values, the processing element being adapted to:

[0228] Obtain a set of encoded weight values ​​corresponding to a set of output weight values, wherein the set of output weight values ​​includes multiple weight values ​​associated with the neural network;

[0229] Select weight values ​​from the set of output weight values ​​that correspond to the weight values ​​of the multiple sets of output weight values;

[0230] For the weight values ​​of the selected group of codes from the multiple groups of codes:

[0231] The codebook index and multiple centroid indices are obtained from the weight values ​​of the encoding of the selected group;

[0232] Identify one of a plurality of codebooks from the obtained codebook index for weight values ​​used in encoding the selected group, each codebook representing a discrete set of centroid values;

[0233] Using the identified codebook, multiple centroid values ​​for weight values ​​used in the encoding of the selected group are identified from the multiple centroid indices. These centroid values ​​are derived from multiple centroid values ​​derived from a discrete set of centroid values ​​represented by the identified codebook.

[0234] Apply the scaling factor to the selected group; and

[0235] The set of output weight values ​​corresponds to the weight values ​​of the set of codes. For each set of weight values ​​in the plurality of sets of codes, the set of output weight values ​​includes weight values ​​corresponding to a plurality of centroid values ​​of the corresponding identification scaled by a corresponding scaling factor.

Claims

1. A method for compressing a set of input weight values ​​to generate a set of encoded weight values, the method comprising: Obtain a set of input weight values, the set of input weight values ​​including multiple weight values ​​associated with the neural network; Select multiple sets of input weight values ​​from the set of input weight values; For a selected group among the multiple sets of input weight values: A scaling factor is applied to the selected group to produce scaled input weight values ​​from the input weight values ​​in the group; Identify one of a plurality of codebooks, each codebook representing a discrete set of centroid values; For each of the multiple scaled input weight values ​​in the selected group, the scaled input weight value is selected, and a centroid value is identified from the discrete set of centroid values ​​represented by the identified codebook to represent the selected scaled input weight value. The codebook index representing the identified codebook is encoded, wherein the codebook index is one of a plurality of codebook indices representing the codebook among the plurality of codebooks; as well as Encode multiple centroid indices representing the centroid values, the centroid values ​​being identified as input weight values ​​representing the multiple scaling operations; as well as Output a set of encoded weight values ​​corresponding to the set of input weight values. For each set of input weight values, the set of encoded weight values ​​includes data representing the codebook index of the code in the set and data representing the centroid indices corresponding to the multiple input weight values ​​in the set.

2. The method according to claim 1, wherein the method comprises: For the first selected group among the multiple sets of input weight values: A first scaling factor is applied to the first selected group to produce scaled input weight values ​​from the input weight values ​​in the first selected group; Identify the first codebook among the plurality of codebooks; For each of the multiple scaled input weight values ​​in the first selected group, the scaled input weight value is selected, and the centroid value is identified from the discrete set of centroid values ​​represented by the first codebook to represent the selected scaled input weight value. Encode the first codebook index representing the first codebook; as well as Encode the first plurality of centroid indices representing the centroid values, the centroid values ​​being identified as input weight values ​​representing the plurality of scaling in the first selected group; as well as For the second selected group among the multiple sets of input weight values: The second scaling factor is applied to the second selected group to produce scaled weight values ​​from the input weight values ​​in the second selected group; Identify the second codebook among the plurality of codebooks; For each of the multiple scaled input weight values ​​in the second selected group, the scaled input weight value is selected, and a centroid value is identified from the discrete set of centroid values ​​represented by the second codebook to represent the selected scaled input weight value. Encode the second codebook index that represents the second codebook; as well as The second plurality of centroid indices representing the centroid values ​​are encoded, and the centroid values ​​are identified as input weight values ​​representing the plurality of scaling in the second selected group.

3. A method for decompressing a set of output weight values ​​from a set of encoded weight values, the method comprising: Obtain a set of encoded weight values ​​corresponding to a set of output weight values, wherein the set of output weight values ​​includes multiple weight values ​​associated with the neural network; Select weight values ​​from the set of output weight values ​​that correspond to the weight values ​​of the multiple sets of output weight values; For the weight values ​​of the selected group of codes from the multiple groups of codes: The codebook index and multiple centroid indices are obtained from the weight values ​​of the encoding of the selected group; Identify one of a plurality of codebooks from the obtained codebook index for weight values ​​used in encoding the selected group, each codebook representing a discrete set of centroid values; Using the identified codebook, a plurality of centroid values ​​for weight values ​​used in the encoding of the selected group are identified from the plurality of centroid indices. The centroid values ​​are derived from a plurality of centroid values ​​derived from a discrete set of centroid values ​​represented by the identified codebook. as well as Apply the scaling factor to the selected group; as well as The set of output weight values ​​corresponds to the weight values ​​of the set of codes. For each set of weight values ​​in the plurality of sets of codes, the set of output weight values ​​includes weight values ​​corresponding to a plurality of centroid values ​​of the corresponding identification scaled by a corresponding scaling factor.

4. The method according to claim 3, wherein the method comprises: For the weight values ​​of the first selected group of codes among the multiple groups of codes: The first codebook index and the first plurality of centroid indices are obtained from the weight values ​​of the encoding of the first selected group; Identify the first codebook from the first codebook index; The first centroid values ​​are identified from the first plurality of centroid indices using the first codebook to determine the weight values ​​for the encoding of the first selected group. as well as Apply the first scaling factor to the selected group; as well as For the weight values ​​of the second selected group of codes among the multiple groups of codes: The second codebook index and the second and a plurality of centroid indices are obtained from the weight values ​​of the encoding of the second selected group; Identify the second codebook from the second codebook index; The second codebook is used to identify a second plurality of centroid values ​​from the second plurality of centroid indices for the weight values ​​of the encoding of the second selected group; as well as Apply the second scaling factor to the selected group.

5. The method according to claim 1 or claim 3, wherein the method comprises: Different scaling factors are applied, and for each of the multiple sets of input weight values, the codebook is identified from the codebook of the same group.

6. The method of claim 1 or claim 3, wherein the number of input weight values ​​in each group is an integer multiple of the width of the Single Instruction Multiple Data (SIMD) vector.

7. The method of claim 1 or claim 3, wherein the number of the plurality of codebooks is such that the plurality of codebooks are adapted to the processor's local register file.

8. The method according to claim 1 or claim 3, wherein the codebook comprises a codebook having a non-uniformly distributed centroid.

9. A method for generating multiple codebooks, the multiple codebooks being used to quantize multiple sets of input weight values ​​to generate multiple sets of encoded weight values, the method comprising: Obtain a set of input weight values, the set of output weight values ​​including multiple weight values ​​associated with one or more neural networks; Apply the grouping structure to the set of input weight values; The scaling factor is applied to the grouping structure of the corresponding group to generate the input weight value for scaling of each corresponding group; as well as For each of the multiple codebooks to be generated: Identify multiple similar distributions of the scaled input weight values ​​in different groups within the grouping structure of the corresponding groups; Clustering of the scaled input weight values ​​from the multiple similar distributions of different groups in the corresponding discrete clusters to identify a predetermined number of centroid values ​​from the clustered scaled input weight values; as well as Generate a codebook from the identified centroid values.

10. A processing element adapted to decompress a set of output weight values ​​from a set of encoded weight values, the processing element being adapted to: Obtain a set of encoded weight values ​​corresponding to a set of output weight values, wherein the set of output weight values ​​includes multiple weight values ​​associated with the neural network; Select weight values ​​from the set of output weight values ​​that correspond to the weight values ​​of the multiple sets of output weight values; For the weight values ​​of the selected group of codes from the multiple groups of codes: The codebook index and multiple centroid indices are obtained from the weight values ​​of the encoding of the selected group; Identify one of a plurality of codebooks from the obtained codebook index for weight values ​​used in encoding the selected group, each codebook representing a discrete set of centroid values; Using the identified codebook, a plurality of centroid values ​​for weight values ​​used in the encoding of the selected group are identified from the plurality of centroid indices. The centroid values ​​are derived from a plurality of centroid values ​​derived from a discrete set of centroid values ​​represented by the identified codebook. as well as Apply the scaling factor to the selected group; as well as The set of output weight values ​​corresponds to the weight values ​​of the set of codes. For each set of weight values ​​in the plurality of sets of codes, the set of output weight values ​​includes weight values ​​corresponding to a plurality of centroid values ​​of the corresponding identification scaled by a corresponding scaling factor.

Images