Vector quantization-based encoding cache methods, systems, electronic devices, and media

Abstract

This application provides a method, system, electronic device, and medium for caching codebooks based on vector quantization. The method includes: acquiring a codebook to be processed; sorting the codebook to be processed to obtain a target codebook index; obtaining an index boundary based on the target codebook index and a codebook cache; the codebook cache being a cache for storing the codebook to be processed; comparing the index boundary and the target codebook index to obtain a judgment result; and storing the codebook to be processed in the corresponding cache within the codebook cache based on the judgment result. This application improves memory performance and execution efficiency by placing codebook entries at different locations in the GPU's memory hierarchy based on the codebook's usage frequency, thus solving the problems of low efficiency in shared memory and global memory.

Description

Technical Field

[0001] This application belongs to the field of data caching technology, and relates to a codebook caching method based on vector quantization, and particularly to a codebook caching method, system, electronic device and medium based on vector quantization. Background Technology

[0002] With the immense success of Large Language Models (LLMs), neural networks are rapidly scaling, placing enormous strain on current hardware, especially memory systems. Quantization techniques have become crucial when deploying these large models. Quantization reduces the original IEEE-754 half-precision format FP16 data to narrower bit width types, such as FP8 and INT4, significantly reducing memory footprint. These techniques compress each data point as an independent element, ignoring the potential information between elements. Therefore, these methods typically only reach a compression limit of 4 bits; compression to 2 bits or less results in significant precision loss.

[0003] In these scenarios, Vector Quantization (VQ) emerges as a key technique to further reduce the memory footprint of Large Language Models (LLMs). VQ methods capture cross-element information by compressing vectors of multiple elements into a single element. Typically, this cross-element information is collected through clustering, specifically by applying a clustering algorithm to all vectors and using cluster centers to represent nearby vectors. Furthermore, some researchers suggest enhancing reconstruction quality by iteratively processing the residual between the original and quantized data. For large language models, VQ techniques can achieve higher accuracy at 4-bit precision, or maintain equivalent accuracy at 2-bit precision, and some methods can compress the key-value cache in large language models to 1 bit.

[0004] For vector quantization, codebooks are crucial. From an algorithmic perspective, codebooks not only determine how data is compressed and represented but also influence the system's representational capabilities, making them indispensable in vector quantization algorithms. Vector quantization algorithms quantize and dequantize vectors using pre-computed codebooks, a process involving significant memory accesses and data exchanges. Improper codebook storage and access can lead to shared memory conflicts and excessive global memory accesses, reducing the efficiency of parallel computing. Furthermore, mismatches between codebook access frequency and data flow can cause unnecessary memory bandwidth consumption, further increasing latency. Summary of the Invention

[0005] This application provides a codebook caching method, system, electronic device, and medium based on vector quantization, which solves the problems of low running performance and execution efficiency of existing codebook-based vector quantization algorithms.

[0006] In a first aspect, this application provides a codebook caching method based on vector quantization, the method comprising: obtaining a codebook to be processed; sorting the codebook to be processed to obtain a target codebook index; obtaining an index boundary based on the target codebook index and the codebook cache; the codebook cache being a cache for storing the codebook to be processed; performing a judgment comparison based on the index boundary and the target codebook index to obtain a judgment result; and storing the codebook to be processed in the corresponding cache in the codebook cache based on the judgment result.

[0007] In one implementation of the first aspect, sorting the codebook to be processed to obtain the target codebook index includes: sorting the codebook to be processed in descending order according to the access frequency of the codebook to be processed to obtain a first codebook entry of the codebook to be processed; and rearranging the codebook to be processed according to the first codebook entry to obtain the target codebook index of the codebook to be processed.

[0008] In one implementation of the first aspect, the index boundary includes a first index boundary and a second index boundary; the codebook cache includes a first cache, a second cache, and a third cache; the first cache is used to store codebooks in the codebook to be processed whose target codebook index is less than the first index boundary; the second cache is used to store codebooks in the codebook to be processed whose target codebook index is greater than or equal to the first index boundary and less than the second index boundary; and the third cache is used to store codebooks in the codebook to be processed whose target codebook index is greater than or equal to the second index boundary.

[0009] In one implementation of the first aspect, obtaining the index boundary based on the target codebook index and the codebook cache includes: obtaining the most frequently accessed codebook index and the least frequently accessed codebook index based on the target codebook index; establishing the index boundary based on the most frequently accessed codebook index, the least frequently accessed codebook index, and the codebook cache; wherein the most frequently accessed codebook index is less than a first index boundary of the index boundary; and the least frequently accessed codebook index is greater than or equal to a second index boundary of the index boundary.

[0010] In one implementation of the first aspect, the judgment comparison based on the index boundary and the target codebook index to obtain the judgment result includes: if the target codebook index in the codebook to be processed is less than the first index boundary, the judgment result is to store the codebooks in the codebook to be processed whose target codebook index is less than the first index boundary in the first cache.

[0011] In one implementation of the first aspect, the judgment comparison based on the index boundary and the target codebook index to obtain the judgment result includes: if the target codebook index in the codebook to be processed is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary, then the judgment result is to store the codebook in the codebook to be processed whose target codebook index is greater than or equal to the first index boundary and whose target codebook index is less than the second index boundary in the second cache.

[0012] In one implementation of the first aspect, the judgment comparison based on the index boundary and the target codebook index to obtain the judgment result includes: if the target codebook index in the codebook to be processed is greater than or equal to the second index boundary, the judgment result is to store the codebooks in the codebook to be processed whose target codebook index is greater than or equal to the second index boundary in the third cache.

[0013] Secondly, this application provides a codebook caching system based on vector quantization. The system includes: an acquisition module for acquiring a codebook to be processed; a sorting module for sorting the codebook to be processed to obtain a target codebook index; a boundary establishment module for obtaining an index boundary based on the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed; a judgment module for judging and comparing the index boundary and the target codebook index to obtain a judgment result; and a storage module for storing the codebook to be processed in the corresponding cache of the codebook cache based on the judgment result.

[0014] Thirdly, this application provides an electronic device, the electronic device comprising: a memory storing a computer program; and a processor communicatively connected to the memory, which executes the above-described vector quantization-based encoding cache method when the computer program is invoked.

[0015] Fourthly, this application provides a computer-readable storage medium storing a computer program thereon, which, when executed by an electronic device, implements the above-described vector quantization-based encoding cache method.

[0016] As described above, the vector quantization-based encoding cache method, system, electronic device, and medium of this application have the following beneficial effects:

[0017] This application obtains a target codec index by sorting the codecs to be processed; obtains an index boundary based on the target codec index and a codec cache; the codec cache is a cache for storing the codecs to be processed; a judgment comparison is performed based on the index boundary and the target codec index to obtain a judgment result; and the codecs to be processed are stored in the corresponding cache within the codec cache based on the judgment result. This application improves memory operation performance and execution efficiency by placing codec entries at different locations in the GPU's memory hierarchy based on the frequency of codec usage, thus solving the problems of low efficiency in shared memory and global memory.

[0018] This application solves the performance bottleneck problem of traditional vector quantization algorithms in large language model inference by combining the resource information of different GPUs and the specific access characteristics of the encoder when applied to different large language models with different vector quantization parameter configurations, dynamically storing data in the GPU memory hierarchy, optimizing data access patterns and reducing memory access latency. Attached Figure Description

[0019] Figure 1 The diagram shows an application scenario of the vector quantization process and optional parameter configuration described in the embodiments of this application.

[0020] Figure 2 The diagram shown is a flowchart illustrating the vector quantization-based encoding caching method described in an embodiment of this application.

[0021] Figure 3 The diagram shows the overall architecture combining hardware co-design, software system architecture, and user interface design as described in the embodiments of this application.

[0022] Figure 4 The diagram shown is a software flow diagram of the software system architecture described in the embodiments of this application.

[0023] Figure 5 The diagram shows the performance of the attention operator described in this application embodiment, which places the code in global memory and shared memory.

[0024] Figure 6 The diagram shown is a latency analysis of the shared memory codebook storage method described in this application embodiment.

[0025] Figure 7 This diagram illustrates the heat distribution of this entry encoded in the AQLM-3 algorithm described in this application embodiment.

[0026] Figure 8The image shown is an illustration of the acceleration effect of VQ CodeBook Cache as described in the embodiments of this application.

[0027] Figure 9 The diagram shown is a schematic representation of the structure of the vector quantization-based encoding cache system described in this application embodiment.

[0028] Figure 10 The diagram shown is a structural schematic of the electronic device described in an embodiment of this application.

[0029] Component designation explanation

[0030] 2. Encoding-based caching system based on vector quantization

[0031] 21 Acquisition Module

[0032] 22 Sorting Module

[0033] 23 Boundary Establishment Module

[0034] 24 Judgment Module

[0035] 25 Storage Modules

[0036] 3 Electronic devices

[0037] 31 Memory

[0038] 32 processors

[0039] Steps S1~Sn Detailed Implementation

[0040] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0041] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of this application. Therefore, the drawings only show the components related to this application and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0042] The following embodiments of this application provide a codebook caching method, system, electronic device, and medium based on vector quantization, which solves the problems of low running performance and execution efficiency of existing codebook-based vector quantization algorithms.

[0043] There are three main existing technical solutions for large-scale model vector quantization: codebook storage in global memory, codebook storage in shared memory, and manual composite tuning of vector quantization algorithms for different configurations. These three solutions will be discussed in more detail below.

[0044] In one embodiment, codebook storage in global memory involves storing the codebook directly in the GPU's global memory. Global memory is a memory region in the GPU architecture used to store large amounts of data, providing data access capabilities for all threads, thread blocks, and streaming multiprocessors (SMs). Under this storage strategy, the required codebook data is loaded from global memory when performing vector quantization or dequantization operations. Due to the large capacity of global memory, this method can easily handle large-scale codebook datasets. However, access to global memory is relatively slow, which can become a performance bottleneck, especially in scenarios requiring frequent access to codebook data.

[0045] In one embodiment, codebook storage in shared memory, compared to a global memory storage scheme, places codebook data in the GPU's shared memory. Shared memory is a fast on-chip memory residing on each SM (Software Module), allowing threads within the same thread block to efficiently share and access data. By storing codebook data in shared memory, data access latency can be significantly reduced because shared memory access is much faster than global memory. This approach is particularly suitable for applications requiring frequent and rapid access to codebook data, improving overall processing speed and efficiency. However, shared memory has limited capacity, and simultaneous access to shared memory by too many thread blocks can lead to resource contention and performance degradation.

[0046] In one embodiment, manual composite tuning of the VQ algorithm for different configurations is performed. The performance and efficiency of the VQ algorithm can be optimized by adjusting multiple parameters, including vector size, number of codebook entries, and number of quantization residuals. Manual composite tuning is a meticulous optimization method that requires adjusting and optimizing these parameters according to specific application requirements and hardware environment. This method involves a deep understanding of vector quantization algorithms and an accurate assessment of the performance impact under different configurations. Through manual tuning, the optimal quantization strategy can be customized for specific application scenarios to achieve the required performance and accuracy. However, the manual tuning process can be very time-consuming and complex, requiring specialized knowledge and extensive experience. Furthermore, as the algorithm and hardware environment change, continuous retuning may be necessary.

[0047] The three technical solutions described above—global memory codebook storage, shared memory codebook storage, and manual composite tuning of vector quantization algorithms for different configurations—offer varying levels of flexibility and performance, suitable for different application scenarios and needs. The global memory storage solution is suitable for situations with large amounts of codebook data but low access frequency, while the shared memory storage solution is suitable for scenarios requiring fast and frequent access to codebook data. Manual composite tuning, on the other hand, provides the possibility for advanced applications that require fine-grained control and optimization of VQ algorithm performance.

[0048] Although VQ compresses storage and computation through vector quantization, in unoptimized attention computation using vector quantization, regardless of whether the encoder is stored in global memory or shared memory, its performance is often inferior to FP16 attention computation without vector idealization. The reasons are analyzed below:

[0049] Storing the codebook in global memory reduces computational and storage costs, but the resulting speedup cannot offset the memory access overhead of reading the codebook from global memory. Global memory access is relatively slow, making memory access a bottleneck during vector quantization and dequantization operations. Furthermore, frequent readings of codebook data from global memory increase power consumption and latency, impacting overall computational efficiency. In large-scale parallel computing scenarios, this performance bottleneck may be even more pronounced, causing overall performance to fall short of expectations.

[0050] While storing codebooks in shared memory can significantly improve data access speed, this approach also has several drawbacks. First, shared memory has limited capacity; large codebooks can consume excessive resources, impacting the execution of other computational tasks. Second, too many threads accessing shared memory simultaneously can lead to resource contention and even conflicts, reducing memory access efficiency. Furthermore, excessive use of shared memory may limit the number of parallel threads, affecting GPU resource utilization.

[0051] Manual composite tuning of VQ algorithms for different configurations, while capable of customizing optimal quantization strategies for specific scenarios, has significant drawbacks. First, the manual tuning process is extremely time-consuming and complex, requiring specialized knowledge and extensive experience. This not only increases development and maintenance costs but also raises the technical barrier to entry. Second, as algorithms and hardware environments change, continuous retuning may be necessary, leading to additional workload. Furthermore, manual tuning often struggles to adapt to diverse VQ algorithms and hardware platforms, limiting its applicability. In rapidly iterating and ever-changing application environments, this method may fall short of meeting the demands for efficiency and flexibility.

[0052] From an algorithmic perspective, the codebook is indispensable in vector quantization because it not only determines how data is compressed and represented but also affects the system's representational capabilities. From an architecture performance optimization perspective, optimizing the codebook's storage location and access patterns (such as caching frequently accessed entries in registers) is key to improving the performance of vector quantization algorithms on general-purpose graphics processing units (GPUs).

[0053] This application aims to improve the performance and execution efficiency of large language models using a vector quantization algorithm based on the codebook through joint optimization of the algorithm and architecture. It is worth noting that the vector quantization algorithm has different parameter configurations in large language models.

[0054] Figure 1 The diagram illustrates an application scenario of the vector quantization process and optional parameter configuration described in this application. Figure 1 As shown, throughout the vector quantization process, the vector size, the number of encoded entries (#Entry), and the residual are configurable, denoted as VQ.<x,y,z> Where x, y, and z correspond to the vector width, the number of codebook entries, and the residual, respectively. The calculation and placement strategies for the optimal codebook differ under different parameter configurations. In particular, this application can adapt to various parameter configurations of the vector quantization algorithm.

[0055] The main innovation of this application lies in the method for determining the specific placement strategy of the codebook based on the GPU hardware resources and the parameter configuration of the codebook in a specific LLM vector quantization algorithm, as well as the user-friendly abstract interface (API) that interacts with both hardware and software. This application addresses the performance bottleneck of traditional vector quantization algorithms in large language model inference by combining the resource information of different GPUs and the specific access characteristics of the codebook when applied to different large language models with different vector quantization parameter configurations. It dynamically stores, optimizes data access patterns, and reduces memory access latency within the GPU memory hierarchy.

[0056] The technical solutions in the embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0057] like Figure 2 As shown, this embodiment provides a method for encoding and caching based on vector quantization, the method including the following steps S1 to S5.

[0058] Step S1: Obtain the codebook to be processed.

[0059] Specifically, a codebook is a set of finite vectors (called codewords or code vectors) used to approximate the input vectors in a k-dimensional vector space. In particular, each codeword is a k-dimensional vector, and the set of all codewords constitutes the codebook. The codebook is an indispensable part of vector quantization algorithms, reducing storage and processing costs by approximating high-dimensional input data as low-dimensional codewords.

[0060] Step S2: Sort the codebooks to be processed to obtain the target codebook index.

[0061] Specifically, the target codebook index is an index of codebook entries obtained by sorting and rearranging the codebooks to be processed in descending order. The sorting of the codebooks to be processed to obtain the target codebook index is performed during the offline phase of the analysis.

[0062] In one embodiment, this application ensures that the index of the most frequently accessed entry is 0 and the index of the least frequently accessed entry is the maximum value by sorting the codebooks to be processed to obtain target codebook indices. All quantized data will use these new target codebook indices.

[0063] In one embodiment of this application, sorting the codebook to be processed to obtain the target codebook index includes the following steps S21 to S22.

[0064] Step S21: Sort the codebook to be processed in descending order according to the access frequency of the codebook to be processed, so as to obtain the first codebook entry of the codebook to be processed.

[0065] Step S22: Rearrange the codebook to be processed according to the first codebook entry to obtain the target codebook index of the codebook to be processed.

[0066] Step S3: Obtain the index boundary based on the target codebook index and the codebook cache. The codebook cache is a cache that stores the codebook to be processed.

[0067] Specifically, the Codebook Cache is designed in collaboration with the GPU hardware, taking into account the varying access latency characteristics of different memory levels within the GPU. The Codebook Cache stores codebooks at different memory levels based on the access frequency of different entries. Specifically, this application saves on shared memory usage by storing infrequently accessed codebook entries in off-chip global memory, and stores the most frequently used entries in thread-local registers to eliminate shared memory conflicts.

[0068] In one embodiment of this application, the index boundary includes a first index boundary and a second index boundary; the encoded cache includes a first cache, a second cache, and a third cache.

[0069] The first cache is used to store encodings whose target encoding index is less than the first index boundary.

[0070] The second cache is used to store encodings in the encodings to be processed whose target encoding index is greater than or equal to the first index boundary and less than the second index boundary.

[0071] The third cache is used to store encodings in the encodings to be processed whose target encoding index is greater than or equal to the second index boundary.

[0072] In one embodiment of this application, obtaining the index boundary based on the target codebook index and the codebook cache includes the following steps S31 to S32.

[0073] Step S31: Obtain the most frequently accessed codebook index and the least frequently accessed codebook index based on the target codebook index.

[0074] Step S32: Establish the index boundary based on the most frequently accessed codebook index, the least frequently accessed codebook index, and the codebook cache; wherein, the most frequently accessed codebook index is less than the first index boundary of the index boundary; and the least frequently accessed codebook index is greater than or equal to the second index boundary of the index boundary.

[0075] Step S4: Compare and judge based on the index boundary and the target encoded index to obtain the judgment result.

[0076] In one embodiment of this application, the judgment and comparison based on the index boundary and the target codebook index to obtain the judgment result includes step S41A: if the target codebook index in the codebook to be processed is less than the first index boundary, the judgment result is to store the codebooks in the codebook to be processed whose target codebook index is less than the first index boundary in the first cache.

[0077] In one embodiment of this application, the judgment comparison based on the index boundary and the target codebook index to obtain the judgment result includes step S41B: if the target codebook index in the codebook to be processed is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary, then the judgment result is to store the codebook in the codebook to be processed whose target codebook index is greater than or equal to the first index boundary and whose target codebook index is less than the second index boundary in the second cache.

[0078] In one embodiment of this application, the judgment comparison based on the index boundary and the target codebook index to obtain the judgment result includes step S41C: if the target codebook index in the codebook to be processed is greater than or equal to the second index boundary, the judgment result is to store the codebooks in the codebook to be processed whose target codebook index is greater than or equal to the second index boundary in the third cache.

[0079] Step S5: Based on the judgment result, store the codebook to be processed in the corresponding cache in the codebook cache.

[0080] This application obtains a target codec index by sorting the codecs to be processed; obtains an index boundary based on the target codec index and a codec cache; the codec cache is a cache for storing the codecs to be processed; a judgment comparison is performed based on the index boundary and the target codec index to obtain a judgment result; and the codecs to be processed are stored in the corresponding cache within the codec cache based on the judgment result. This application improves memory operation performance and execution efficiency by placing codec entries at different locations in the GPU's memory hierarchy based on the frequency of codec usage, thus solving the problems of low efficiency in shared memory and global memory.

[0081] The main innovation of this application lies in the method for determining the specific placement strategy of the codebook based on the GPU hardware resources and the parameter configuration of the codebook in a specific LLM vector quantization algorithm, as well as the user-friendly abstract interface (API) that interacts with both hardware and software. This application addresses the performance bottleneck of traditional vector quantization algorithms in large language model inference by combining the resource information of different GPUs and the specific access characteristics of the codebook when applied to different large language models with different vector quantization parameter configurations. It dynamically stores, optimizes data access patterns, and reduces memory access latency within the GPU memory hierarchy.

[0082] In one embodiment, this application describes in detail the specific design and implementation of this application through hardware co-design, software system architecture and user interface design.

[0083] Figure 3The diagram shows the overall architecture combining hardware co-design, software system architecture, and user interface design as described in the embodiments of this application. Figure 3 As shown, the overall architecture mainly consists of three parts: hardware co-design, software system design, and user interface design. Hardware co-design forms the practical basis of this application, combining hardware resources and performance characteristics to achieve hierarchical and adaptive codebook storage. Software system design is the specific algorithm for adaptive hierarchical storage. User interface design shields the underlying implementation, providing users with a unified view of the codebook cache. The following sections will elaborate on these three aspects: hardware co-design, software system design, and user interface design.

[0084] In one embodiment, this application, based on the varying access latency characteristics of GPU memory levels, co-designs a codebook cache with the GPU hardware. The codebook cache can store entries at different memory levels according to their access frequency. Specifically, this application saves shared memory usage by storing infrequently accessed codebook entries in off-chip global memory, and eliminates shared memory conflicts by storing the most frequently used entries in thread-local registers.

[0085] In one embodiment, the cache implementation relies on a tagged array or look-up table, which may increase latency and storage overhead. In the cache implementation of this application, a reordering-based static mapping mechanism is employed (see [reference]). Figure 4 As shown in the figure, this mechanism is very lightweight and configurable.

[0086] Figure 4 The diagram shown is a software flow diagram illustrating the software system architecture described in an embodiment of this application. For example... Figure 4 As shown, the software process implementation method of the software system architecture in this application mainly includes the following steps S61 to S64.

[0087] Step S61: This application first sorts the codebook entries in descending order according to access frequency to obtain codebook entries ordered by access frequency. The sorting of the codebook entries is completed in the offline analysis phase.

[0088] Step S62: Then, the sorted code entries are rearranged offline. The sorting and rearrangement of code entries are completed in the analysis-based offline phase, thus ensuring that the index of the most frequently accessed entries is 0 and the index of the least frequently accessed entries is the maximum value. All quantified data will use these new indices.

[0089] Step S63: Next, establish two index boundaries: Nreg and Nshared. In this application, the index boundaries are established based on the hardware resources such as the CodeBookCache obtained by co - designing with the GPU according to the different access latencies of the GPU's memory hierarchy and the codebook entries after offline rearrangement. Regarding the meanings of the two boundaries Nreg and Nshared, in this application, the first Nreg entries are allocated to thread - local registers, the subsequent entries up to Nshared are stored in shared memory, and any remaining entries are stored in global memory.

[0090] Step S64: For each codebook entry, during the de - quantization process at runtime, locating the codebook entry involves simple index comparison. If the index < Nreg, this application locates the codebook entry in the register. If Nreg <= index < Nshared, this application locates the codebook entry in shared memory. If the index ≥ Nshared, the codebook entry is located in global memory.

[0091] In one embodiment, this application provides the following APIs for users to use the "codebook cache" (hereinafter referred to as CB) described in the embodiments of this application. The API explanations are as follows:

[0092] CB cached ,n reg，shared ←Load(CB,Slack)

[0093] Entry←Access(CB cached ,n eg，shared ,CB,Index)

[0094] CB←Switch(New CB Pointer)

[0095] The first API is Load(CB, Slack). The Load API loads the codebook stored in global memory into the cache. The Load API accepts both the codebook and the memory slack, and returns the codebook cached across the memory hierarchy and two access boundaries. The Load API performs initial offline sorting of access frequencies and offline rearrangement of codebook entries based on the specific content of the codebook and the memory slack information in the hardware co - design, generating an initialized codebook cache configuration scheme for subsequent access.

[0096] The second API is Access(CB cached ,n reg,sharedThe Access API (CB, Index) allows users to access specific entries during the dequantization process. The Access API accepts cached codebooks and codebooks stored in global memory, along with an index used to locate entries. It also uses two boundaries to determine entry positions. Furthermore, although this application uses a pre-defined heuristic to configure these boundaries, users can still override them. The Access API performs online indexing based on the existing codebook caching configuration and returns the results. Then, based on the specific entry accessed by the user, it increases the corresponding codebook entry index frequency and dynamically updates the codebook index boundaries.

[0097] The last API is Switch, which is very useful when training different codebooks for different parts of a tensor. The Switch API conveniently switches to a new codebook based on the specific tensor part the user is working with. The Switch API automatically saves the current codebook's metadata (including hardware co-configuration information and codebook index boundaries) and the codebook itself to a specific area of ​​Global Memory, and then switches the new codebook into the codebook cache. For the new codebook cache, the Switch API uses a similar approach to the Load API to initialize and configure the new codebook.

[0098] Wherein, CB represents the codebook itself in the input codebook cache; the cached index indicates that the codebook has already been processed by the "codebook" cache; nreg represents the register index boundary found during the execution of the Load API, shared represents the shared memory index boundary found during the execution of the Load API; reg represents a register; shared represents GPU shared memory; slack represents the memory slack factor; Entry represents the codebook entry returned by accessing the codebook cache through the Access API; Index represents the codebook entry number to be accessed; and New CB Pointer represents the pointer to the new codebook to which the access needs to be made.

[0099] In one embodiment, this application uses an attention operator in Llamma-7B with parameters configured as 32 heads, 128 dimensions, and VQ<4,8,1> to illustrate the optimization principle of this application. Llamma-7B is a large-scale foundational language model released by Meta AI.

[0100] In existing VQ-LLM operator implementations, most logarithmic methods store the entire codebook in global memory. This leads to slow global memory communication time dominating operator latency, resulting in poor performance. More advanced shared memory-based optimization methods store the entire massive codebook in shared memory, failing to consider the access frequency of codebook entries or fully utilize registers as a high-level, fast storage medium. Shared memory-based methods are primarily constrained by shared memory conflicts and communication bottlenecks between shared memory and global memory, and between shared memory and registers. The VQ-LLM operator, however, is a method that optimizes the inference efficiency and performance of large language model (LLM) systems using vector quantization (VQ) techniques.

[0101] Figure 5 The diagram shows the performance of the attention operator described in this application embodiment, which places the code in global memory and shared memory. For example... Figure 5 As shown, VQ-attn-GC / SC are attention operators that place the codec in global memory and shared memory, respectively. Figure 5 This indicates that in a real-world production environment, both VQ algorithms are slower than the baseline version using the FP16 data format. VQ-attn-GC / SC is a variant of the attention mechanism based on vector quantization, primarily used to optimize the inference efficiency and performance of large language models (LLMs).

[0102] according to Figure 5 It can be seen that the poor performance of the global memory version of the operator is due to the low global memory bandwidth; placing the code in global memory makes access to global memory a bottleneck. According to... Figure 6 As is known, the poor performance of shared memory version operators is due to a combination of factors caused by the specific access frequency of the codebook, including low SM utilization, low shared memory utilization, shared memory bank conflicts, shared memory-global memory communication bottlenecks, and shared memory-register communication bottlenecks. Figure 6 The diagram shown is a latency analysis of the shared memory codebook storage method described in this application embodiment.

[0103] Because shared memory resources are limited, storing a large number of codebook entries can lead to a surge in shared memory usage, which limits the number of thread blocks that can run simultaneously on each streaming multiprocessor (SM), thereby reducing the utilization of GPU computing resources. Figure 6As shown, the shared-memory codebook storage method exhibits a computational utilization decrease of over 30% compared to the FP16 benchmark, primarily due to excessive shared memory usage. The random codebook entry access patterns in the shared-memory codebook storage method lead to significant shared-memory bank conflicts. These bank conflicts severely hinder the parallelism of memory accesses and increase memory access latency. In the shared-memory codebook storage method, the number of codebook entries far exceeds the number of shared-memory banks, and the access patterns are random, making it extremely difficult to reduce bank conflicts using common reordering or padding strategies. Furthermore, the numerous register accesses and data accesses between shared memory and global memory in the shared-memory codebook storage method significantly impact vector quantization performance.

[0104] This application addresses the inefficiency of shared memory and global memory by placing codebook entries at different locations in the GPU's memory hierarchy based on the frequency of codebook usage.

[0105] Figure 7 This diagram illustrates the heat distribution of this entry encoded using the AQLM-3 algorithm described in this application embodiment. Figure 7 As shown, different entries in the codebook do indeed exhibit varying degrees of "popularity" in terms of access frequency. Figure 7 This example demonstrates an implementation of the AQLM-3 algorithm. Over half of the code entries have an access frequency below average, indicating that placing these low-frequency code entries in shared memory offers little benefit. Twenty-six popular code entries have an access frequency above the average plus three standard deviations (mean plus three standard deviations), suggesting these 26 entries are more susceptible to unavoidable shared memory conflicts. Here, "code entry" refers to the code itself. The AQLM-3 algorithm is an advanced large-scale language model (LLM) compression technique based on additive quantization (AQ). It boasts advantages such as efficient inference, accuracy preservation, and high flexibility, and is widely used in terminal device deployments and various task support applications.

[0106] in accordance with Figure 3The overall structure of this application is shown. This application evaluates the vector quantization-based encoding cache method described in this application on an NVIDIA RTX 4090 24GB GPU. The evaluated computational kernels include various VQ-enhanced GeMM (matrix multiplication), GeMV (matrix-vector multiplication), and FlashDecoding. The evaluated VQ configurations include QuiP#-4, AQLM-3, GPTVQ-2, and CQ-2 / 4, where the suffix numbers indicate the equivalent bit width. The first two kernels use weighted quantization, and the last kernel uses KV cache quantization. This application sets the shape of these kernels according to the Llama-7B and Llama-65B models.

[0107] It's important to note that QuiP#-4 is a model compression technique based on Quantization-aware Training (QAT). It simulates the quantization process during training, allowing the model to better adapt to the accuracy loss caused by quantization during inference. AQLM-3 is a large language model (LLM) compression technique based on Additive Quantization (AQ). It represents data points by the sum of multiple codebook vectors, achieving efficient model compression. GPTVQ-2 is a technique based on Generative Product of Experts Vector Quantization (GPTQ). It combines generative models and vector quantization to improve the model's inference efficiency and performance. CQ-2 / 4 is a technique combining contrastive learning and quantization. It learns data representation through contrastive learning and reduces model storage requirements and computational complexity through quantization.

[0108] Figure 8 The image shown is an illustration of the acceleration effect of VQ CodeBook Cache as described in the embodiments of this application. Figure 8 As shown, the figure illustrates the latency reduction of the optimized version compared to the unoptimized version under various VQ configurations. The vertical axis represents the percentage of latency reduction, and the horizontal axis shows the specific configurations for different kernels and batch sizes, including tasks such as GeMM, GeMV, and Attention. The bar charts in the figure represent the performance of the Llama-7B and Llama-65B models, with solid sections representing Llama-7B and diagonally filled sections representing Llama-65B.

[0109] In the computationally intensive GeMM (matrix multiplication) operation, the latency reduction is significant across different configurations (such as QuiP#-4, AQLM-3, and GPTVQ-2). Specifically, the latency reduction of Llama-65B is close to 60%, while that of Llama-7B is around 50%. Particularly noteworthy is the high latency reduction of Llama-65B, reaching approximately 65% ​​in the QuiP#-4 configuration, significantly outperforming Llama-7B's approximately 55%.

[0110] In the memory-intensive GeMV (matrix-vector multiplication) operation, tests were conducted at batch sizes of 1 (BS1) and 16 (BS16). The Llama-65B showed a more significant latency reduction of approximately 45% at a batch size of 16, while the Llama-7B maintained a reduction of approximately 40%. Furthermore, at a batch size of 1, the difference in latency reduction was smaller, with the Llama-65B showing approximately 35% and the Llama-7B approximately 30%.

[0111] In operational tests using the Attention mechanism, latency reductions were observed at sequence lengths of 1k and 4k. At a sequence length of 1k, Llama-65B showed a latency reduction of nearly 40%, while Llama-7B saw a reduction of approximately 35%. At a longer sequence length of 4k, the latency reduction of Llama-65B further increased to approximately 50%, while Llama-7B saw a reduction of approximately 45%.

[0112] Overall, on average, the Llama-65B reduces latency by approximately 50% across all configurations, while the Llama-7B reduces latency by approximately 45%. This indicates that the Llama-65B's optimization becomes more significant with increasing batch size and sequence length in weighted quantization and KV cached quantization tasks, demonstrating a clear performance advantage in handling computationally intensive and memory-intensive tasks.

[0113] It should be noted that LLaMA-7B and LLaMA-65B are two different versions of the LLaMA series models released by Meta AI, and they differ significantly in terms of the number of parameters, the amount of training data, and application scenarios.

[0114] LLaMA-7B Parameter Count: LLaMA-7B has 7 billion parameters. Training Data Volume: LLaMA-7B is trained on approximately 1 trillion tokens. Application Scenarios: Due to its smaller size and lower computational resource requirements, LLaMA-7B may be more suitable for running in resource-constrained environments, such as on a single GPU. It is suitable for applications that require high performance but have limited budgets, such as personal research projects, small businesses, or individual developers.

[0115] Number of parameters in LLaMA-65B: LLaMA-65B has 65 billion parameters. Training data volume: LLaMA-65B is trained on a much larger dataset. While the exact number of tokens isn't directly mentioned, the entire LLaMA series training dataset contains approximately 1.4TB of tokens after tokenization. Application scenarios: Due to its massive scale and powerful performance, LLaMA-65B is better suited for large-scale tasks requiring high performance and accuracy, such as complex natural language processing tasks and large-scale data analysis. It performs exceptionally well in multiple benchmark tests and is highly competitive compared to the industry-leading models Chinchilla-70B and PaLM-540B.

[0116] The scope of protection of the vector quantization-based encoding caching method described in this application is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principles of this application is included within the scope of protection of this application.

[0117] This application also provides a codebook caching system based on vector quantization. The codebook caching system based on vector quantization can implement the codebook caching method based on vector quantization described in this application. However, the implementation device of the codebook caching method based on vector quantization described in this application includes, but is not limited to, the structure of the codebook caching system based on vector quantization listed in this embodiment. All structural modifications and substitutions of the prior art made in accordance with the principles of this application are included within the protection scope of this application.

[0118] like Figure 9 As shown, this embodiment provides an encoding cache system based on vector quantization. The system 2 includes: an acquisition module 21, a sorting module 22, a boundary establishment module 23, a judgment module 24, and a storage module 25.

[0119] The acquisition module 21 is used to acquire the codebook to be processed;

[0120] The sorting module 22 is used to sort the codebooks to be processed to obtain the target codebook index;

[0121] The boundary establishment module 23 is used to obtain the index boundary based on the target encoding index and the encoding cache; the encoding cache is a cache that stores the encoding to be processed;

[0122] The judgment module 24 is used to perform a judgment comparison based on the index boundary and the target encoded index to obtain a judgment result;

[0123] The storage module 25 is used to store the codebook to be processed in the corresponding cache in the codebook cache according to the judgment result.

[0124] It should be noted that the functions or operations of the acquisition module 21, sorting module 22, boundary establishment module 23, judgment module 24 and storage module 25 described in the embodiments of this application correspond one-to-one with the steps in the above-described caching method for encoding based on vector quantization, and therefore will not be repeated here.

[0125] In the embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0126] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of this application, depending on actual needs. For example, the functional modules / units in the various embodiments of this application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0127] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0128] like Figure 10 As shown, this embodiment provides an electronic device, the electronic device 3 including: a memory 31 and a processor 32.

[0129] The memory 31 stores a computer program;

[0130] The processor 32 is communicatively connected to the memory 31, and executes the above-described vector quantization-based encoding cache method when calling the computer program.

[0131] This application also provides a computer-readable storage medium. Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0132] This application embodiment may also provide a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application embodiment are generated. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0133] When the computer program product is executed by a computer, the computer performs the method described in the foregoing method embodiments. The computer program product can be a software installation package; when the foregoing method is required, the computer program product can be downloaded and executed on the computer.

[0134] In summary, the vector quantization-based encoding buffer method, system, electronic device, and medium described in this application have the following beneficial effects:

[0135] This application obtains a target codec index by sorting the codecs to be processed; obtains an index boundary based on the target codec index and a codec cache; the codec cache is a cache for storing the codecs to be processed; a judgment comparison is performed based on the index boundary and the target codec index to obtain a judgment result; and the codecs to be processed are stored in the corresponding cache within the codec cache based on the judgment result. This application improves memory operation performance and execution efficiency by placing codec entries at different locations in the GPU's memory hierarchy based on the frequency of codec usage, thus solving the problems of low efficiency in shared memory and global memory.

[0136] This application solves the performance bottleneck problem of traditional vector quantization algorithms in large language model inference by combining the resource information of different GPUs and the specific access characteristics of the encoder when applied to different large language models with different vector quantization parameter configurations, dynamically storing data in the GPU memory hierarchy, optimizing data access patterns and reducing memory access latency.

[0137] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0138] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A method for encoding and caching based on vector quantization, characterized in that, The method includes: Obtain the codebook to be processed; The codebooks to be processed are sorted to obtain a target codebook index; the codebooks to be processed are sorted in descending order according to their access frequency to obtain a first codebook entry; the codebooks to be processed are rearranged according to the first codebook entry to obtain a target codebook index. The index boundaries are obtained based on the target codebook index and the codebook cache; the most frequently accessed codebook index and the least frequently accessed codebook index are obtained based on the target codebook index; the index boundaries are established based on the most frequently accessed codebook index, the least frequently accessed codebook index, and the codebook cache; wherein, the most frequently accessed codebook index is less than a first index boundary of the index boundaries; the least frequently accessed codebook index is greater than or equal to a second index boundary of the index boundaries; the index boundaries include the first index boundary and the second index boundary; the codebook cache is a cache for storing the codebooks to be processed; The judgment result is obtained by comparing the index boundary and the target encoded index. Based on the judgment result, the codebook to be processed is stored in the corresponding cache in the codebook cache.

2. The encoding cache method based on vector quantization according to claim 1, characterized in that, The encoded cache includes a first cache, a second cache, and a third cache; The first cache is used to store encodings in the encodings to be processed whose target encoding index is less than the first index boundary; The second cache is used to store encodings in the encodings to be processed whose target encoding index is greater than or equal to the first index boundary and less than the second index boundary; The third cache is used to store encodings whose target encoding index is greater than or equal to the second index boundary.

3. The encoding cache method based on vector quantization according to claim 2, characterized in that, The judgment result is obtained by comparing the index boundary and the target encoded index. If the target codebook index in the codebook to be processed is less than the first index boundary, then the determination result is to store the codebooks in the codebook to be processed whose target codebook index is less than the first index boundary in the first cache.

4. The encoding cache method based on vector quantization according to claim 2, characterized in that, The judgment result is obtained by comparing the index boundary and the target encoded index. If the target codebook index in the codebook to be processed is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary, then the determination result is to store the codebook in the codebook to be processed whose target codebook index is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary in the second cache.

5. The encoding cache method based on vector quantization according to claim 2, characterized in that, The judgment result is obtained by comparing the index boundary and the target encoded index. If the target codebook index in the codebook to be processed is greater than or equal to the second index boundary, the determination result is to store the codebooks in the codebook to be processed whose target codebook index is greater than or equal to the second index boundary in the third cache.

6. A codebook caching system based on vector quantization, characterized in that, The system includes: The acquisition module is used to acquire the codebook to be processed. The sorting module is used to sort the codebook to be processed to obtain a target codebook index; sort the codebook to be processed in descending order according to the access frequency of the codebook to be processed to obtain a first codebook entry of the codebook to be processed; and rearrange the codebook to be processed according to the first codebook entry to obtain a target codebook index of the codebook to be processed. A boundary establishment module is used to obtain an index boundary based on the target codebook index and the codebook cache; obtain the most frequently accessed codebook index and the least frequently accessed codebook index based on the target codebook index; and establish the index boundary based on the most frequently accessed codebook index, the least frequently accessed codebook index, and the codebook cache; wherein the most frequently accessed codebook index is less than a first index boundary of the index boundary; the least frequently accessed codebook index is greater than or equal to a second index boundary of the index boundary; the index boundary includes a first index boundary and a second index boundary; and the codebook cache is a cache for storing the codebooks to be processed. The judgment module is used to perform a judgment comparison based on the index boundary and the target encoded index to obtain a judgment result; The storage module is used to store the codebook to be processed in the corresponding cache in the codebook cache according to the judgment result.

7. An electronic device, characterized in that, The electronic device includes: A memory that stores a computer program; The processor, which is communicatively connected to the memory, executes the encoding cache method based on vector quantization as described in any one of claims 1 to 5 when calling the computer program.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by an electronic device, the program implements the vector quantization-based encoding cache method as described in any one of claims 1 to 5.

Images