Codebook caching method and system based on vector quantization, electronic equipment and medium

Abstract

The invention provides a vector quantization-based codebook caching method and system, electronic equipment and a medium. The method comprises the following steps of: obtaining a to-be-processed codebook; sorting the to-be-processed codebooks to obtain a target codebook index; obtaining an index boundary according to the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed; performing judgment and comparison according to the index boundary and the target codebook index to obtain a judgment result; and storing the to-be-processed codebook in a corresponding cache in the codebook cache according to the judgment result. According to the use frequency of the codebook, the entries of the codebook are placed at different positions of the memory hierarchy of the GPU, so that the memory operation performance and the execution efficiency are improved, and the problem of low efficiency of a shared memory and a global memory is solved.

Description

Technical Field

[0001] The present application belongs to the technical field of data caching, and relates to a codebook caching method based on vector quantization, and in particular to a codebook caching method, system, electronic device and medium based on vector quantization. Background Art

[0002] With the huge success of Large Language Models (LLMs), neural networks are rapidly scaling, placing tremendous pressure on current hardware, especially memory systems. Quantization techniques become critical when deploying these large models. Quantization reduces the original IEEE-754 half-precision format FP16 data to narrower bit width types such as FP8 and INT4, significantly reducing memory usage. These techniques treat each data point as an independent element for compression, ignoring the potential information between elements. As a result, these methods typically only achieve a compression limit of 4 bits; compression to 2 bits or less can result in significant accuracy loss.

[0003] In these scenarios, vector quantization (VQ) emerges as a key technology to further reduce the memory footprint of large language models (LLM). Vector quantization methods capture cross-element information by compressing vectors of multiple elements into a single element. Typically, this cross-element information is collected through clustering, where a clustering algorithm is applied to all vectors and the cluster centers are used to represent nearby vectors. In addition, some researchers have suggested enhancing the quality of reconstruction by iteratively processing the residual between the original data and the quantized data. For large language models, vector quantization technology can achieve higher accuracy at 4 bits of precision, or maintain equivalent accuracy at 2 bits of precision, and some methods are able to compress the KV cache in large language models to 1 bit.

[0004] Codebooks are crucial for vector quantization. From an algorithmic perspective, codebooks not only determine how data is compressed and represented, but also affect the system's representation capabilities, and are indispensable in vector quantization algorithms. Vector quantization algorithms quantize and dequantize vectors using pre-computed codebooks, a process that involves a large amount of memory access and data exchange. If the codebook is not stored and accessed properly, it will cause shared memory conflicts and excessive global memory accesses, thereby reducing the efficiency of parallel computing. In addition, the inconsistency between the access frequency of the codebook and the data flow will cause unnecessary memory bandwidth consumption, further increasing latency. Summary of the invention

[0005] The present application provides a codebook caching method, system, electronic device and medium based on vector quantization, which are used to solve the problems of low operating performance and execution efficiency of codebook-based vector quantization algorithms in the prior art.

[0006] In a first aspect, the present application provides a codebook cache method based on vector quantization, the method comprising: obtaining a codebook to be processed; sorting the codebook to be processed to obtain a target codebook index; obtaining an index boundary according to the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed; performing a judgment and comparison based on the index boundary and the target codebook index to obtain a judgment result; storing the codebook to be processed in a corresponding cache in the codebook cache according to the judgment result.

[0007] In an implementation manner of the first aspect, sorting the codebook to be processed to obtain a target codebook index includes: sorting the codebook to be processed in descending order according to an access frequency of the codebook to be processed to obtain a first codebook entry of the codebook to be processed; and rearranging the codebook to be processed according to the first codebook entry to obtain a target codebook index of the codebook to be processed.

[0008] In an implementation of the first aspect, the index boundary includes a first index boundary and a second index boundary; the codebook cache includes a first cache, a second cache and a third cache; the first cache is used to store codebooks in which the target codebook index in the codebook to be processed is less than the first index boundary; the second cache is used to store codebooks in which the target codebook index in the codebook to be processed is greater than or equal to the first index boundary and less than the second index boundary; the third cache is used to store codebooks in which the target codebook index in the codebook to be processed is greater than or equal to the second index boundary.

[0009] In an implementation of the first aspect, obtaining an index boundary according to the target codebook index and the codebook cache includes: obtaining a most frequently accessed codebook index and a least frequently accessed codebook index according to the target codebook index; establishing the index boundary according to the most frequently accessed codebook index, the least frequently accessed codebook index and the codebook cache; wherein the most frequently accessed codebook index is less than a first index boundary of the index boundary; and the least frequently accessed codebook index is greater than or equal to a second index boundary of the index boundary.

[0010] In an implementation of the first aspect, performing a judgment and comparison based on the index boundary and the target codebook index to obtain a judgment result includes: if the target codebook index in the codebook to be processed is less than the first index boundary, the judgment result is to store the codebook in the codebook to be processed whose target codebook index is less than the first index boundary in the first cache.

[0011] In an implementation of the first aspect, performing a judgment and comparison based on the index boundary and the target codebook index to obtain a judgment result includes: if the target codebook index in the codebook to be processed is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary, then the judgment result is to store the codebook in the codebook to be processed whose target codebook index is greater than or equal to the first index boundary and whose target codebook index is less than the second index boundary in the second cache.

[0012] In an implementation of the first aspect, performing a judgment and comparison based on the index boundary and the target codebook index to obtain a judgment result includes: if the target codebook index in the codebook to be processed is greater than or equal to the second index boundary, then the judgment result is to store the codebook whose target codebook index in the codebook to be processed is greater than or equal to the second index boundary in the third cache.

[0013] In a second aspect, the present application provides a codebook cache system based on vector quantization, the system comprising: an acquisition module for acquiring a codebook to be processed; a sorting module for sorting the codebook to be processed to obtain a target codebook index; a boundary establishment module for obtaining an index boundary based on the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed; a judgment module for performing a judgment and comparison based on the index boundary and the target codebook index to obtain a judgment result; a storage module for storing the codebook to be processed in a corresponding cache in the codebook cache based on the judgment result.

[0014] In a third aspect, the present application provides an electronic device, comprising: a memory storing a computer program; and a processor communicatively connected to the memory, for executing the above-mentioned vector quantization-based codebook caching method when calling the computer program.

[0015] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by an electronic device, implements the above-mentioned vector quantization-based codebook caching method.

[0016] As described above, the codebook caching method, system, electronic device and medium based on vector quantization described in the present application have the following beneficial effects:

[0017] The present application sorts the codebook to be processed to obtain the target codebook index; obtains the index boundary according to the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed; makes a judgment and comparison according to the index boundary and the target codebook index to obtain a judgment result; stores the codebook to be processed in the corresponding cache in the codebook cache according to the judgment result. The present application improves the memory operation performance and execution efficiency by placing the codebook entries at different locations of the GPU's memory hierarchy according to the frequency of use of the codebook, and solves the problem of low efficiency of shared memory and global memory.

[0018] This application combines resource information of different GPUs and specific access characteristics of the codebook when different vector quantization parameter configurations are applied to different large language models, dynamically stores and optimizes data access patterns and reduces memory access latency in the GPU memory hierarchy, thereby solving the performance bottleneck problem of traditional vector quantization algorithms in large language model reasoning. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Shown is a schematic diagram of an application scenario of the vector quantization process and optional parameter configuration described in an embodiment of the present application.

[0020] Figure 2 Shown is a flow chart of a codebook caching method based on vector quantization described in an embodiment of the present application.

[0021] Figure 3 Shown is a schematic diagram of the overall architecture combining the hardware collaborative design, software system architecture and user interface design described in an embodiment of the present application.

[0022] Figure 4 Shown is a software flow diagram of the software system architecture described in an embodiment of the present application.

[0023] Figure 5 The following is a diagram showing the performance effect of the attention operator when the codebook is placed in the global memory and the shared memory as described in an embodiment of the present application.

[0024] Figure 6 Shown is a time delay diagram of the shared memory codebook storage method described in an embodiment of the present application.

[0025] Figure 7 Shown is a schematic diagram of the heat distribution of the items encoded in the AQLM-3 algorithm described in the embodiment of the present application.

[0026] Figure 8Shown is a diagram of the VQ CodeBook Cache acceleration effect described in an embodiment of the present application.

[0027] Fig. 9 Shown is a schematic diagram of the structure of a codebook cache system based on vector quantization described in an embodiment of the present application.

[0028] Fig.10 Shown is a schematic diagram of the structure of an electronic device described in an embodiment of the present application.

[0029] Component number description

[0030] 2 Codebook Cache System Based on Vector Quantization

[0031] 21 Get Module

[0032] 22 Sorting Module

[0033] 23 Boundary Establishment Module

[0034] 24 Judgment module

[0035] 25 Storage Module

[0036] 3 Electronic devices

[0037] 31 Memory

[0038] 32 processors

[0039] S1~Sn Steps DETAILED DESCRIPTION

[0040] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0041] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application, and thus the drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed at will, and the component layout may also be more complicated.

[0042] The following embodiments of the present application provide a codebook caching method, system, electronic device and medium based on vector quantization, which solve the problems of low operating performance and execution efficiency of codebook-based vector quantization algorithms in the prior art.

[0043] There are three main technical solutions for large model vector quantization, namely codebook storage in global memory, codebook storage in shared memory, and manual compound tuning of vector quantization algorithms for different configurations. These three technical solutions will be described in more detail below.

[0044] In one embodiment, codebook storage in global memory, this method involves storing the codebook directly in the global memory of the GPU. Global memory is a memory area in the GPU architecture used to store large amounts of data, which provides data access capabilities for all threads, thread blocks, and streaming multiprocessors (SM: Streaming Multiprocessors). Under this storage strategy, when vector quantization or dequantization operations are performed, the required codebook data is loaded from the global memory. Due to the large capacity of the global memory, this method can easily handle large-scale codebook data sets. However, the access speed of the global memory is relatively slow, which may become a performance bottleneck, especially in scenarios where frequent access to codebook data is required.

[0045] In one embodiment, the codebook storage in shared memory, compared with the global memory storage scheme, the shared memory storage scheme places the codebook data in the shared memory of the GPU. Shared memory is a fast on-chip memory located on each SM, which allows threads within the same thread block to efficiently share and access data. By storing the codebook data in shared memory, data access latency can be significantly reduced because the access speed of shared memory is much faster than that of global memory. This method is particularly suitable for applications that require frequent and fast access to codebook data, which can improve overall processing speed and efficiency. However, the capacity of shared memory is limited, and too many thread blocks accessing shared memory at the same time may cause resource contention and performance degradation.

[0046] In one embodiment, manual compound tuning of VQ algorithms with different configurations is performed, and the performance and efficiency of the VQ algorithm can be optimized by adjusting multiple parameters, including vector size, number of codebook entries, number of quantization residuals, etc. Manual compound tuning is a meticulous optimization method that requires adjustment and optimization of these parameters according to specific application requirements and hardware environment. This method involves an in-depth understanding of the vector quantization algorithm and an accurate assessment of the performance impact under different configurations. Through manual tuning, the optimal quantization strategy can be customized for a specific application scenario to achieve the required performance and accuracy requirements. However, the manual tuning process can be very time-consuming and complicated, requiring professional knowledge and rich experience. In addition, as the algorithm and hardware environment change, it may be necessary to constantly re-tune.

[0047] The three technical solutions described above, codebook storage in global memory, codebook storage in shared memory, and manual compound tuning of vector quantization algorithms for different configurations, provide different levels of flexibility and performance, suitable for different application scenarios and requirements. The global memory storage solution is suitable for situations where the codebook data volume is large but the access frequency is not high, the shared memory storage solution is suitable for scenarios where codebook data needs to be accessed quickly and frequently, and manual compound tuning provides the possibility for advanced applications that require fine control and optimization of VQ algorithm performance.

[0048] Although VQ compresses storage and computation through vector quantization, in unoptimized attention calculations using vector quantization, whether the codebook is stored in global memory or shared memory, its performance is often not as good as FP16 attention calculations that do not use vector idealization. The reasons are analyzed as follows:

[0049] Codebook storage in global memory. The codebook is stored in global memory. The acceleration benefits brought by reducing the amount of computation and storage cannot cover the memory access overhead brought by reading the codebook from global memory. The access speed of global memory is relatively slow, which causes memory access to become a bottleneck in the entire computing process when performing vector quantization and dequantization operations. In addition, frequent reading of codebook data from global memory increases power consumption and latency, thereby affecting the overall computing efficiency. In large-scale parallel computing scenarios, this performance bottleneck may be more obvious, resulting in overall performance failing to meet expectations.

[0050] Codebook storage in shared memory,Although storing the codebook in shared memory can significantly improve data access speed, this approach also has some problems.,First, the capacity of shared memory is limited. When the codebook data is large, it may occupy too much shared memory resources, thus affecting the execution of other computing tasks.,Second, too many thread blocks accessing shared memory at the same time may cause resource competition,or even bank conflicts, which will reduce memory access efficiency.,In addition, excessive use of shared memory may limit the number of parallel threads,affecting the computing resource utilization of the GPU.

[0051] Manual compound tuning of VQ algorithms with different configurations. Although manual compound tuning can customize the optimal quantization strategy for specific scenarios, its shortcomings cannot be ignored. First, the manual tuning process is very time-consuming and complicated, requiring professional knowledge and rich experience. This not only increases the cost of development and maintenance, but also raises the technical threshold. Secondly, as the algorithm and hardware environment change, it may be necessary to constantly re-tune, which will result in additional workload. In addition, manual tuning is often difficult to adapt to diverse VQ algorithms and hardware platforms, which limits its scope of application. In a rapidly iterating and changing application environment, this approach may be difficult to meet the needs of efficiency and flexibility.

[0052] For vector quantization, from an algorithmic perspective, the codebook not only determines how data is compressed and represented, but also affects the system's representation capabilities, and is indispensable in the vector quantization algorithm. From the perspective of architecture performance optimization, optimizing the storage location and access mode of the codebook (such as caching frequently accessed entries in registers) is the key to improving the performance of the vector quantization algorithm on general-purpose graphics processing units (GPUs).

[0053] This application aims to improve the performance and efficiency of large language models under codebook-based vector quantization algorithms by jointly optimizing algorithms and architectures. It is worth noting that the vector quantization algorithm has different parameter configurations in large language models.

[0054] Figure 1 The diagram shows an application scenario of the vector quantization process and optional parameter configuration described in the embodiment of the present application. Figure 1 As shown in the figure, during the entire vector quantization process, the vector size, the number of codebook entries (#Entry), and the residual are configurable, denoted as VQ<x,y,z> , where x, y and z correspond to the vector width, the number of codebook entries and the residual respectively. The calculation and placement strategies of the optimal codebook under different parameter configurations are also different. In particular, the present application can adapt to various parameter configurations of the vector quantization algorithm.

[0055] The main innovation of this application is the method of determining the specific placement strategy of the codebook according to the GPU hardware resource amount and the parameter configuration of the codebook in the vector quantization algorithm under a specific LLM, as well as the user-friendly abstract interface (API) that interacts with software and hardware in a collaborative manner. This application combines the resource information of different GPUs and the specific access characteristics of the codebook when different vector quantization parameter configurations are applied to different large language models, dynamically stores in the GPU memory hierarchy, optimizes data access patterns, and reduces memory access latency, thereby solving the performance bottleneck problem of traditional vector quantization algorithms in large language model reasoning.

[0056] The technical solutions in the embodiments of the present application will be described in detail below in conjunction with the drawings in the embodiments of the present application.

[0057] like Figure 2 As shown, this embodiment provides a codebook caching method based on vector quantization, and the method includes the following steps S1 to S5.

[0058] Step S1: Obtain the codebook to be processed.

[0059] Specifically, a codebook is a set of finite vectors (called codewords or code vectors) that are used to approximate the vectors in the input k-dimensional vector space. Specifically, each codeword is a k-dimensional vector, and the set of all codewords constitutes the codebook. The codebook is an integral part of the vector quantization algorithm, which reduces storage and processing costs by approximating high-dimensional input data as low-dimensional codewords.

[0060] Step S2: sort the codebooks to be processed to obtain target codebook indexes.

[0061] Specifically, the target codebook index is a codebook entry index obtained by sorting and rearranging the codebook to be processed in descending order. Sorting the codebook to be processed to obtain the target codebook index is completed based on the offline stage of analysis.

[0062] In one embodiment, the present application ensures that the index of the most frequently accessed entry is 0 and the index of the least frequently accessed entry is the maximum value by sorting the codebook to be processed to obtain the target codebook index. All quantized data will use these new target codebook indexes.

[0063] In one embodiment of the present application, the sorting of the codebooks to be processed to obtain a target codebook index includes the following steps S21 to S22.

[0064] Step S21: sort the codebook to be processed in descending order according to the access frequency of the codebook to be processed to obtain a first codebook entry of the codebook to be processed.

[0065] Step S22: rearrange the codebook to be processed according to the first codebook entry to obtain a target codebook index of the codebook to be processed.

[0066] Step S3: Acquire an index boundary according to the target codebook index and the codebook cache. The codebook cache is a cache for storing the codebook to be processed.

[0067] Specifically, the CodeBook Cache is designed in collaboration with the GPU hardware based on the different access latency characteristics of the GPU's memory hierarchy. The CodeBook Cache stores the codebooks at different memory levels based on the access frequency of different entries in the codebook. Specifically, the present application saves shared memory usage by storing rarely accessed codebook entries in off-chip global memory, and stores the most frequently used entries in thread-local registers to eliminate shared memory bank conflicts.

[0068] In one embodiment of the present application, the index boundary includes a first index boundary and a second index boundary; the codebook cache includes a first cache, a second cache, and a third cache.

[0069] The first cache is used to store the codebook whose target codebook index in the codebook to be processed is less than the first index boundary.

[0070] The second cache is used to store the codebook whose target codebook index in the codebook to be processed is greater than or equal to the first index boundary and less than the second index boundary.

[0071] The third cache is used to store the codebook whose target codebook index is greater than or equal to the second index boundary in the codebook to be processed.

[0072] In an embodiment of the present application, obtaining an index boundary according to the target codebook index and the codebook cache includes the following steps S31 to S32.

[0073] Step S31: Obtain a most frequently accessed codebook index and a least frequently accessed codebook index according to the target codebook index.

[0074] Step S32, establishing the index boundary according to the most frequently accessed codebook index, the least frequently accessed codebook index and the codebook cache; wherein the most frequently accessed codebook index is less than a first index boundary of the index boundary; and the least frequently accessed codebook index is greater than or equal to a second index boundary of the index boundary.

[0075] Step S4: Perform judgment and comparison based on the index boundary and the target codebook index to obtain a judgment result.

[0076] In one embodiment of the present application, a judgment and comparison is performed based on the index boundary and the target coding book index to obtain a judgment result, including step S41A. If the target coding book index in the coding book to be processed is smaller than the first index boundary, the judgment result is to store the coding book in the coding book to be processed whose target coding book index is smaller than the first index boundary in the first cache.

[0077] In one embodiment of the present application, a judgment and comparison is performed based on the index boundary and the target codebook index to obtain a judgment result, including step S41B. If the target codebook index in the codebook to be processed is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary, then the judgment result is to store the codebook in the codebook to be processed whose target codebook index is greater than or equal to the first index boundary and whose target codebook index is less than the second index boundary in the second cache.

[0078] In one embodiment of the present application, a judgment and comparison is performed based on the index boundary and the target coding book index to obtain a judgment result, including step S41C. If the target coding book index in the coding book to be processed is greater than or equal to the second index boundary, the judgment result is to store the coding book whose target coding book index in the coding book to be processed is greater than or equal to the second index boundary in the third cache.

[0079] Step S5: storing the codebook to be processed in a corresponding buffer in the codebook buffer according to the judgment result.

[0080] The present application sorts the codebook to be processed to obtain the target codebook index; obtains the index boundary according to the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed; makes a judgment and comparison according to the index boundary and the target codebook index to obtain a judgment result; stores the codebook to be processed in the corresponding cache in the codebook cache according to the judgment result. The present application improves the memory operation performance and execution efficiency by placing the codebook entries at different locations of the GPU's memory hierarchy according to the frequency of use of the codebook, and solves the problem of low efficiency of shared memory and global memory.

[0081] The main innovation of this application is the method of determining the specific placement strategy of the codebook according to the GPU hardware resource amount and the parameter configuration of the codebook in the vector quantization algorithm under a specific LLM, as well as the user-friendly abstract interface (API) that interacts with software and hardware in a collaborative manner. This application combines the resource information of different GPUs and the specific access characteristics of the codebook when different vector quantization parameter configurations are applied to different large language models, dynamically stores in the GPU memory hierarchy, optimizes data access patterns, and reduces memory access latency, thereby solving the performance bottleneck problem of traditional vector quantization algorithms in large language model reasoning.

[0082] In one embodiment, the present application describes in detail the specific design and implementation of the present application through hardware collaborative design, software system architecture and user interface design.

[0083] Figure 3The overall architecture diagram of the hardware collaborative design, software system architecture and user interface design described in the embodiment of this application is shown. Figure 3 As shown, the overall architecture mainly includes three parts: hardware co-design, software system design, and user interface design. Hardware co-design is the practical basis of this application, combining hardware resources and performance characteristics to adaptively store the codebook in a hierarchical manner. Software system design is the specific algorithm for adaptive hierarchical storage. User interface design shields the underlying implementation and provides users with a unified view of the codebook cache. The following will explain the three aspects of hardware co-design, software system design, and user interface design.

[0084] In one embodiment, the present application designs a codebook cache in collaboration with GPU hardware based on the different access latency characteristics of the GPU memory hierarchy. The codebook cache can store different entries at different memory levels according to their access frequencies. Specifically, the present application stores rarely accessed codebook entries in off-chip global memory to save shared memory usage, and stores the most frequently used entries in thread local registers to eliminate shared memory bank conflicts.

[0085] In one embodiment, the cache implementation relies on a tagged array or a look-up table, which may add additional latency and storage overhead. In the code cache implementation of the present application, the present application adopts a static mapping mechanism based on reordering (see Figure 4 ), this mechanism is very lightweight and configurable.

[0086] Figure 4 The software flow diagram of the software system architecture described in the embodiment of the present application is shown. Figure 4 As shown, the software process implementation method of the software system architecture of the present application mainly includes the following steps S61 to S64.

[0087] Step S61: The present application first sorts the codebook entries in descending order according to the access frequency to obtain the codebook entries in order of access frequency. The sorting of the codebook entries is completed in an offline stage based on analysis.

[0088] Step S62: Then, the sorted codebook entries are rearranged offline. The sorting and rearrangement of the codebook entries are completed in the offline phase based on analysis, so as to ensure that the index of the most frequently accessed entry is 0 and the index of the least frequently accessed entry is the maximum value. All quantized data will use these new indexes.

[0089] Step S63: Next, establish two index boundaries: Nreg and Nshared. In this application, the index boundaries are established based on the hardware resources such as the CodeBookCache obtained by co-designing with the GPU according to the different access latencies of the memory hierarchy of the GPU and the codebook entries after offline rearrangement. Regarding the meanings of the two boundaries Nreg and Nshared, in this application, the first Nreg entries are allocated to the thread-local registers, and the subsequent entries up to Nshared are stored in the shared memory, while any remaining entries are stored in the global memory.

[0090] Step S64: For each codebook entry, during the dequantization process at runtime, locating the codebook entry involves simple index comparison. If the index < Nreg, this application locates the codebook entry in the register. If Nreg <= index < Nshared, this application locates the codebook entry in the shared memory. If the index ≥ Nshared, the codebook entry is located in the global memory.

[0091] In one embodiment, this application provides the following APIs for users to use the "codebook cache" (hereinafter referred to as CB) described in the embodiments of this application. The API explanations are as follows:

[0092] CB cached ,n reg，shared ←Load(CB,Slack)

[0093] Entry←Access(CB cached ,n eg，shared ,CB,Index)

[0094] CB←Switch(New CB Pointer)

[0095] The first API is Load(CB, Slack). The Load API loads the codebook stored in the global memory into the cache. The Load API accepts both the codebook and the Memory Slack, and returns the codebook cached across the memory hierarchy and two access boundaries. The Load API performs initial offline sorting of the access frequencies and offline rearrangement of the codebook entries based on the specific content of the codebook and the memory slack information in the hardware co-design to generate an initialized codebook cache configuration scheme for subsequent access.

[0096] The second API is Access(CB cached ,n reg,shared, CB, Index), Access API allows users to access specific entries during the dequantization process. Access API accepts cached codebooks and codebooks stored in global memory, as well as indexes for locating entries. At the same time, Access API also uses two boundaries to determine the entry location. In addition, although this application uses a preset heuristic method to configure these boundaries, users can still overwrite them. Access API will perform online indexing according to the existing codebook cache configuration scheme and return the results. Then, according to the specific entries accessed by the user, the corresponding codebook entry index frequency is increased, and the codebook index boundaries are dynamically updated.

[0097] The last API is Switch. The Switch API is very useful when the algorithm trains different codebooks for different parts of the tensor. The Switch API makes it easy to switch to a new codebook based on the specific part of the tensor that the user is processing. The Switch API automatically saves the metadata of the current code cache (including hardware co-configuration information and codebook index boundaries) and the codebook itself to a specific area of ​​Global Memory, and switches the new codebook into the codebook cache. For the new codebook cache, the Switch API uses a similar method to the Load API to initialize the configuration of the new codebook.

[0098] Among them, CB represents the codebook itself of the input codebook cache; the cached subscript indicates that the codebook has been processed by the "codebook" cache; nreg represents the register index boundary found during the execution of the Load API, and shared represents the shared memory index boundary found during the execution of the Load API; reg represents register; shared represents GPU shared memory; slack represents the memory slack coefficient; Entry represents the codebook entry returned by accessing the codebook cache through the Access API; Index represents the codebook entry number that needs to be accessed; New CB Pointer represents the pointer to the new codebook that needs to be transferred to.

[0099] In one embodiment, the present application uses the attention operator with parameter configuration of 32 heads, 128 dimensions, and VQ<4,8,1> in Llamma-7B as an embodiment to explain the optimization principle of the present application. Among them, Llamma-7B is a large basic language model released by Meta AI.

[0100] In the existing VQ-LLM operator implementation, most logarithmic methods store the codebook completely in global memory, which causes the slower global memory communication time to completely dominate the operator latency, resulting in poor performance; the more advanced shared memory-based optimization methods store the huge codebook completely in shared memory, without considering the access frequency of the codebook entries, and do not fully utilize the register as an advanced fast storage medium. The shared memory-based method is mainly subject to shared memory bank conflicts and communication bottlenecks between shared memory-global memory and shared memory-register. Among them, the VQ-LLM operator is a method that optimizes the efficiency and performance of large language model (LLM) reasoning through vector quantization (VQ) technology.

[0101] Figure 5 The performance effect diagram of the attention operator with the codebook placed in the global memory and shared memory as described in the embodiment of this application is shown. Figure 5 As shown in Figure 2, VQ-attn-GC / SC are attention operators that place the codebook in global memory and shared memory respectively. Figure 5 It means that in actual production environments, the execution speed of both VQ algorithms is slower than the baseline version using FP16 data format. Among them, VQ-attn-GC / SC is a vector quantization-based attention mechanism variant, which is mainly used to optimize the reasoning efficiency and performance of large language models (LLM).

[0102] according to Figure 5 It can be seen that the reason for the poor performance of the global memory version of the operator is that the global memory bandwidth is too low. Placing the codebook in the global memory makes the access to the global memory a bottleneck. Figure 6 As far as we know, the reason for the poor performance of the shared memory version of the operator comes from the complex factors brought about by the specific access frequency of the codebook, including low SM utilization, low shared memory utilization, shared memory bank conflicts, shared memory-global memory communication bottlenecks and shared memory-register communication bottlenecks. Figure 6 Shown is a time delay diagram of the shared memory codebook storage method described in an embodiment of the present application.

[0103] Since shared memory resources are limited, storing a large number of codebook entries will cause a surge in shared memory usage, which limits the number of thread blocks that can run simultaneously on each streaming multiprocessor (SM), thereby reducing the computational resource utilization of the GPU. Figure 6As shown in the figure, the shared memory codebook storage method has a computational utilization reduction of more than 30% compared to the FP16 benchmark, mainly due to the excessive shared memory usage. The random codebook entry access pattern in the shared memory codebook storage method leads to significant shared memory bank conflicts. Shared memory bank conflicts can seriously hinder the parallelism of memory access and increase memory access latency. In the shared memory codebook storage method, since the number of codebook entries far exceeds the number of shared memory banks and the access pattern is random, it is very difficult to reduce bank conflicts through common reordering or filling strategies. In addition, the large number of register accesses and data accesses between shared memory and global memory in the shared memory codebook storage method will greatly affect the performance of vector quantization.

[0104] This application solves the problem of low efficiency of shared memory and global memory by placing codebook entries at different locations of the GPU's memory hierarchy according to the frequency of use of the codebook.

[0105] Figure 7 The diagram shows the heat distribution of the items encoded in the AQLM-3 algorithm described in the embodiment of the present application. Figure 7 As shown, different entries in the codebook do show different degrees of “heat” in terms of access frequency. Figure 7 An embodiment of the AQLM-3 algorithm is shown. More than half of the codebook entries have access frequencies lower than the average, which indicates that there is little benefit in placing codebook entries with access frequencies lower than the average in shared memory. Among them, 26 hot codebook entries have access frequencies higher than the average plus three standard deviations (mean plus three times the standard deviation), which indicates that these 26 hot codebook entries are more susceptible to inevitable shared bank conflicts. Among them, the codebook entries are codebook entries. The AQLM-3 algorithm is an advanced large language model (LLM) compression technology based on additive quantization (AQ). It has the advantages of efficient reasoning, accuracy retention, and strong flexibility, and is widely used in terminal device deployment and multiple task support.

[0106] in accordance with Figure 3The overall structure of the present application is shown. The present application evaluates the vector quantization-based codebook cache method described in the present application on the NVIDIA RTX 4090 24GB GPU. The evaluated computing kernels include various VQ-enhanced GeMM (matrix multiplication), GeMV (matrix-vector multiplication) and FlashDecoding. The evaluated VQ configurations include QuiP#-4, AQLM-3, GPTVQ-2 and CQ-2 / 4, where the suffix numbers represent the equivalent bit width. The first two kernels use weight quantization, and the last kernel uses KV cache quantization. The present application sets the shapes of these kernels according to the Llama-7B and Llama-65B models.

[0107] It should be noted that QuiP#-4 is a model compression technology based on quantization-aware training (QAT), which simulates the quantization process during training so that the model can better adapt to the accuracy loss caused by quantization during reasoning. AQLM-3 is a large language model (LLM) compression technology based on additive quantization (AQ), which represents data points by the sum of multiple codebook vectors to achieve efficient model compression. GPTVQ-2 is a technology based on generative product of experts vector quantization (GPTQ), which aims to improve the reasoning efficiency and performance of the model by combining generative models and vector quantization technology. CQ-2 / 4 is a technology based on contrastive learning and quantization, which learns the representation of data through contrastive learning and reduces the storage requirements and computational complexity of the model through quantization.

[0108] Figure 8 The VQ CodeBook Cache acceleration effect diagram described in the embodiment of this application is shown. Figure 8 As shown in the figure, it shows the latency reduction of the optimized version under various VQ configurations relative to the unoptimized version. The vertical axis represents the percentage of latency reduction, and the horizontal axis shows the specific configurations for different kernels and batch sizes, including GeMM, GeMV, and Attention tasks. The bar charts in the figure represent the performance of the Llama-7B and Llama-65B models, where the solid part represents Llama-7B and the diagonal filled part represents Llama-65B.

[0109] In the computationally intensive operation of GeMM (matrix multiplication), the latency reduction of different configurations (such as QuiP#-4, AQLM-3, and GPTVQ-2) is quite significant. Specifically, the latency reduction of Llama-65B is close to 60%, while the latency reduction of Llama-7B is around 50%. In particular, under the QuiP#-4 configuration, the latency reduction rate of Llama-65B is as high as about 65%, which is significantly better than the approximately 55% of Llama-7B.

[0110] In the memory-intensive operation GeMV (matrix-vector multiplication), tests were conducted at batch sizes of 1 (BS1) and 16 (BS16). Llama-65B showed a more significant latency reduction of about 45% at batch size 16, while Llama-7B remained at about 40%. In addition, at batch size 1, the latency reduction difference between the two was smaller, about 35% for Llama-65B and about 30% for Llama-7B.

[0111] In the operation test under the Attention mechanism, latency reduction was observed at 1k and 4k sequence lengths. When the sequence length is 1k, the latency reduction of Llama-65B is close to 40%, while that of Llama-7B is about 35%; at a longer sequence length of 4k, the latency reduction of Llama-65B is further improved to about 50%, while that of Llama-7B is about 45%.

[0112] On average, the average latency reduction of Llama-65B under all configurations is about 50%, while the average latency reduction of Llama-7B is about 45%. This shows that the optimization effect of Llama-65B on weight quantization and KV cache quantization tasks is more significant as the batch size and sequence length increase, and it has obvious performance advantages when processing computationally intensive and memory intensive tasks.

[0113] It should be noted that LLaMA-7B and LLaMA-65B are two different versions of the LLaMA series of models released by Meta AI. They have significant differences in the number of parameters, amount of training data, and application scenarios.

[0114] Number of parameters of LLaMA-7B: LLaMA-7B has 7 billion parameters. Amount of training data: LLaMA-7B is trained on about 1 trillion tokens. Application scenarios: Due to its smaller size and lower computing resource requirements, LLaMA-7B may be more suitable for running in resource-constrained environments, such as a single GPU. It is suitable for applications that require high performance but have a limited budget, such as personal research projects, small businesses, or individual developers.

[0115] Number of parameters of LLaMA-65B: LLaMA-65B has 65 billion parameters. Amount of training data: LLaMA-65B is trained on a larger dataset. The specific number of tokens is not directly mentioned, but the training dataset of the entire LLaMA series contains about 1.4T tokens after tokenization. Application scenarios: Due to its large scale and powerful performance, LLaMA-65B is more suitable for large-scale tasks that require high performance and high accuracy, such as complex natural language processing tasks, large-scale data analysis, etc. It performs well in multiple benchmarks and is also very competitive compared with the industry's best models Chinchilla-70B and PaLM-540B.

[0116] The protection scope of the codebook cache method based on vector quantization described in the embodiment of the present application is not limited to the execution order of the steps listed in the present embodiment. All solutions implemented by adding, reducing or replacing steps in the prior art based on the principles of the present application are included in the protection scope of the present application.

[0117] The embodiment of the present application also provides a codebook cache system based on vector quantization, which can implement the codebook cache method based on vector quantization described in the present application. However, the implementation device of the codebook cache method based on vector quantization described in the present application includes but is not limited to the structure of the codebook cache system based on vector quantization listed in the present embodiment. All structural deformations and replacements of the prior art made according to the principles of the present application are included in the protection scope of the present application.

[0118] like Fig. 9 As shown, this embodiment provides a codebook cache system based on vector quantization, and the system 2 includes: an acquisition module 21, a sorting module 22, a boundary establishment module 23, a judgment module 24 and a storage module 25.

[0119] The acquisition module 21 is used to obtain the codebook to be processed;

[0120] The sorting module 22 is used to sort the codebook to be processed to obtain a target codebook index;

[0121] The boundary establishing module 23 is used to obtain an index boundary according to the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed;

[0122] The judgment module 24 is used to perform judgment and comparison according to the index boundary and the target codebook index to obtain a judgment result;

[0123] The storage module 25 is used to store the codebook to be processed in a corresponding buffer in the codebook buffer according to the judgment result.

[0124] It should be noted that the functions or operations of the acquisition module 21, sorting module 22, boundary establishment module 23, judgment module 24 and storage module 25 described in the embodiment of the present application correspond one-to-one to the steps in the above-mentioned coding book caching method based on vector quantization, so they will not be repeated here.

[0125] In the several embodiments provided in the present application, it should be understood that the disclosed system, device or method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of modules / units is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or units can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules or units, which can be electrical, mechanical or other forms.

[0126] The modules / units described as separate components may or may not be physically separated, and the components displayed as modules / units may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules / units may be selected according to actual needs to achieve the purpose of the embodiments of the present application. For example, the functional modules / units in the various embodiments of the present application may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0127] Those of ordinary skill in the art should further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0128] like Fig.10 As shown, this embodiment provides an electronic device, and the electronic device 3 includes: a memory 31 and a processor 32.

[0129] The memory 31 stores a computer program;

[0130] The processor 32 is in communication with the memory 31 and executes the above-mentioned codebook caching method based on vector quantization when calling the computer program.

[0131] The present application embodiment also provides a computer-readable storage medium. A person of ordinary skill in the art can understand that all or part of the steps in the method for implementing the above embodiment can be completed by a program to instruct the processor, and the program can be stored in a computer-readable storage medium, and the storage medium is a non-transitory medium, such as a random access memory, a read-only memory, a flash memory, a hard disk, a solid-state hard disk, a magnetic tape, a floppy disk, an optical disc, and any combination thereof. The above storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid-state hard disk (SSD)), etc.

[0132] The present application embodiment may also provide a computer program product, the computer program product including one or more computer instructions. When the computer instructions are loaded and executed on a computing device, the process or function described in the embodiment of the present application is generated in whole or in part. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer or data center.

[0133] When the computer program product is executed by a computer, the computer executes the method described in the above method embodiment. The computer program product may be a software installation package, and when the above method is required, the computer program product may be downloaded and executed on a computer.

[0134] In summary, the method, system, electronic device and medium for caching codebooks based on vector quantization described in the present application have the following beneficial effects:

[0135] The present application sorts the codebook to be processed to obtain the target codebook index; obtains the index boundary according to the target codebook index and the codebook cache; the codebook cache is a cache for storing the codebook to be processed; makes a judgment and comparison according to the index boundary and the target codebook index to obtain a judgment result; stores the codebook to be processed in the corresponding cache in the codebook cache according to the judgment result. The present application improves the memory operation performance and execution efficiency by placing the codebook entries at different locations of the GPU's memory hierarchy according to the frequency of use of the codebook, and solves the problem of low efficiency of shared memory and global memory.

[0136] This application combines resource information of different GPUs and specific access characteristics of the codebook when different vector quantization parameter configurations are applied to different large language models, dynamically stores and optimizes data access patterns and reduces memory access latency in the GPU memory hierarchy, thereby solving the performance bottleneck problem of traditional vector quantization algorithms in large language model reasoning.

[0137] The descriptions of the processes or structures corresponding to the above-mentioned figures have different emphases. For parts that are not described in detail in a certain process or structure, please refer to the relevant descriptions of other processes or structures.

[0138] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A codebook caching method based on vector quantization, characterized in that: The method comprises: Get the codebook to be processed; Sorting the codebooks to be processed to obtain target codebook indexes; Acquire an index boundary according to the target codebook index and a codebook cache; the codebook cache is a cache for storing the codebook to be processed; Performing a judgment and comparison based on the index boundary and the target codebook index to obtain a judgment result; According to the judgment result, the codebook to be processed is stored in a corresponding cache in the codebook cache.

2. The codebook caching method based on vector quantization according to claim 1, characterized in that: The sorting of the to-be-processed codebooks to obtain target codebook indexes comprises: Sorting the codebook to be processed in descending order according to the access frequency of the codebook to be processed to obtain a first codebook entry of the codebook to be processed; The codebook to be processed is rearranged according to the first codebook entry to obtain a target codebook index of the codebook to be processed.

3. The codebook caching method based on vector quantization according to claim 1, characterized in that: The index boundary includes a first index boundary and a second index boundary; the codebook cache includes a first cache, a second cache and a third cache; The first cache is used to store the codebook whose target codebook index in the to-be-processed codebook is smaller than the first index boundary; The second cache is used to store the codebook whose target codebook index in the codebook to be processed is greater than or equal to the first index boundary and less than the second index boundary; The third cache is used to store the codebook whose target codebook index is greater than or equal to the second index boundary in the codebook to be processed.

4. The codebook caching method based on vector quantization according to claim 3, characterized in that: Acquiring an index boundary according to the target codebook index and the codebook cache includes: Obtaining a most frequently accessed codebook index and a least frequently accessed codebook index according to the target codebook index; Establishing the index boundary according to the most frequently accessed codebook index, the least frequently accessed codebook index and the codebook cache; wherein the most frequently accessed codebook index is less than a first index boundary of the index boundary; The least frequently accessed codebook index is greater than or equal to a second index boundary of the index boundary.

5. The codebook caching method based on vector quantization according to claim 3, characterized in that: The comparison between the index boundary and the target codebook index to obtain a judgment result includes: If the target codebook index in the codebook to be processed is smaller than the first index boundary, the judgment result is to store the codebook whose target codebook index in the codebook to be processed is smaller than the first index boundary in the first cache.

6. The codebook caching method based on vector quantization according to claim 3, characterized in that: The comparison between the index boundary and the target codebook index to obtain a judgment result includes: If the target codebook index in the codebook to be processed is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary, the judgment result is to store the codebook whose target codebook index in the codebook to be processed is greater than or equal to the first index boundary and the target codebook index is less than the second index boundary in the second cache.

7. The codebook caching method based on vector quantization according to claim 3, characterized in that: The comparison between the index boundary and the target codebook index to obtain a judgment result includes: If the target codebook index in the codebook to be processed is greater than or equal to the second index boundary, the judgment result is to store the codebook whose target codebook index in the codebook to be processed is greater than or equal to the second index boundary in the third cache.

8. A codebook cache system based on vector quantization, characterized in that: The system comprises: An acquisition module is used to obtain the codebook to be processed; A sorting module, used for sorting the codebook to be processed to obtain a target codebook index; A boundary establishment module, used for acquiring an index boundary according to the target codebook index and a codebook cache; the codebook cache is a cache for storing the codebook to be processed; A judgment module, used for performing judgment and comparison according to the index boundary and the target codebook index to obtain a judgment result; A storage module is used to store the codebook to be processed in a corresponding cache in the codebook cache according to the judgment result.

9. An electronic device, characterized in that: The electronic device comprises: A memory storing a computer program; A processor is communicatively connected to the memory, and executes the codebook caching method based on vector quantization according to any one of claims 1 to 7 when calling the computer program.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by an electronic device, it implements the codebook caching method based on vector quantization as described in any one of claims 1 to 7.

Images