A graphics processor, a key-value cache read-write method, a device, and a storage medium

Abstract

The application discloses a kind of graphic processor, key value cache reading and writing method, equipment and storage medium, graphic processor includes: instruction analysis unit, vector compression unit, key value cache unit, data management unit and vector decompression unit.Instruction analysis unit determines access instruction parameter by parsing instruction.Vector compression unit determines vector compression data by vector compression.Key value cache unit stores vector compression data.Data management unit retrieves target vector compression data corresponding to target vector from key value cache unit.Vector decompression unit is used to decompress reconstruction target vector compression data, and obtain target vector.The application successfully realizes that key value vector is kept in compressed form in the whole process of writing, caching and reading out, effectively reduces the pressure of on-chip cache capacity and on-chip interconnection bandwidth when key value cache reading is carried out in graphic processor, and improves key value cache reading and writing efficiency, improves the task execution efficiency and quality of graphic processing.

Description

Technical Field

[0001] This application belongs to the field of graphics processor technology, and in particular relates to a graphics processor, a key-value cache read / write method, a device, and a storage medium. Background Technology

[0002] With technological advancements, Graphics Processing Units (GPUs) are widely used in scenarios such as large language models, multimodal models, and long-context reasoning tasks. However, during the decoding phase, GPUs need to continuously read and write the key and value vectors corresponding to tokens, forming a massive, frequently accessed key-value cache (KV Cache), leading to excessive consumption of on-chip cache capacity and on-chip interconnect bandwidth.

[0003] Compression of key-value caches can effectively alleviate the pressure on on-chip cache capacity and on-chip interconnect bandwidth. However, traditional compression methods mostly choose off-chip storage for compression, and the compression and decompression operations have high latency, making it difficult to meet the real-time requirements of graphics processor key-value reading. In actual cache read and write operations, it still seriously affects cache capacity and consumes a lot of bandwidth, severely impacting the key-value cache read and write rate, thereby reducing the graphics processor's task execution efficiency and affecting the normal operation of the graphics processor.

[0004] Therefore, how to effectively improve the read and write efficiency of key-value cache on graphics processors, and reduce the pressure on on-chip cache capacity and on-chip interconnect bandwidth, are important issues that urgently need to be addressed. Summary of the Invention

[0005] This application provides a graphics processor, a key-value cache read / write method, a device, and a storage medium, which can effectively improve the key-value cache read / write efficiency on the graphics processor and reduce the pressure on on-chip cache capacity and on-chip interconnect bandwidth.

[0006] A first aspect of this application provides a graphics processor, including an instruction parsing unit, a vector compression unit, a key-value cache unit, a data management unit, and a vector decompression unit, wherein: The instruction parsing unit is used to parse key-value cache access instructions and determine the access instruction parameters; The vector compression unit is used to perform vector compression on the vector to be written when the access instruction parameter is the instruction parameter for a write access, and to determine the vector compressed data corresponding to the vector to be written. Key-value cache units are used to store vector compressed data; The data management unit is used to retrieve the target vector compressed data from the key-value cache unit when the access instruction parameter is a read access instruction parameter. The target vector compressed data is the vector compressed data corresponding to the target vector to be read. The vector decompression unit is used to decompress and reconstruct the target vector compressed data to obtain the target vector.

[0007] A second aspect of this application provides a key-value cache read / write method, applied to a graphics processor, the method comprising: Retrieve the key-value cache access instruction and determine the access instruction parameters; When the access instruction parameter is a write access instruction parameter, the vector to be written is compressed based on the access instruction parameter, and the compressed vector data corresponding to the vector to be written is determined and stored. When the access instruction parameter is a read access instruction parameter, the target vector compressed data is retrieved and determined based on the access instruction parameter. The target vector compressed data is the vector compressed data corresponding to the target vector to be read. The target vector is obtained by decompressing and reconstructing the compressed data.

[0008] A third aspect of this application provides an electronic device, the electronic device comprising: at least one graphics processor; and a memory communicatively connected to the at least one graphics processor; wherein the memory stores a computer program executable by the at least one graphics processor, the computer program being executed by the at least one graphics processor to enable the at least one graphics processor to execute the key-value cache read / write method of the second aspect.

[0009] A fourth aspect of this application provides a computer-readable storage medium storing computer instructions that cause a graphics processor to execute the key-value cache read / write method of the second aspect.

[0010] The graphics processor provided in this application includes an instruction parsing unit, a vector compression unit, a key-value cache unit, a data management unit, and a vector decompression unit. The instruction parsing unit can parse key-value cache access instructions to determine the access instruction parameters. The vector compression unit can determine the vector compressed data corresponding to the vector to be written by vector compression when the access instruction parameters are write access instruction parameters. The key-value cache unit can store the vector compressed data. The data management unit can retrieve the target vector compressed data corresponding to the target vector from the key-value cache unit when the access instruction parameters are read access instruction parameters. The vector decompression unit can be used to decompress and reconstruct the target vector compressed data to obtain the target vector.

[0011] The graphics processor provided in this application implements the compression, decompression, and access management of the key-value cache through a dedicated hardware unit, enabling physical hardware separation of the key-value cache read / write path and end-to-end compressed state residence and transmission. The graphics processor provided in this application successfully maintains the compressed form of the key-value vector throughout the entire process of writing, caching, and reading, effectively reducing the pressure on the on-chip cache capacity and on-chip interconnect bandwidth when the graphics processor reads the key-value cache, improving the key-value cache read / write efficiency, and enhancing the task execution efficiency and quality of graphics processing. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is one of the schematic diagrams of the structure of a graphics processor provided in one embodiment of this application; Figure 2 A second schematic diagram of the structure of a graphics processor provided in one embodiment of this application; Figure 3 A flowchart illustrating a key-value cache read / write method provided in one embodiment of this application; Figure 4 This is a schematic diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0014] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0015] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0016] It should be noted that the acquisition, storage, use, and processing of data in the technical solution of this application all comply with the relevant provisions of national laws and regulations. In the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solution of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0017] With technological advancements, Graphics Processing Units (GPUs) are widely used in scenarios such as large language models, multimodal models, and long-context reasoning tasks. However, during the decoding phase, GPUs need to continuously read and write the key and value vectors corresponding to tokens, forming a massive, frequently accessed key-value cache (KV Cache), leading to excessive consumption of on-chip cache capacity and on-chip interconnect bandwidth.

[0018] Compression of key-value caches can effectively alleviate the pressure on on-chip cache capacity and on-chip interconnect bandwidth. However, traditional compression methods mostly choose off-chip storage for compression, and the compression and decompression operations have high latency, making it difficult to meet the real-time requirements of graphics processors when reading key values. In addition, traditional cache compression methods mostly perform compression and decompression operations on the software side. This approach makes it difficult to balance throughput, energy consumption, and hardware implementation complexity, resulting in high latency in cache write processes and making it unsuitable for word-by-word meta-streaming online scenarios.

[0019] In traditional compression techniques, compressed data is typically only cached in off-chip memory. Uncompressed data remains and is transmitted and resides in the on-chip caches, shared buffers, and on-chip interconnects at various levels of the graphics processor. This results in significant cache capacity consumption and high interconnect throughput, leading to substantial on-chip bandwidth usage. Furthermore, compressing key-value vectors using a general-purpose cache array on the graphics processor significantly increases chip control complexity and power consumption, while also posing timing convergence risks.

[0020] Furthermore, traditional compression methods lack a clear decoupling mapping between the logical block address and the physical compression address of key-value vectors. This makes it difficult to uniformly and accurately manage the granularity of cache allocation for compressed data on the graphics processing unit (GPU). Maintaining the corresponding metadata is prone to confusion, resulting in low vector update efficiency and high cache replacement overhead, severely impacting the GPU's ability to perform tasks. In some traditional compression methods, the same compression method is often used for both key and value vectors, failing to flexibly select the appropriate compression method based on the different sensitivities of different vectors in terms of inner product precision and reconstruction accuracy. This significantly affects the accuracy of data compression and reconstruction.

[0021] Based on the aforementioned technical problems, embodiments of this application provide a graphics processor, a key-value cache read / write method, a device, and a storage medium. The graphics processor includes: an instruction parsing unit, a vector compression unit, a key-value cache unit, a data management unit, and a vector decompression unit. The instruction parsing unit can parse key-value cache access instructions to determine access instruction parameters. The vector compression unit can determine the vector compressed data corresponding to the vector to be written by vector compression when the access instruction parameters are write access instruction parameters. The key-value cache unit can store the vector compressed data. The data management unit can retrieve the target vector compressed data corresponding to the target vector from the key-value cache unit when the access instruction parameters are read access instruction parameters. The vector decompression unit can be used to decompress and reconstruct the target vector compressed data to obtain the target vector.

[0022] The graphics processor provided in this application implements the compression, decompression, and access management of the key-value cache through a dedicated hardware unit, enabling physical hardware separation of the key-value cache read / write path and end-to-end compressed state residence and transmission. The graphics processor provided in this application successfully maintains the compressed form of the key-value vector throughout the entire process of writing to the cache and reading the vector, effectively reducing the pressure on the on-chip cache capacity and on-chip interconnect bandwidth when the graphics processor reads the key-value cache, improving the key-value cache read / write efficiency, and enhancing the task execution efficiency and quality of graphics processing.

[0023] It should be noted that the specific application scenarios of the graphics processor, key-value cache read / write method, device and storage medium provided in the embodiments of this application are not limited. The technical solutions provided in the embodiments of this application can be flexibly applied to various actual scenarios that require key-value vector caching and reading according to actual needs.

[0024] For example, in practical scenarios involving long-context reasoning in large language models using graphics processing units (GPUs), such as tasks requiring the processing of multi-lexical contexts, like document summarization, multi-turn dialogue history understanding, and code-assisted generation, the GPU provided in this application can efficiently compress and decompress key-value vectors through the aforementioned dedicated units configured for key-value caching. Furthermore, the compressed key-value vector data is always stored and transmitted in a compressed state within the GPU's various levels of cache. The GPU provided in this application effectively reduces the cache pressure and bandwidth consumption during context reasoning, achieves efficient key-value vector reading and writing, fully guarantees the accuracy of long-context reasoning, and improves model throughput and energy efficiency.

[0025] It should be noted that the application scenarios described in the above embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will understand that with the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0026] Figure 1 This is a schematic diagram of the structure of a graphics processor provided in one embodiment of this application. Figure 1 As shown, the graphics processor 100 includes an instruction parsing unit 101, a vector compression unit 102, a key-value cache unit 103, a data management unit 104, and a vector decompression unit 105.

[0027] The instruction parsing unit 101 can accurately parse the received key-value cache access instruction, thereby determining whether the key-value cache access instruction is a write instruction or a read instruction for the key-value vector, and determining the access instruction parameters contained in the instruction.

[0028] In the embodiments provided in this application, dedicated read / write instructions for reading and writing key-value vectors can be set in the instruction system, micro-operating system, and equivalent control interface on the graphics processor. The software side can convert key-value write or read operations in the model on the graphics processor into dedicated read / write instructions. Furthermore, the hardware side can no longer treat the key-value cache as an ordinary tensor cache object, but instead determine it as a dedicated storage object with a logical block address space and compressed physical layout, and store it in a dedicated key-value cache unit 103 after compression by the vector compression unit 102.

[0029] In some embodiments, during the execution of inference tasks by the graphics processor, when the inference model obtains new terms or user-input terms, it can determine the key vector and value vector corresponding to the term through attention calculation. The graphics processor can then generate a key-value cache access instruction for writing to the key-value cache for that term. When the inference model is generating new terms, it performs attention calculation between the current query vector and the key values ​​of all historical terms, which generates a key-value cache access instruction for reading from the key-value cache.

[0030] In some embodiments, the aforementioned key-value cache access instruction may specifically include, but is not limited to: logical block address, layer identifier, header identifier, token range identifier, key / value type identifier, logical length, compression mode, and return target path. The logical block address may represent the address information of the key-value cache data to be accessed, specifically, for example, a number decoupled from the physical storage address in the software logical view. In some embodiments, the logical block address may be composed of a combination of information such as layer identifier, attention header identifier, token identifier, key-value type identifier (kv_type), and sub-block identifier (sub_block_id).

[0031] The layer identifier indicates the layer where the key-value cache data to be accessed resides. When performing key-value vector reads or writes, this index can be used to locate the corresponding layer's cache partition or address space, such as the layer in the Transformer model. The header identifier indicates the attention header corresponding to the key-value cache data to be accessed within its layer.

[0032] A lexical range identifier can indicate a continuous range of lexical terms to be read or written, specifically, from the start lexical index to the end lexical index. Based on the lexical range identifier, efficient reading of key-value vectors corresponding to multiple lexical terms within a certain range can be achieved. The key / value type identifier can indicate whether the key-value cache data to be accessed corresponds to a key vector or a value vector.

[0033] The logical length represents the total number of dimensions of the key-value vector to be accessed. The compression mode represents the compression or decompression method corresponding to the accessed key-value cache data. The return target path represents the hardware target to which the decompressed key-value vector should be sent back, such as the general-purpose register file on the graphics processor, the tensor core buffer, etc. It should be noted that in some embodiments, the logical length of the logical vector block, the compressed payload length, and the cache allocation granularity may not be completely consistent with each other. For example, a logical vector block with a length of 128 dimensions can be packaged into several consecutive bytes and a norm field in a 3-bit compression mode during subsequent compression, while it can be stored using half-byte alignment in a 4-bit compression mode.

[0034] In some embodiments, when the key-value cache access instruction is a write instruction for a key-value vector, it may include the logical key-value block address, layer identifier, header identifier, key / value type identifier, logical length, and compression mode as described in the above embodiments. In some embodiments, key-value vector writing is mostly performed on a single term, and the term range identifier can be flexibly set according to the actual application scenario. The return target path is used to point to the decompressed transmission target and is not used during the writing process. When the key-value cache access instruction is a read instruction for a key-value vector, it may include all the instruction parameters described in the above embodiments. The instruction parsing unit 101 can accurately parse the key-value cache access instruction to determine the multiple instruction access parameters for writing or reading key-value vectors in the above embodiments, thereby providing accurate guidance for the subsequent compression process.

[0035] The vector compression unit 102 can perform vector compression processing on the vector to be written pointed to by the access instruction parameter when the key-value cache access instruction is a write instruction for the key-value vector, and determine the compressed vector data of the vector to be written. The vector to be written is a key vector or a value vector.

[0036] In the embodiments provided in this application, the graphics processor may further include a vector partitioning unit 106. Before the vector to be written is compressed by the vector compression unit 102, the vector to be written can be divided into vector blocks by the vector partitioning unit 106 to determine at least one logical vector block of the vector to be written. In some embodiments, a logical vector block may exactly correspond to the key vector or value vector of a single word in an attention head, or it may correspond to a sub-block of a vector. The partitioning length or sub-block length of the vector partitioning unit 106 can be flexibly set according to actual needs and application scenarios. In some embodiments, the sub-block length may be 32-dimensional, 64-dimensional, 128-dimensional, 256-dimensional, etc.

[0037] The purpose of vector splitting is to enable the subsequent compression process to be executed at a uniform granularity, and to configure corresponding metadata for each logical vector block, thereby achieving efficient variable-length storage and laying the foundation for efficient access in the subsequent vector decompression process.

[0038] For each logical vector block after the vector to be written is segmented, the vector compression unit 102 can perform compression processing one by one to obtain the vector compressed data corresponding to each logical vector block, thereby forming the vector compressed data of the complete vector to be written.

[0039] In the above embodiments, vector partitioning is performed using vector partitioning units, simplifying the compression operation from variable-length vector processing to pipelined processing of uniform-granularity logical blocks. This significantly reduces the control complexity and area overhead of the graphics processor during subsequent compression processing. Based on this embodiment, the cache management of compressed data can be processed at a finer logical block granularity, significantly improving the flexibility of data storage and management and cache utilization.

[0040] Regarding the specific processing procedure of vector compression unit 102 for compressing the vector or logic vector block to be written. In the embodiments provided in this application, vector compression unit 102 can perform multi-stage compression processing on the vector or logic vector block to be written, including but not limited to: norm extraction and normalization processing, orthogonal transformation and non-uniform quantization, etc.

[0041] Specifically, taking a logical vector block as an example, the vector compression unit 102 can extract the norm of the logical vector block to be written, determine the corresponding original vector norm, such as the L2 norm, and normalize the logical vector block to obtain the corresponding normalized vector.

[0042] In some embodiments, the above normalization process can be specifically referred to as formulas (1) and (2): (1) (2) in, This indicates a vector or logical vector block to be written. Indicates the vector or logical vector block to be written. The modulus value. The vector to be written can be accurately calculated using formula (2). The corresponding normalized vector .

[0043] Then, for the normalized vector, the vector compression unit 102 can perform orthogonal transformation processing to determine the corresponding transformed vector. In some embodiments, a structured orthogonal transformation or an approximate orthogonal transformation can be performed on the normalized vector, making the vector components of each dimension more suitable for subsequent dimension-by-dimensional independent quantization, while reducing the implementation complexity of the image processor. Specifically, the structured orthogonal transformation can be selected from transformation methods such as random sign flipping, Walsh-Hadamard transformation, butterfly network transformation, coordinate permutation, etc., or a combination of multiple transformation methods in the above examples. For example, random sign flipping and Walsh-Hadamard transformation can be performed first, and then a second random sign flipping can be performed to obtain the transformed vector. In other embodiments, orthogonal transformation processing can also be implemented by pre-generating orthogonal matrices, butterfly networks, or reconfigurable shuffle networks, which can be flexibly selected according to actual needs and application scenarios.

[0044] In some embodiments, the above orthogonal transformation process can be specifically referred to as shown in formula (3): (3) in, Represents the transformed vector. It can be a random orthogonal rotation matrix, used to normalize vectors. Perform rotation processing without changing the relative structure and positional relationships between data within the vector.

[0045] Furthermore, for the transformed vector, the vector compression unit 102 can perform non-uniform quantization on the vector components of each dimension in the transformed vector according to a pre-constructed preset codebook, so as to determine the codebook index corresponding to each dimension of the vector component in the preset codebook. The preset codebook can be pre-set according to the vector dimension and quantization bit width during the GPU chip design or model deployment stage. The vector dimension can affect the range of numerical distribution covered by the codebook, and the quantization bit width can affect the codebook size. In some embodiments, the non-uniform quantization or the preset codebook can be selected with bit widths of 2 bits, 3 bits, or 4 bits, etc., and can be flexibly selected and set according to actual needs and application scenarios.

[0046] Next, the vector compression unit 102 can package the codebook indices corresponding to multiple vector components to determine the codebook index data of the vector to be written, and at the same time determine the metadata corresponding to the codebook index data. Then, the codebook index data and metadata can be used as the vector compressed data corresponding to the vector or logical vector block to be written.

[0047] In some embodiments, the packaging process can be based on preset format requirements for packaging codebook indexes. For example, when the compression width is 4 bits, it can be packaged in half-byte mode; when it is 3 bits, it can be packaged based on continuous byte streams; and when it is 2 bits, it can be packaged based on four-value encoding.

[0048] The metadata is used by the subsequent data management unit 104 to locate and parse the codebook index data when retrieving the target vector compressed data during vector decompression. In some embodiments, the metadata corresponding to the vector compressed data may specifically include, but is not limited to, the following: compressed bit, format ID, logical length, payload bytes, physical class, codebook ID, transform ID, norm field, dwell position field, exception bit or overflow bit, etc.

[0049] The compression flags mentioned above indicate whether the data pointed to by the metadata is compressed or uncompressed; for example, they can be binary flags. The format identifier indicates the specific encoding format of the vector compressed data, which can include quantization bit width, whether residual encoding is included, and the type of residual encoding. The logical length indicates the number of vector dimensions of the vector or logical vector block to be written before compression.

[0050] The physical payload length indicates the actual number of bytes in the vector compressed data, such as the length of the complete bitstream. The physical allocation category indicates the allocation type of the vector compressed data in the physical cache slots on the hardware, such as standard slots or long block slots. The codebook identifier indicates the index of the non-uniform quantization codebook used to obtain the vector compressed data, such as the codebook number.

[0051] The transformation identifier indicates the orthogonal transformation type used when performing orthogonal transformations on the normalized vector. The norm field indicates the original vector norm obtained by norm extraction on the vector or logical vector block to be written; specifically, it can be a scalar floating-point number, and in some embodiments, it may also include a residual norm field. The resident location field indicates the physical storage level corresponding to the vector compressed data; specifically, it can be a dedicated buffer line, cache line, sector, or sub-block set. The exception bit or overflow bit indicates whether there is numerical overflow or anomaly during the compression and quantization process of the vector or logical vector block to be written. Overflow can refer to a vector component exceeding the maximum coverage range of a preset codebook, and anomaly can refer to a vector component value being infinite or non-numerical. In other embodiments, the metadata may also include parameters such as an original bit width fallback flag, a version number field, a check field, a sector mask, a sub-block bitmap, or a physical offset field, which can be flexibly set according to actual needs and application scenarios.

[0052] In some embodiments, the process of determining the codebook index can be specifically referred to as shown in formula (4): (4) in, Indicates the codebook index. This indicates matching the transformed vector from the codebook. The corresponding representative value. The codebook is similar to a lookup table, dividing the real number axis into several non-uniform intervals, each assigned a unique index number. For the input real number (i.e., the transformed vector), it determines whether the number falls within an interval and returns the corresponding index number as the codebook index.

[0053] The above embodiments efficiently compress key-value vectors through a vector compression unit, effectively reducing the computational burden on the general-purpose computing cores of the graphics processor during the vector compression process. This embodiment achieves high-precision compression of the vector to be written through multi-stage processing of logical vector blocks. Furthermore, the binding storage of codebook index data and metadata allows for precise location of the compressed data corresponding to the target vector during data decompression. It also supports efficient transfer and parsing of compressed data between cache levels, significantly improving the read / write efficiency of the key-value cache on the graphics processor.

[0054] In the embodiments provided in this application, considering that when compressing key vectors and value vectors, since the core operation of key vectors in actual attention analysis is inner product operation, the accuracy requirement of direction information is high. That is, during the compression process, the impact of quantization on the accuracy of inner product needs to be considered in the non-uniform quantization stage.

[0055] Based on this, residual analysis can be introduced when the vector to be written is a key vector, so as to correct the residual error that may be caused by the decrease in inner product precision due to quantization during subsequent vector decompression. Specifically, the vector compression unit 102 can first perform non-uniform quantization on the vector components of each dimension in the orthogonally transformed vector corresponding to the key vector or logic vector block to determine the initial codebook index corresponding to each vector component.

[0056] Then, for each vector component, the vector compression unit 102 can perform a codebook lookup based on a preset codebook index to reconstruct the vector component before quantization and determine the reconstructed vector corresponding to the initial codebook index. Furthermore, based on the reconstructed vector and the original vector component, the residual vector of the vector component in this quantization process can be determined.

[0057] Next, the vector compression unit 102 can determine the residual code corresponding to the residual vector by quantizing the residual vector, and then determine the codebook index corresponding to the vector component based on the initial codebook index and the residual code. In some embodiments, the residual vector can be quantized using methods such as one-bit sign quantization, random projection quantization, or Quantized Johnson-Lindenstrauss Transform (QJLT) to determine the residual code. The specific method can be flexibly selected according to actual needs and application scenarios.

[0058] In some embodiments, the specific processing steps for vector reconstruction of the initial codebook index and determination of the residual index can be referred to as formulas (5) to (8): (5) (6) (7) (8) in, This indicates a table lookup using the codebook. Determine the codebook index Corresponding to the vector before quantization, This represents the reconstructed vector obtained through inverse transformation and norm recovery. This represents the residual vector between the reconstructed vector and the original vector or logical vector block to be written. Represents the residual vector The residual code obtained after random projection and one-bit sign quantization. Let N be a Gaussian random matrix, where each element follows a random order N(0, 1). This refers to the sign function.

[0059] Through the above embodiments and corresponding formulas, the key vector can be quantized in two stages. In the first stage, the transformed key vector is quantized dimension by dimension. In the second stage, the residual vector after the first stage of quantization is accurately calculated, thereby obtaining a codebook index that includes the quantization result of the vector itself and the residual code. This ensures that the inner product accuracy is fully guaranteed when the key vector is decompressed.

[0060] For value vectors, in the embodiments provided in this application, the mean square error of value vectors is an important factor to consider during attention analysis. In order to ensure the reconstruction accuracy of value vectors during decompression and improve the compression rate, when the vector to be written is a value vector, the residual analysis process in the above embodiments can be omitted. The value vector or the corresponding logical vector block is quantized one-dimensionally in the first stage of the above embodiments, and the determined initial codebook index is used as the codebook index for subsequent packaging.

[0061] In some embodiments, the total bit width corresponding to the codebook index determined by the above embodiments can be the same for the key vector and the value vector. Based on the above embodiments, the codebook index of the key vector can be composed of an initial codebook index obtained by one-stage dimension-wise quantization and a residual code obtained by two-stage residual analysis. For example, a 3-bit initial codebook index combined with a 1-bit residual code results in a 4-bit codebook index. The codebook index of the value vector can be directly composed of the initial codebook index obtained by dimension-wise quantization, and compared to the key vector, the codebook level corresponding to the initial codebook index of the value vector can be increased. For example, the bit width corresponding to the initial codebook index of the value vector in the above example can be 4 bits.

[0062] In other embodiments, the key vector and value vector can also be compressed with different average bit widths. For example, the key vector can be compressed using a 4-bit or 8-bit format, while the value vector can be compressed using a 2-bit, 3-bit, or 4-bit format, thereby further balancing the different precision requirements and bandwidth usage of the key and value vectors. In some embodiments, the compression bit width of the key and value vectors can be flexibly selected as either symmetric or asymmetric based on the hardware capabilities of the model or graphics processor.

[0063] In the above embodiments, a two-stage compression path of quantization plus residual compensation is adopted when determining the codebook index for the key vector. This effectively prevents the loss of directional details in the vector components of the key vector during the dimension-wise non-uniform quantization process, and significantly improves the inner product accuracy of the key vector when performing attention calculations after subsequent decompression. This embodiment fully guarantees the compression benefits while effectively reducing the impact of quantization errors on the accuracy of attention analysis, ensuring the processing quality when performing corresponding tasks such as contextual reasoning on a graphics processor.

[0064] The key-value cache unit 103 is a hardware cache module established on the graphics processor in this application specifically for storing compressed vector data corresponding to key-value vectors. The compressed vector data obtained after processing by the vector compression unit 102 can be stored in the key-value cache unit 103, providing data support for subsequent key-value cache readings. It should be noted that in some embodiments, the metadata determined by the compression process described above may not be stored in the key-value cache unit 103. The codebook index data is stored in the key-value cache unit 103, while the corresponding metadata can be written to metadata SRAM or a sidecar array to provide data location assistance for subsequent target vector retrieval processes.

[0065] In the embodiments provided in this application, a cache hierarchy interface unit 107 may also be provided on the graphics processor. The cache hierarchy interface unit 107 can be used to connect the key-value cache unit 103 and other cache modules of the graphics processor, such as multi-level cache modules (e.g., L1 cache and L2 cache), on-chip interconnect modules, and graphics processor-to-off-chip storage. Furthermore, the cache hierarchy interface unit 107 can also be used to connect the shared cache on the graphics processor and the aforementioned other cache modules.

[0066] Based on the cache hierarchy interface unit 107, the compressed vector data stored in the key-value cache unit 103 can be efficiently and stably transmitted between the various cache modules of the graphics processor in a compressed state, such as performing data movement, swapping in, swapping out, eviction, and write-back operations. This effectively reduces the excessive occupation of the graphics processor's on-chip cache capacity and on-chip bandwidth during key-value cache read and write processes.

[0067] When the instruction parsing unit 101 determines that the key-value cache access instruction is a read instruction for the target vector, the data management unit 104 can retrieve the compressed target vector data from the key-value cache unit 103 based on the access instruction parameters in the read instruction. The compressed target vector data can be the compressed vector data corresponding to the target vector to be read in the key-value cache unit 103.

[0068] In the embodiments provided in this application, the data management unit 104 can accurately retrieve and determine the metadata corresponding to the target vector based on instruction parameters such as the aforementioned logical block address in the read instruction. Furthermore, it can accurately locate the target vector compressed data corresponding to the target vector from the key-value cache unit 103 based on parameters such as the residence location field, physical load length, and format identifier in the metadata. Further, the target vector compressed data can subsequently be input into the vector decompression unit 105 for high-precision vector decompression and reconstruction.

[0069] As described above, the vector decompression unit 105 can decompress and reconstruct the target vector compressed data retrieved by the data management unit 104, thereby obtaining a complete and accurate target vector.

[0070] In the embodiments provided in this application, the decompression process performed by the vector decompression unit 105 on the target vector compressed data can correspond to the compression process performed by the vector compression unit 102 in the above embodiments, that is, it is deduced in the reverse direction of the compression process: after the compressed data is unpacked, codebook lookup is performed, inverse non-uniform quantization is performed, inverse orthogonal transformation is performed, and norm recovery is performed. For the key vector, residual recovery can also be performed on the reconstructed vector according to the residual encoding in the codebook index.

[0071] Specifically, the vector decompression unit 105 can unpack the codebook index data in the target vector compressed data to determine the target codebook index corresponding to each dimension vector component in the target vector or logical vector block. Then, based on the corresponding preset codebook, a codebook lookup table is performed on the target codebook index to determine the dequantized target reconstruction vector. Finally, an inverse orthogonal transform is performed on the target reconstruction vector to obtain the inverse transform vector corresponding to the target vector.

[0072] Finally, the norm of the inverse transform vector can be restored based on the vector norm in the metadata corresponding to the target vector compressed data to obtain the restored vector. In some embodiments, for the restored vector where the target vector is a value vector, as can be seen from the above compression process, the key to the reconstruction process of the value vector lies in the reconstruction accuracy. Therefore, the compression process does not require residual analysis, and there is no corresponding residual encoding in the codebook index. Therefore, the restored vector after decompression and reconstruction of the value vector by the vector decompression unit 105 can be directly output as the target vector.

[0073] For the recovered vector where the target vector is the key vector, in the embodiments provided in this application, based on the above compression process, it is known that the key vector reconstruction process needs to focus on the inner product accuracy during subsequent attention analysis. Therefore, during the compression process, residual analysis needs to be performed based on dimension-by-dimensional quantization, and residual codes corresponding to the residual vectors need to be added to the codebook index.

[0074] Based on this, during the vector reconstruction process of the key vector, the vector decompression unit 105 can determine the residual components corresponding to each dimension of the vector components based on the residual encoding in the target codebook index of the key vector. Furthermore, the vector decompression unit 105 can perform residual correction on the recovered vector of the key vector based on the residual components, and the corrected vector can be output as the target vector.

[0075] In some embodiments, the specific residual correction process for the target vector that is the key vector can be referred to as formulas (9) to (11): (9) (10) (11) in, For residual vectors The modulus, Indicates the dimension parameter. Represents the residual vector The original number of dimensions, To use formula (10) based on the residual vector The calculated residual compensation vector, The recovered vector can be obtained by formula (11) for the residual compensation vector. and the recovered vector Summing yields the target vector, which is the key vector.

[0076] The residual correction process described above effectively compensates for the directional error in the first-stage quantization process of the key vector during compression, accurately restoring the directional details lost during the dimension-wise non-uniform quantization process. Based on this embodiment, the accuracy of the inner product is significantly improved when performing attention inner product calculations on the target vector that serves as the key vector, thereby enhancing the accuracy and generation quality of the graphics processor when performing corresponding tasks (such as attention analysis within a contextual scene).

[0077] It should be noted that the decompressed and reconstructed target vector can be directly output to the attention calculation unit or tensor processing unit on the graphics processor. The general cache does not need to store the complete vector data for a long time, thus effectively reducing the capacity occupation of the on-chip cache.

[0078] Through the decompression processing of the vector decompression unit 105, the decompression operation can be separated from the general computing core of the graphics processor and configured with a dedicated hardware path, eliminating the latency jitter and computing power occupation caused by software-side decompression. Furthermore, each decompression step in the decompression process strictly corresponds to the compression process, ensuring data accuracy while achieving accurate restoration of the target vector. This effectively reduces the on-chip cache capacity and on-chip interconnect bandwidth occupation of the graphics processor, while improving the task execution efficiency and quality of the graphics processor.

[0079] In addition to the above, in the embodiments provided in this application, besides the multiple units mentioned above, the graphics processor may also include multiple auxiliary functional units. Specifically, these may include, but are not limited to, a hotspot buffer unit 108, a bit width allocation unit 109, and a compression mode control unit 110. In some embodiments, the hotspot buffer unit 108, the bit width allocation unit 109, and the compression mode control unit 110 may be established separately, or they may be integrated into a single auxiliary functional unit, which can be flexibly set according to actual needs and application scenarios.

[0080] The hotspot buffer unit 108 can cache hotspot vectors for a preset short period of time. Its specific hardware location can be set near the vector decompression unit 105 on the graphics processor or near the attention entry buffer. Hotspot vectors are target vectors or logical vector blocks whose reading frequency exceeds a preset frequency threshold. The hotspot buffer unit 108 can fully cache such vector data, thereby avoiding repeated decompression due to high-frequency reading and further improving the key-value cache read rate.

[0081] Before the vector compression unit 102 determines the codebook index data of the vector to be written in the above embodiment, the bit width allocation unit 109 divides the multidimensional components of the vector to be written into high bit width and low bit width according to the root mean square error after non-uniform quantization of the vector to be written, thereby determining the high bit width dimension vector and low bit width dimension vector in the vector to be written, and the quantization bit width corresponding to the high bit width dimension vector is greater than the quantization bit width corresponding to the low bit width dimension vector.

[0082] The channels corresponding to high-bit-width dimension vectors are high-bit-width outlier channels, and a higher compression bit width can be selected for corresponding vector compression processing. As mentioned above regarding the non-uniform distribution of high and low bit widths, the graphics processor provided in this application can support non-integer average bit width configurations. For example, for a 128-dimensional vector block, based on the root mean square error, 64 of the corresponding dimensions can be designated as high-bit-width outlier channels, using 4-bit quantization during compression, while the remaining 64 channels use 3-bit quantization, forming an average 3.5-bit configuration. In other embodiments, some high-amplitude channels, predefined position channels, or important channels can also be set as high-bit-width outlier channels, while the remaining channels maintain a low bit width, thereby achieving an average 2.5-bit, 3.5-bit, or other fractional bit-width compression mode. This allows the vector compression process to be flexibly and dynamically adjusted, effectively balancing accuracy requirements and computational power consumption.

[0083] The compression mode control unit 110 can flexibly switch between compression modes. These compression modes include at least a compression-resident mode, a bypass mode, and a hybrid mode. The compression-resident mode means that the compression processing described in the above embodiments is performed by the vector compression unit 102, and the vector data is transmitted in a compressed state throughout the entire link. The bypass mode means that some paths do not perform compression processing after obtaining key-value vectors, and the original vector data is stored. The hybrid mode means that the compression-resident mode and the bypass mode can be switched or combined according to the actual compression standard; for example, the main path is compression-resident, while hot paths temporarily retain complete vector data. In some embodiments, the actual compression standard may include, but is not limited to, compression benefits, vector access frequency, model type, and the hardware capabilities of the graphics processor. The compression mode control unit 110 can flexibly switch between the compression-resident mode and the bypass mode according to the actual compression standard to effectively balance vector accuracy requirements and bandwidth usage.

[0084] Regarding the hardware layout on the graphics processor, in the embodiments provided in this application, in order to further improve the key value cache read and write speed, the key value cache unit 103 can be set within a preset range around the attention data access path on the graphics processor, and directly connected to the vector compression unit 102, the data management unit 104 and the vector decompression unit 105.

[0085] This layout places the key-value cache unit adjacent to the attention data access path and directly connects it to the vector compression unit, data management unit, and vector decompression unit, creating a tightly coupled physical path on the graphics processor dedicated to key-value cache read / write. This embodiment significantly reduces the access latency for vector compressed data write and read, avoids unnecessary timing overhead, and enables low-latency, smooth connections between vector compression, metadata retrieval, and decompression reconstruction, thereby effectively improving key-value cache read / write throughput and energy efficiency.

[0086] Additionally, in the embodiments provided in this application, a path control unit 111 may also be provided on the graphics processor. The path control unit 111 can output the target vector obtained after decompression and reconstruction by the vector decompression unit 105. Specifically, it can send the decompressed and reconstructed target vector to other consumer units on the graphics processor, such as streaming multiprocessors, tensor cores, attention pipelines, and vector processing units, so that each unit can perform corresponding tasks based on the target vector.

[0087] When partial writing or content modification of the key-value cache is involved, in the embodiments provided in this application, the data management unit 104 can accurately locate the old compressed vector data to be modified based on instruction parameters. Then, the vector decompression unit 105 performs vector decompression and reconstruction to obtain the target vector to be modified. After modifying or replacing the corresponding logical elements, the modified vector can be recompressed by the vector compression unit 102 and stored in the key-value cache unit 103, achieving high-efficiency and high-precision key-value cache modification. In some embodiments, when a write-back or cache replacement occurs, the modified and recompressed codebook index data and metadata can be written back to other cache modules of the graphics processor or connected off-chip storage.

[0088] Based on the above embodiments, the graphics processor provided in this application will be comprehensively described. For details, please refer to [link / reference needed]. Figure 2 The structure diagram is shown below.

[0089] Figure 2 This is a schematic diagram of the structure of a graphics processor provided in one embodiment of this application.

[0090] like Figure 2 As shown, the graphics processor includes an instruction parsing unit 101, a vector compression unit 102, a key-value cache unit 103, a data management unit 104, a vector decompression unit 105, a vector partitioning unit 106, a cache level interface unit 107, a hotspot buffer unit 108, a bit width allocation unit 109, a compression mode control unit 110, and a path control unit 111.

[0091] The instruction parsing unit 101 analyzes access instructions to determine the access instruction parameters for key-value reading or writing. The vector compression unit 102 compresses the vector to be written during vector writing and can also divide logical vector blocks through the vector partitioning unit 106. The key-value caching unit 103 stores the compressed vector data and effectively reduces the on-chip cache capacity occupied by the graphics processor by setting up a dedicated cache.

[0092] The data management unit 104 performs precise retrieval of the target vector compressed data during vector reading, while the vector decompression unit 105 accurately reconstructs the target vector through a decompression process corresponding to the compression process. The cache-level interface unit 107 is responsible for the efficient transmission of compressed vector data in compressed form to various cache modules on the graphics processor. The bit-width allocation unit 109 is used for flexible bit-width configuration during compression. The compression mode control unit 110 is used for flexible switching of compression modes. The path control unit 111 is used to transmit the reconstructed vector during vector reading to the consumption module on the graphics processor for execution of corresponding tasks.

[0093] In some embodiments, this application also provides a computing system that can be equipped with the aforementioned graphics processor. The computing system may include the graphics processor described in the above embodiments, as well as a central processing unit, main memory, off-chip video memory (or high-bandwidth memory), etc., connected to the graphics processor. The computing system provided in this application embodiment can be used to perform application scenarios requiring high-dimensional key-value cache access, such as large language model inference, long context question answering, retrieval enhancement generation, multimodal inference, and vector retrieval.

[0094] The above describes the specific implementation of the graphics processor provided in this application. The graphics processor provided in this application implements the compression, decompression, and access management of the key-value cache through a dedicated hardware unit, enabling physical hardware separation of the key-value cache read / write path and end-to-end compressed state residence and transmission. The graphics processor provided in this application successfully maintains the compressed form of the key-value vector throughout the entire process of writing, caching, and reading, effectively reducing the pressure on the on-chip cache capacity and on-chip interconnect bandwidth when reading the key-value cache in the graphics processor, and improving the key-value cache read / write efficiency, thereby improving the task execution efficiency and quality of graphics processing. Furthermore, different compression strategies are selected for key vectors and value vectors to fully meet the compression accuracy and attention analysis requirements of different vector types.

[0095] Based on the same technical concept as the aforementioned graphics processor, this application also provides a key-value cache read / write method, which can be applied to the aforementioned graphics processor. For details, please refer to... Figure 3 The flowchart shown is shown.

[0096] Figure 3 This is a flowchart illustrating a key-value cache read / write method according to one embodiment of this application. Figure 3 As shown, the key-value cache read / write method provided in this application embodiment includes steps S301 to S304.

[0097] S301: Obtain the key-value cache access instruction and determine the access instruction parameters.

[0098] S302: When the access instruction parameter is a write access instruction parameter, the vector to be written is compressed based on the access instruction parameter, and the compressed vector data corresponding to the vector to be written is determined and stored.

[0099] S303: When the access instruction parameter is a read access instruction parameter, retrieve and determine the target vector compressed data based on the access instruction parameter.

[0100] Among them, the target vector compressed data is the vector compressed data corresponding to the target vector to be read.

[0101] S304: Decompress and reconstruct the target vector compressed data to obtain the target vector.

[0102] In some embodiments, before performing vector compression on the vector to be written based on access instruction parameters, determining the vector-compressed data corresponding to the vector to be written, and storing it, the method further includes: The vector to be written is divided into vector blocks to determine at least one logical vector block of the vector to be written. Based on the access command parameters, vector compression is performed on the vector to be written, and the vector compression data corresponding to the vector to be written is determined and stored, including: Perform vector compression on each logical vector block of the vector to be written, and determine the vector compressed data corresponding to each logical vector block.

[0103] In some embodiments, vector compression is performed on the vector to be written based on access instruction parameters to determine the vector-compressed data corresponding to the vector to be written, including: Norm extraction is performed on the vector to be written to determine its vector norm, and normalization is performed on the vector to be written to obtain the normalized vector. Perform an orthogonal transformation on the normalized vector to determine the transformed vector to be written into the vector. Based on a pre-defined codebook, non-uniform quantization is performed on the vector components of multiple dimensions in the transformed vector to determine the codebook index corresponding to each vector component. The codebook indexes corresponding to multiple vector components are packaged into data to determine the codebook index data of the vector to be written, as well as the metadata corresponding to the codebook index data. The codebook index data and metadata are used as the vector compression data corresponding to the vector to be written. The metadata is used by the data management unit to locate and parse the codebook index data during the process of retrieving the target vector compression data.

[0104] In some embodiments, based on a preset codebook, non-uniform quantization is performed on the vector components of multiple dimensions in the transformed vector to determine the codebook index corresponding to each vector component, including: When the vector to be written is a key vector, non-uniform quantization is performed on the vector components of multiple dimensions in the transformed vector of the key vector to determine the initial codebook index corresponding to each vector component of the key vector. For each vector component, based on the preset codebook, a codebook lookup is performed on the initial codebook index to determine the reconstructed vector corresponding to the initial codebook index; Based on the reconstructed vector and vector components, determine the residual vector of the vector components; The residual vector is quantized to determine the residual code corresponding to the residual vector, and the codebook index corresponding to the vector component is determined based on the initial codebook index and the residual code.

[0105] In some embodiments, the target vector compressed data is decompressed and reconstructed to obtain the target vector, including: Unpack the codebook index data in the compressed target vector data to determine the target codebook index corresponding to the multi-dimensional vector components in the target vector. Based on the preset codebook, the target codebook index is used to perform a codebook lookup to determine the target reconstructed vector of the target vector. Perform an orthogonal inverse transformation on the target reconstruction vector to obtain the inverse transformation vector of the target vector; Based on the vector norm corresponding to the compressed data of the target vector, the norm of the inverse transform vector is restored to obtain the restored vector, and the target vector is determined based on the restored vector.

[0106] In some embodiments, the target vector is determined based on the recovered vector, including: When the target vector is a key vector, the residual components of the multidimensional vector components in the key vector are determined based on the residual coding in the target codebook index of the key vector. Based on the residual components, residual correction is performed on the recovered vector, and the corrected vector is used as the target vector.

[0107] In some embodiments, the method further includes: Vector compressed data is transmitted between the graphics processor's key-value cache unit, off-chip storage, multi-level cache module, shared cache module, and on-chip interconnect module through a cache hierarchy interface unit.

[0108] In some embodiments, the method further includes: The hotspot vector is cached for a preset short period of time through the hotspot buffer unit. The hotspot vector is the target vector or logical vector block whose reading frequency exceeds the preset frequency threshold. Before determining the codebook index data of the vector to be written, the high-bit-width dimension vector and the low-bit-width dimension vector in the vector to be written are determined by the root mean square error of the vector to be written after non-uniform quantization, and the quantization bit width corresponding to the high-bit-width dimension vector is greater than the quantization bit width corresponding to the low-bit-width dimension vector. Compression mode switching is performed through the compression mode control unit. The compression modes include at least compression resident mode, bypass mode and hybrid mode. Compression resident mode means that vector compression is performed through the vector compression unit. Bypass mode means that vector compression is not performed. Hybrid mode means that compression resident mode and bypass mode are switched or combined according to the actual compression standard.

[0109] Figure 4 This is a schematic diagram of the structure of an electronic device 400 provided for an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or claimed herein.

[0110] like Figure 4 As shown, the electronic device 400 includes at least one graphics processor 100 and a memory, such as a read-only memory (ROM) 402 and a random access memory (RAM) 403, communicatively connected to the at least one graphics processor 100. The memory stores computer programs executable by the at least one graphics processor. The graphics processor 100 can perform various appropriate actions and processes based on the computer program stored in the ROM 402 or loaded from storage unit 408 into the RAM 403. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The graphics processor 100, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0111] Multiple components in electronic device 400 are connected to input / output (I / O) interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of monitors, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0112] The graphics processor 100 can be any dedicated graphics processing component with processing and computing capabilities. The graphics processor 100 executes the various methods and processes described above, such as the key-value cache read / write method mentioned above.

[0113] In some embodiments, the key-value cache read / write method described above can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 400 via read-only memory (ROM) 402 and / or communication unit 409. When the computer program is loaded into random access memory (RAM) 403 and executed by graphics processor 100, one or more steps of the key-value cache read / write method described above can be performed. Alternatively, in other embodiments, graphics processor 100 can be configured to perform the key-value cache read / write method described above by any other suitable means (e.g., by means of firmware).

[0114] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementation in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable graphics processor, which may be an application-specific programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0115] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to the graphics processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the graphics processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0116] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0117] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0118] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include: Local Area Networks (LANs), Wide Area Networks (WANs), blockchain networks, and the Internet.

[0119] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0120] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A graphics processor, characterized in that, It includes an instruction parsing unit, a vector compression unit, a key-value caching unit, a data management unit, and a vector decompression unit, wherein: The instruction parsing unit is used to parse key-value cache access instructions and determine access instruction parameters; The vector compression unit is used to perform vector compression on the vector to be written when the access instruction parameter is a write access instruction parameter, and to determine the vector compressed data corresponding to the vector to be written. The key-value cache unit is used to store the vector compressed data; The data management unit is used to retrieve target vector compressed data from the key-value cache unit when the access instruction parameter is a read access instruction parameter. The target vector compressed data is the vector compressed data corresponding to the target vector to be read. The vector decompression unit is used to decompress and reconstruct the target vector compressed data to obtain the target vector; The vector compression unit is further configured to, when the vector to be written is a key vector, perform non-uniform quantization on the vector components of multiple dimensions in the vector after orthogonal transformation of the key vector, and determine the initial codebook index corresponding to each vector component of the key vector; for each vector component, perform a codebook lookup on the initial codebook index based on a preset codebook to determine the reconstructed vector corresponding to the initial codebook index; determine the residual vector of the vector component based on the reconstructed vector and the vector component; perform quantization processing on the residual vector to determine the residual code corresponding to the residual vector; and determine the codebook index corresponding to the vector component based on the initial codebook index and the residual code. The vector compression unit is further configured to, when the vector to be written is a value vector, perform dimension-by-dimensional quantization on the value vector and use the determined initial codebook index as the codebook index of the value vector. The vector decompression unit is further configured to, when the target vector is a key vector, determine the residual components corresponding to the multi-dimensional vector components in the key vector based on the residual encoding in the codebook index corresponding to the key vector; and perform residual correction on the recovered vector of the key vector based on the residual components, and use the corrected vector as the target vector. The vector decompression unit is further configured to, when the target vector is a value vector, perform vector decompression and reconstruction on the value vector, and use the recovered vector after decompression and reconstruction of the value vector as the target vector.

2. The graphics processor according to claim 1, characterized in that, The graphics processor also includes a vector partitioning unit; The vector partitioning unit is used to partition the vector to be written into vector blocks and determine at least one logical vector block of the vector to be written. The vector compression unit is used to perform vector compression on each logical vector block of the vector to be written, and to determine the vector compressed data corresponding to each logical vector block.

3. The graphics processor according to claim 1 or 2, characterized in that, The vector compression unit is used to extract the norm of the vector to be written, determine the vector norm of the vector to be written, and normalize the vector to be written to obtain the normalized vector of the vector to be written; perform orthogonal transformation on the normalized vector to determine the transformed vector of the vector to be written; perform non-uniform quantization on the vector components of multiple dimensions in the transformed vector based on a preset codebook to determine the codebook index corresponding to each vector component; package the codebook indices corresponding to multiple vector components to determine the codebook index data of the vector to be written, and determine the metadata corresponding to the codebook index data; use the codebook index data and the metadata as the vector compression data corresponding to the vector to be written, and the metadata is used by the data management unit to locate and parse the codebook index data during the process of retrieving the target vector compression data.

4. The graphics processor according to claim 3, characterized in that, The vector decompression unit is used to unpack the codebook index data in the target vector compressed data to determine the target codebook index corresponding to the multi-dimensional vector components in the target vector; based on the preset codebook, perform a codebook lookup on the target codebook index to determine the target reconstructed vector of the target vector; perform an inverse orthogonal transformation on the target reconstructed vector to obtain the inverse transformation vector of the target vector; perform norm recovery on the inverse transformation vector based on the vector norm corresponding to the target vector compressed data to obtain the recovered vector; and determine the target vector based on the recovered vector.

5. The graphics processor according to claim 1, characterized in that, A cache hierarchy interface unit is provided between the key-value cache unit and the shared cache module on the graphics processor, the multi-level cache module and the on-chip interconnect module on the graphics processor, and the off-chip storage corresponding to the graphics processor. The cache layer interface unit is used for transmitting the vector compressed data between the key-value cache unit, the off-chip storage, the multi-level cache module, the shared cache module, and the on-chip interconnect module.

6. The graphics processor according to claim 1, characterized in that, The graphics processor also includes at least one of a hotspot buffer unit, a bit width allocation unit, and a compression mode control unit; The hotspot buffer unit is used to cache hotspot vectors for a preset short period of time. The hotspot vectors are target vectors or logical vector blocks whose reading frequency exceeds a preset frequency threshold. The bit width allocation unit is used to determine the high bit width dimension vector and the low bit width dimension vector in the vector to be written based on the root mean square error of the vector to be written after non-uniform quantization before the vector compression unit determines the codebook index data of the vector to be written. The quantization bit width corresponding to the high bit width dimension vector is greater than the quantization bit width corresponding to the low bit width dimension vector. The compression mode control unit is used to switch compression modes. The compression modes include at least a compression resident mode, a bypass mode, and a hybrid mode. The compression resident mode indicates that vector compression is performed through the vector compression unit. The bypass mode indicates that vector compression is not performed. The hybrid mode indicates that the compression resident mode and the bypass mode are switched or combined according to the actual compression standard.

7. The graphics processor according to claim 1, characterized in that, The key-value caching unit is located within a preset range around the attention data access path on the graphics processor, and the key-value caching unit is connected to the vector compression unit, the data management unit, and the vector decompression unit.

8. A key-value cache read / write method, characterized in that, The method is applied to a graphics processor, and the method includes: Retrieve the key-value cache access instruction and determine the access instruction parameters; When the access instruction parameter is a write access instruction parameter, the vector to be written is vector compressed based on the access instruction parameter, and the vector compressed data corresponding to the vector to be written is determined and stored. When the access instruction parameter is a read access instruction parameter, the target vector compressed data is retrieved and determined based on the access instruction parameter. The target vector compressed data is the vector compressed data corresponding to the target vector to be read. The target vector is obtained by decompressing and reconstructing the compressed data of the target vector. The step of performing vector compression on the vector to be written based on the access instruction parameters includes: When the vector to be written is a key vector, non-uniform quantization is performed on the vector components of multiple dimensions in the vector after orthogonal transformation of the key vector to determine the initial codebook index corresponding to each vector component of the key vector; for each vector component, a codebook lookup table is performed on the initial codebook index based on a preset codebook to determine the reconstructed vector corresponding to the initial codebook index; based on the reconstructed vector and the vector component, the residual vector of the vector component is determined; the residual vector is quantized to determine the residual code corresponding to the residual vector, and the codebook index corresponding to the vector component is determined according to the initial codebook index and the residual code; When the vector to be written is a value vector, the value vector is quantized dimension by dimension, and the determined initial codebook index is used as the codebook index of the value vector. The process of decompressing and reconstructing the compressed data of the target vector to obtain the target vector includes: When the target vector is a key vector, the residual components corresponding to the multi-dimensional vector components in the key vector are determined based on the residual encoding in the codebook index corresponding to the key vector; based on the residual components, the recovered vector of the key vector is residual corrected, and the corrected vector is used as the target vector. When the target vector is a value vector, the value vector is decompressed and reconstructed, and the recovered vector after decompression and reconstruction is used as the target vector.

9. An electronic device, characterized in that, The electronic device includes: at least one graphics processor; and a memory communicatively connected to the at least one graphics processor; wherein the memory stores a computer program executable by the at least one graphics processor, the computer program being executed by the at least one graphics processor to enable the at least one graphics processor to perform the key-value cache read / write method of claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the graphics processor to execute the key-value cache read / write method of claim 8.

Images