Key value cache grouping quantification method and device, storage medium and electronic equipment

Abstract

The invention discloses a key value cache grouping quantification method and device, a storage medium and electronic equipment. When the key value cache grouping quantification method provided by the specification is adopted to quantify key value cache data, key vector and value vector data are divided based on channel dimensions to obtain a plurality of grouped data; determining quantization parameters of the grouped data, and performing asymmetric quantization on elements in the grouped data based on the quantization parameters; finally, quantization results and corresponding quantization parameter partitions may be stored in physical blocks. According to the method, on the premise of ensuring the model generation precision, the video memory occupation of key value cache is greatly compressed, and meanwhile, the reasoning throughput is improved. Through fusion of dynamic channel grouping quantization and implicit inverse quantization, a key value cache quantization solution capable of guaranteeing the generation precision is provided for edge device end-side deployment, and the technical defect that the occupation scale of a video memory is too large when the occupation amount of the key value cache video memory is linearly increased along with the sequence length in the autoregression decoding process of a large language model is relieved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a key-value cache group quantization method, device, storage medium, and electronic device. Background Art

[0002] The memory usage of the dynamically constructed key-value cache in the autoregressive decoding mechanism increases linearly with sequence length. This excessive memory usage has become a bottleneck for long-context processing. Existing technologies alleviate memory pressure by reducing the numerical precision of the key-value cache, but this faces a dilemma in quantization strategies: per-word quantization ignores differences in channel dimensional distribution, leading to distortion of attention weights and error accumulation; while per-channel quantization improves accuracy, it violates the timing constraints of streaming decoding because it requires global parameter updates.

[0003] The hardware limitations of consumer devices further exacerbate this problem. Traditional quantization schemes are not sufficiently compatible with hardware computing units, making it difficult to support traditional quantization schemes while meeting the stringent resource requirements of both generation quality and real-time inference.

[0004] Therefore, how to achieve quantification that takes into account both accuracy and resource requirements to break through the bottleneck of video memory usage of key-value cache is an urgent problem to be solved. Summary of the Invention

[0005] This specification provides a key-value cache group quantization method, device, storage medium and electronic device to at least partially solve the above-mentioned problems existing in the prior art.

[0006] This manual adopts the following technical solutions:

[0007] This specification provides a key-value cache grouping quantization method, including:

[0008] Divide the vector to be processed based on the channel dimension to obtain several grouped data;

[0009] Determine the quantization parameter of each group of data according to the maximum and minimum values ​​of the elements contained in the group of data;

[0010] For each group of data, quantize each element contained in the group of data using the quantization parameter of the group of data;

[0011] Creating a physical block for storing quantization results, wherein the physical block includes a parameter area and a data area;

[0012] The quantization parameter of each grouped data is stored in the parameter area of ​​the physical block, and the quantization result of each grouped data is stored in the data area of ​​the physical block.

[0013] Optionally, each packet of data contains the same number of channels.

[0014] Optionally, determining the quantization parameter of each group of data according to the maximum value and the minimum value of the elements contained in the group of data specifically includes:

[0015] For each group of data, determining the difference between the maximum value and the minimum value of the elements contained in the group of data;

[0016] A quantization parameter is determined according to a difference between the maximum value and the minimum value and a predetermined quantization result range.

[0017] Optionally, a range of quantification results is predetermined, specifically including:

[0018] Determine the data type and number of bits of the quantization result;

[0019] The range of the quantization result is determined according to the data type and the number of bits.

[0020] Optionally, quantizing each element contained in the grouped data using a quantization parameter of the grouped data specifically includes:

[0021] For each element included in the grouped data, determining a difference between the element and a minimum value of the elements included in the grouped data;

[0022] A quantized value of the element is obtained according to a difference between the element and a minimum value of elements included in the grouped data and a quantization parameter of the grouped data.

[0023] Optionally, the data types of the quantization parameters of the grouped data are the same, the data types of the quantization results of the grouped data are the same, and the data types of the quantization parameters and the quantization results are different.

[0024] Optionally, after quantizing each element contained in each group of data by using the quantization parameter of the group of data, the method further includes:

[0025] In response to any element being called, the element is restored according to the quantization value of the element and the quantization parameter of the grouped data where the element is located.

[0026] This specification provides a key-value cache group quantization device, the device comprising:

[0027] A partitioning module is used to partition the vector to be processed based on the channel dimension to obtain a number of grouped data;

[0028] A determination module, configured to determine a quantization parameter of each group of data according to a maximum value and a minimum value of elements contained in the group of data;

[0029] A quantization module, configured to quantize each element contained in each group of data in turn using the quantization parameter of the group of data;

[0030] A creation module, configured to create a physical block for storing quantization results, wherein the physical block includes a parameter area and a data area;

[0031] The storage module is used to store the quantization parameter of each grouped data in the parameter area of ​​the physical block, and store the quantization result of each grouped data in the data area of ​​the physical block.

[0032] This specification provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned key-value cache grouping quantization method.

[0033] This specification provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned key-value cache grouping quantization method is implemented.

[0034] At least one of the above technical solutions adopted in this specification can achieve the following beneficial effects:

[0035] In the key-value cache group quantization method provided in this specification, a vector to be processed is divided based on the channel dimension to obtain a number of grouped data; a quantization parameter of the group of data is determined based on the maximum and minimum values ​​of the elements contained in each group of data; for each group of data, the quantization parameter of the group of data is used to quantize each element contained in the group of data; a physical block for storing the quantization results is created, and the physical block includes a parameter area and a data area; the quantization parameter of each group of data is stored in the parameter area of ​​the physical block, and the quantization result of each group of data is stored in the data area of ​​the physical block.

[0036] When the key-value cache group quantization method provided in this specification is used to quantize the key-value cache data, the key vector and value vector data can be divided based on the channel dimension to obtain a number of grouped data; the quantization parameters of each grouped data are determined respectively, and based on this, the elements in each grouped data are asymmetric quantized; finally, the quantization results and the corresponding quantization parameter partitions can be stored in physical blocks. This method can greatly compress the video memory occupancy of the key-value cache while ensuring the accuracy of model generation, and at the same time improve the inference throughput. By fusing dynamic channel group quantization and implicit dequantization, a key-value cache quantization solution that can guarantee generation accuracy is provided for edge device side deployment, alleviating the technical defect of excessive video memory occupancy when the video memory occupancy of the key-value cache increases linearly with the sequence length during autoregressive decoding of large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of this specification and constitute a part of this specification. The exemplary embodiments and descriptions of this specification are used to explain this specification and do not constitute an improper limitation of this specification. In the drawings:

[0038] Figure 1 This is a flowchart of a key-value cache grouping quantization method in this specification;

[0039] Figure 2 This is a schematic diagram of partition storage in a physical block in this specification;

[0040] Figure 3 This is a schematic diagram of a key-value cache group quantization device provided in this specification;

[0041] Figure 4 The corresponding Figure 1 Schematic diagram of electronic equipment. DETAILED DESCRIPTION

[0042] To make the purpose, technical solutions, and advantages of this specification more clear, the technical solutions of this specification will be clearly and completely described below in conjunction with the specific embodiments of this specification and the corresponding drawings. Obviously, the embodiments described are only part of the embodiments of this specification, not all of the embodiments. Based on the embodiments in this specification, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0043] The technical solutions provided by the embodiments of this specification are described in detail below with reference to the accompanying drawings.

[0044] Figure 1 The following is a flowchart of a key-value cache group quantization method in this specification, which specifically includes the following steps:

[0045] S100: Divide the vector to be processed based on the channel dimension to obtain a number of grouped data.

[0046] All steps in the key-value cache grouping quantization method provided in this specification can be implemented by any electronic device with computing capabilities, such as a terminal, a server, and the like.

[0047] This method is mainly used to quantize the key-value cache generated by a large generative model when executing computational tasks. Before quantization, all key-value caches in the large model can be obtained and the key vectors and value vectors in them are used as the vectors to be processed.

[0048] In this step, the vector to be processed can be divided based on the channel dimension to obtain several groups of different grouped data, each of which contains vector data of several channels. The channel of the key vector and the value vector is the feature dimension d of the vector. k , used to represent the abstract features of the corresponding word (token) at different levels. Generally speaking, the larger the number of channels, the more complex the encoded features and the meaning that can be represented.

[0049] The key vector and value vector in a token's key-value cache can be divided and quantized separately, and the processing methods adopted can be the same.

[0050] When the vector to be processed is divided, in order to make the subsequent quantization processing operation more efficient, the number of channels contained in each grouped data obtained by the division can be the same. The number of channels contained in each grouped data can be set according to specific needs, and this specification does not impose any specific restrictions on this.

[0051] For example, suppose the key vector t of the current token is K ∈R d , the channel dimension d = 1024; in this case, it can be divided into G = d / g = 32 groups according to the fixed grouping granularity g = 32. In other words, the key vector with 1024 channels is divided so that each group contains 32 channels, resulting in 32 grouped data.

[0052] The above embodiment is based on the key vector t K Taking the embodiment as an example, when the vector to be processed is a value vector t V When the data are grouped, they can also be grouped in the same way, which will not be described in detail in this manual.

[0053] In this method, each group of data obtained by division will be quantized independently in the subsequent steps, and the group data will not affect each other, thus realizing asymmetric quantization.

[0054] S102: Determine a quantization parameter of each group of data according to the maximum value and the minimum value of the elements contained in the group of data.

[0055] After completing the grouping of the vectors to be processed in step S100 and obtaining a number of grouped data, the independent quantization of each grouped data can be started in this step. First, the quantization parameters for each grouped data can be determined. Generally, the quantization parameters may include a scaling factor s and a zero point z. The scaling factor s defines the mapping ratio of floating-point values ​​to integers, and the zero point z represents the corresponding value of 0 in the pre-quantized data in the quantized data.

[0056] In this method, the quantization parameter of each group of data can be determined based on the maximum and minimum values ​​of the elements included in the group of data. Specifically, for each group of data, the difference between the maximum and minimum values ​​of the elements included in the group of data can be determined; and the quantization parameter can be determined based on the difference between the maximum and minimum values ​​and a predetermined range of quantization results.

[0057] In one embodiment, t (i) represents the i-th group data, s i 、z i denote the scaling factor and zero point of the i-th group of grouped data respectively. Then the quantization parameter of the i-th group of grouped data can be expressed by the following formula:

[0058]

[0059] Among them, α i =max(t (i) ), that is, t (i) The maximum value in β i =min(t (i) ), that is, t (i) The minimum value in b bit Indicates the number of bits of data obtained after quantization, Characterizes the range of quantitative results. For example, when α i =2.8,β i =-1.2, b bit =4, there is s i =0.2667, z i =5.

[0060] Generally, the data type of the data before quantization is a data type with a longer number of bits, such as a floating-point number; the data type of the data after quantization is a data type with a shorter number of bits, such as an integer.

[0061] In this method, the range of the quantization results can be predetermined according to the quantization requirements. Specifically, the data type and the number of bits of the quantization results can be determined; and the range of the quantization results can be determined according to the data type and the number of bits.

[0062] Following the above embodiment, when the type of the quantization result is determined to be integer and the number of bits is determined to be 4, that is, when the quantization obtains data of type INT4, the quantization result range can be determined to be 2 4 -1.

[0063] S104: quantizing each element included in each group of data using the quantization parameter of the group of data.

[0064] After the quantization parameter of each group of data is determined in step S102 , the elements in each group of data may be quantized based on the quantization parameter in this step.

[0065] During quantization, each grouped data element is quantized using its own quantization parameter. Specifically, for each element in the grouped data, the difference between the element and the minimum value of the elements in the grouped data is determined; the quantized value of the element is obtained based on the difference between the element and the minimum value of the elements in the grouped data and the quantization parameter of the grouped data.

[0066] The quantization process of each element can be expressed as the following formula:

[0067]

[0068] Among them, x represents a single element in the grouped data, Quant(x) represents the quantization result of element x, β i Indicates the minimum value in the grouped data where x is located. Indicates the value range of the quantization result.

[0069] At this point, asymmetric quantization of all data in the grouped data is completed in this step, that is, the quantization of each key vector and value vector, i.e., the key-value cache, is completed. Each grouped data is quantized using its own quantization parameter, and they do not affect each other.

[0070] Continuing with the above embodiment, when FP16 type data is quantized into INT4 type data, when the number of channels is 1024 and each divided packet data contains 32 channel data, the total memory usage after quantization is:

[0071]

[0072] Compared with the original memory usage of FP16 (16384 bits), the memory compression rate is 31.25%.

[0073] S106: Create a physical block for storing the quantization result, wherein the physical block includes a parameter area and a data area.

[0074] After the data in each key-value cache is quantized through the aforementioned steps of the method, each quantization result and quantization parameter can be stored in this step in the form of a correspondence between the quantization result and the quantization parameter.

[0075] According to the change of data volume, any number of physical blocks for storing quantization results can be created, and this specification does not impose any specific limitation on this.

[0076] S108: storing the quantization parameter of each grouped data into the parameter area of ​​the physical block, and storing the quantization result of each grouped data into the data area of ​​the physical block.

[0077] In this step, the quantization results of each grouped data and the corresponding quantization parameters can be stored together in the physical block created in step S106. The parameter area is used to store the quantization parameters of each grouped data, and the data area is used to store the quantization results of each grouped data.

[0078] Figure 2 This specification provides a schematic diagram of partition storage in a physical block. Figure 2 In the illustrated embodiment, a parameter area and a data area can be created within a physical block. The parameter area stores quantization parameters as 16-bit floating-point numbers, while the data area stores quantization results as 4-bit integers. si and zi represent the scaling factor and zero point of the i-th group of grouped data, respectively, and Quant(t(i)) represents the quantization results for all elements in the i-th group of grouped data.

[0079] The data type of the quantization parameter and quantization result provided in the above embodiment is only one of a variety of optional data types, and other data types may be used according to specific requirements. It is worth mentioning that the data type of each quantization parameter stored in the physical block should be the same, and the data type of each quantization result should also be the same, while the data types of the quantization parameter and the quantization result are different.

[0080] It is worth mentioning that during the storage process, in order to facilitate subsequent calls and searches, the quantization parameters and quantization results corresponding to each grouped data should be sorted and stored in the same order in the parameter area and data area of ​​the physical block. Figure 2 In the specific embodiment shown, the parameter area and the data area are stored in the order of 1, 2, 3, ..., 32, and the two sides correspond to each other.

[0081] Additionally, when the original key-value cache data needs to be called, the corresponding quantization parameter can be used to perform a dequantization operation on the quantization result to restore the original data. That is, in response to any element being called, the element is restored based on the quantization value of the element and the quantization parameter of the grouped data in which the element is located.

[0082] At the same time, by fusing inverse quantization and matrix multiplication in the computational pipeline, explicit data conversion overhead can be avoided to obtain the required attention score. The calculation formula can be shown as follows:

[0083]

[0084] Among them, K fp16 represents the original data, K int4 Indicates the quantization result, sK 、z K represents the quantization parameter, Q fp16 Denotes the corresponding Query, and AttnScore is the attention score to be calculated. The above dequantization process does not require explicit data conversion and can obtain the original data and the required attention score for calculation.

[0085] When the key-value cache group quantization method provided in this specification is used to quantize the key-value cache data, the key vector and value vector data can be divided based on the channel dimension to obtain a number of grouped data; the quantization parameters of each grouped data are determined respectively, and based on this, the elements in each grouped data are asymmetric quantized; finally, the quantization results and the corresponding quantization parameter partitions can be stored in physical blocks. This method can greatly compress the video memory occupancy of the key-value cache while ensuring the accuracy of model generation, and at the same time improve the inference throughput. By fusing dynamic channel group quantization and implicit dequantization, a key-value cache quantization solution that can guarantee generation accuracy is provided for edge device side deployment, alleviating the technical defect of excessive video memory occupancy when the video memory occupancy of the key-value cache increases linearly with the sequence length during autoregressive decoding of large language models.

[0086] The above is the key-value cache grouping quantization method provided in this specification. Based on the same idea, this specification also provides a corresponding key-value cache grouping quantization device, such as Figure 3 shown.

[0087] Figure 3 This is a schematic diagram of a key-value cache group quantization device provided in this specification, specifically including:

[0088] A division module 200 is used to divide the vector to be processed based on the channel dimension to obtain a plurality of grouped data;

[0089] A determination module 202 is configured to determine a quantization parameter of each group of data based on a maximum value and a minimum value of elements contained in the group of data;

[0090] The quantization module 204 is configured to quantize each element of each group of data using the quantization parameter of the group of data in turn;

[0091] A creation module 206 is configured to create a physical block for storing quantization results, wherein the physical block includes a parameter area and a data area;

[0092] The storage module 208 is configured to store the quantization parameter of each group of data into the parameter area of ​​the physical block, and store the quantization result of each group of data into the data area of ​​the physical block.

[0093] Optionally, each packet of data contains the same number of channels.

[0094] Optionally, the determination module 202 is specifically configured to determine, for each group of data, a difference between a maximum value and a minimum value of elements contained in the group of data; and determine a quantization parameter according to the difference between the maximum value and the minimum value and a predetermined quantization result range.

[0095] Optionally, the apparatus further includes a pre-determination module 210, specifically configured to determine a data type and a number of bits of the quantization result; and determine a range of the quantization result according to the data type and the number of bits.

[0096] Optionally, the quantization module 204 is specifically used to determine, for each element contained in the grouped data, the difference between the element and the minimum value of the elements contained in the grouped data; and obtain the quantization value of the element based on the difference between the element and the minimum value of the elements contained in the grouped data and the quantization parameter of the grouped data.

[0097] Optionally, the data types of the quantization parameters of the grouped data are the same, the data types of the quantization results of the grouped data are the same, and the data types of the quantization parameters and the quantization results are different.

[0098] Optionally, the apparatus further includes a restoration module 212, which is specifically configured to, in response to any element being called, restore the element according to the quantization value of the element and the quantization parameter of the grouped data where the element is located.

[0099] This specification also provides a computer-readable storage medium, which stores a computer program that can be used to execute the above Figure 1 Provides a key-value cache grouping quantization method.

[0100] This manual also provides Figure 4 The schematic structure diagram of the electronic device shown in FIG. Figure 4 As mentioned above, at the hardware level, the electronic device includes a processor, an internal bus, a network interface, a memory and a non-volatile memory, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The key-value cache grouping quantization method. Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of software and hardware, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0101] Improvements to a technology can be clearly distinguished as either hardware improvements (for example, improvements to circuit structures such as diodes, transistors, and switches) or software improvements (improvements to process flows). However, with technological advancements, many process flow improvements today can now be considered direct improvements to hardware circuit structures. Designers almost always program the improved process flow into the hardware circuit to obtain the corresponding hardware circuit structure. Therefore, it cannot be said that a process flow improvement cannot be implemented using a hardware module. For example, a programmable logic device (PLD), such as a field programmable gate array (FPGA), is an integrated circuit whose logical function is determined by user programming. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to hire a chip manufacturer to design and manufacture a dedicated integrated circuit chip. Moreover, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly done using "logic compiler" software. This is similar to the software compiler used when developing programs. Before compilation, the original code must also be written in a specific programming language, called a hardware description language (HDL). There is not just one HDL, but many, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art will also understand that by simply programming the method flow in one of these hardware description languages ​​and then programming it into an integrated circuit, a hardware circuit that implements the logic method flow can be easily obtained.

[0102] The controller can be implemented in any suitable manner. For example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also know that in addition to implementing the controller in a purely computer-readable program code format, the controller can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, such a controller can be considered a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component. Or even, the devices for implementing various functions can be considered as both software modules that implement the method and structures within the hardware component.

[0103] The systems, devices, modules, or units described in the above embodiments may be implemented by computer chips or entities, or by products having certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0104] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0105] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0106] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0107] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0108] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0109] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0110] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0111] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0112] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0113] Those skilled in the art will appreciate that the embodiments of this specification may be provided as methods, systems, or computer program products. Thus, this specification may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0114] This specification may be described in the general context of computer-executable instructions, such as program modules, executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. This specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media, including storage devices.

[0115] The various embodiments in this specification are described in a progressive manner. Similar parts between the various embodiments can be referred to in conjunction with each other. Each embodiment focuses on the differences between the other embodiments. In particular, the system embodiments are generally similar to the method embodiments, so the description is relatively simple. For relevant parts, refer to the description of the method embodiments.

[0116] The foregoing is merely an embodiment of the present invention and is not intended to limit the present invention. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention are intended to be included within the scope of the claims of this application.

Claims

1. A key-value cache grouping quantization method, characterized in that: include: Divide the vector to be processed based on the channel dimension to obtain several grouped data; Determine the quantization parameter of each group of data according to the maximum and minimum values ​​of the elements contained in the group of data; For each group of data, quantize each element contained in the group of data using the quantization parameter of the group of data; Creating a physical block for storing quantization results, wherein the physical block includes a parameter area and a data area; The quantization parameter of each grouped data is stored in the parameter area of ​​the physical block, and the quantization result of each grouped data is stored in the data area of ​​the physical block.

2. The method according to claim 1, wherein Each group of data contains the same number of channels.

3. The method according to claim 1, wherein Determine the quantization parameters of each group of data based on the maximum and minimum values ​​of the elements contained in the group of data, specifically including: For each group of data, determining the difference between the maximum value and the minimum value of the elements contained in the group of data; A quantization parameter is determined according to a difference between the maximum value and the minimum value and a predetermined quantization result range.

4. The method according to claim 3, wherein Predetermine the range of quantitative results, including: Determine the data type and number of bits of the quantization result; The range of the quantization result is determined according to the data type and the number of bits.

5. The method according to claim 1, wherein Quantizing each element contained in the grouped data using the quantization parameter of the grouped data, specifically including: For each element included in the grouped data, determining a difference between the element and a minimum value of the elements included in the grouped data; A quantized value of the element is obtained according to a difference between the element and a minimum value of elements included in the grouped data and a quantization parameter of the grouped data.

6. The method according to claim 1, wherein The data types of the quantization parameters of the grouped data are the same, the data types of the quantization results of the grouped data are the same, and the data types of the quantization parameters and the quantization results are different.

7. The method according to claim 1, wherein After quantizing each element contained in each group of data by using the quantization parameter of the group of data, the method further includes: In response to any element being called, the element is restored according to the quantization value of the element and the quantization parameter of the grouped data where the element is located.

8. A key-value cache group quantization device, characterized in that: include: A partitioning module is used to partition the vector to be processed based on the channel dimension to obtain a number of grouped data; A determination module, configured to determine a quantization parameter of each group of data according to a maximum value and a minimum value of elements contained in the group of data; A quantization module, configured to quantize each element contained in each group of data in turn using the quantization parameter of the group of data; A creation module, configured to create a physical block for storing quantization results, wherein the physical block includes a parameter area and a data area; The storage module is used to store the quantization parameter of each grouped data in the parameter area of ​​the physical block, and store the quantization result of each grouped data in the data area of ​​the physical block.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 7 is implemented.

Images