Self-attention mechanism calculation structure based on hybrid quantization precision key value cache
By designing a computing structure based on mixed quantization accuracy in the self-attention mechanism, dynamically adjusting the quantization accuracy and optimizing the key-value cache, the problems of low computational efficiency and low storage resource utilization of the self-attention mechanism are solved, and the large language model is realized Efficient computing and resource optimization.
Patent Information
- Application Number
- CN202411075896.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-07
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-07
AI Technical Summary
The self-attention mechanism has high computational complexity and storage requirements in deep learning models, especially in low computing efficiency when processing large-scale data. Traditional hybrid quantization accuracy technology has failed to effectively solve the problems of key-value cache and nonlinear unit computing time.
Design a self-attention mechanism calculation structure based on mixed quantization accuracy, including input data quantization module, self-attention mechanism calculation module, n-m inverse quantization operation module and key-value cache module. By dynamically adjusting the quantization accuracy, the key-value cache is optimized. Storage and computing efficiency, achieving calculation difference-load difference matching to reduce data transmission delay and energy consumption.
It significantly reduces the computational complexity and storage overhead, improves the computing efficiency and resource utilization of the self-attention mechanism, realizes the balance between inference accuracy and computing efficiency, and is suitable for efficient computing of large language models.
Smart Images

Figure CN119047527A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of artificial intelligence and integrated circuits. Specifically, the present invention discloses a self-attention mechanism computing structure based on a mixed quantization precision key-value cache and an implementation method thereof, which belongs to the technical field of computing, inference or counting. Background Art
[0002] As an important technology in deep learning, the self-attention mechanism has been widely used in natural language processing, computer vision and other fields. In the self-attention mechanism, each element of the input sequence is associated with all other elements in the sequence to generate a set of attention weights, which are used for weighted summation to capture the relationship between elements in the sequence. This mechanism performs well in improving model performance and capturing long-distance dependencies, but it also brings huge computational and storage requirements.
[0003] As the scale of deep learning models continues to expand, the computational complexity and storage requirements of the models also increase. In particular, in the self-attention mechanism, due to the existence of nonlinear functions and the need to calculate and store a large number of attention weights, the consumption of computing resources and storage resources has increased significantly, and the computational efficiency is low, especially when processing large-scale data. The problem of low computational efficiency is particularly obvious. This poses a huge challenge to resource-limited devices, such as mobile devices and embedded systems. To meet these challenges, researchers have proposed a variety of optimization methods, including model compression, quantization, and hardware acceleration. Among them, quantization technology has received widespread attention because it can significantly reduce the bit width of model parameters, thereby reducing computing and storage requirements.
[0004] Hybrid quantization precision is a method that combines high-precision and low-precision representations. By using different quantization precisions in different parts, the purpose of minimizing computing and storage resource consumption while ensuring model performance is achieved. However, the direct application of hybrid quantization precision technology in the self-attention mechanism still faces many technical difficulties. First, the calculation process of the self-attention mechanism involves a large number of matrix multiplication and addition operations and nonlinear functions. How to efficiently implement these operations under hybrid quantization precision is a key issue. Secondly, data conversion and key-value cache (KVcache) between different quantization precisions are also a complex process. When storing the key matrix and value matrix of historical data, traditional data caching methods fail to fully utilize the characteristics of their quantization precision, resulting in a waste of storage resources. How to effectively manage and schedule these cache resources to reduce the delay and energy consumption caused by data transmission is also a problem that needs to be solved. There are some existing solutions, such as precision adjustment of weight matrices, interlaced linear arrangement strategies, and in-memory computing unit design for hybrid precision neural networks. However, most of the above hybrid precision quantization methods do not consider the key-value cache problem existing in large language models, the time problem for nonlinear unit calculations, and the loading time problem of hybrid quantization data. In the reasoning process of large language models, how to reduce the computational complexity and achieve a balance between computational efficiency and reasoning accuracy while ensuring the model reasoning accuracy is a technical problem that needs to be solved urgently.
[0005] In response to the above problems, the present invention proposes a self-attention mechanism calculation structure and implementation method based on mixed quantization precision key-value cache. Summary of the invention
[0006] The purpose of the present invention is to address the deficiencies of the above-mentioned background technology. On the basis of considering the key-value cache, an efficient mixed quantization precision self-attention mechanism computing architecture is designed to optimize the storage problem of the key-value cache, improve the utilization efficiency of the computing time of the self-attention mechanism, balance the time difference between data calculation and data loading, and achieve the purpose of accelerating the calculation of the self-attention mechanism and optimizing the utilization of storage resources, thereby solving the technical problems of low computing efficiency and low storage resource utilization of the existing self-attention mechanism computing structure.
[0007] The present invention adopts the following technical solutions to achieve the above-mentioned invention object:
[0008] A self-attention mechanism computing structure based on mixed quantization precision key-value cache, comprising: an input data quantization module, a self-attention mechanism computing module, an nm anti-quantization operation module, a calculation difference-load difference matching module and a key-value cache module;
[0009] An input data quantization module, used for quantizing the input data into an n-bit query matrix, an n-bit key matrix and an m-bit value matrix;
[0010] A self-attention mechanism calculation module is used to calculate a softmax function value with an accuracy of n bits according to an n-bit query matrix and an n-bit key matrix, and to perform a multiplication operation according to an m-bit value matrix and a softmax function value with an accuracy of m bits to obtain a calculation result of the self-attention mechanism;
[0011] nm dequantization operation module, used to dequantize the softmax function value with an accuracy of n bits into a softmax function value with an accuracy of m bits, and obtain the calculation period difference between the key matrix and the value matrix;
[0012] A calculation difference-loading difference matching module matches the loading cycle difference according to the calculation cycle difference of the key matrix and the value matrix, and selects a hybrid quantization precision scheme with the best quantization model performance from the hybrid quantization precision schemes that meet the loading cycle difference. The hybrid quantization precision scheme with the best quantization model performance includes an n-bit key matrix and an m-bit value matrix, where n and m are positive integers.
[0013] The key-value cache module is used to cache the hybrid quantization precision solution with the best performance of the quantization model, and loads the n-bit key matrix and the m-bit value matrix into the autonomous mechanism calculation module according to the loading cycle difference.
[0014] As a further optimization scheme for the computational structure of the self-attention mechanism based on the hybrid quantization precision key-value cache, the nm dequantization operation module dequantizes the softmax function value with an accuracy of n bits to the softmax function value with an accuracy of m bits. The expression is: Among them, the input data is the n-bit softmax function value, the quantization factor 1 is the factor of the quantization key matrix of the input data quantization module, the quantization factor 2 is the factor of the quantization value matrix of the input data quantization module, and the output data is the m-bit softmax function value.
[0015] As a further optimization scheme for the computational structure of the self-attention mechanism based on mixed quantization precision key-value cache, the computational period difference between the key matrix and the value matrix is the sum of the Softmax function processing period and the nm dequantization operation period.
[0016] As a further optimization scheme for the calculation structure of the self-attention mechanism based on the hybrid quantization precision key-value cache, the calculation difference-loading difference matching module matches the loading cycle difference according to the calculation cycle difference of the key matrix and the value matrix. Specifically, the loading cycle difference is calculated under the constraint that the calculation cycle difference ≥ the loading cycle difference.
[0017] As a further optimization scheme for the calculation structure of the self-attention mechanism based on the hybrid quantization precision key-value cache, the softmax function is Among them, x is all the data in a row of the product result of the query matrix and the transposed matrix of the key matrix, and x i is the i-th data in x, and n is the total number of data included in x.
[0018] As a further optimization solution for the calculation structure of the self-attention mechanism based on the mixed quantization precision key-value cache, the quantization model performance includes but is not limited to: model compression precision and inference precision.
[0019] As a further optimization solution for the calculation structure of the self-attention mechanism based on the mixed quantization precision key-value cache, the calculation difference-loading difference matching module matches the loading cycle difference according to the calculation cycle difference between the key matrix and the value matrix, and selects the mixed quantization precision scheme with the optimal quantization model performance from the mixed quantization precision schemes that meet the loading cycle difference. Specifically: initialize n and m as different quantization precision values, n < m, start the first iteration of the optimal mixed quantization precision scheme, obtain the first iteration result of the optimal quantization precision scheme, broadcast the corresponding mixed quantization precision n and m of the first iteration result to the input data quantization module, the self-attention mechanism calculation module, the n-m dequantization operation module and the key-value cache module, and then perform subsequent iterative optimization of the optimal mixed quantization precision scheme according to the loading cycle difference updated by the n-m dequantization operation module.
[0020] As a further optimization solution for the calculation structure of the self-attention mechanism based on the mixed quantization precision key-value cache, during the first iteration of the optimal mixed quantization precision scheme, the self-attention mechanism calculation module calculates the softmax function value with n-bit precision according to the n-bit query matrix output by the input data quantization module, the n-bit key matrix output by the input data quantization module, and the n-bit key matrix calculation precision loaded by the key-value cache module, and performs a multiplication operation according to the m-bit value matrix output by the input data quantization module, the m-bit value matrix loaded by the key-value cache module, and the softmax function value with m-bit precision.
[0021] As a further optimization solution for the calculation structure of the self-attention mechanism based on the mixed quantization precision key-value cache, during the subsequent iterative optimization of the optimal mixed quantization precision scheme, the self-attention mechanism calculation module calculates the softmax function value with n-bit precision according to the n-bit query matrix and the n-bit key matrix output by the input data quantization module, and performs a multiplication operation according to the m-bit value matrix output by the input data quantization module and the softmax function value with m-bit precision.
[0022] The present invention adopts the above technical solutions and has the following beneficial effects:
[0023] (1) The present invention provides a self-attention mechanism computing structure based on hybrid quantization precision key-value cache, which is suitable for efficient computing of large language model reasoning. By including a key-value cache module based on hybrid quantization precision, an nm dequantization operation module and a calculation difference-load difference matching module, a reasonable trade-off between accuracy, computing performance and resource utilization is achieved, and the technical advantages of low power consumption, high energy efficiency and low latency are achieved. By dynamically adjusting the hybrid quantization precision technology, the computational complexity is significantly reduced while ensuring the reasoning accuracy of the large language model, thereby achieving the best balance between reasoning accuracy and computational efficiency.
[0024] (2) The computing structure proposed in the present invention uses n quantization precision and m quantization precision to store the key matrix and value matrix of historical data respectively according to the time requirements and quantization precision requirements of the key matrix and value matrix for different self-attention mechanism calculation parts, which effectively reduces the storage overhead, reduces the computational complexity, and improves the computational efficiency of the self-attention mechanism.
[0025] (3) The computing structure proposed in the present invention not only improves the accuracy of model reasoning, but also significantly reduces computing delay and energy consumption by optimizing the computing strategy for dynamically adjusting quantization accuracy. It is suitable for the self-attention mechanism in a variety of complex computing scenarios and provides a more efficient and accurate solution for neural network operations. BRIEF DESCRIPTION OF THE DRAWINGS
[0026] Figure 1 This is a block diagram of the computing structure of the self-attention mechanism based on mixed quantization precision key-value cache disclosed in the present invention.
[0027] Figure 2 The figure is a schematic diagram of the input and output data flow of the nm inverse quantization operation module disclosed in the present invention.
[0028] Figure 3 It is a schematic diagram of the input and output data flow of the calculation difference-loading difference matching module and the nm quantization precision solution set disclosed in the present invention.
[0029] Figure 4 A schematic diagram of the calculation difference-load difference matching module disclosed in the present invention generating a quantization scheme for key-value caching.
[0030] Figure 5 The present invention discloses a flowchart of a method for implementing a self-attention mechanism computing structure based on a mixed quantization precision key-value cache. DETAILED DESCRIPTION
[0031] The technical solution of the invention is described in detail below in conjunction with the accompanying drawings. It should be understood that these embodiments are only used to illustrate the invention and are not used to limit the scope of the invention. After reading the invention, various equivalent modifications made to the invention by those skilled in the art all fall within the scope defined by the claims attached to the invention.
[0032] The present invention provides a self-attention mechanism computing structure based on mixed quantization precision key-value cache, such as Figure 1 As shown, it specifically includes a self-attention mechanism calculation module, an input data quantization module, a key-value cache module of mixed quantization precision, an nm dequantization operation module, and a calculation difference-loading difference matching module. The time requirement for the key matrix and the value matrix to participate in the calculation of the self-attention mechanism, that is, the calculation cycle difference, is matched to the loading cycle difference of the key matrix loading and the value matrix loading, so as to realize the dynamic adjustment of the calculation accuracy and model compression of the self-attention mechanism.
[0033] The self-attention mechanism calculation module is used to complete the core calculation function of the self-attention mechanism. The input data includes the query matrix Query and key matrix Key of n quantization precision obtained from the input data quantization module and the value matrix Value of m quantization precision.
[0034] The input data quantization module is used to calculate the query matrix Query, key matrix Key and value matrix Value of the new data, and quantize Query and Key with n quantization precision, quantize Value with m quantization precision, and record the quantization factor 1 for obtaining the n quantization precision Key, and record the quantization factor 2 for obtaining the m quantization precision Value.
[0035] Key-value caching is a method for processing large language model inference by using storage space in exchange for computing resources. In the self-attention mechanism, the key matrix and value matrix have different computing requirements in time. The key-value cache module with mixed quantization precision is used to store the key matrix Key and value matrix Value of historical data. Among them, the key matrix of historical data is stored with n quantization precision, and the value matrix of historical data is stored with m quantization precision, such as Figure 4 As shown, the dotted line part represents the loading cycle difference between the loading value matrix and the loading key matrix, and the processing process of the Softmax function and the nm inverse quantization operation module is the calculation cycle difference caused by the key matrix participating in the calculation and the value matrix participating in the calculation.
[0036] The hybrid quantization accuracy selection of the key-value cache module is determined by the time requirements of the self-attention mechanism calculation formula of the key matrix and the value matrix in the large language model, while considering the inference accuracy and performance of the large language model, so as to achieve a balance between inference accuracy and key-value cache model compression. Specifically, the calculation difference-load difference module dynamically adjusts the quantization accuracy of the key matrix and the value matrix by evaluating the time requirements of the large language model in the self-attention mechanism calculation to adapt to different computing loads and performance requirements. In this process, the quantization accuracy selection of the key matrix and the value matrix must not only meet the time constraints of the self-attention mechanism calculation formula, but also ensure the accuracy and overall performance of the large language model during the inference process. By comprehensively considering these factors, the calculation difference-load difference module can find the best balance between inference accuracy and model compression, thereby minimizing the use of storage and computing resources while ensuring efficient computing.
[0037] The time requirements of the self-attention mechanism calculation formula of the key matrix and the value matrix in the large language model are different. Specifically, in the self-attention mechanism calculation formula, after the calculation requirements of the key matrix, it needs to go through the processing process including the Softmax function and the nm dequantization operation, and then generate the calculation requirements of the value matrix. In this process, the Softmax function is responsible for standardizing the weights of the key matrix to ensure that its sum is 1, so that the attention score can correctly reflect the importance of each element of the key matrix. Subsequently, the nm dequantization operation module will convert the standardized key matrix data from n quantization precision to m quantization precision to match the data precision of the value matrix for subsequent calculations. After the above processing steps, the calculation structure will generate the calculation requirements of the value matrix. Since the key matrix needs to go through multiple steps before generating the calculation requirements of the value matrix, this leads to differences in the calculation time requirements of the key matrix and the value matrix.
[0038] The nm inverse quantization operation module is used to calculate the Softmax (Q×K T ) is used to perform an inverse quantization operation of m quantization precision on the preliminary result of n quantization precision after quantization, and output the calculated period difference between the key matrix and the value matrix. The specific steps of the inverse quantization operation include: first, calculate Softmax (Q × K T ) is input into the nm inverse quantization operation module. The nm inverse quantization operation module converts the preliminary result of n quantization precision into the result of m quantization precision according to the quantization factor 1 and the quantization factor 2 through the preset inverse quantization formula, such as Figure 2 shown.
[0039] The Softmax function is shown in the following formula:
[0040]
[0041] Among them, x is all the data in a row of the matrix product result of the transposed query and key, e x To perform exponential operation on all data of x, Accumulate all the exponents of a row in the matrix product result of the transposed query and key.
[0042] The dequantization formula is as follows:
[0043]
[0044] In this conversion process, the nm dequantization operation module will adjust the data representation and accuracy level according to the difference between the n quantization accuracy and the m quantization accuracy to ensure the integrity and accuracy of the data during the conversion process. T ) is dequantized and output in the form of m quantization precision, and further calculated with the value matrix Value of m quantization precision, thus completing the entire self-attention mechanism calculation process. Through the above dequantization operation, it is ensured that the data is converted from n quantization precision to m quantization precision with minimal precision loss, and the consistency and accuracy of the data are maintained. While maintaining data accuracy, it is possible to improve the flexibility and adaptability of calculations, provide a high-precision data foundation for subsequent calculation steps, and ensure the accuracy of the overall calculation results.
[0045] In the calculation difference-load difference matching module, since there is a calculation cycle difference between the process of calculating the key matrix and the process of calculating the value matrix, the load cycle difference is matched based on this calculation cycle difference. On the basis of considering the model performance, the calculation difference-load difference matching module is guided to select the optimal mixed precision quantization scheme in the nm quantization precision scheme set, such as Figure 3 As shown, the mixed quantization precision scheme is stored in the key-value cache module. The formula for matching the loading cycle difference is as follows:
[0046] Calculation cycle difference = (Softmax processing cycle) + (nm dequantization operation cycle)
[0047]
[0048] Calculation cycle difference ≥ loading cycle difference
[0049] The calculation difference-load difference matching module effectively manages storage resources through different quantization precisions to balance storage efficiency and calculation precision. Specifically, the calculation difference-load difference matching module determines the load cycle difference between the key matrix and the value matrix according to the calculation cycle difference of the output of the nm anti-quantization operation module, by analyzing the calculation requirements and response time of the key matrix and the value matrix when executing the self-attention mechanism. On this basis, the calculation difference-load difference matching module dynamically adjusts the hybrid quantization precision scheme adopted in the key-value cache module, so that the calculation structure can flexibly adapt to the accuracy requirements and performance requirements at different calculation stages. Among them, the key matrix selects n quantization precision storage, which can significantly reduce the storage space occupation while ensuring sufficient accuracy, thereby improving data retrieval and access speed to ensure the calculation accuracy under high precision requirements; the value matrix selects m quantization precision storage to provide accurate data storage and calculation results to ensure the reliability and consistency of the results in complex calculation processes, so as to improve the efficiency of data processing and overall energy efficiency. By using different quantization precision storage methods for the key matrix and the value matrix, the calculation difference-load difference matching module realizes intelligent selection and balance between different quantization precisions, and optimizes the storage and calculation performance of the key-value cache module.
[0050] The present invention provides a self-attention mechanism calculation structure based on hybrid quantization precision key-value cache. Through hybrid quantization precision technology, the calculation complexity and data transmission volume are reduced without significantly affecting the reasoning accuracy performance of the large language model, and the calculation efficiency of the self-attention mechanism is improved. According to the time requirements and quantization precision requirements of the key matrix and the value matrix for different parts of the self-attention calculation, n quantization precision and m quantization precision are used to store the key matrix and value matrix of historical data respectively, which effectively reduces the storage overhead. By dynamically selecting the hybrid quantization precision scheme, the calculation complexity is significantly reduced while ensuring the reasoning accuracy of the large language model, and the best balance between reasoning accuracy and calculation efficiency is achieved.
[0051] The present invention also proposes a method for implementing a self-attention mechanism computing structure based on a mixed quantization precision key-value cache. Figure 5 As shown, the method includes the following five steps.
[0052] Step 1: First, initialize the two quantization precision parameters used in the calculation structure, namely n and m. Specifically, the quantization precision n and m represent different precision levels, which will be used for the quantization processing of Query, Key and Value in subsequent steps. The purpose of initializing the quantization precision is to provide basic parameters for subsequent calculations. During the initialization process, different quantization precision values can be set and initialized according to specific application requirements and hardware conditions. Usually, the quantization precision n is smaller than the quantization precision m.
[0053] Step 2: After receiving new data, the input data is processed according to the pre-trained weight parameters to calculate the corresponding Query and Key results of n quantization precision, and the Value results of m quantization precision. In this process, the Query and Key results of n quantization precision are used to characterize the characteristics of the input data, while the Value results of m quantization precision are used to represent the values in subsequent calculations.
[0054] Step 3: Substitute the Query and Key with n quantization precision and the Value with m quantization precision obtained in step 2 into the calculation formula of the self-attention mechanism, and after the nm dequantization operation module, preliminarily obtain the calculation cycle difference between the Key and Value calculations. The purpose of this step is to make preliminary calculations so that the calculation structure can be adjusted and optimized more finely in subsequent steps according to the specific calculation cycle difference.
[0055] Step 4: Input the calculation cycle difference calculated in step 3 into the calculation difference-loading difference matching module, match the calculation cycle difference and the loading cycle difference according to the calculation requirements of the current self-attention mechanism, select the corresponding quantization scheme from the nm quantization precision scheme set, and dynamically adjust the quantization schemes corresponding to different quantization precisions in the system.
[0056] Step 5: After selecting a specific load cycle difference, it may involve multiple hybrid quantization precision schemes for key matrices and value matrices. According to the performance requirements such as model inference accuracy, the optimal hybrid quantization precision scheme is further selected and updated to the key-value cache module. Specifically, (1) according to the values of n and m in the optimal hybrid quantization precision scheme, the quantization precision n of the key matrix and the quantization precision m of the value matrix are applied to the new data calculation, involving the quantization processing of Query, Key and Value; (2) the quantization precision n of the key matrix and the quantization precision m of the value matrix are respectively applied to the input data quantization module, the self-attention mechanism calculation module, the nm inverse quantization operation module and the key-value cache module to update the quantization result of the new data, the inverse quantization output result of the nm inverse quantization operation module, the self-attention mechanism calculation result, and the load cycle difference; (3) in the calculation difference-load difference matching module, according to the load cycle difference, the corresponding hybrid quantization precision scheme is selected, and different quantization precision schemes are dynamically adjusted and optimized, and the dynamically adjusted and optimized hybrid precision scheme is stored in the key-value cache module.
[0057] Step 6: Repeat steps 2 to 5, and continuously optimize and select the hybrid quantization precision scheme until the optimal scheme is found in the nm quantization precision scheme set. Finally, the calculation function of the self-attention mechanism calculation structure based on the hybrid quantization precision key-value cache is completed. In the process of first determining the optimal hybrid quantization precision scheme, the self-attention mechanism calculation is performed according to the n-bit query matrix Query, the n-bit key matrix Key, and the m-bit value matrix Value output by the input data quantization module; in the subsequent iterative optimization process of the optimal hybrid quantization precision scheme, the query matrix data in the self-attention mechanism calculation comes from the n-bit query matrix Query output by the input data quantization module, the key matrix data in the self-attention mechanism calculation comes from the n-bit key matrix Key output by the input data quantization module and the n-bit key matrix historical data loaded by the key-value cache module, and the value matrix data in the self-attention mechanism calculation comes from the m-bit value matrix Value output by the input data quantization module and the m-bit value matrix Value loaded by the key-value cache module.
[0058] In the above steps, the reasoning accuracy and computing performance of the large language model are gradually improved by continuously adjusting and optimizing the quantization precision parameters. Specifically, in each iteration, the Query, Key, and Value results of the input data at different quantization precisions are first recalculated, and the loading strategy in the calculation difference-loading difference matching module is adjusted according to the change in the calculation cycle difference. Subsequently, based on the model reasoning accuracy and performance requirements, an appropriate hybrid quantization precision scheme is selected and optimized to ensure that the system can operate efficiently under different computing loads. Through multiple rounds of iterative calculations and optimizations, the optimal quantization precision configuration is gradually approached, so that the system can maintain high computing accuracy while maximizing the overall computing performance and energy efficiency, and finally realize an efficient and accurate self-attention mechanism computing structure based on hybrid quantization precision key-value cache.
[0059] In the above steps, the reasoning accuracy and computing performance of the large language model are gradually improved by continuously adjusting and optimizing the quantization precision parameters. Specifically, in each iteration, the query, key, and value results of the new input data at different quantization precisions are first recalculated, and the mixed quantization precision scheme in the calculation difference-load difference matching module is adjusted according to the change in the calculation cycle difference. Subsequently, the appropriate mixed quantization precision scheme is selected and optimized in combination with the model reasoning accuracy and performance requirements. Through multiple rounds of iterative calculations and optimizations, the optimal quantization precision configuration is gradually approached, so that the system can maintain high calculation accuracy while maximizing the overall computing performance and energy efficiency, and finally realize an efficient and accurate self-attention mechanism calculation structure based on mixed quantization precision key-value cache.
[0060] Below, an example is given.
[0061] The following is an example of a self-attention mechanism computing structure based on a mixed quantization precision key-value cache with n=4bit, m=8bit to verify the computing function of the computing structure proposed in the invention.
[0062] First, calculate the initial calculation cycle difference. By default, n = 12bit, m = 12bit. Calculate Query, Key and Value based on the new data. The calculation cycle difference between Key participating in the self-attention mechanism operation and Value participating in the self-attention mechanism operation is 2000 clock units. The calculation cycle difference includes the delay of the Softmax function operation and the delay of the nm dequantization operation module.
[0063] Match the loading cycle difference. Figure 2 As shown, a matching mixed quantization precision scheme is selected from the 8bit-8bit quantization precision scheme set, and the number of loaded data per cycle is set to 25 and the number of loaded data is set to 10000.
[0064] Select the appropriate nm quantization precision scheme. From the calculation, we know that mn≤5, so the available schemes include Key (1bit), Value (6bit); Key (2bit), Value (7bit); Key (3bit), Value (8bit); Key (4bit), Value (9bit); Key (5bit), Value (10bit); Key (6bit), Value (11bit); Key (7bit), Value (12bit) and other mixed quantization precision schemes.
[0065] Introduce model inference accuracy and other performance as reference indicators. The above scheme is used for model quantization, and the model compression and inference accuracy of each quantization model are comprehensively evaluated to determine the optimal mixed precision quantization scheme as Key (5bit), Value (10bit).
[0066] The selected optimal hybrid quantization precision scheme is updated to the key-value cache module. (1) The 5-bit quantization precision of the key matrix and the 10-bit quantization precision of the value matrix are applied to the new data calculation, involving the quantization processing of Query, Key and Value; (2) When performing 5-bit-10-bit inverse quantization operations, the 5-bit quantization precision of the key matrix and the 10-bit quantization precision of the value matrix are applied to the corresponding inverse quantization modules respectively to update the 5-bit-10-bit inverse quantization operations; (3) In the calculation difference-load difference matching module, according to the selection of the load cycle difference, the data of different quantization precisions are dynamically adjusted and optimized to improve the overall computing performance and accuracy.
[0067] like Figure 5The process shown in Table 1 is the iterative hybrid quantization precision scheme evaluation process. Using Key (5bit) and Value (10bit) as the basic hybrid quantization precision scheme, repeatedly matching the loading cycle difference, selecting the appropriate nm quantization precision scheme, introducing model inference accuracy and other performance as reference indicators, dynamically adjusting the optimal hybrid quantization precision scheme Key (4bit) and Value (8bit), and iteratively optimizing the hybrid quantization precision scheme process is shown in Table 1.
[0068]
[0069]
[0070] Table 1
[0071] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above specific embodiments. The above specific embodiments and the description in the specification are only for further illustrating the principles and preparation effects of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of the present invention is defined by the claims and their equivalents.
Claims
1. A self-attention mechanism computing structure based on mixed quantization precision key-value cache, It is characterized in that include: An input data quantization module, used for quantizing the input data into an n-bit query matrix, an n-bit key matrix and an m-bit value matrix; A self-attention mechanism calculation module is used to calculate a softmax function value with an accuracy of n bits according to an n-bit query matrix and an n-bit key matrix, and to perform a multiplication operation according to an m-bit value matrix and a softmax function value with an accuracy of m bits to obtain a calculation result of the self-attention mechanism; nm dequantization operation module, used for dequantizing the softmax function value with the precision of n bits into the softmax function value with the precision of m bits, and obtaining the calculation period difference between the key matrix and the value matrix; A calculation difference-loading difference matching module, which matches the loading cycle difference according to the calculation cycle difference of the key matrix and the value matrix, and selects a hybrid quantization precision scheme with the best quantization model performance from the hybrid quantization precision schemes that meet the loading cycle difference, wherein the hybrid quantization precision scheme with the best quantization model performance includes an n-bit key matrix and an m-bit value matrix, where n and m are positive integers; and, The key-value cache module is used to cache the hybrid quantization precision solution with the best quantization model performance, and loads the n-bit key matrix and the m-bit value matrix into the autonomous mechanism calculation module according to the loading cycle difference.
2. According to claim 1, a self-attention mechanism computing structure based on mixed quantization precision key-value cache, It is characterized in that The nm dequantization operation module dequantizes the softmax function value with an accuracy of n bits to a softmax function value with an accuracy of m bits. The expression is: Among them, the input data is the n-bit softmax function value, the quantization factor 1 is the factor of the quantization key matrix of the input data quantization module, the quantization factor 2 is the factor of the quantization value matrix of the input data quantization module, and the output data is the m-bit softmax function value.
3. According to claim 2, a self-attention mechanism computing structure based on mixed quantization precision key-value cache, It is characterized in that The calculation period difference between the key matrix and the value matrix is the sum of the Softmax function processing period and the nm dequantization operation period.
4. According to claim 3, a self-attention mechanism computing structure based on mixed quantization precision key-value cache, It is characterized in that The calculation difference-loading difference matching module matches the loading cycle difference according to the calculation cycle difference of the key matrix and the value matrix, specifically: the loading cycle difference is calculated under the constraint that the calculation cycle difference ≥ the loading cycle difference, 5. According to claim 4, a self-attention mechanism computing structure based on mixed quantization precision key-value cache, It is characterized in that The softmax function is Where x is all the data in one row of the transposed matrix product of the query matrix and the key matrix. i is the i-th data in x, and n is the total number of data contained in x.
6. According to claim 5, a self-attention mechanism computing structure based on mixed quantization precision key-value cache, It is characterized in that The quantized model performance includes but is not limited to: model compression accuracy and inference accuracy.
7. A self-attention mechanism computing structure based on mixed quantization precision key-value cache according to any one of claims 1 to 6, It is characterized in that The calculation difference-loading difference matching module matches the loading period difference according to the calculation period difference between the key matrix and the value matrix, and selects the mixed quantization precision scheme with the optimal quantization model performance from the mixed quantization precision schemes that meet the loading period difference. Specifically: initialize n and m as different quantization precision values, where n < m, start the first iteration of the optimal mixed quantization precision scheme, obtain the result of the first iteration of the optimal quantization precision scheme, broadcast the corresponding mixed quantization precision n and m of the first iteration result to the input data quantization module, the self-attention mechanism calculation module, the n-m dequantization operation module, and the key-value cache module, and then perform subsequent iterative optimization of the optimal mixed quantization precision scheme according to the loading period difference updated by the n-m dequantization operation module.
8. The self-attention mechanism calculation structure based on a mixed quantization precision key-value cache according to claim 7, wherein, in the first iteration process of the optimal mixed quantization precision scheme, the self-attention mechanism calculation module calculates the softmax function value with an n-bit precision according to the n-bit query matrix output by the input data quantization module, the n-bit key matrix output by the input data quantization module, and the n-bit key matrix loaded by the key-value cache module, and performs a multiplication operation according to the m-bit value matrix output by the input data quantization module, the m-bit value matrix loaded by the key-value cache module, and the softmax function value with an m-bit precision.
9. The self-attention mechanism calculation structure based on a mixed quantization precision key-value cache according to claim 7, wherein, in the subsequent iterative optimization process of the optimal mixed quantization precision scheme, the self-attention mechanism calculation module calculates the softmax function value with an n-bit precision according to the n-bit query matrix and the n-bit key matrix output by the input data quantization module, and performs a multiplication operation according to the m-bit value matrix output by the input data quantization module and the softmax function value with an m-bit precision.
Citation Information
Patent Citations
Data processing acceleration method of dynamic sparse attention mechanism capable of being realized by hardware
CN113918882A
Log merge tree key value storage system and related method and related equipment
CN114840134A
Accelerated reasoning method and system for cognitive model question and answer prediction
CN117910576A
Method and computing device for quantitative tuning of residual structure of self-attention model
CN118153634A
Deep-learning-based method for predicting residual life of aero-engine
WO2024045377A1
Cited By
Large model reasoning optimization method and device, computer equipment and storage medium
CN120633851A
KV cache data quantification device and method
CN121031682A
