Local perception lookup table lookup method

Abstract

The invention relates to the field of data lookup, and discloses a locality perception lookup table lookup method, which comprises the following steps of: S1, reordering between rows: clustering and reordering the rows of an input vector and a weight matrix according to the same quantized value to enable elements with the same quantized value to be continuously distributed; for a common matrix GeMV scene, cache locality is improved through rearrangement and subsequent blocking operation, and repeated loading of a lookup table LUT is reduced. And the data locality of the GeMV scene is remarkably improved through inter-line reordering and a virtual mapping strategy. On the basis of reordering of input vector element values, elements with the same value are aggregated, so that the LUT line loading times are reduced to 256 times at most from being in direct proportion to the input dimension d, and frequent replacement of the LUT lines in the WRAM is avoided; meanwhile, a weight matrix is divided into BR * BC sub-matrixes, the characteristic that vertical adjacent rows share an accumulator is utilized, multi-row sub-matrix calculation is loaded at a time, and virtual rearrangement maintains the row matching relation between an input vector and the weight matrix through index mapping.

Description

Technical Field

[0001] This invention relates to the field of data retrieval technology, specifically to a locality-aware lookup table retrieval method. Background Technology

[0002] A naive GeMV kernel implementation based on lookup tables runs very slowly because in-memory computing hardware typically has two levels of storage. The first level is the working random access memory (WRAM) within the processor core of the in-memory computing hardware itself, which is inaccessible to the host CPU. The second level is on-chip random access memory (MRAM) accessible to the host CPU. Since the size of the lookup table exceeds the capacity of the in-memory computing hardware's WRAM, random access to the MRAM is required for each operation.

[0003] In-memory computing hardware has limited random access memory (RAM) capacity, while large models often involve large amounts of computational data, leading to frequent cache misses. Due to process and area limitations, the computational power of in-memory computing components is often limited. To handle the numerous multiplication operations in large models, current technologies typically use lookup tables instead of direct multiplication calculations. However, this table lookup approach, which trades memory access for computation, increases memory access pressure, making reducing memory access overhead a pressing issue. Lookup table-based matrix-vector multiplication operators dominate LLM inference. For all elements with the same value in a vector, a row in the lookup table (LUT) will be searched repeatedly; reducing these repetitions is an effective way to lower memory access overhead. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a locality-aware lookup table method, which solves the problem of handling a large number of multiplication operations in large models. Existing technologies typically use lookup tables instead of direct multiplication calculations. However, the lookup table method, which trades memory access for computation, exacerbates the memory access pressure.

[0005] To achieve the above objectives, the present invention provides a locality-aware lookup table lookup method through the following technical solution, characterized by comprising the following steps: S1, Inter-row reordering: The rows of the input vector and weight matrix are reordered according to the same quantization value, so that the elements with the same quantization value are distributed continuously; For the GeMV scenario of ordinary matrix, this reordering and subsequent block operation improve the locality of the cache and reduce the repeated loading of the lookup table LUT. S2, Block Loading: Divide the weight matrix after S1 reordering into sub-matrix blocks of size BR×BC and load them into the secondary storage MRAM; at the same time, based on the LUT row requirements determined after S1 reordering, perform LUT row loading operation and combine the sub-matrix blocks to perform LUT lookup and calculation. S3, Product Value Preloading: Based on the LUT lookup logic of S1, the product result of elements with the same value is identified in the sparse matrix SpMV operation; and the corresponding row in the LUT is loaded into the first-level storage WRAM according to the input vector after S1 reordering, with each row being loaded only once; S4. In-row reordering: Based on the product results identified in S3, for the sparse matrix SpMV scenario corresponding to the LLM weights of the large language model with sparse pruning, the elements in each row of the sparse matrix are sorted according to the product value and mapped to 256 predefined buckets. Register operations are optimized through reordering and bucket storage to further reduce memory access overhead. S5, Bucket Accumulation: Based on the reordered matrix constructed in S4 and 256 predefined bucket structures, the common product value in the LUT is read in bucket order, the element positions in the bucket are traversed and the common product value is accumulated to the corresponding position in the result vector until all buckets are processed, and the calculation of one row of the sparse matrix is ​​completed.

[0006] Preferably, step S1, inter-row reordering, further includes the following steps: dividing the weight matrix required for LLM inference into several sub-matrices of size BR×BC, and performing subsequent calculations on a sub-matrix basis to avoid repeated data copying caused by direct calculation of the complete matrix; Wherein, the weight matrix is ​​the multiplicand, BR is the block row, and BC is the block column.

[0007] Preferably, step S1, inter-row reordering, further includes: based on the partitioning of the submatrix, clustering elements with the same quantization value according to the element quantization value of the input vector through reordering to form a reordering order of the input vector; this reordering order aims to reduce the number of repeated loadings of LUT rows, and needs to adapt to the multiplication calculation logic of the input vector and the weight matrix, that is, the i-th dimension of the input vector is multiplied by the i-th row of the weight matrix, and the direct multiplication is replaced by LUT table lookup; Here, the input vector is the multiplier; the rows of the LUT correspond to the quantized values ​​of the input vector elements, the columns correspond to the values ​​of the weight matrix elements, and the intersection of the rows and columns is the product result.

[0008] Preferably, step S1, inter-row reordering, further includes: reordering the quantized values ​​of the input vector by clustering, performing virtual reordering on the rows of the weight matrix containing each submatrix, ensuring that elements with the same quantized values ​​are continuously distributed after reordering, and that the multiplication logic between the input vector and the weight matrix is ​​not misaligned.

[0009] Preferably, step S2, block loading, specifically includes: based on the LUT row requirements determined after rearrangement in S1 and the size of the weight submatrix BR×BC after virtual rearrangement, calculating the required LUT rows and loading them into the primary storage WRAM, while simultaneously loading the weight submatrix blocks after virtual rearrangement into the secondary storage MRAM; performing LUT lookup between the LUT rows in the primary storage WRAM and the weight submatrix blocks in the secondary storage MRAM, obtaining the multiplication results and accumulating them, and finally updating the result vector of the matrix-vector multiplication.

[0010] Preferably, step S3, product value preloading, specifically includes: in the sparse matrix SpMV operation using the standard compressed sparse row (CSR) format, based on the LUT lookup logic of S1, the quantized value of the input vector corresponding to the LUT row is used to identify the product result of elements with the same value within the sparse matrix row; and the standard CSRS spMV requires 5d load / store operations. Based on the identified product result, the input vector is reordered according to S1, and the corresponding row in the LUT is loaded into the first-level storage WRAM, with each row loaded only once to reduce repeated memory accesses; Where d is the dimension of the input vector, the operation includes reading value-position pairs, reading the product value, reading the result value, calculating, and writing back the result vector, and 5 is the number of load / store operations per multiplication operation. Preferably, step S4, in-row reordering, specifically includes: based on the product result identified in S3, reordering the element indices within the rows of the CSR format sparse matrix according to the product value to generate a reordered sparse matrix; simultaneously, mapping the reordered elements to 256 predefined buckets, each bucket corresponding to a quantization value, and the number of different quantization values ​​in the matrix not exceeding 256, with the buckets storing the position information of the elements, including the positions corresponding to those without element values.

[0011] Preferably, step S5, bucket accumulation, specifically includes: based on the reordering matrix constructed in S4 and 256 predefined bucket structures, traversing all buckets in a preset order; when switching to a new bucket, reading the common product value of the quantized values ​​corresponding to that bucket from the LUT, then traversing all element positions within the bucket, and accumulating the common product value to the corresponding position in the result vector; repeating this process until all buckets are processed, completing the calculation of one row of the sparse matrix.

[0012] This invention provides a locality-aware lookup table lookup method. It has the following advantages: 1. This invention significantly improves data locality in GeMV scenarios through inter-row reordering and virtual mapping strategies. Based on the reordering of input vector element values, elements with the same value are aggregated, reducing the number of LUT row loadings from proportional to the input dimension d to a maximum of 256 times, avoiding frequent replacement of LUT rows in WRAM. At the same time, the weight matrix is ​​divided into BR×BC submatrices, and the characteristic of vertically adjacent rows sharing accumulators is utilized to load multiple rows of submatrices for calculation at one time, reducing the overhead of result vector access by 40%-50%. Virtual reordering maintains the row matching relationship between the input vector and the weight matrix through index mapping, avoiding physical data migration and ensuring the correctness of the calculation logic. Furthermore, LUT rows are only loaded into high-speed WRAM to adapt to high-frequency access, and weight submatrices are stored in large-capacity MRAM to adapt to batch storage, reducing cross-level data migration by 60% and fully leveraging the performance advantages of different storage media.

[0013] 2. This invention reduces the number of load / store operations of the standard CSR from 5d to 3d+256 by reordering the elements in the CSR row according to the product value and allocating them to bucket storage. When d≥1024, the memory access operations are reduced by more than 40%, eliminating the repeated reading of the "value" in the value-position pair, and compressing the product value reading from d times to 256 times. At the same time, elements with the same product value are stored in the same bucket. When switching buckets, the corresponding row of the LUT is read at once, and it can be continuously accumulated to the result vector. The number of product value switching in the register is reduced by 80%, and the amount of data exchanged between the register and memory is reduced by 50%. Moreover, the bucket structure naturally supports efficient processing of positions without element values. Even in the LLM weight matrix with a sparsity of >90%, it can still maintain a stable memory access optimization effect.

[0014] 3. This invention achieves efficient collaboration between GeMV and SpMV through a unified locality-aware framework. The result vectors of SpMV and GeMV are naturally merged according to the computational logic, without the need for additional data transformation steps, reducing the end-to-end latency of the complete matrix-vector multiplication by 25%. Furthermore, the inter-row and intra-row reordering strategies share the LUT access logic, reducing the complexity of algorithm design, while ensuring that the LUT loading of dense and sparse parts does not conflict, making the marginal benefits of locality optimization more significant, and providing key technical support for real-time inference of large models. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the overall solution process and three-dimensional architecture of the present invention. Figure 2 This is the GeMV and interline reordering diagram of the present invention; Figure 3 This is a schematic diagram of I / O operations in the SpMV scenario without reordering, as described in this invention. Figure 4 This is a schematic diagram of the rearranged sparse matrix format in the SpMV scenario of the present invention; Figure 5 This is a schematic diagram of the I / O operations after reordering in the SpMV scenario according to the present invention; Figure 6 This is a schematic diagram of the mapping and matrix partitioning of the LLM inference layer of the present invention; Figure 7 This is a flowchart of the method of the present invention. Detailed Implementation

[0016] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0017] Example: Please see the appendix Figure 1 -Appendix Figure 7 This invention provides a locality-aware lookup table lookup method, characterized by the following steps: S1. Inter-row reordering: The rows of the input vector and weight matrix are reordered according to the same quantization value, so that elements with the same quantization value are continuously distributed. For the ordinary matrix GeMV scenario, this reordering and subsequent block operation improve cache locality and reduce repeated loading of lookup table LUTs. The inter-row reordering step S1 also includes the following steps: Divide the weight matrix required for large language model LLM inference into several sub-matrices of size BR×BC. Subsequent calculations are performed on a sub-matrix basis to avoid repeated data copying caused by direct calculation of the complete matrix. The inter-row reordering step S1 also includes: Based on the sub-matrix partitioning, according to the quantization value of the elements of the input vector, the elements with the same quantization value are clustered by reordering to form the reordering order of the input vector. This reordering order aims to reduce the number of repeated loading of LUT rows and needs to adapt to the multiplication calculation logic of the input vector and the weight matrix, that is, the input vector's first row... The i-th dimension is multiplied by the i-th row of the weight matrix, and the LUT lookup table is used to replace the direct multiplication. The inter-row reordering in step S1 also includes: reordering the order of clustering based on the quantization values ​​of the input vector, and performing virtual reordering on the rows of the weight matrix containing each submatrix to ensure that the elements with the same quantization value are continuously distributed after reordering, and that the multiplication calculation logic of the input vector and the weight matrix is ​​not misaligned. The core function of "virtual reordering" is to avoid misalignment of the multiplication calculation between the input vector and the weight matrix after reordering. If only the input vector is reordered without reordering the weight matrix, it will lead to "the i-th dimension of the input vector being multiplied by the weight matrix that is not in the i-th row", which will destroy the original calculation logic. Virtual reordering can ensure that the correspondence between the two is consistent with that before reordering. Moreover, this virtual reordering only adjusts the matching relationship of the weight matrix rows through index mapping, without physically moving the weight matrix data. This can save data migration time and accurately align the reordering order of the input vector, ensuring the accuracy of the calculation results.

[0018] In this system, the input vector is the multiplier; the rows of the LUT correspond to the quantized values ​​of the input vector elements, the columns correspond to the values ​​of the weight matrix elements, and the intersection of the rows and columns is the product result. The weight matrix is ​​the multiplicand, BR is the block row, and BC is the block column. The complete calculation of LLM requires multiplying the input vector with the weight matrix. Directly calculating the complete matrix will result in the inefficiency of repeatedly copying duplicate data to the cache. After the matrix is ​​divided into blocks, only a part of the multiplier (input vector) and multiplicand (weight matrix) is taken each time, which can increase data locality and further avoid repeated copying to improve system efficiency. All subsequent reordering operations are also carried out on the calculation of the submatrix.

[0019] S2, Block Loading: The weight matrix after S1 reordering is divided into sub-matrix blocks of size BR×BC and loaded into the secondary storage MRAM; simultaneously, based on the LUT row requirements determined after S1 reordering, LUT row loading is performed, and LUT lookup and calculation are combined with the sub-matrix blocks. Step S2, block loading, specifically includes: based on the LUT row requirements determined after S1 reordering and the BR×BC size of the virtually reordered weight sub-matrix, the required LUT rows are calculated and loaded into the primary storage WRAM, and the virtually reordered weight sub-matrix blocks are loaded into the secondary storage MRAM; the LUT rows in the primary storage WRAM and the weight sub-matrix blocks in the secondary storage MRAM... The block performs LUT lookup, retrieves and accumulates the multiplication results, and finally updates the matrix-vector multiplication result. The primary storage WRAM is a dedicated storage space within the device's own processor core, inaccessible to the host CPU. The secondary storage MRAM is a shared on-chip storage space accessible to the host CPU. MRAM is larger, allowing all LUTs to be placed there. WRAM is faster, and some LUTs can only be read and calculated by the processor core after being moved into WRAM. This loading method aligns with the fast memory access speed of WRAM, adapting to the high-frequency read requirements of LUTs. MRAM has a large capacity, adapting to the storage requirements of submatrixes, further reducing the memory access overhead during data loading.

[0020] S3. Product Value Preloading: Based on the LUT lookup logic of S1, the product results of elements with the same value are identified in the sparse matrix SpMV operation; and the corresponding rows in the LUT are loaded into the primary storage WRAM according to the input vector after reordering in S1, with each row being loaded only once. The product value preloading step S3 specifically includes: in the sparse matrix SpMV operation using the standard compressed sparse row (CSR) format, based on the LUT lookup logic of S1, the quantized value of the input vector corresponding to the LUT row is used to identify the product results of elements with the same value within the sparse matrix row; and the standard CSR SpMV requires 5d load / storage operations. Based on the identified product results, the corresponding rows in the LUT are loaded into the primary storage WRAM according to the input vector after reordering in S1, with each row being loaded only once to reduce repeated memory accesses; Where d is the dimension of the input vector, the operation includes reading value-position pairs, reading product values, reading result values, calculating, and writing back the result vector, and 5 is the number of load / store operations for a single multiplication operation.

[0021] S4. In-row reordering: Based on the product results identified in S3, for the sparse matrix SpMV scenario corresponding to the LLM weights of a large language model with sparse pruning, the elements in each row of the sparse matrix are sorted according to their product values ​​and mapped to 256 predefined buckets. Register operations are optimized through reordering and bucket storage to further reduce memory access overhead. Step S4, in-row reordering, specifically includes: based on the product results identified in S3, the element indices in the rows of the CSR format sparse matrix are reordered according to the product value to generate a reordered sparse matrix; at the same time, the reordered elements are mapped to 256 predefined buckets, each bucket corresponding to a quantization value, and the number of different quantization values ​​in the matrix does not exceed 256. The buckets store the position information of the elements, including the positions corresponding to the absence of element values.

[0022] S5, Bucket Accumulation: Based on the reordered matrix constructed in S4 and 256 predefined bucket structures, the common product value in the LUT is read in bucket order. The positions of elements within the bucket are traversed, and the common product value is accumulated to the corresponding position in the result vector until all buckets are processed, completing the calculation of one row of the sparse matrix. The bucket accumulation step S5 specifically includes: based on the reordered matrix constructed in S4 and 256 predefined bucket structures, all buckets are traversed in the preset order; when switching to a new bucket, the common product value of the corresponding quantized value of the bucket is read from the LUT, and then all element positions within the bucket are traversed, and the common product value is accumulated to the corresponding position in the result vector; this process is repeated until all buckets are processed, completing the calculation of one row of the sparse matrix. The process only requires 3d+256 load / store operations, which is less than the 5d load / store operations of the standard CSRspMV. Here, 3d represents the number of operations to read element positions, read result values, and write back the result vector; 256 represents the number of times the product value is read when traversing all buckets; and 256 represents the total number of rows in the LUT. Specifically, compared to the standard CSR SpMV, this step saves 2d operations by only reading element positions instead of complete value-position pairs, thus avoiding the repeated reading of values ​​in value-position pairs. By utilizing the total number of LUT rows of 256, the number of times the product value is read is reduced from d to 256. Since the input vector dimension d in LLM inference is usually much larger than 256, 3d + 256 must be less than 5d, which can significantly reduce the memory access overhead in the SpMV scenario. After the calculation is completed, the result vector of SpMV is merged with the result vector of GeMV in S1 to form the complete matrix-vector multiplication result required for LLM inference.

[0023] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A locality-aware lookup table lookup method, characterized in that, Includes the following steps: S1, Inter-row reordering: The rows of the input vector and weight matrix are reordered according to the same quantization value, so that the elements with the same quantization value are distributed continuously; For the GeMV scenario of ordinary matrix, this reordering and subsequent block operation improve the locality of the cache and reduce the repeated loading of the lookup table LUT. S2, Block Loading: Divide the weight matrix after S1 reordering into sub-matrix blocks of size BR×BC and load them into the secondary storage MRAM; at the same time, based on the LUT row requirements determined after S1 reordering, perform LUT row loading operation and combine the sub-matrix blocks to perform LUT lookup and calculation. S3, Product Value Preloading: Based on the LUT lookup logic of S1, the product result of elements with the same value is identified in the sparse matrix SpMV operation; and the corresponding row in the LUT is loaded into the first-level storage WRAM according to the input vector after S1 reordering, with each row being loaded only once; S4. In-row reordering: Based on the product results identified in S3, for the sparse matrix SpMV scenario corresponding to the LLM weights of the large language model with sparse pruning, the elements in each row of the sparse matrix are sorted according to the product value and mapped to 256 predefined buckets. Register operations are optimized through reordering and bucket storage to further reduce memory access overhead. S5, Bucket Accumulation: Based on the reordered matrix constructed in S4 and 256 predefined bucket structures, the common product value in the LUT is read in bucket order, the element positions in the bucket are traversed and the common product value is accumulated to the corresponding position in the result vector until all buckets are processed, and the calculation of one row of the sparse matrix is ​​completed.

2. The locality-aware lookup table search method according to claim 1, characterized in that, The step S1 inter-row reordering also includes the following steps: dividing the weight matrix required for LLM inference into several sub-matrices of size BR×BC, and performing subsequent calculations on a sub-matrix basis to avoid repeated data copying caused by direct calculation of the complete matrix; Wherein, the weight matrix is ​​the multiplicand, BR is the block row, and BC is the block column.

3. The locality-aware lookup table search method according to claim 2, characterized in that, The step S1 inter-row reordering further includes: based on the partitioning of the submatrix, according to the element quantization values ​​of the input vector, the elements with the same quantization values ​​are clustered by reordering to form the reordering order of the input vector; the reordering order aims to reduce the number of repeated loading of LUT rows, and needs to adapt to the multiplication calculation logic of the input vector and the weight matrix, that is, the i-th dimension of the input vector is multiplied by the i-th row of the weight matrix, and the direct multiplication is replaced by LUT table lookup; Here, the input vector is the multiplier; the rows of the LUT correspond to the quantized values ​​of the input vector elements, the columns correspond to the values ​​of the weight matrix elements, and the intersection of the rows and columns is the product result.

4. The locality-aware lookup table search method according to claim 1, characterized in that, The step S1 inter-row reordering further includes: reordering the clustering order of the quantized values ​​of the input vector, performing virtual reordering on the rows of the weight matrix containing each submatrix, ensuring that the elements with the same quantized values ​​are continuously distributed after reordering, and that the multiplication calculation logic between the input vector and the weight matrix is ​​not misaligned.

5. The locality-aware lookup table search method according to claim 1, characterized in that, The step S2, block loading, specifically includes: based on the LUT row requirements determined after rearrangement in S1 and the size of the weight submatrix BR×BC after virtual rearrangement, calculating the required LUT rows and loading them into the primary storage WRAM, while simultaneously loading the weight submatrix blocks after virtual rearrangement into the secondary storage MRAM; performing LUT lookup between the LUT rows in the primary storage WRAM and the weight submatrix blocks in the secondary storage MRAM, obtaining the multiplication results and accumulating them, and finally updating the result vector of the matrix-vector multiplication.

6. The locality-aware lookup table search method according to claim 1, characterized in that, The product value preloading step S3 specifically includes: in the sparse matrix SpMV operation using the standard compressed sparse row (CSR) format, based on the LUT lookup logic of S1, the quantized value of the input vector corresponding to the LUT row is used to identify the product result of elements with the same value in the sparse matrix row; and the standard CSRS spMV requires 5d load / store operations. Based on the identified product result, the input vector is reordered according to S1, and the corresponding row in the LUT is loaded into the first-level storage WRAM, with each row loaded only once to reduce repeated memory accesses; Where d is the dimension of the input vector, the operation includes reading value-position pairs, reading product values, reading result values, calculating, and writing back the result vector, and 5 is the number of load / store operations for a single multiplication operation.

7. The locality-aware lookup table search method according to claim 1, characterized in that, The specific steps of S4, in-row reordering, include: based on the product results identified in S3, reordering the element indices within the rows of the CSR format sparse matrix according to the product value to generate a reordered sparse matrix; simultaneously, mapping the reordered elements to 256 predefined buckets, each bucket corresponding to a quantization value, and the number of different quantization values ​​in the matrix does not exceed 256. The buckets store the position information of the elements, including the positions corresponding to those without element values.

8. The locality-aware lookup table search method according to claim 1, characterized in that, The specific steps of S5, bucket accumulation, include: based on the reordering matrix constructed in S4 and 256 predefined bucket structures, traversing all buckets in a preset order; when switching to a new bucket, reading the common product value of the quantized values ​​corresponding to that bucket from the LUT, then traversing all element positions within the bucket, and accumulating the common product value to the corresponding position in the result vector; repeating this process until all buckets are processed, completing the calculation of one row of the sparse matrix.

Images