Memory allocation method and device, computer device and storage medium

Abstract

Embodiments of the present application disclose a memory allocation method and device, computer equipment and a storage medium, wherein the method reduces the data loading times of one of the input matrices by reconstructing the calculation sequence, thereby significantly reducing the dependence on the external memory bandwidth, cooperating with the fixed-size temporary storage space, effectively breaking through the bandwidth bottleneck under the condition of limited hardware cost, and the introduced three-buffer intelligent scheduling strategy completely eliminates the transmission conflicts in the high-speed memory buffer access, avoids the additional delay caused thereby, solves the two core problems of bandwidth limitation and transmission conflicts under the premise of saving hardware resources, and realizes the leap-forward improvement of the matrix multiplication calculation performance.

Description

Technical Field

[0001] This application relates to, but is not limited to, the field of computer technology, and in particular to a memory allocation method, apparatus, computer device, and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technologies such as large language models, the number of model parameters is growing exponentially, making matrix multiplication (GEMM) the most significant computational overhead and performance bottleneck in large model inference. In dedicated AI accelerator designs, while the processing power of computing units continues to improve, the data bandwidth between off-chip dynamic random access memory (DDR) and on-chip computing cores is growing relatively slowly, resulting in data transfer speeds lagging significantly behind computation speeds. Therefore, in bandwidth-constrained scenarios, efficiently organizing matrix multiplication calculations to minimize data transfer overhead has become a key challenge for improving the overall performance of AI accelerators.

[0003] In existing technologies, block matrix multiplication is widely used. This approach divides large input matrices A and B into multiple smaller sub-blocks and performs the matrix multiplication step by step through algebraic operations between the sub-blocks. This method aims to adapt the computational data size to a limited on-chip high-speed buffer (such as SRAM) and reduce the number of accesses to off-chip DDR through data multiplexing. However, this approach requires a high-speed buffer capable of storing an entire row of sub-block data from matrix A. When the K-dimensionality of the matrix (i.e., the column width of A or the row height of B) is large, the required buffer capacity increases linearly, leading to a sharp increase in chip area and cost, making it difficult to implement in resource-constrained embedded or edge AI accelerators.

[0004] Another common optimization strategy is to partition matrix A along its M-dimensional axis, prefetching an entire row of sub-blocks from DDR to a high-speed buffer for temporary storage. In subsequent computations, different sub-blocks of matrix B are continuously read from DDR to perform a series of sub-block multiplication and addition operations with the data in matrix A in the buffer. This method aims to reduce the number of times matrix A is repeatedly loaded. However, this approach lacks sufficient consideration of the internal architecture characteristics of the high-speed buffer. A typical high-speed buffer consists of multiple banks, each of which typically supports only one access operation (read or write) within the same clock cycle. Under traditional scheduling methods, when writing new sub-block data of matrix B to a bank while simultaneously needing to read intermediate results from the same bank for accumulation, an access conflict occurs, causing pipeline stalls and additional latency. This transmission conflict caused by improper scheduling further restricts the improvement of matrix multiplication efficiency, exacerbating the bandwidth bottleneck problem. Summary of the Invention

[0005] In view of this, embodiments of this application provide at least one memory allocation method, apparatus, computer device, and storage medium.

[0006] The technical solution of this application embodiment is implemented as follows: On one hand, embodiments of this application provide a memory allocation method, the method comprising: The first and second matrices to be calculated are divided according to the preset first and second sub-block sizes to obtain multiple first sub-blocks and multiple second sub-blocks; Allocate temporary storage space in the high-speed memory buffer that is related to the row dimension and variable parameter product of the first sub-block; Based on the reconstructed calculation order, the first and second sub-blocks currently required are sequentially transferred from external memory to the matrix calculation unit; The updated intermediate result is obtained by accumulating the calculation result of the matrix calculation unit with the intermediate result stored in the temporary storage space, and the updated intermediate result is written back to the corresponding position in the temporary storage space. By employing a triple-buffered memory allocation strategy and a transmission scheduling strategy, the input operation of the second sub-block to the high-speed memory buffer and the input and output operation of the intermediate result between the high-speed memory buffer and the vector calculation unit are coordinated to eliminate transmission conflicts and repeat the above process until the matrix multiplication calculation of all the first sub-blocks and the second sub-blocks is completed, and the final result matrix is ​​obtained.

[0007] On the other hand, embodiments of this application provide another memory allocation apparatus, the method comprising: The processing module is used to divide the first matrix and the second matrix to be calculated according to the preset first sub-block size and second sub-block size to obtain multiple first sub-blocks and multiple second sub-blocks; Allocate temporary storage space in the high-speed memory buffer that is related to the row dimension and variable parameter product of the first sub-block; Based on the reconstructed calculation order, the first and second sub-blocks currently required are sequentially transferred from external memory to the matrix calculation unit; The updated intermediate result is obtained by accumulating the calculation result of the matrix calculation unit with the intermediate result stored in the temporary storage space, and the updated intermediate result is written back to the corresponding position in the temporary storage space. The allocation module is used to coordinate the input operation of the second sub-block to the high-speed memory buffer and the input and output operation of the intermediate result between the high-speed memory buffer and the vector calculation unit through a memory allocation strategy and transmission scheduling strategy based on three buffers, so as to eliminate transmission conflicts and repeat the above process until the matrix multiplication calculation of all the first sub-blocks and the second sub-blocks is completed to obtain the final result matrix.

[0008] In another aspect, embodiments of this application provide a computer device, including a memory and a processor. The memory stores a computer program that can run on the processor, and when the processor executes the program, it implements some or all of the steps in the memory allocation method described above.

[0009] In another aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements some or all of the steps in the memory allocation method described above.

[0010] In another aspect, embodiments of this application provide a computer program including computer-readable code, wherein when the computer-readable code is run in a computer device, the processor in the computer device performs some or all of the steps for implementing the above-described memory allocation method.

[0011] In another aspect, embodiments of this application provide a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described memory allocation method.

[0012] This application embodiment reduces the number of times data is loaded into one of the input matrices by reconstructing the calculation order, thereby significantly reducing the dependence on external memory bandwidth. Combined with a fixed-size temporary storage space, it effectively breaks through the bandwidth bottleneck under limited hardware costs. At the same time, the introduced three-buffer intelligent scheduling strategy completely eliminates transmission conflicts in high-speed memory buffer access, avoiding the additional latency caused by this. While saving hardware resources, it synergistically solves the two core problems of bandwidth limitation and transmission conflict, and achieves a leapfrog improvement in matrix multiplication calculation performance.

[0013] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of this application. Attached Figure Description

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application.

[0015] Figure 1 A schematic diagram illustrating the implementation process of a memory allocation method provided in an embodiment of this application; Figure 2 A schematic diagram of the principle of a prior art provided for an embodiment of this application; Figure 3 One of the schematic diagrams illustrating the principle of a memory allocation method provided in this application embodiment; Figure 4 A second schematic diagram illustrating the principle of a memory allocation method provided in this application embodiment; Figure 5 The third schematic diagram illustrating the principle of a memory allocation method provided in this application embodiment; Figure 6 This is a schematic diagram of the composition structure of a memory allocation device provided in an embodiment of this application; Figure 7 This is a schematic diagram of the hardware entity of a computer device provided in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application are further described in detail below with reference to the accompanying drawings and embodiments. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0018] The terms “first / second / third” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first / second / third” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0019] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used herein is for descriptive purposes only and is not intended to limit the scope of this application.

[0020] In the field of large-scale model inference, matrix multiplication (GEMM) computation is the main overhead of inference time. As the number of parameters in large models grows exponentially, the scale of matrix multiplication computation continues to expand. The problem of data transfer speed from DDR memory to matrix computation units being constrained by bandwidth is becoming increasingly prominent. Bandwidth bottleneck has become the core constraint limiting the performance breakthrough of AI accelerators. Therefore, achieving efficient acceleration of matrix multiplication computation in bandwidth-constrained scenarios is a key direction for performance optimization of dedicated AI accelerators.

[0021] To improve computational parallelism, block matrix multiplication techniques are widely used. For example... Figure 2 As shown, block matrix multiplication refers to dividing a matrix into several small sub-matrix blocks and obtaining the final matrix multiplication result through algebraic operations on these sub-matrix blocks.

[0022] The mathematical expressions for block matrix multiplication are as follows: (1) and (2): (1)

[0023] Where M, K, and N represent matrices A and B as M rows and K columns and K rows and N columns, respectively; subM, subK, and subN represent the size of the submatrices; A_{ij} and B_{ij} represent the submatrices with indices i and j.

[0024] Where C represents the result matrix, and C_{ij} represents the result matrix submatrix with indices i and j.

[0025] A single matrix computation requires transferring one sub-block from DDR to the matrix computation unit for both matrix A and matrix B. If a bandwidth bottleneck exists, the transfer time of the sub-block from DDR becomes the main overhead time for matrix multiplication. To solve this problem, a common approach is to partition matrix A along the M-dimensional plane, dividing each row... The matrix multiplication calculation is performed by transferring data from DDR to a high-speed memory buffer, which reduces transmission time by only transferring sub-blocks of matrix B from DDR each time. However, this approach has two major drawbacks: first, it requires a massive amount of high-speed memory buffer space, and the space requirement increases linearly with the dimension of matrix K, resulting in extremely high hardware costs; second, it does not consider the architectural characteristics of the high-speed memory buffer and lacks a reasonable buffer scheduling mechanism, leading to additional transmission access conflicts during data transfer and further limiting the improvement of matrix multiplication performance.

[0026] In current AI acceleration chip architectures, large-scale matrix multiplication calculations are constrained by bandwidth bottlenecks, with data transmission overhead accounting for a very high percentage. Traditional solutions not only require massive amounts of high-speed memory hardware but also suffer from frequent transmission conflicts due to a lack of optimized buffer scheduling strategies, further reducing computational efficiency. Existing technologies struggle to overcome both bandwidth bottlenecks and transmission conflicts simultaneously within limited hardware costs, thus limiting improvements in matrix multiplication performance.

[0027] This application proposes a variable-order matrix multiplication computation architecture and memory allocation technology based on 3-buffer intelligent scheduling. By changing the matrix multiplication computation order and using 3-buffer intelligent scheduling, the space allocation and data transmission logic of the high-speed memory buffer are optimized. Under limited hardware resources, the bandwidth bottleneck and transmission conflict problems are completely solved, and a leapfrog improvement in matrix multiplication performance is achieved.

[0028] This application provides a memory allocation method, which can be executed by the processor of a computer device. The computer device can refer to a server, laptop computer, tablet computer, desktop computer, smart TV, set-top box, mobile device (such as a mobile phone, portable video player, personal digital assistant, dedicated messaging device, portable gaming device), or other similar computer equipment. Figure 1 This is a schematic diagram illustrating the implementation process of a memory allocation method provided in an embodiment of this application, as shown below. Figure 1 As shown, the method includes: Step 101: Divide the first matrix and the second matrix to be calculated according to the preset first sub-block size and second sub-block size to obtain multiple first sub-blocks and multiple second sub-blocks.

[0029] In this embodiment, the first matrix and the second matrix refer to two input matrices that need to be multiplied. The first sub-block size and the second sub-block size are preset parameters used to define the dimensions of the small data blocks into which the large matrix is ​​cut in the row and column directions. Partitioning refers to the process of logically or physically dividing the original large matrix into multiple smaller, regular data blocks according to these size parameters.

[0030] It receives two matrices to be computed as input. Based on pre-configured partitioning parameters (e.g., dividing the first matrix into M1xK1 sub-blocks along the MxK dimension, and the second matrix into K1xN1 sub-blocks along the KxN dimension), it performs a matrix partitioning operation. This operation generates two sets of sub-blocks: one consisting of multiple first sub-blocks (from the first matrix), and the other consisting of multiple second sub-blocks (from the second matrix). This partitioning forms the basis for subsequent block matrix multiplication calculations.

[0031] Step 102: Allocate temporary storage space in the high-speed memory buffer that is related to the row dimension and variable parameter product of the first sub-block.

[0032] In this embodiment, the high-speed memory buffer is a storage area located inside the AI ​​accelerator chip, with access speeds far exceeding those of external DDR memory. The temporary storage space is a specially designated area within this buffer used to store intermediate results during matrix multiplication calculations. The variable parameter is a configurable integer that determines how many second sub-blocks and one first sub-block will be used consecutively in a single calculation loop; this parameter directly affects the size of the temporary storage space.

[0033] A specific size of storage area is dynamically allocated or statically reserved in the high-speed memory buffer as temporary space. The size of this space is related to two factors: the dimension of the first sub-block in the row direction (e.g., M1) and the variable parameter L. Specifically, the required size of the temporary space is proportional to the product of M1 and L. The purpose of allocating this space is to temporarily store the intermediate accumulation results of some matrix multiplications, avoiding frequent access to external memory and thus alleviating bandwidth pressure.

[0034] Step 103: Based on the reconstructed calculation order, the first sub-block and the second sub-block currently required are sequentially transferred from the external memory to the matrix calculation unit.

[0035] In this embodiment, the reconstructed computation order refers to a novel computation sequence that differs from the traditional row-by-row or column-by-column order. It optimizes data reuse by changing the order in which the first and second sub-blocks are loaded and computed. External memory typically refers to off-chip DDR memory. The matrix computation unit is a hardware module in an AI accelerator dedicated to performing matrix multiplication operations.

[0036] Data is scheduled according to an optimized, unconventional process. Specifically, instead of loading all the necessary data at once, a specific first sub-block and a series of specific second sub-blocks are sequentially transferred from external DDR memory to the local registers or cache of the matrix computation unit, based on the needs of the current computation stage. The core feature of this sequence is that within a period defined by the variable parameter L, a first sub-block is reused and computed separately with L consecutive second sub-blocks, thereby reducing the number of times the first sub-block is read from external memory.

[0037] Step 104: Based on the calculation result of the matrix calculation unit, perform an accumulation operation with the intermediate result stored in the temporary storage space to obtain the updated intermediate result, and write the updated intermediate result back to the corresponding position in the temporary storage space.

[0038] In this embodiment, the intermediate result is the cumulative sum of partial matrix multiplication calculations, stored in temporary storage space. The accumulation operation refers to adding the newly calculated sub-block multiplication to the existing intermediate result. The updated intermediate result is the new value obtained after completing this accumulation.

[0039] After the matrix calculation unit completes the multiplication of the current first sub-block and the current second sub-block, it generates a partial result. It then reads the previously stored intermediate result corresponding to the current output position from a specified location in the temporary storage space. Next, (possibly with the assistance of the vector calculation unit) it adds this newly calculated partial result to the read intermediate result. Finally, it writes the updated intermediate result obtained from the addition operation back to the same specified location in the temporary storage space. This step achieves the gradual accumulation of calculation results.

[0040] Step 105: Using a memory allocation strategy and transmission scheduling strategy based on triple buffering, coordinate the input operation of the second sub-block to the high-speed memory buffer and the input and output operation of the intermediate result between the high-speed memory buffer and the vector calculation unit to eliminate transmission conflicts and repeat the above process until the matrix multiplication calculation of all the first sub-blocks and the second sub-blocks is completed to obtain the final result matrix.

[0041] In this embodiment, the triple-buffered memory allocation and transmission scheduling strategy is a method that uses three independent storage banks to coordinate data inflow and outflow. The vector computation unit is a hardware module responsible for performing vectorized operations such as accumulation. Transmission conflicts refer to access contention that occurs when multiple hardware modules attempt to read or write to the same storage bank simultaneously, leading to performance degradation.

[0042] A smart scheduling strategy is employed to manage data access to the high-speed memory buffer (typically composed of multiple banks). This strategy alternately allocates the buffer area for storing the second sub-block and the temporary space for storing intermediate results across the three banks. The operation of storing the second sub-block from the outside into a specific bank is precisely scheduled to be time-staggered with the operation of transferring intermediate results between the vector computation unit and the other two banks. Specifically, it ensures that at any given time, read and write operations to a bank will not occur simultaneously, and access requests to the same bank from different hardware will not conflict. This cyclical scheduling mechanism eliminates waiting and conflict times during data transfer. Steps 103 and 104 are repeated until all combinations of the first and second sub-blocks have been computed, and the accumulated result in the final temporary storage space is the final result matrix of the entire matrix multiplication.

[0043] This application embodiment reduces the number of times data is loaded into one of the input matrices by reconstructing the calculation order, thereby significantly reducing the dependence on external memory bandwidth. Combined with a fixed-size temporary storage space, it effectively breaks through the bandwidth bottleneck under limited hardware costs. At the same time, the introduced three-buffer intelligent scheduling strategy completely eliminates transmission conflicts in high-speed memory buffer access, avoiding the additional latency caused by this. While saving hardware resources, it synergistically solves the two core problems of bandwidth limitation and transmission conflict, and achieves a leapfrog improvement in matrix multiplication calculation performance.

[0044] In some implementations, step 101 includes: Step 1011: Obtain the total row dimension, total column dimension of the first matrix, and total column dimension of the second matrix.

[0045] In this embodiment, the total row dimension refers to the total number of elements in the matrix in the vertical direction, i.e., the number of rows in the matrix. The total column dimension refers to the total number of elements in the matrix in the horizontal direction, i.e., the number of columns in the matrix. In this step, it is necessary to obtain the number of rows and columns of the first matrix, as well as the number of columns of the second matrix. These dimensions are the basic parameters for subsequent matrix partitioning and calculation.

[0046] The dimension information of the first and second matrices is read from memory or a specified data interface. Specifically, metadata describing the shape of the matrices is parsed or received, and the total number of rows (M dimensions) and total number of columns (K dimensions) of the first matrix, as well as the total number of columns (N dimensions) of the second matrix, are extracted. These values ​​are typically integers and stored as variables in registers or memory, providing input for subsequent block-based computations.

[0047] Step 1012: Based on the computing power and memory bandwidth constraints of the hardware computing unit, determine the first block value of the first sub-block size in the row direction and the second block value in the column direction, and determine the third block value of the second sub-block size in the column direction.

[0048] In this embodiment, the computing power of the hardware computing unit refers to the maximum number of floating-point operations or the processing power of a fixed-size matrix multiplication that a processor specifically designed to perform matrix multiplication (such as a matrix computing unit in an AI accelerator) can complete per unit time. Memory bandwidth constraint refers to the maximum data transfer rate achievable by the physical bus when data is transferred between memory (such as DDR) and the computing unit or high-speed buffer. The block value is an integer that defines the size of a sub-block in a specific dimension when a large matrix is ​​divided into sub-blocks.

[0049] The calculations are performed based on the hardware specifications and performance model. First, the computation cycle of the matrix computation unit handling matrix multiplication of sub-blocks of a specific size is evaluated. Simultaneously, the bandwidth parameters of the memory controller are queried. Based on these hardware constraints, an optimization algorithm is executed. This algorithm aims to minimize the overall computation and data transfer time by solving for the optimal block parameters: the size of the first sub-block in the row direction (first block value, e.g., m), the size in the column direction (second block value, e.g., k), and the size of the second sub-block in the column direction (third block value, e.g., n). These block values ​​ensure that the sub-block size can be efficiently computed by the hardware while also being compatible with memory bandwidth, reducing the risk of data transfer becoming a bottleneck.

[0050] Step 1013: Divide the first matrix evenly according to the first block value and the second block value to generate the plurality of first sub-blocks with regular shapes.

[0051] In this embodiment, uniform division refers to the process of dividing a large matrix into several smaller, identically sized sub-matrix blocks in both row and column directions according to a specified block size. Regular shape means that each sub-block obtained after division is a complete rectangular matrix, with its number of rows and columns determined by the block size.

[0052] The first matrix is ​​divided based on the first block value (m) and the second block value (k). Starting from the top-left element of the first matrix, it is divided into groups of m consecutive rows and k consecutive columns. For the edges of the matrix that may be less than m rows or k columns, padding or special processing is used to ensure that all sub-blocks have regular shapes. After the division is completed, a logical grid index is generated to identify and manage each generated first sub-block (e.g., A11, A12, ...), and the metadata of these sub-blocks (such as starting memory address and size) is recorded for subsequent scheduling and transmission.

[0053] Step 1014: Divide the second matrix evenly according to the second block value and the third block value to generate the plurality of second sub-blocks with regular shapes.

[0054] In this embodiment, the second matrix is ​​another input matrix in the matrix multiplication operation. The third block value is determined in step 1012 and is used to divide the second matrix into sub-block sizes in the column direction.

[0055] The second matrix is ​​partitioned based on the second block value (k) and the third block value (n). Since matrix multiplication requires the number of columns in the first matrix to be equal to the number of rows in the second matrix, the number of rows (K dimensions) of the second matrix is ​​implicitly determined by the number of columns in the first matrix (the second block value k). Starting from the top left corner of the second matrix, every k consecutive rows are grouped together, and every n consecutive columns are grouped together. Similarly, boundary cases are handled to ensure the regularity of the sub-block shapes. After partitioning, indexes and metadata for all second sub-blocks (e.g., B11, B12, …) are generated and managed.

[0056] Step 1015: The partitioning process ensures that the first sub-block and the second sub-block match in size along a common dimension to satisfy the algebraic rules of matrix multiplication.

[0057] In this embodiment, the common dimension in matrix multiplication specifically refers to the column dimension of the first matrix and the row dimension of the second matrix, and these two dimensions must be equal in algebraic operations. Size matching means that when performing sub-block matrix multiplication, the number of columns of the first sub-block must be strictly equal to the number of rows of the second sub-block being multiplied; this is a prerequisite for matrix multiplication to be performed.

[0058] After partitioning, a validation check is performed. The column size of the first sub-block (i.e., the second partition value k) is compared with the row size of the second sub-block (this size is guaranteed by the partitioning process and is also equal to the second partition value k). These two values ​​are confirmed to be completely equal. If they are equal, the partitioning process is valid, ensuring that any first sub-block can be used for a valid matrix multiplication with its corresponding second sub-block. If a mismatch occurs due to incorrect parameter settings, an error will be reported, and the process will terminate or the partitioning parameters will be readjusted.

[0059] This application's embodiments achieve automated and optimized block partitioning of large-scale input matrices. This process comprehensively considers the computational power and memory bandwidth limitations of the underlying hardware, generating appropriately sized sub-blocks, thus laying the foundation for subsequent implementation of variable-order matrix multiplication calculations based on 3-buffer intelligent scheduling.

[0060] In some implementations, step 102 includes: Step 1021: Determine the row dimension of the temporary storage space based on the first block value of the first sub-block in the row direction.

[0061] In this embodiment, the temporary storage space is a contiguous memory region specifically allocated within the high-speed memory buffer of the AI ​​accelerator for storing intermediate results of matrix multiplication calculations. The row dimension refers to the size or length of this temporary storage space along the row direction in its two-dimensional structure when data is organized in memory.

[0062] First, the block information of matrix A to be calculated is parsed to obtain the block size of the first sub-block in the row direction of matrix A, i.e., the first block value. This first block value is directly used as the row dimension parameter of the temporary storage space. This operation determines the number of rows of the intermediate result matrix that the temporary storage space needs to logically accommodate, and this number of rows is consistent with the number of rows of the sub-blocks in the row group currently participating in the calculation of matrix A.

[0063] Step 1022: Determine the column dimension of the temporary storage space based on the product of the variable parameter and the third block value of the second sub-block in the column direction. The variable parameter is used to control the number of the second sub-blocks involved in a single calculation loop.

[0064] In this embodiment, the variable parameter is a user-configurable integer parameter used to control the number of sub-blocks loaded from matrix B and involved in the calculation in a single calculation loop. The column dimension refers to the size or length of the temporary storage space in the column direction of its two-dimensional structure when data is organized in memory.

[0065] The system reads the preset variable parameter L and parses the block information of matrix B to obtain the block size of the second sub-block in the column direction of matrix B, i.e., the third block value. It then performs a multiplication operation to calculate the product of the variable parameter L and the third block value. This product result is used as the column dimension parameter of the temporary storage space. This operation determines the number of columns of the intermediate result matrix that the temporary storage space logically needs to hold, which is equal to the total number of columns of the multiple sub-blocks of matrix B processed in a single calculation loop.

[0066] Step 1023: Based on the determined row and column dimensions, apply for a contiguous storage area within a specific physical address range of the high-speed memory buffer as the temporary storage space. The temporary storage space is used to store the intermediate results generated when a single row group of the first matrix is ​​multiplied by a column group of the second matrix.

[0067] In this embodiment, the high-speed memory buffer is a high-speed, low-latency storage unit inside an AI accelerator chip, typically composed of multiple memory banks, used to cache data frequently accessed by the computing unit. A specific physical address range refers to a physically contiguous, unoccupied range of memory addresses within the high-speed memory buffer.

[0068] Based on the row dimension determined in step 1021 and the column dimension determined in step 1022, the total storage capacity required for the temporary storage space is calculated. A sufficiently large and physically contiguous storage region is located within the free memory area of ​​the high-speed memory buffer. A request is made to the memory management unit, marking this contiguous physical address region as occupied, and a logical temporary storage space data structure is established to map it to the memory management unit. This temporary storage space is designated to store all intermediate results generated during multiple rounds of sub-block multiplication and accumulation operations between a single row group of matrix A and a column group of matrix B.

[0069] This application embodiment accurately calculates the minimum memory space size required to store intermediate results by combining matrix block parameters and configurable computation loop parameters, and efficiently allocates contiguous physical storage in a high-speed memory buffer, ensuring that intermediate results can be stored and accessed compactly and continuously in subsequent variable-order matrix multiplication calculations.

[0070] In some implementations, step 103 includes: Step 1031: Using the row grouping of the first matrix as the outer loop unit, in each loop, one of the first sub-blocks belonging to the same row group is loaded from the external memory into the input buffer of the matrix calculation unit.

[0071] In this embodiment, external memory refers to a storage device located outside the AI ​​accelerator chip, with a large capacity but relatively slow access speed, such as DDR memory. The matrix computation unit is a core hardware module within the AI ​​accelerator specifically designed to perform matrix multiplication operations. The input buffer is a high-speed memory area within or adjacent to the matrix computation unit, used to temporarily store input data that will be used in the computation.

[0072] First, the first matrix is ​​grouped row-wise. Each row group serves as the basic unit of the outer loop. At the beginning of each outer loop, a first sub-block belonging to the current row group (e.g., a sub-block A_ij of matrix A) is read from external memory and transferred and stored in the input buffer dedicated to the matrix computation unit. This first sub-block will be reused in subsequent calculations within this outer loop.

[0073] Step 1032: While the first sub-block resides in the matrix calculation unit, traverse the multiple second sub-blocks corresponding to the current column group in an inner loop manner.

[0074] In this embodiment, the inner loop is a loop structure nested within the outer loop. In the current context, it refers to the process of traversing the sequence of second matrix sub-blocks that need to be multiplied after the first sub-block has been loaded.

[0075] Once the first sub-block is successfully loaded and resides in the input buffer of the matrix computation unit, an inner loop is initiated. This inner loop iterates sequentially through all the second sub-blocks (e.g., sub-block B_jk of matrix B) within the column group of the second matrix corresponding to the current computation task. This traversal process ensures that the first sub-block can be paired with each necessary second sub-block in the column group for computation in turn.

[0076] Step 1033: Load each of the second sub-blocks traversed from the external memory into another input buffer of the matrix calculation unit.

[0077] In this embodiment, the other input buffer refers to a separate buffer in the matrix computation unit used to store another input matrix data. Typically, the matrix computation unit is equipped with at least two input buffers, used to store the left operand (first matrix sub-block) and right operand (second matrix sub-block) of the multiplication, respectively.

[0078] During the inner loop traversal, a data loading operation is performed each time a new second sub-block is pointed to. The currently traversed second sub-block is read from external memory and transferred and stored in another input buffer of the matrix computation unit. At this point, both input sub-blocks participating in a single matrix multiplication kernel operation are in place, and the matrix computation unit can perform the multiplication calculation of these two sub-blocks.

[0079] This embodiment of the application significantly reduces the number of times the first matrix is ​​repeatedly loaded from external memory by grouping the first matrix by row and fixing its sub-blocks in the computing unit, and then cyclically loading multiple sub-blocks of the second matrix for computation. This data reuse strategy effectively reduces the dependence on external memory bandwidth and alleviates the bandwidth bottleneck.

[0080] In some implementations, step 104 includes: Step 1041: Perform multiplication and addition operations on the currently loaded first sub-block and second sub-block using the matrix calculation unit to generate a partial result matrix.

[0081] In this embodiment, the matrix computation unit is a dedicated hardware module in the AI ​​accelerator, specifically designed for efficiently performing matrix multiplication operations. The first sub-block and the second sub-block are smaller data blocks partitioned from the original large matrix, representing portions of matrix A and matrix B, respectively. The partial result matrix is ​​an intermediate product matrix obtained by the matrix computation unit after performing multiplication operations on the currently loaded first and second sub-blocks, which has not yet been accumulated with historical results.

[0082] The control matrix computation unit takes the first and second sub-blocks, which have been loaded from its high-speed memory buffer into the unit's internal registers, as input. Following the matrix multiplication algorithm, the unit performs numerous multiplication-addition operations in parallel to compute the complete result of multiplying these two sub-blocks. This result, called the partial result matrix, corresponds to the product of a specific combination of sub-blocks in the current iteration, but it is not yet the corresponding part of the final output matrix, as this part may be composed of the sum of multiple such multiplications.

[0083] Step 1042: Read the intermediate result matrix block corresponding to the current second sub-block column position from the temporary storage space.

[0084] In this embodiment, the temporary storage space is a pre-allocated storage area within the high-speed memory buffer, used to store intermediate accumulation results during matrix multiplication. The intermediate result matrix block is a data block stored in the temporary storage space, representing the sum of all calculated partial results for a specific sub-region of the final output matrix up to the current calculation step.

[0085] Based on the column position information of the currently processed second sub-block in the original matrix B, determine which intermediate result matrix block in the temporary storage space should be accumulated to corresponding to the partial result of this calculation. Then, initiate a read operation to retrieve the previously calculated and stored intermediate result matrix block from the specific storage address corresponding to that column position in the temporary storage space, and transfer it to the input buffer of the vector calculation unit to prepare data for subsequent accumulation operations.

[0086] Step 1043: In the vector calculation unit, the partial result matrix is ​​added element by element to the read intermediate result matrix block to generate an updated intermediate result matrix block.

[0087] In this embodiment, the vector computation unit is a dedicated hardware module in an AI accelerator used to perform element-wise operations (such as addition and multiplication), capable of efficiently processing vector or matrix data. Element-wise addition refers to the operation of adding the values ​​at corresponding positions of two matrices of the same dimension to obtain a new matrix of the same dimension.

[0088] The control vector computation unit receives a partial result matrix from the matrix computation unit and an intermediate result matrix block read from temporary storage space. The vector computation unit performs operations in parallel, adding each element in the partial result matrix to the corresponding element in the intermediate result matrix block. This operation accumulates the computation results of the current iteration onto the historical sum, generating an updated intermediate result matrix block containing the sum of all relevant partial results up to the current time.

[0089] Step 1044: Write the updated intermediate result matrix block back to the storage address in the temporary storage space that is the same as the read location.

[0090] In this embodiment, the storage address is a specific memory location identifier in the temporary storage space used to locate and access data blocks. The updated intermediate result matrix block is an intermediate data block containing the latest calculation results, obtained after an accumulation operation.

[0091] The updated intermediate result matrix block generated by the vector computation unit is written back to the temporary storage space of the high-speed memory buffer through the memory controller. The target address of the write-back operation is exactly the same as the source address of the read operation performed in step 1042, that is, overwriting the original old intermediate result. This operation ensures that the latest accumulation result is always stored in the temporary storage space, providing the correct basic data for the accumulation operation in the next round of computation.

[0092] The embodiments of this application decompose large matrix multiplication into iterative calculations of small data blocks, and dynamically add and update partial results with accumulated intermediate results in each iteration, which greatly reduces the number of data accesses to slow external memory.

[0093] In some implementations, step 105 includes: Step 1051: Divide the physical storage unit in the high-speed memory buffer used to store input data and intermediate results into at least three independent storage blocks.

[0094] In this embodiment, the high-speed memory buffer is a high-speed storage area within the AI ​​accelerator, used to temporarily store data to be processed by the computing unit or intermediate results generated by the computation. Its access speed is much higher than that of external DDR memory. The physical storage unit is the basic storage circuit unit constituting the high-speed memory buffer. A storage bank is a logical or physical partitioned area within the high-speed memory buffer with independent read / write ports; each storage bank can independently perform data access operations.

[0095] The physical storage units in the high-speed memory buffer used to store input data (such as sub-blocks of matrix B) and intermediate results from matrix multiplication are logically or physically divided into at least three independent memory banks. This division is based on the characteristics of the hardware architecture; for example, the address space or physical circuitry of the high-speed memory buffer (typically composed of multiple banks) is configured as at least three independent storage modules that can operate in parallel. After division, each memory bank has an independent access path, laying the hardware foundation for subsequent parallel data scheduling.

[0096] Step 1052: Assign different storage bodies as target operation objects for the input operation of the second sub-block and the output and input operations of the intermediate results.

[0097] In this embodiment, the second sub-block refers to a data block of matrix B transferred from external DDR memory to a high-speed memory buffer. An incoming operation refers to the process of writing data from a data transfer unit (such as a DMA controller) to the high-speed memory buffer. An intermediate result refers to the partial sum and result generated by the matrix calculation unit performing multiplication and addition operations on sub-blocks A and B of matrix A. An outgoing operation refers to the process of reading data from the high-speed memory buffer to the vector calculation unit. In this step, the incoming operation also refers to the process of writing the updated intermediate result from the vector calculation unit back to the high-speed memory buffer. The target operation object refers to the specific memory bank specified for a particular data transfer or access operation.

[0098] Different memory banks are assigned as the operation targets for three types of key data transfer operations to achieve spatial separation of operations. Specifically, the second sub-block of matrix B, which needs to be loaded from DDR to the high-speed memory buffer, is allocated to be written to a specific memory bank (the first target memory bank). Simultaneously, the old intermediate results that need to be read from the high-speed memory buffer for accumulation calculations by the vector computation unit are allocated to be read from another different memory bank (the second target memory bank). Furthermore, the new intermediate results generated after the vector computation unit completes the accumulation are allocated to be written to a third memory bank that is different from the first two (the third target memory bank). This allocation ensures that the physical storage resources accessed by the three types of data flow operations do not overlap at any given time.

[0099] Step 1053: Establish scheduling rules so that read and write operations on the same memory bank do not overlap in time within any given operation cycle.

[0100] In this embodiment, the scheduling rule is a set of predefined or dynamically generated instruction sequences executed by a controller or scheduler, used to specify the execution order and timing of different data transmission and access operations on the timeline. An operation cycle refers to the minimum time unit or clock cycle for performing one basic data transmission or processing operation. A read operation refers to the operation of reading data from memory. A write operation refers to the operation of writing data to memory. Temporal overlap refers to two or more operations contending for the same hardware resource within the same operation cycle or time period.

[0101] Establish and solidify a set of scheduling rules. The core constraint of these rules is to ensure that read and write operations are not initiated simultaneously on the same storage bank within any given operation cycle. This means that for a specific storage bank, the time periods during which data is written to it and the time periods during which data is read from it must be completely staggered by the scheduler, with no overlap. This rule aims to fundamentally avoid hardware access conflicts and pipeline stalls caused by concurrent reads and writes to the same storage port, thereby eliminating the additional latency resulting from these operations.

[0102] Step 1054: According to the scheduling rules, the control data transport unit writes the second sub-block into the first target storage while simultaneously reading the old intermediate result from the second target storage to the vector calculation unit, and writes the updated intermediate result output from the vector calculation unit into the third target storage. The three target storages rotate in a preset sequence to ensure continuous and conflict-free transmission of the data stream.

[0103] In this embodiment, the data transport unit is a dedicated hardware module in the AI ​​accelerator responsible for transporting data between DDR memory, high-speed memory buffers, and computing units, such as a direct memory access controller. The vector computing unit is a hardware computing core specifically designed to perform vector or matrix operations (such as multiplication and addition). The preset sequence is a predefined, cyclical list used to determine the rotation pattern of the three target storage units (first, second, and third) in different computing cycles, specifying which physical storage unit they are mapped to. Data flow refers to the path and order of data transmission between storage units and computing units. Continuous, conflict-free transmission means that data can move continuously between components according to computing needs without waiting or blocking due to resource contention.

[0104] According to the established scheduling rules, the operation timing of the data transport unit and the vector computation unit is precisely controlled. In a typical operation cycle, the data transport unit writes a second sub-block of matrix B to the first target storage designated for the current cycle. Simultaneously, it reads the intermediate result stored in the previous cycle from the second target storage and transmits it to the vector computation unit. Upon receiving this intermediate result, the vector computation unit adds it to the computation results of the currently active sub-block of matrix A and the newly received sub-block of matrix B, generating an updated intermediate result. This updated intermediate result is then written to the third target storage. The roles of the first, second, and third target storage units are not fixed to any three physical storage units, but rather rotate according to a preset cyclic sequence (e.g., for three storage units Bank1, Bank2, and Bank3, the sequence might be Bank1->Bank2->Bank3->Bank1...) after each operation cycle. This rotation mechanism ensures that the three data flows—data inflow (matrix B sub-block), data outflow (old intermediate result), and data write-back (new intermediate result)—can proceed continuously and in parallel, without accessing the same storage at any time. This achieves continuous and conflict-free data transmission throughout the entire computation process.

[0105] This application provides a parallel hardware foundation by dividing the data into multiple independent storage banks. Then, it allocates independent storage targets to concurrent data streams and establishes strict read / write timing rules to avoid conflicts. Finally, it achieves seamless data stream connection by dynamically rotating the roles of the storage banks. All steps work together, and the overall technical effect is: during matrix multiplication calculations, it completely eliminates transmission access conflicts caused by concurrent reads and writes to the same storage bank within the high-speed memory buffer, minimizing data transfer time and improving the actual utilization efficiency of hardware bandwidth.

[0106] In some implementations, step 1053 includes: Step 10531: Within the operation cycle, a first priority time period is allocated for the data write operation of the second sub-block, a second priority time period is allocated for the data read operation of the intermediate result, and a third priority time period is allocated for the data write operation of the updated intermediate result. The first priority time period, the second priority time period, and the third priority time period are arranged sequentially in time and do not overlap with each other.

[0107] In this application embodiment, an operation cycle refers to the smallest repeated time unit for executing a complete data processing flow. In this application, one operation cycle typically corresponds to processing a sub-block of matrix B and completing one accumulation and write-back of intermediate results. A priority period refers to an exclusive time window allocated for a specific type of data operation within an operation cycle. During this period, only operations of that priority are executed to avoid conflicts when different types of operations access the same hardware resources.

[0108] A complete operation cycle is divided into three consecutive, non-overlapping time periods on the timeline. In the first priority period, the current sub-block of matrix B is written from external memory (such as DDR) to a high-speed memory buffer. Then, in the second priority period, the intermediate results needed for the current accumulation are read from the high-speed memory buffer. Finally, in the third priority period, the new intermediate results, updated by the matrix calculation unit, are written back to the high-speed memory buffer. This time-sequencing enforces the order of read and write operations, isolating potentially concurrent accesses to the same memory bank in terms of time.

[0109] Step 10532: Arrange the at least three memory banks in numerical order, and perform a modulo operation on the memory bank numbers according to the current sub-block sequence number. Based on the operation result, dynamically map the specific numbers of the first target memory bank, the second target memory bank, and the third target memory bank to ensure that the memory bank number being operated on is unique within any priority time period.

[0110] In this embodiment, a memory bank is the basic physical unit (Bank) constituting a high-speed memory buffer. Each memory bank can be read or written independently, but at any given time, only one operation (read or write) can be performed on a memory bank. Modulo operation is a mathematical operation that returns the remainder when one number is divided by another. In this step, the modulo operation is performed on the total number of memory banks (at least 3) using the currently processed sub-block number to determine the starting offset for cyclic use. Dynamic mapping refers to an allocation strategy where the target memory bank's number is not fixed but is calculated and determined in real time based on the currently processed sub-block number.

[0111] First, the storage banks of at least three high-speed memory buffers are arranged in physical order. When processing each sub-block of matrix B, a modulus value is calculated based on the sub-block's index. Based on this modulus value, the specific indexes of the three target storage banks are dynamically determined: a first target storage bank for writing to the sub-block of matrix B, a second target storage bank for reading old intermediate results, and a third target storage bank for writing new intermediate results. The mapping rules are specially designed to ensure that the accessed storage bank index is unique within any priority period defined in step 10531. For example, when writing a sub-block of matrix B to a storage bank with a certain index in the first priority period, the intermediate results to be read in the second priority period will never be located in the same storage bank, thus achieving complete decoupling of read and write operations in time and space.

[0112] This application enables efficient and conflict-free scheduling of input, intermediate result reading, and update write-back for sub-blocks of matrix B within a limited high-speed buffer space consisting of only three memory banks. It completely eliminates the additional latency caused by memory access conflicts, allowing data transfer to continue at maximum bandwidth. Simultaneously, this scheduling strategy, combined with variable-order matrix multiplication, significantly reduces the data transfer requirements for matrix A, thereby overcoming the overall bottleneck of memory bandwidth in computational performance and achieving a significant improvement in matrix multiplication efficiency under limited hardware costs.

[0113] In some implementations, the core innovation of this application lies in the deep synergy between "3-buffer intelligent scheduling technology" and "computation order reconstruction," which achieves a breakthrough in bandwidth bottlenecks through extremely small temporary high-speed memory buffer space, while completely eliminating transmission conflicts. Its overall technical solution is as follows: Figure 3 As shown, this application only requires allocating memory in a high-speed memory buffer. temp space ( Figure 3 As shown (In the extreme case) storing the intermediate results of matrix multiplication can overcome the bandwidth bottleneck. The overall calculation steps are as follows: Step 1: Transfer sub-block A11 of matrix A to the matrix calculation unit.

[0114] Step 2: Transfer sub-block B11 of matrix B to the matrix calculation unit.

[0115] Step 3: Store the matrix multiplication result of sub-block A11 and sub-block B11 into the high-speed memory buffer D11.

[0116] Step 4: Repeat Step 2 and Step 3 L times in the direction of B11, B12, ..., B1L to fill the temp space of the high-speed memory buffer.

[0117] Step 5: Transfer sub-block A12 of matrix A to the matrix calculation unit.

[0118] Step 6: Transfer sub-block B21 of matrix B to the matrix calculation unit.

[0119] Step 7: Add the matrix multiplication result of sub-block A12 and sub-block B21 to the intermediate result D11 and then store it in the high-speed memory buffer D11.

[0120] Step 8: Repeat Step 6 and Step 7 L times in the direction of B21, B22, ..., B2L to fill the temp space of the high-speed memory buffer.

[0121] Step 9: Following the directions A11, A12, ..., A1k, repeat Step 5, Step 6, Step 7, and Step 8 k times to finally obtain the first row of matrix A. With matrix The result of matrix multiplication.

[0122] Step 10: Repeat Step 1, Step 2, Step 3, Step 4, and Step 9 m times to obtain the matrix. With matrix The result of matrix multiplication.

[0123] Step 11: Repeat Step 10 yields the matrix. With matrix The result of matrix multiplication.

[0124] It is not difficult to see from the above calculation steps of this application that completing one matrix With matrix Matrix multiplication requires matrix A to be transmitted. Matrix B is transmitted m times. Here, L is a variable parameter; the larger L is, the larger the required temp space, and the fewer transmissions matrix A needs. To obtain the theoretical limit of the temp space required to overcome the bandwidth bottleneck, this application provides a calculation method based on a specific case. Assume the data transmission bandwidth is 51.2 GB / s, the data transmission efficiency is 70%, matrix A is a bfloat16 data type with each value occupying 2 bytes, matrix B is a quantized 4-bit data type with each value occupying 0.5 bytes, and the matrix calculation unit performs one bfloat16 data type matrix... With matrix The computation time required for matrix multiplication is given by the following formula (3): (3) Where subM and subK represent the number of rows and columns of the left submatrix, and subN represents the number of columns of the right submatrix.

[0125] It is not difficult to see from calculation steps 4 and 8 that the overall matrix With matrix Matrix multiplication consists of multiple such small computational structures, that is, matrix A transmits a... The sub-blocks and L elements of matrix B The sub-blocks are used for matrix multiplication. Therefore, when this computational structure breaks through the bandwidth bottleneck, the overall matrix multiplication will also break through the bandwidth bottleneck. The computation transmission time and matrix computation time are given by the following formulas (4) and (5): (4) (5) Where subM and subK represent the number of rows and columns of the left submatrix, subN represents the number of columns of the right submatrix, and L is a constant coefficient.

[0126] Here it is set After simplification, where P is a constant coefficient, the simplified results are as follows: (6) and (7): (6) (7) Where subM and subK represent the number of rows and columns of the left submatrix, and L and P are constant coefficients.

[0127] Breaking through the bandwidth bottleneck will lead to... Substituting the simplified time formula above, we can obtain the following inequalities: (8), (9), (10), and (11): (8) (9) (10) (11) Where subM and subK represent the number of rows and columns of the left submatrix, and L and P are constant coefficients.

[0128] As can be seen from the above formula, in this case, only design is required. satisfy Simultaneously, by designing L and P to satisfy the above formula, the bandwidth bottleneck can be overcome. Since different hardware only changes the values ​​of some parameters, based on the process of this application, a design that meets the requirements can be created. , Furthermore, by providing a temp space that satisfies condition L, matrix multiplication can be accelerated under bandwidth bottlenecks. Moreover, this application does not restrict the dimension of the matrix, and the required temp space will not increase with the increase of dimension, which effectively saves hardware costs.

[0129] In the design of AI acceleration chips, high-speed memory buffers typically consist of multiple banks, each with the hardware characteristic of only being able to read from or write to one type of hardware at a time. Traditional solutions, lacking targeted memory allocation strategies, suffer from additional transmission time in computation steps like Step 4 and Step 8 of this application due to transmission conflicts (reading and writing to the same bank simultaneously, or reading or writing to different hardware concurrently). However, the 3-buffer intelligent scheduling technology proposed in this application completely eliminates transmission conflicts by precisely allocating memory space and scheduling data storage and transmission timing, providing a crucial guarantee for improving matrix multiplication performance.

[0130] The main process of data transmission is as follows Figure 4 As shown, this can be approximated as performing a matrix multiplication of a sub-block after each transmission of matrix B, then adding the result of the matrix multiplication to the intermediate result in temp and putting it back into temp. Therefore, there are two main types of transmission conflicts affecting transmission time in the high-speed memory cache, namely matrix B... High-speed memory buffer and high-speed memory buffer Two types of conflicts between vector computation units.

[0131] like Figure 5 The 3-buffered intelligent scheduling technology shown can effectively resolve the two conflicts mentioned above. Let the banks of the three high-speed memory buffers be numbered 1, 2, and 3. The specific transmission process is as follows: First, determine if L modulo 3 is 1. If the modulo is not 1, the sub-blocks of matrix B are stored in the corresponding bank's high-speed memory buffer in the order of bank number 1, 2, 3, 1, 2, 3, 1, 2, ..., and the intermediate results are stored in the corresponding bank's temp in the order of bank number 3, 1, 2, 3, 1, 2, 3, 1, ...; if the modulo is 1, the sub-blocks of matrix B are stored in the corresponding bank's high-speed memory buffer in the order of bank number 1, 2, 3, 1, 2, 3, 1, 2, ..., 2, and the intermediate results are stored in the corresponding bank's temp in the order of bank number 3, 1, 2, 3, 1, 2, 3, 1, ..., 1.

[0132] After implementing this 3-buffered intelligent scheduling strategy, when a sub-block of matrix B is passed to bank1, bank3 outputs the intermediate result; when a sub-block of matrix B is passed to bank2, bank1 outputs the intermediate result, while the vector computation unit passes the intermediate result to bank3; when a sub-block of matrix B is passed to bank3, bank2 outputs the intermediate result, while the vector computation unit passes the intermediate result to bank1; when a sub-block of matrix B is passed to bank1, bank3 outputs the intermediate result, while the vector computation unit passes the intermediate result to bank2. This process is repeated until the result of the matrix multiplication is calculated. In this way, the input of sub-blocks and the input and output of intermediate results in the high-speed memory buffer do not occur in the same bank, thus resolving the additional transmission time loss caused by transmission conflicts and ultimately achieving accelerated variable-order matrix multiplication based on 3-buffered intelligent scheduling.

[0133] Based on the foregoing embodiments, this application provides a memory allocation device, which includes the included units and the modules included in each unit, which can be implemented by a processor in a computer device; of course, it can also be implemented by specific logic circuits; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0134] Figure 6 This is a schematic diagram of the composition structure of a memory allocation device provided in an embodiment of this application, as shown below. Figure 6 As shown, the memory allocation device 20 includes: Processing module 201 is used to divide the first matrix and the second matrix to be calculated according to the preset first sub-block size and second sub-block size to obtain multiple first sub-blocks and multiple second sub-blocks; Allocate temporary storage space in the high-speed memory buffer that is related to the row dimension and variable parameter product of the first sub-block; Based on the reconstructed calculation order, the first and second sub-blocks currently required are sequentially transferred from external memory to the matrix calculation unit; The updated intermediate result is obtained by accumulating the calculation result of the matrix calculation unit with the intermediate result stored in the temporary storage space, and the updated intermediate result is written back to the corresponding position in the temporary storage space. The allocation module 202 is used to coordinate the input operation of the second sub-block to the high-speed memory buffer and the input and output operation of the intermediate result between the high-speed memory buffer and the vector calculation unit through a memory allocation strategy and transmission scheduling strategy based on three buffers, so as to eliminate transmission conflicts and repeat the above process until the matrix multiplication calculation of all the first sub-blocks and the second sub-blocks is completed to obtain the final result matrix.

[0135] In some implementations, the allocation module 202 is further configured to: The physical storage units in the high-speed memory buffer used to store input data and intermediate results are divided into at least three independent storage blocks; Different storage bodies are allocated as target operation objects for the input operation of the second sub-block and the output and input operations of the intermediate results, respectively. Establish scheduling rules so that read and write operations on the same memory bank do not overlap in time within any given operation cycle; According to the scheduling rules, while the control data transport unit writes the second sub-block into the first target storage, it reads the old intermediate result from the second target storage into the vector calculation unit, and writes the updated intermediate result output from the vector calculation unit into the third target storage. The three target storages rotate in a preset sequence to ensure continuous and conflict-free transmission of the data stream.

[0136] In some implementations, the allocation module 202 is further configured to: Within the operation cycle, a first priority time period is allocated for the data write operation of the second sub-block, a second priority time period is allocated for the data read operation of the intermediate result, and a third priority time period is allocated for the data write operation of the updated intermediate result. The first priority time period, the second priority time period, and the third priority time period are arranged sequentially in time and do not overlap with each other. The at least three memory banks are arranged in numerical order, and a modulo operation is performed on the memory bank numbers according to the current sub-block sequence number. The specific numbers of the first target memory bank, the second target memory bank, and the third target memory bank are dynamically mapped according to the operation result to ensure that the memory bank number being operated on is unique within any priority time period.

[0137] In some implementations, the processing module 201 is further configured to: Obtain the total row dimension and total column dimension of the first matrix, and the total column dimension of the second matrix; Based on the computing power and memory bandwidth constraints of the hardware computing unit, the first block size in the row direction and the second block size in the column direction are determined, and the third block size in the column direction of the second sub-block is determined. The first matrix is ​​uniformly divided according to the first block value and the second block value to generate the plurality of first sub-blocks with regular shapes; The second matrix is ​​uniformly divided according to the second block value and the third block value to generate a plurality of second sub-blocks with regular shapes. The division process ensures that the first sub-block and the second sub-block match in size in the common dimension to satisfy the algebraic rules of matrix multiplication.

[0138] In some implementations, the processing module 201 is further configured to: The row dimension of the temporary storage space is determined based on the first block value of the first sub-block in the row direction; The column dimension of the temporary storage space is determined by multiplying the variable parameter with the third block value of the second sub-block in the column direction. The variable parameter is used to control the number of the second sub-blocks involved in a single calculation loop. Based on the determined row and column dimensions, a contiguous storage area is allocated within a specific physical address range of the high-speed memory buffer as the temporary storage space. The temporary storage space is used to store the intermediate results generated when a single row group of the first matrix is ​​multiplied by a column group of the second matrix.

[0139] In some implementations, the processing module 201 is further configured to: Using the row grouping of the first matrix as the outer loop unit, each loop loads one of the first sub-blocks belonging to the same row group from the external memory into the input buffer of the matrix calculation unit; While the first sub-block resides in the matrix calculation unit, the multiple second sub-blocks corresponding to the current column group are traversed in an inner loop manner; Each of the second sub-blocks traversed is loaded from the external memory into another input buffer of the matrix calculation unit.

[0140] In some implementations, the processing module 201 is further configured to: The matrix calculation unit performs multiplication and addition operations on the currently loaded first sub-block and second sub-block to generate a partial result matrix. Read the intermediate result matrix block corresponding to the current column position of the second sub-block from the temporary storage space; In the vector calculation unit, the partial result matrix is ​​added element by element to the read intermediate result matrix block to generate an updated intermediate result matrix block; Write the updated intermediate result matrix block back to the same storage address in the temporary storage space as the read location.

[0141] This application embodiment reduces the number of times data is loaded into one of the input matrices by reconstructing the calculation order, thereby significantly reducing the dependence on external memory bandwidth. Combined with a fixed-size temporary storage space, it effectively breaks through the bandwidth bottleneck under limited hardware costs. At the same time, the introduced three-buffer intelligent scheduling strategy completely eliminates transmission conflicts in high-speed memory buffer access, avoiding the additional latency caused by this. While saving hardware resources, it synergistically solves the two core problems of bandwidth limitation and transmission conflict, and achieves a leapfrog improvement in matrix multiplication calculation performance.

[0142] The descriptions of the apparatus embodiments above are similar to those of the method embodiments above, and have similar beneficial effects. In some embodiments, the functions or modules included in the apparatus provided in this application can be used to perform the methods described in the method embodiments above. For technical details not disclosed in the apparatus embodiments of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0143] It should be noted that, in the embodiments of this application, if the above-described memory allocation method is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the related technology, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, mobile hard drives, read-only memory (ROM), magnetic disks, or optical disks. Thus, the embodiments of this application are not limited to any specific hardware, software, or firmware, or any combination of hardware, software, and firmware.

[0144] This application provides a computer device including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the program, it implements some or all of the steps in the above-described method.

[0145] This application provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements some or all of the steps in the above-described method. The computer-readable storage medium can be transient or non-transient.

[0146] This application provides a computer program including computer-readable code, wherein when the computer-readable code is executed in a computer device, a processor in the computer device performs some or all of the steps in the above-described method.

[0147] This application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program. When the computer program is read and executed by a computer, it implements some or all of the steps in the above-described method. This computer program product can be implemented specifically through hardware, software, or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium; in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc.

[0148] It should be noted that the descriptions of the various embodiments above tend to emphasize the differences between them, while their similarities or commonalities can be referred to interchangeably. The descriptions of the above embodiments of the device, storage medium, computer program, and computer program product are similar to the descriptions of the above method embodiments and have similar beneficial effects. For technical details not disclosed in the embodiments of the device, storage medium, computer program, and computer program product of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0149] It should be noted that, Figure 7 This is a schematic diagram of a hardware entity of a computer device in an embodiment of this application, such as... Figure 7 As shown, the hardware entity of the computer device 700 includes: one or more processors 701, a communication interface 702, and a memory 703, wherein: Processor 701 typically controls the overall operation of computer device 700.

[0150] Communication interface 702 enables computer devices to communicate with other terminals or servers over a network.

[0151] The memory 703 is configured to store instructions and applications executable by the processor 701, and can also cache data to be processed or already processed (e.g., image data, audio data, voice communication data, and video communication data) in the processor 701 and various modules in the computer device 700. It can be implemented using flash memory or random access memory (RAM). Data transfer between the processor 701, the communication interface 702, and the memory 703 can be performed via bus 704. Only one processor is shown in the figure; each processor 700 includes one or more cores.

[0152] It should be noted that the computer device may include multiple processors 701, and each processor 701 can interact with each other through aggregated communication methods such as all-to-all, all-gather, or all-reduce. The processors 701 may be central processing units (CPUs), graphics processing units (GPUs), embedded neural network processing units (NPUs), tensor processing units (TPUs), data processing units (DPUs), accelerated processing units (APUs), floating-point processing units (FPUs), or application-specific integrated circuits (ASICs). The processors may also be single-core or multi-core processors. The processor may consist of a CPU and hardware chips. The hardware chips may be ASICs, PLDs, or combinations thereof. The PLDs may be complex programmable logic devices (CPLDs), FPGAs, generic array logic (GALs), or any combination thereof. The processor can also be implemented using logic devices with built-in processing logic, such as FPGAs or digital signal processors (DSPs).

[0153] The communication interface 702 can be a wired interface or a wireless interface, used to communicate with other modules or devices. The wired interface can be an Ethernet interface, a local interconnect network (LIN), etc., and the wireless interface can be a cellular network interface or a wireless LAN interface, etc.

[0154] Memory 703 can be non-volatile memory, such as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Memory 703 can also be volatile memory, which can be random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synclink dynamic random access memory (SLDRAM), and direct rambus RAM (DRRAM), direct rambus DRAM (DRDRAM), and rambus DRAM.

[0155] The 704 bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc.

[0156] It should be understood that the phrase "one embodiment" or "an embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above steps / processes do not imply a sequential order of execution; the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above embodiments of this application are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0157] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0158] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods, such as: multiple units or components can be combined, or integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the various components shown or discussed can be through some interfaces, and the indirect coupling or communication connection between devices or units can be electrical, mechanical, or other forms.

[0159] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units. They may be located in one place or distributed across multiple network units. Some or all of the units may be selected to achieve the purpose of this embodiment according to actual needs.

[0160] In addition, each functional unit in the various embodiments of this application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the integrated unit can be implemented in hardware or in the form of hardware plus software functional units.

[0161] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0162] Alternatively, if the integrated units described above are implemented as software functional modules and sold or used as independent products, they can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the part that contributes to related technologies, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as mobile storage devices, ROM, magnetic disks, or optical disks.

[0163] The above description is merely an embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A memory allocation method, characterized in that, The method includes: The first and second matrices to be calculated are divided according to the preset first and second sub-block sizes to obtain multiple first sub-blocks and multiple second sub-blocks; Allocate temporary storage space in the high-speed memory buffer that is related to the row dimension and variable parameter product of the first sub-block; Based on the reconstructed calculation order, the first and second sub-blocks currently required are sequentially transferred from external memory to the matrix calculation unit; The updated intermediate result is obtained by accumulating the calculation result of the matrix calculation unit with the intermediate result stored in the temporary storage space, and the updated intermediate result is written back to the corresponding position in the temporary storage space. By using a three-buffered memory allocation strategy and a transmission scheduling strategy, the input operation of the second sub-block to the high-speed memory buffer and the input and output operation of the intermediate result between the high-speed memory buffer and the vector calculation unit are coordinated to eliminate transmission conflicts and repeat the above process until the matrix multiplication calculation of all the first sub-blocks and the second sub-blocks is completed to obtain the final result matrix. The method of coordinating the input operation of the second sub-block to the high-speed memory buffer and the input and output operation of the intermediate result between the high-speed memory buffer and the vector computing unit through a memory allocation strategy and transmission scheduling strategy based on a triple buffer includes: The physical storage units in the high-speed memory buffer used to store input data and intermediate results are divided into at least three independent storage blocks; Different storage bodies are allocated as target operation objects for the input operation of the second sub-block and the output and input operations of the intermediate results, respectively. Establish scheduling rules so that read and write operations on the same memory bank do not overlap in time within any given operation cycle; According to the scheduling rules, while the control data transport unit writes the second sub-block into the first target storage, it reads the old intermediate result from the second target storage into the vector calculation unit, and writes the updated intermediate result output from the vector calculation unit into the third target storage. The three target storages are rotated in a preset sequence to ensure continuous and conflict-free transmission of the data stream. The establishment of scheduling rules, which ensures that read and write operations on the same memory bank do not overlap in time within any given operation cycle, includes: Within the operation cycle, a first priority time period is allocated for the data write operation of the second sub-block, a second priority time period is allocated for the data read operation of the intermediate result, and a third priority time period is allocated for the data write operation of the updated intermediate result. The first priority time period, the second priority time period, and the third priority time period are arranged sequentially in time and do not overlap with each other. The at least three memory banks are arranged in numerical order, and a modulo operation is performed on the memory bank numbers according to the current sub-block sequence number. The specific numbers of the first target memory bank, the second target memory bank, and the third target memory bank are dynamically mapped according to the operation result to ensure that the memory bank number being operated on is unique within any priority time period.

2. The method according to claim 1, characterized in that, The process of dividing the first and second matrices to be calculated according to preset first and second sub-block sizes to obtain multiple first sub-blocks and multiple second sub-blocks includes: Obtain the total row dimension and total column dimension of the first matrix, and the total column dimension of the second matrix; Based on the computing power and memory bandwidth constraints of the hardware computing unit, the first block size in the row direction and the second block size in the column direction are determined, and the third block size in the column direction of the second sub-block is determined. The first matrix is ​​uniformly divided according to the first block value and the second block value to generate the plurality of first sub-blocks with regular shapes; The second matrix is ​​uniformly divided according to the second block value and the third block value to generate a plurality of second sub-blocks with regular shapes. The division process ensures that the first sub-block and the second sub-block match in size in the common dimension to satisfy the algebraic rules of matrix multiplication.

3. The method according to claim 2, characterized in that, The step of allocating temporary storage space in the high-speed memory buffer related to the row dimension and variable parameter product of the first sub-block includes: The row dimension of the temporary storage space is determined based on the first block value of the first sub-block in the row direction; The column dimension of the temporary storage space is determined by multiplying the variable parameter with the third block value of the second sub-block in the column direction. The variable parameter is used to control the number of the second sub-blocks involved in a single calculation loop. Based on the determined row and column dimensions, a contiguous storage area is allocated within a specific physical address range of the high-speed memory buffer as the temporary storage space. The temporary storage space is used to store the intermediate results generated when a single row group of the first matrix is ​​multiplied by a column group of the second matrix.

4. The method according to claim 1, characterized in that, The step of sequentially transferring the currently required first sub-block and second sub-block from external memory to the matrix calculation unit according to the reconstructed calculation order includes: Using the row grouping of the first matrix as the outer loop unit, each loop loads one of the first sub-blocks belonging to the same row group from the external memory into the input buffer of the matrix calculation unit; While the first sub-block resides in the matrix calculation unit, the multiple second sub-blocks corresponding to the current column group are traversed in an inner loop manner; Each of the second sub-blocks traversed is loaded from the external memory into another input buffer of the matrix calculation unit.

5. The method according to claim 1, characterized in that, The process of accumulating the calculation result based on the matrix calculation unit with the intermediate result stored in the temporary storage space to obtain the updated intermediate result, and writing the updated intermediate result back to the corresponding location in the temporary storage space, includes: The matrix calculation unit performs multiplication and addition operations on the currently loaded first sub-block and second sub-block to generate a partial result matrix. Read the intermediate result matrix block corresponding to the current column position of the second sub-block from the temporary storage space; In the vector calculation unit, the partial result matrix is ​​added element by element to the read intermediate result matrix block to generate an updated intermediate result matrix block; Write the updated intermediate result matrix block back to the same storage address in the temporary storage space as the read location.

6. A memory allocation device, characterized in that, The device includes: The processing module is used to divide the first matrix and the second matrix to be calculated according to the preset first sub-block size and second sub-block size to obtain multiple first sub-blocks and multiple second sub-blocks; Allocate temporary storage space in the high-speed memory buffer that is related to the row dimension and variable parameter product of the first sub-block; Based on the reconstructed calculation order, the first and second sub-blocks currently required are sequentially transferred from external memory to the matrix calculation unit; The updated intermediate result is obtained by accumulating the calculation result of the matrix calculation unit with the intermediate result stored in the temporary storage space, and the updated intermediate result is written back to the corresponding position in the temporary storage space. The allocation module is used to coordinate the input operation of the second sub-block to the high-speed memory buffer and the input and output operation of the intermediate result between the high-speed memory buffer and the vector calculation unit through a memory allocation strategy and transmission scheduling strategy based on three buffers, so as to eliminate transmission conflicts and repeat the above process until the matrix multiplication calculation of all the first sub-blocks and the second sub-blocks is completed to obtain the final result matrix. The allocation module is specifically used for: The physical storage units in the high-speed memory buffer used to store input data and intermediate results are divided into at least three independent storage blocks; Different storage bodies are allocated as target operation objects for the input operation of the second sub-block and the output and input operations of the intermediate results, respectively. Establish scheduling rules so that read and write operations on the same memory bank do not overlap in time within any given operation cycle; According to the scheduling rules, while the control data transport unit writes the second sub-block into the first target storage, it reads the old intermediate result from the second target storage into the vector calculation unit, and writes the updated intermediate result output from the vector calculation unit into the third target storage. The three target storages are rotated in a preset sequence to ensure continuous and conflict-free transmission of the data stream. Within the operation cycle, a first priority time period is allocated for the data write operation of the second sub-block, a second priority time period is allocated for the data read operation of the intermediate result, and a third priority time period is allocated for the data write operation of the updated intermediate result. The first priority time period, the second priority time period, and the third priority time period are arranged sequentially in time and do not overlap with each other. The at least three memory banks are arranged in numerical order, and a modulo operation is performed on the memory bank numbers according to the current sub-block sequence number. The specific numbers of the first target memory bank, the second target memory bank, and the third target memory bank are dynamically mapped according to the operation result to ensure that the memory bank number being operated on is unique within any priority time period.

7. A computer device comprising a memory and a processor, the memory storing a computer program executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the memory allocation method according to any one of claims 1 to 5.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the memory allocation method according to any one of claims 1 to 5.

Images