Data processing method and data processing system based on memory partitioning
By using a memory-blocking data processing method, the matrix is divided into global, locally shared, and thread-private storage spaces for computation, which solves the problem of high computational cost in matrix multiplication operations in neural network models and improves computational efficiency and hardware utilization efficiency.
Patent Information
- Application Number
- CN202511294588.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-09-10
- Publication Date
- 2025-12-30
AI Technical Summary
Neural network models require frequent matrix multiplication operations during training and inference, resulting in high computational costs and hardware requirements. Existing technologies struggle to perform matrix multiplication operations efficiently.
A memory-blocking data processing method is adopted, which stores the matrix in the global storage space and performs matrix multiplication operations by dividing it into multiple local shared and thread-private storage spaces, thereby improving efficiency by utilizing the access characteristics of different memory levels.
By reducing the number of reads from global memory, the efficiency of matrix multiplication operations is improved, while reducing the hardware burden and computational cost.
Smart Images
Figure CN121235005A_ABST
Abstract
Description
Technical Field
[0001] The following description relates to a data processing method, and more specifically, to a data processing system based on memory block partitioning. Background Technology
[0002] With the widespread adoption of neural network models, many electronic devices have become intelligent, such as smartphones, drones, and self-driving cars. While neural network models have made progress in many applications, they involve a large number of matrix multiplication operations in both training and inference processes. These matrix multiplication operations typically require frequent access to hardware (e.g., memory) and incur high computational costs, making the application of neural network models a hardware-intensive process. Summary of the Invention
[0003] According to one aspect, a memory-based data processing method is provided, the data processing method comprising: storing a first matrix and a second matrix from a neural network model to be subjected to matrix multiplication operations in a global storage space in the memory space of a storage system, wherein the global storage space is globally shared; reading a plurality of first partition matrices and a plurality of second partition matrices from the global storage space, and writing the plurality of first partition matrices and the plurality of second partition matrices into a local shared storage space in the memory space of the storage system, wherein the plurality of first partition matrices are generated by partitioning a first matrix, and the plurality of second partition matrices are generated by partitioning a second matrix, wherein the local shared storage space is locally shared by multiple threads; and reading the plurality of first partition matrices and the plurality of second partition matrices from the local shared storage space, and performing matrix multiplication operations between the first matrix and the second matrix by performing corresponding matrix multiplication operations on the plurality of first partition matrices and the plurality of second partition matrices.
[0004] According to at least one example embodiment, the memory space of the storage system further includes a thread-private storage space, wherein the thread-private storage space is private to one of the plurality of threads.
[0005] According to at least one example embodiment, the step of reading the plurality of first partition matrices and the plurality of second partition matrices from a local shared storage space includes: reading from the local shared storage space a plurality of first sub-partition matrices and a plurality of second sub-partition matrices generated by partitioning each of the plurality of first partition matrices and each of the plurality of second partition matrices respectively.
[0006] According to at least one example embodiment, the step of performing a matrix multiplication operation between a first matrix and a second matrix by performing corresponding matrix multiplication operations on the plurality of first sub-partition matrices and the plurality of second sub-partition matrices includes: writing the plurality of first sub-partition matrices and the plurality of second sub-partition matrices into a thread-private storage space; and reading the plurality of first sub-partition matrices and the plurality of second sub-partition matrices from the thread-private storage space, and performing a matrix multiplication operation between the first matrix and the second matrix by performing corresponding matrix multiplication operations on the plurality of first sub-partition matrices and the plurality of second sub-partition matrices.
[0007] According to at least one example embodiment, the data processing method further includes: reading a plurality of third partition matrices generated by partitioning a third matrix from a global storage space, and writing the plurality of third partition matrices into a local shared storage space, wherein the result of matrix multiplication of the first matrix and the second matrix can be multiplied with the third matrix; reading a plurality of third sub-partition matrices generated by partitioning the plurality of third partition matrices from the local shared storage space, and writing the plurality of third sub-partition matrices into a thread-private storage space; and reading the plurality of third sub-partition matrices from the thread-private storage space, and performing a matrix multiplication operation between a second result matrix and a third matrix by performing corresponding matrix multiplication operations on the plurality of first result matrices and the plurality of third sub-partition matrices, wherein the plurality of first result matrices are result matrices generated by performing corresponding matrix multiplication operations on the plurality of first sub-partition matrices and the plurality of second sub-partition matrices, and the second result matrix is the result matrix of the matrix multiplication operation between the first matrix and the second matrix.
[0008] According to at least one example embodiment, the Softmax function is applied to the plurality of first result matrices before performing corresponding matrix multiplication operations on the plurality of first result matrices and the plurality of third subpartition matrices.
[0009] According to at least one example embodiment, the first matrix is a one-dimensional query vector in the attention block, the second matrix is the transpose of the one-dimensional key vector in the attention block corresponding to the one-dimensional query vector, and the third matrix is a one-dimensional value vector in the attention block corresponding to the one-dimensional query vector and the one-dimensional key vector.
[0010] According to at least one example embodiment, the step of reading a plurality of first partition matrices and a plurality of second partition matrices generated by partitioning a first matrix and a second matrix respectively from global storage space includes: reading the plurality of first partition matrices generated by partitioning a one-dimensional query vector from global storage space in an outer loop, and reading the plurality of second partition matrices generated by partitioning a transpose matrix of a one-dimensional key vector from global storage space in an inner loop, wherein the step of reading a plurality of third partition matrices generated by partitioning a third matrix from global storage space includes: reading the plurality of third partition matrices generated by partitioning a one-dimensional value vector from global storage space in an inner loop.
[0011] According to at least one example embodiment, the data processing method further includes: writing the result matrix of the matrix multiplication operation between the second result matrix and the third matrix into the global storage space in the outer loop.
[0012] According to at least one example embodiment, the first matrix and the second matrix are matrices in the attention block of a Transformer-based neural network model, wherein the Transformer-based neural network model is configured to perform one of speech recognition, natural language processing, and video generation.
[0013] According to one aspect, a memory-based data processing system is provided, the data processing system comprising: a storage system configured to store data; and a processor configured to: store a first matrix and a second matrix from a neural network model to be subjected to matrix multiplication operations in a global storage space within the memory space of the storage system, wherein the global storage space is globally shared; read a plurality of first partition matrices and a plurality of second partition matrices from the global storage space, and write the plurality of first partition matrices and the plurality of second partition matrices into a locally shared storage space within the memory space of the storage system, wherein the plurality of first partition matrices are generated by partitioning a first matrix, and the plurality of second partition matrices are generated by partitioning a second matrix, wherein the locally shared storage space is locally shared by multiple threads; and read the plurality of first partition matrices and the plurality of second partition matrices from the locally shared storage space, and perform matrix multiplication operations between the first matrix and the second matrix by performing corresponding matrix multiplication operations on the plurality of first partition matrices and the plurality of second partition matrices.
[0014] According to at least one example embodiment, the memory space of the storage system further includes a thread-private storage space, wherein the thread-private storage space is private to one of the plurality of threads.
[0015] According to at least one example embodiment, the processor is configured to: read from a local shared memory space a plurality of first sub-partition matrices and a plurality of second sub-partition matrices generated by partitioning each of the plurality of first partition matrices and each of the plurality of second partition matrices respectively, and write the plurality of first sub-partition matrices and the plurality of second sub-partition matrices into a thread-private memory space.
[0016] According to at least one example embodiment, the processor is configured to: read the plurality of first sub-partition matrices and the plurality of second sub-partition matrices from thread-private storage space, and perform matrix multiplication operations between the first matrix and the second matrix by performing corresponding matrix multiplication operations on the plurality of first sub-partition matrices and the plurality of second sub-partition matrices.
[0017] According to at least one example embodiment, the processor is further configured to: read from a global storage space a plurality of third partition matrices generated by partitioning a third matrix, and write the plurality of third partition matrices into a local shared storage space, wherein the result of matrix multiplication of the first matrix and the second matrix is capable of matrix multiplication with the third matrix; read from the local shared storage space a plurality of third sub-partition matrices generated by partitioning the plurality of third partition matrices, and write the plurality of third sub-partition matrices into a thread-private storage space; and read from the thread-private storage space the plurality of third sub-partition matrices, and perform a matrix multiplication operation between a second result matrix and a third matrix by performing corresponding matrix multiplication operations on the plurality of first result matrices and the plurality of third sub-partition matrices, wherein the plurality of first result matrices are result matrices generated by performing corresponding matrix multiplication operations on the plurality of first sub-partition matrices and the plurality of second sub-partition matrices, and the second result matrix is the result matrix of the matrix multiplication operation between the first matrix and the second matrix.
[0018] According to at least one example embodiment, the processor is further configured to apply the Softmax function to the plurality of first result matrices before performing corresponding matrix multiplication operations on the plurality of first result matrices and the plurality of third subpartition matrices.
[0019] According to at least one example embodiment, the first matrix is a one-dimensional query vector in the attention block, the second matrix is the transpose of the one-dimensional key vector in the attention block corresponding to the one-dimensional query vector, and the third matrix is a one-dimensional value vector in the attention block corresponding to the one-dimensional query vector and the one-dimensional key vector.
[0020] According to at least one example embodiment, the processor is configured to: read from global storage space in an outer loop the plurality of first partitioning matrices generated by partitioning a one-dimensional query vector, and read from global storage space in an inner loop the plurality of second partitioning matrices generated by partitioning a transpose matrix of a one-dimensional key vector, wherein the processor is configured to: read from global storage space in an inner loop the plurality of third partitioning matrices generated by partitioning a one-dimensional value vector.
[0021] According to at least one example embodiment, the processor is also configured to write the result matrix of the matrix multiplication operation between the second result matrix and the third matrix into the global storage space in the outer loop.
[0022] According to at least one example embodiment, the first matrix and the second matrix are matrices in the attention block of a Transformer-based neural network model, wherein the Transformer-based neural network model is configured to perform one of speech recognition, natural language processing, and video generation. Attached Figure Description
[0023] The above and other aspects will become clearer from the following description of exemplary embodiments in conjunction with the accompanying drawings.
[0024] Figure 1 This is a flowchart illustrating a memory-block-based data processing method according to an example embodiment.
[0025] Figure 2 This is a flowchart illustrating a memory-block-based data processing method according to an example embodiment.
[0026] Figure 3 This is a flowchart illustrating a memory-block-based data processing method according to an example embodiment.
[0027] Figure 4 This is a diagram illustrating a memory-block-based data processing method according to an example embodiment.
[0028] Figure 5 This is a block diagram illustrating a memory-block-based data processing system according to an example embodiment. Detailed Implementation
[0029] In the following, exemplary embodiments will be described in detail with reference to the accompanying drawings. Throughout the specification, the same components are designated by the same reference numerals, and repeated descriptions thereof are omitted. The embodiments described herein are exemplary embodiments, and therefore, this disclosure is not limited thereto, and may be implemented in various other forms. Each embodiment provided in the following description is not excluded from being associated with one or more features of another example, nor is it excluded from being associated with another embodiment, whether provided herein or not but consistent with this disclosure. The same reference numerals may refer to the same components.
[0030] Figure 1 This is a flowchart illustrating a memory-block-based data processing method according to an example embodiment.
[0031] Reference Figure 1 In operation S110, the first and second matrices of the neural network model to be multiplied can be stored in the global storage space of the storage system's memory space. The global storage space is shared globally.
[0032] Memory partitioning can represent memory space comprising multiple memory levels with different access characteristics. In one example embodiment, a memory partitioning-based data processing method may involve dividing data into multiple parts with different access characteristics. Access characteristics may include, but are not limited to, global access, local access, and / or private access. This partitioning can be based on chips with different levels of memory media (e.g., AI chips). For example, the memory levels of such chips may include global storage, local storage, and private storage. The storage space of global storage, local storage, and private storage can decrease from large to small. Furthermore, the access speed of global storage, local storage, and private storage also increases from slow to fast. Additionally, for local storage and private storage, multiple threads can share local storage space, and multiple workgroups can share private storage space.
[0033] Neural network models can include various types of neural network models. In one example, a neural network model can include a Transformer-based neural network model. For example, a Transformer-based neural network model is configured to perform one of speech recognition, natural language processing, and video generation. Transformers are widely recognized as a fundamental class of models, and their emergence has driven rapid development in fields such as large language models, image understanding, and video generation. For example, a neural network model can be a Transformer-based neural network model configured to perform natural language processing on text data, and the first and second matrices used for matrix computation can be matrices generated by the Transformer-based neural network model during the natural language processing of the text data.
[0034] In a non-restrictive example, the first matrix can be the query in the attention block (e.g., but not limited to a one-dimensional query vector), and the second matrix can be the transpose of the key corresponding to the query in the attention block (e.g., but not limited to a one-dimensional key vector). The attention block can be an attention block in various neural network models (e.g., Transformer-based neural network models). In one example, the attention block is a core structure in a Transformer-based neural network model, and its main computation involves multiplying two matrices and performing a softmax calculation to obtain an attention weight score. In the attention block, the matrix Query and the matrix Key are first multiplied (this is the first matrix computation of the attention block) and then a softmax calculation is performed to obtain an attention weight matrix. The attention weight matrix obtained in the previous step is then multiplied by the matrix Value to obtain the final output. Transformer-based neural network models generally contain multiple attention layers. Due to the increasing sequence length of the matrices to be processed, the runtime and memory usage of stacked attention blocks increase quadratically. Therefore, stacked attention blocks are considered a bottleneck for Transformer-type models in terms of fast inference or training. The memory-block-based data processing method according to the example embodiment can help improve the computational performance of stacked attention blocks.
[0035] Multiple threads can be different threads or different cores configured to perform one or more operations on a neural network model. Multithreading or multicore is common in different GPUs, NPUs, or other chips. By utilizing hardware architectures with multi-level memory (e.g., chips), the matrices in the fused attention block are sequentially stored in blocks at different memory levels. Leveraging the faster read performance and sharing mechanisms of different memory levels reduces overall data read time and improves hardware utilization efficiency, resulting in higher performance for the fused attention block implementation.
[0036] Because the global storage space is shared globally, it can be accessed by any application and / or thread. This means that the global storage space can be over-accessed. When the global storage space is accessed too frequently, its access efficiency will decrease, affecting how applications and / or threads use it.
[0037] In operation S120, multiple first partition matrices and multiple second partition matrices can be read from the global storage space, and these matrices can be written to the local shared storage space in the memory space of the storage system. The first matrix can be partitioned into multiple first partition matrices, and the second matrix can be partitioned into multiple second partition matrices. The local shared storage space is shared locally by multiple threads.
[0038] Multiple first partitioning matrices can be submatrices of a first matrix, and multiple second partitioning matrices can be submatrices of a second matrix. The first and second matrices can be partitioned in similarly arbitrary ways. For example, in a non-limiting example, when the first matrix is [0 1 1 0] and the second matrix is [1 0 0 1] T When the first matrix [0 1 1 0] can be divided into multiple first partition matrices [0 1] and [1 0], and the second matrix can be divided into multiple second partition matrices
[10] . T and [0 1] T In another non-restrictive example, when the first matrix is [0 1 1 0] and the second matrix is [1 0 01] T At that time, the first matrix [0 1 1 0] can be divided into multiple first partition matrices [0] and [1 1 0], and the second matrix can be divided into multiple second partition matrices [1] and [0 0 1]. T However, the example embodiments are not limited to the above examples, and the first matrix and the second matrix, as well as the way the first matrix and the second matrix are divided, are not limited to these.
[0039] In operation S130, multiple first partition matrices and multiple second partition matrices can be read from the local shared memory space, and matrix multiplication between the first matrix and the second matrix can be performed by performing corresponding matrix multiplication operations on the multiple first partition matrices and the multiple second partition matrices.
[0040] In the computation of a first matrix of size m×k (e.g., matrix A) multiplied by a second matrix of size k×n (e.g., matrix B) to obtain a matrix C of size m×n, if only global storage is used, in this comparative example, when computing a single element in matrix C, k elements from a single row of matrix A and k elements from a single row of matrix B are needed, and each read of a single element corresponds to one read operation; therefore, 2×k read operations to global storage are required. Since matrix C has m×n elements, computing matrix C requires 2×m×n×k read operations from global storage. Furthermore, each time a single element in matrix C is computed, a write operation to global storage is required; since matrix C has m×n elements, computing matrix C requires m×n write operations to global storage. Therefore, computing matrix C requires 2×m×n×k read operations from global storage and m×n write operations to global storage. For example, the read and write operations can be performed by the processor or under the control of the processor.
[0041] In contrast, the memory-block-based data processing method according to the example embodiment can apply block partitioning from global storage space to local shared storage space to reduce the number of global reads and improve overall execution efficiency.
[0042] Specifically, in one example, a matrix A of size m×k and a matrix B of size k×n can be divided into M×K first partition matrices and K×N second partition matrices, respectively. Each first partition matrix has a size of lm×lk, and each second partition matrix has a size of lk×ln, where m, k, n, M, K, N, lm, lk, and ln are positive integers, m = M×lm, k = K×lk, and n = N×ln. Based on a principle similar to that described in the comparative example above, calculating matrix C requires M×N×K×(lm×lk + lk×ln) global memory read operations. Compared to the 2×m×n×k global read operations in the comparative example, the global read operations of the memory-block-based data processing method according to the example embodiment become 1 / 2×(1 / lm + 1 / ln) times that of the comparative example, thus reducing the number of global reads and improving overall execution efficiency.
[0043] To better understand the example embodiment, the following description uses the first matrix as [0 1 1 0] and the second matrix as [1 0 01]. T The first matrix [0 1 1 0] is divided into multiple first partition matrices [0 1] and [1 0], and the second matrix is divided into multiple second partition matrices [1 0]. T and [0 1] TExamples are described, where m=1, k=4, n=1, M=2, K=1, N=1, lm=1, lk=2, and ln=1.
[0044] In the matrix multiplication operation between the first and second matrices, if (for example, as in the comparison example) only global storage is used, then 0×1, 1×0, 1×0, and 0×1 need to be calculated sequentially. When calculating the first 0×1, 0 needs to be read from the first matrix and 1 needs to be read from the second matrix. Therefore, the number of read operations on global storage is 2. When calculating the first 1×0, 1 needs to be read from the first matrix and 0 needs to be read from the second matrix. Therefore, the number of read operations on global storage is 2. When calculating the second 1×0, 1 needs to be read from the first matrix and 0 needs to be read from the second matrix. Therefore, the number of read operations on global storage is 2. When calculating the first 0×1, 0 needs to be read from the first matrix and 1 needs to be read from the second matrix. Therefore, the number of read operations on global storage is 2. Finally, the result of the matrix multiplication operation between the first and second matrices is written to global storage. Therefore, in this comparison example, the total number of read operations on the global storage space is 2×m×n×k=2×1×1×4=2+2+2+2=8 times, and the total number of write operations on the global storage space is m×n=1 times.
[0045] In contrast, in a memory-block-based data processing method according to an example embodiment (e.g., a data processing method executed by a processor), the first matrix can be divided into multiple first partition matrices, and the second matrix can be divided into multiple second partition matrices. The data processing method can read multiple first partition matrices and multiple second partition matrices from global storage space by applying blocks from global storage space to local shared storage space. Therefore, in performing matrix multiplication between the first and second matrices by performing corresponding matrix multiplication operations on the multiple first partition matrices and multiple second partition matrices, the number of read operations required is equal to M×N×K×(lm×lk+lk×ln) times. For example, the first matrix [0 1 1 0] can be divided into multiple first partition matrices [0 1] and [1 0], and the second matrix can be divided into multiple second partition matrices [1 0]. T and [0 1] T In the case of [0 1], in order to perform matrix multiplication of the first and second matrices, the processing according to the example embodiment requires calculating [0 1] × [1 0]. T +[0 1]×[0 1] T + [1 0]×[1 0] T +[0 1]×[1 0] TTherefore, the number of times elements are read from the multiple first partition matrices and multiple second partition matrices in the global storage space is equal to M×N×K×(lm×lk+lk×ln)=4 times. Compared with the comparative example, the memory-block-based data processing method according to the example embodiment reduces the number of global reads and improves the overall execution efficiency.
[0046] In the example above, the first matrix is divided into multiple first partition matrices of the same size, and the second matrix is divided into multiple second partition matrices of the same size. However, the partitioning method is not limited to this, and the first and second matrices can be partitioned in any way. For example, the first matrix can be partitioned such that the multiple first partition matrices have different sizes, and the second matrix can be partitioned such that the multiple second partition matrices have different sizes.
[0047] Figure 2 This is a flowchart illustrating a memory-block-based data processing method according to an example embodiment.
[0048] The memory space of the storage system may also include thread-private memory space, which is private to one of a plurality of threads. For example, thread-private memory space can be exclusively accessed by a single thread.
[0049] In other words, the memory-based data processing method according to the example embodiment can further apply the block division from the local shared storage space to the thread-private storage space on the basis of the block division from the global storage space to the local shared storage space.
[0050] In operation S210, multiple first sub-partition matrices and multiple second sub-partition matrices generated by partitioning each of multiple first partition matrices and each of multiple second partition matrices can be read from the local shared memory space.
[0051] The division in operation S210 can be compared with the reference. Figure 1 The partitioning in operation S120 is similar in principle. Therefore, to avoid redundancy, the partitioning will not be described in more detail here, but will instead refer to... Figure 1 The description of the partitioning in operation S120 can also be applied to the partitioning in operation S210.
[0052] In operation S220, multiple first sub-partition matrices and multiple second sub-partition matrices can be written to the thread-private storage space.
[0053] In operation S230, multiple first sub-partition matrices and multiple second sub-partition matrices are read from the thread-private storage space, and matrix multiplication between the first matrix and the second matrix is performed by performing corresponding matrix multiplication operations on the multiple first sub-partition matrices and multiple second sub-partition matrices.
[0054] Combination Figure 1 The description of operation S130 states that a matrix A of size m×k (i.e., the first matrix) and a matrix B of size k×n (i.e., the second matrix) can be divided into M×K first partition matrices and K×N second partition matrices, respectively. Each first partition matrix has a size of lm×lk, and each second partition matrix has a size of lk×ln, where m, k, n, M, K, N, lm, lk, and ln are positive integers, m = M×lm, k = K×lk, and n = N×ln. Completing the calculation of matrix C requires M×N×K×(lm×lk + lk×ln) read operations from the global memory space. Figure 2 In operation S240, with combination Figure 1 The principle described in operation S130 is similar. If the size of each first partition matrix is lm×lk and the size of each second partition matrix is lk×ln, without applying the block division from the local shared memory space to the thread private memory space, then completing the matrix multiplication operation of the first partition matrix and the second partition matrix requires 2×lm×ln×lk read operations and lm×ln write operations to the local shared memory space.
[0055] In one example, a first partition matrix of size lm × lk and a second partition matrix of size lk × ln can be divided into M1 × K1 first sub-partition matrices and K1 × N1 second sub-partition matrices, respectively. Each first sub-partition matrix has a size of rm × rk, and each second sub-partition matrix has a size of rk × rn, where lm, lk, ln, M1, K1, N1, rm, rk, and rn are positive integers, lm = M1 × rm, lk = K1 × rk, and ln = N1 × rn. Completing the calculation of the first and second partition matrices requires M1 × N1 × K1 × (rm × rk + rk × rn) read operations on the local shared memory space. Compared to the 2 × lm × ln × lk read operations on the local shared memory space in the comparison example without applying the block partitioning from the local shared memory space to the thread-private memory space, the global read operation of the memory-block-based data processing method according to the example embodiment becomes 1 / 2 × (1 / rm + 1 / rn) times that of the comparison example, thus reducing the number of shared reads and improving overall execution efficiency.
[0056] Figure 3This is a flowchart illustrating a memory-block-based data processing method according to an example embodiment.
[0057] In one example embodiment, the matrix multiplication operation of the memory-block-based data processing method according to the example embodiment may involve a first matrix × a second matrix × a third matrix. The matrix multiplication operation of the first matrix × the second matrix has been referred to... Figure 1 and Figure 2 The process has already been described and will not be repeated here. The result of multiplying the first matrix by the second matrix can be called the matrix multiplication result, and the principle of matrix multiplication between the first and second matrices can be applied to the principle of matrix multiplication between the matrix multiplication result and the third matrix. Therefore, the matrix multiplication operation between the matrix multiplication result and the third matrix will be briefly described below.
[0058] Reference Figure 3 In operation S310, multiple third partition matrices generated by partitioning the third matrix are read from the global storage space, and these matrices are written to the local shared storage space. The result of matrix multiplication of the first and second matrices can be multiplied with the third matrix.
[0059] In a non-limiting example, the first matrix is the query (e.g., but not limited to a one-dimensional query vector) in the attention block, the second matrix is the transpose of the key (e.g., but not limited to a one-dimensional key vector) corresponding to the query in the attention block, and the third matrix is the value (e.g., but not limited to a one-dimensional value vector) corresponding to the query and the key in the attention block. The attention block can be an attention block in various neural network models (e.g., Transformer-based neural network models). For example, Figure 1 The operation of reading multiple first partition matrices and multiple second partition matrices generated by partitioning the first matrix and the second matrix respectively from the global storage space may include: reading multiple first partition matrices generated by partitioning the one-dimensional query vector from the global storage space in the outer loop, and reading multiple second partition matrices generated by partitioning the transpose matrix of the one-dimensional key vector from the global storage space in the inner loop. Furthermore, Figure 3The operation of reading multiple third partition matrices generated by partitioning the third matrix from global storage includes: within the inner loop, reading multiple third partition matrices generated by partitioning the one-dimensional value vector from global storage. In one example, the outer and inner loops represent that the three matrices in the attention block need to be multiplied twice, requiring the three matrix data to be divided into blocks and calculated separately at different memory levels. The reading and calculation of the Key and Value matrices are performed in the inner loop, while the reading and calculation of the Query and output matrices are performed in the outer loop. Because of the use of data partitioning techniques at different levels, the three matrices Query, Key, and Value in the attention block can reuse data more efficiently in each memory level. For the Key and Value matrices, the total number of loop iterations in the inner loop within a workgroup can also be reduced, while in a single loop, vector computation and data rearrangement techniques can be better combined to perform computation on small blocks of data, thereby improving parallelism.
[0060] The division in operation S320 can be compared with the reference. Figure 1 The partitioning in operation S120 is similar in principle. Therefore, to avoid redundancy, the partitioning will not be described in more detail here, but will instead refer to... Figure 1 The description of the partitioning in operation S120 can also be applied to the partitioning in operation S320.
[0061] In operation S320, multiple third sub-partition matrices generated by partitioning multiple third partition matrices are read from the local shared memory space, and the multiple third sub-partition matrices are written to the thread-private memory space.
[0062] The division in operation S320 can be compared with the reference. Figure 2 The partitioning in operation S220 is similar in principle. Therefore, to avoid redundancy, the partitioning will not be described in more detail here, but will instead refer to... Figure 2 The description of the partitioning in operation S220 can also be applied to the partitioning in operation S320.
[0063] In operation S330, multiple third sub-partition matrices are read from the thread-private storage space, and matrix multiplication between the second result matrix and the third matrix is performed by performing corresponding matrix multiplication operations on multiple first result matrices and multiple third sub-partition matrices. The multiple first result matrices are the result matrices generated by performing corresponding matrix multiplication operations on multiple first sub-partition matrices and multiple second sub-partition matrices, and the second result matrix is the result matrix of the matrix multiplication operation between the first matrix and the second matrix.
[0064] With reference Figure 1 and Figure 2The principle described is similar. According to the example embodiment, the memory-based data processing method can reduce the number of reads from the global storage space and the number of reads from the local shared storage space by applying two levels of memory partitioning (i.e., partitioning from the global storage space to the local shared storage space and partitioning from the local shared storage space to the thread-private storage space).
[0065] In one example, the Softmax function (or flexible maximum function) can be applied to multiple first result matrices before performing the corresponding matrix multiplication operations on multiple first result matrices and multiple third subpartition matrices.
[0066] Furthermore, the result matrix of the matrix multiplication operation between the second and third result matrices can also be written to the global storage space in the outer loop.
[0067] Figure 4 This is a diagram illustrating a memory-block-based data processing method according to an example embodiment.
[0068] The memory-block-based data processing method according to the example embodiment provides block partitioning from global storage space to local shared storage space and from local shared storage space to thread-private storage space. These two types of partitioning constitute a two-dimensional memory partitioning technique. The memory-block-based data processing method according to the example embodiment can combine this two-dimensional memory partitioning technique with FlashAttention to form a two-dimensional partitioning technique-optimized attention implementation method, 2DTilingAttention. The attention implementation method 2DTilingAttention can be applied to different types of graphics processing units (GPUs) or artificial intelligence (AI) accelerators with multi-level storage.
[0069] Reference Figure 4 The memory-based data processing method according to the example embodiment may include G2LTiling and L2PTiling. G2LTiling and L2PTiling refer to the block application from global storage space to local shared storage space and the block application from local shared storage space to thread-private storage space, respectively.
[0070] exist Figure 4 In this context, the key K can be the key matrix in the attention block, the value V can be the value matrix in the attention block, and the query Q can be the query matrix in the attention block. Figure 4In the example, the key K, value V, and query Q are shown as one-dimensional matrices (i.e., at least one of the number of rows and the number of columns is 1). However, the key K, value V, and query Q are not limited to this and can be matrices of any size. The transpose of the key K matrix can have d × Nkv columns, where d and Nkv can be positive integers. The query Q can have Nq × d rows, where Nq and d can be positive integers. The value V can have Nkv × d rows, where Nkv and d can be positive integers.
[0071] In the memory-based data processing method according to the example embodiment, the storage space may include global storage space, local shared storage space and thread-private storage space by applying block application from global storage space to local shared storage space and block application from local shared storage space to thread-private storage space.
[0072] In one example embodiment, query Q may correspond to a first matrix, the transpose of key K may correspond to a second matrix, and value V may correspond to a third matrix. Referring to the above... Figures 1 to 3 Similar to the first, second, and third matrices described, the query Q, the transpose of key K (or key K), and the value V are stored in global storage. The transpose of key K and value V can be read in the inner loop, and the query Q can be read in the outer loop. Furthermore, the query Q, the transpose of key K, and value V can be partitioned into multiple first partition matrices, multiple second partition matrices, and multiple third partition matrices in a manner similar to partitioning the first, second, and third matrices. For example, the number of multiple first partition matrices, multiple second partition matrices, and multiple third partition matrices can be d. The size of each first partition matrix, each second partition matrix, and each third partition matrix can be Np×1, 1×Nkv, and Nkv×1, respectively. Multiple first partition matrices, multiple second partition matrices, and multiple third partition matrices can be stored in local shared storage (e.g., local memory). Multiple first partition matrices, multiple second partition matrices, and multiple third partition matrices can be further partitioned into multiple first subpartition matrices, multiple second subpartition matrices, and multiple third subpartition matrices. Multiple first sub-partition matrices, multiple second sub-partition matrices, and multiple third sub-partition matrices can be read from local shared memory space into thread-private memory space (e.g., registers). Then, matrix multiplication can be performed on the registers based on the corresponding first sub-partition matrices, the corresponding second sub-partition matrices, and the corresponding third sub-partition matrices, and the final calculation result (e.g., softmax(QK)) can be obtained based on the result of the matrix multiplication. TIn the private storage space, a single thread processes multiple results (i.e., multiple workgroups). Data arrangement allows multiple workgroups to share the same input, reducing memory reads in the private storage space. Computation results can be output to global storage in the outer loop.
[0073] Because of the use of data partitioning techniques at different levels, the three matrices of the attention block—query Q, key K, and value V—can reuse data more effectively in each level of memory. This also reduces the total number of loop iterations within the inner loop of a workgroup for both key K and value V, while allowing for better integration of vector computation and data rearrangement techniques for computing small data blocks within a single loop, thereby improving parallelism.
[0074] According to the memory-block-based data processing method in the example embodiment, more data is first read into the local shared memory in each thread, and then the data in the local shared memory is divided into blocks. More data is loaded into the private registers of each thread. Combined with vector operations and data rearrangement, the calculation of small matrices is accelerated.
[0075] According to the memory-block-based data processing method of the example embodiment, for a query Q, key K, and value V, the number of loops in a workgroup is reduced because more data is copied to the local shared memory and copied from the local shared memory to the private storage in vector form in each thread.
[0076] According to the memory-block-based data processing method in the example embodiment, the performance of small matrix multiplication is improved. Firstly, by leveraging the inherent characteristics of tiling different memory levels, more data is reused locally, thereby reducing read / write operations from higher-level memory. Secondly, by applying block-based storage from locally shared memory to thread-private memory within each thread, the computation of small matrices can have richer data retrieval methods and better utilize vector operations.
[0077] According to the memory-block-based data processing method in the example embodiment, in the calculation of the fused attention block, since the calculation of Softmax involves conditional or comparison instructions, and because the addition of the two-level block technology allows for the processing of more data under a single conditional instruction in a single thread, and multiple comparison instructions can better utilize data parallelism, the total number of conditional instructions is reduced and the execution of the total comparison instructions is accelerated.
[0078] According to the memory-block-based data processing method in the example embodiment, the calculation of small matrix multiplications in the attention block kernel is improved by re-accessing local shared memory, allowing data to make greater use of vector operations to operate on continuous data, thereby improving computational performance.
[0079] Figure 5 This is a block diagram illustrating a memory-block-based data processing system according to an example embodiment.
[0080] Reference Figure 5 The memory-block-based data processing system 500 may include a storage system 510 and a processor 520. The memory-block-based data processing system 500 may be a reference... Figures 1 to 4 At least one of the data processing systems described herein is used to perform a method for memory-block-based data processing.
[0081] The memory space of storage system 510 includes a global storage space and a local shared storage space. The global storage space is shared globally and configured to store a first matrix and a second matrix in the neural network model to be subjected to matrix multiplication operations. The local shared storage space is shared locally by multiple threads. For example, the global storage space may correspond to a first-level memory as hardware (e.g., global memory), and the local shared storage space may correspond to a second-level memory as hardware (e.g., local memory).
[0082] Processor 520 divides the first matrix into multiple first partition matrices and the second matrix into multiple second partition matrices. Processor 520 reads the multiple first partition matrices and multiple second partition matrices generated by dividing the first matrix and the second matrix respectively from global storage space, and writes the multiple first partition matrices and multiple second partition matrices into local shared storage space.
[0083] The processor 520 reads multiple first partition matrices and multiple second partition matrices from the local shared memory space, and performs matrix multiplication operations between the first matrix and the second matrix by performing corresponding matrix multiplication operations on the multiple first partition matrices and multiple second partition matrices.
[0084] In the computation of a first matrix of size m×k (e.g., matrix A) multiplied by a second matrix of size k×n (e.g., matrix B) to obtain a matrix C of size m×n, if only global storage is used, in the comparative example, computing matrix C requires 2×m×n×k read operations from global storage and m×n write operations to global storage. Compared to the 2×m×n×k global read operations in the comparative example, by using the conversion and use of the matrix stored in global storage to the partitioned matrix stored in local shared storage, the number of global read operations in the memory-block-based data processing system according to the example embodiment becomes 1 / 2×(1 / lm+1 / ln) times that of the comparative example. Therefore, the number of global reads is reduced, the overall execution efficiency is improved, and the degradation rate of global storage as hardware is reduced.
[0085] In addition, storage system 510 may also include thread-private storage space. The thread-private storage space is private to one of a plurality of threads. For example, the thread-private storage space may correspond to third-level memory as hardware (e.g., registers, by way of example only).
[0086] Processor 520 divides the first partition matrix into multiple first sub-partition matrices and the second partition matrix into multiple second sub-partition matrices. Processor 520 reads the multiple first sub-partition matrices and multiple second sub-partition matrices generated by partitioning the first partition matrix and the second partition matrix respectively from the local shared memory space, and writes the multiple first sub-partition matrices and multiple second sub-partition matrices into the thread-private memory space.
[0087] As described above, the local shared memory space can be configured to store the first partition matrix and the second partition matrix. The thread-private memory space can be configured to store multiple first sub-partition matrices generated by partitioning the first partition matrix and multiple second sub-partition matrices generated by partitioning the second partition matrix. In one example, processor 520 can read a first portion of data from the first partition matrix from the local shared memory space as one of the multiple first sub-partition matrices and write that sub-partition matrix into the thread-private memory space. Furthermore, processor 520 can read a second portion of data from the first partition matrix from the local shared memory space as another of the multiple first sub-partition matrices and write that other sub-partition matrix into the thread-private memory space. Similarly, processor 120 can read other portions of data from the local shared memory space as other sub-partition matrices among the multiple first sub-partition matrices and write these other sub-partition matrices into the thread-private memory space. Processor 520 can process the second partition matrix in a similar manner to processing the first partition matrix.
[0088] The processor 520 can read multiple first sub-partition matrices and multiple second sub-partition matrices from the thread-private storage space, and perform matrix multiplication operations between the first matrix and the second matrix by performing corresponding matrix multiplication operations on the multiple first sub-partition matrices and multiple second sub-partition matrices.
[0089] Similar to the reduction in global read counts described above, the memory-block-based data processing system according to the example embodiment reduces the number of shared reads in shared read operations, improves overall execution efficiency, and slows down the degradation rate of the local shared memory space as hardware by using the conversion and use of a matrix stored in the local shared memory space to a partitioned matrix stored in the thread-private memory space.
[0090] The memory-block-based data processing system according to the example embodiment can handle tasks corresponding to computationally intensive applications by employing a hardware architecture corresponding to multiple levels of memory (e.g., first-level memory, second-level memory, and third-level memory), thus leveraging the read performance advantages of different memory levels. Therefore, it can reduce the number of reads on relatively high-level memory, thereby reducing the degradation rate of relatively high-level memory and extending hardware lifespan.
[0091] Processor 520 can be any type of processor (e.g., graphics processor, central processing unit, etc.). Processor 520 can be an execution reference. Figures 1 to 4 At least one of the processors described in the document is based on a memory-blocking data processing method. Therefore, to avoid redundancy, repeated descriptions will be omitted.
[0092] Storage system 510 can be configured to store data. For example, storage system 510 may include one or more volatile memory devices (e.g., read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory), one or more non-volatile memory devices (e.g., flash memory devices), and one or more memory controllers. Storage system 510 may include any similar memory. The storage space of storage system 510 may include global storage space, local shared storage space, and / or thread-private storage space. The storage space of storage system 510 may be a reference... Figures 1 to 4 At least one description of the storage space is required. Therefore, to avoid redundancy, duplicate descriptions will be omitted.
[0093] The foregoing description is illustrative of exemplary embodiments and should not be construed as limiting them. Although some exemplary embodiments have been described, those skilled in the art will readily understand that many modifications can be made to the exemplary embodiments without substantially departing from the novel teachings and advantages of the exemplary embodiments of the inventive concept. Therefore, all such modifications are intended to be included within the scope of the exemplary embodiments of the inventive concept as defined in the claims. It will therefore be understood that the foregoing description is illustrative of various exemplary embodiments and should not be construed as limiting to the specific exemplary embodiments disclosed, and modifications to the disclosed exemplary embodiments, as well as other exemplary embodiments, are intended to be included within the scope of the appended claims.
Claims
1. A data processing method based on memory partitioning, the data processing method comprising: storing, in a global storage space in a memory space of a storage system, a first matrix and a second matrix in a neural network model to be subjected to a matrix multiplication operation, wherein the global storage space is globally shared; reading a plurality of first partitioned matrices and a plurality of second partitioned matrices from the global storage space and writing the plurality of first partitioned matrices and the plurality of second partitioned matrices to a locally shared storage space in the memory space of the storage system, wherein the plurality of first partitioned matrices are generated by partitioning the first matrix and the plurality of second partitioned matrices are generated by partitioning the second matrix, and wherein the locally shared storage space is locally shared by a plurality of threads; reading the plurality of first partitioned matrices and the plurality of second partitioned matrices from the locally shared storage space; and performing the matrix multiplication operation between the first matrix and the second matrix by performing respective matrix multiplication operations on the plurality of first partitioned matrices and the plurality of second partitioned matrices.
2. The data processing method of claim 1, wherein, The memory space of the storage system further comprises a thread private storage space, wherein the thread private storage space is exclusively accessed by one thread of the plurality of threads.
3. The data processing method of claim 2, wherein, The step of reading the plurality of first partitioned matrices and the plurality of second partitioned matrices from the locally shared storage space comprises reading a plurality of first sub-partitioned matrices and a plurality of second sub-partitioned matrices from the locally shared storage space, wherein the plurality of first sub-partitioned matrices and the plurality of second sub-partitioned matrices are generated by partitioning the plurality of first partitioned matrices and the plurality of second partitioned matrices, respectively.
4. The data processing method of claim 3, wherein, The step of performing the matrix multiplication operation between the first matrix and the second matrix comprises: writing the plurality of first sub-partitioned matrices and the plurality of second sub-partitioned matrices to the thread private storage space; reading the plurality of first sub-partitioned matrices and the plurality of second sub-partitioned matrices from the thread private storage space; and performing the matrix multiplication operation between the first matrix and the second matrix by performing respective matrix multiplication operations on the plurality of first sub-partitioned matrices and the plurality of second sub-partitioned matrices.
5. The data processing method of claim 3, wherein, The data processing method further comprises: reading a plurality of third partitioned matrices generated by partitioning a third matrix from the global storage space and writing the plurality of third partitioned matrices to the locally shared storage space, wherein a result of the matrix multiplication operation of the first matrix and the second matrix is capable of being subjected to a matrix multiplication operation with the third matrix; reading a plurality of third sub-partitioned matrices generated by partitioning the plurality of third partitioned matrices from the locally shared storage space and writing the plurality of third sub-partitioned matrices to the thread private storage space; and reading the plurality of third sub-partitioned matrices from the thread private storage space and performing a matrix multiplication operation between a second result matrix and the third matrix by performing respective matrix multiplication operations on a plurality of first result matrices and the plurality of third sub-partitioned matrices, wherein the plurality of first result matrices are result matrices generated by performing respective matrix multiplication operations on the plurality of first sub-partitioned matrices and the plurality of second sub-partitioned matrices, and the second result matrix is a result matrix of the matrix multiplication operation between the first matrix and the second matrix.
6. The data processing method of claim 5, further comprising: The Softmax function is applied to the plurality of first result matrices before performing the respective matrix multiplication operations on the plurality of first result matrices and the plurality of third sub-partition matrices.
7. The data processing method of claim 5, wherein, The first matrix is a one-dimensional query vector in an attention block, the second matrix is a transpose matrix of a one-dimensional key vector corresponding to the one-dimensional query vector in the attention block, and the third matrix is a one-dimensional value vector corresponding to the one-dimensional query vector and the one-dimensional key vector in the attention block.
8. The data processing method of claim 7, wherein, The step of reading the plurality of first partition matrices and the plurality of second partition matrices generated by partitioning the first matrix and the second matrix from the global memory space includes: reading the plurality of first partition matrices generated by partitioning the one-dimensional query vector from the global memory space in an outer loop, and reading the plurality of second partition matrices generated by partitioning the transpose matrix of the one-dimensional key vector from the global memory space in an inner loop, and The step of reading the plurality of third partition matrices generated by partitioning the third matrix from the global memory space includes: reading the plurality of third partition matrices generated by partitioning the one-dimensional value vector from the global memory space in the inner loop.
9. The data processing method of claim 8, wherein, The data processing method further includes: writing a result matrix of a matrix multiplication operation between the second result matrix and the third matrix to the global memory space in the outer loop.
10. The data processing method of claim 1, wherein, The first matrix and the second matrix are matrices in an attention block in a Transformer-based neural network model, and The Transformer-based neural network model is configured to perform one of speech recognition, natural language processing, and video generation.
11. A data processing system based on memory partitioning, the data processing system comprising: a storage system configured to store data; and a processor configured to: store a first matrix and a second matrix in a neural network model to be executed in a matrix multiplication operation in a global memory space in a memory space of the storage system, wherein the global memory space is globally shared; read a plurality of first partition matrices and a plurality of second partition matrices from the global memory space, and write the plurality of first partition matrices and the plurality of second partition matrices to a locally shared memory space in the memory space of the storage system, wherein the plurality of first partition matrices are generated by partitioning the first matrix, and the plurality of second partition matrices are generated by partitioning the second matrix, wherein the locally shared memory space is locally shared by a plurality of threads; and read the plurality of first partition matrices and the plurality of second partition matrices from the locally shared memory space, and perform a matrix multiplication operation between the first matrix and the second matrix by performing respective matrix multiplication operations on the plurality of first partition matrices and the plurality of second partition matrices.