A multi-functional DMA for HPC / AI accelerators

Abstract

The application discloses a multifunctional DMA for HPC / AI accelerator, which comprises a channel, a read / write request distribution module and a register array, the channel is connected with the read request distribution module and the write request distribution module respectively, the channel is connected with a source storage unit, the read request distribution module and the write request distribution module are connected with a target storage unit, the channel comprises a normal transmission module, a transpose transmission module and a splicing transmission module, the normal transmission module directly outputs the transmitted data without processing, the transpose transmission module is used for outputting after transpose operation, and the splicing transmission module is used for outputting after splicing operation. The application integrates the normal transmission, the transpose transmission and the splicing transmission functions in the DMA, ensures that different data structures can be correctly spliced, matrix transposition can be processed, the effective data transmission efficiency in the HPC / AI accelerator is improved, and the hardware computing resource utilization rate is increased.

Description

Technical Field

[0001] This invention relates to data storage technology, and more specifically to a multifunctional DMA for HPC / AI accelerators. Background Technology

[0002] Since the advent of deep learning, people's lives have undergone tremendous changes, and the idea of ​​developing a dedicated chip for neural networks has long been deeply ingrained. In 2013, Google's various products and services, such as Google Image Search, Google Photos, Google Cloud Vision API, and Google Translate, all required the use of deep neural networks. With the sheer scale of these applications, Google realized that the rapidly growing computing demands of its millions of servers running day and night necessitated doubling the number of data centers. However, both in terms of cost and computing power, internal centers could no longer be sustained simply by relying on GPUs and CPUs, which spurred the development of AI (Artificial Intelligence) accelerators. High-performance computing (HPC) applications (including high-energy physics research, energy exploration, national economic forecasting, and other high-precision fields) require massive computations; therefore, HPC equipment possesses extremely high computing power, and HPC clusters are often referred to as supercomputers. After decades of continuous development, HPC hardware and software have matured. As a core component of HPC applications, Generalized Matrix Multiplication (GEMM) is also crucial in the field of artificial intelligence. This is because the fundamental structures of deep learning models, such as Multilayer Perceptrons (MLPs), CNNs, and Transformers, all involve GEMM computation. Both HPC and typical AI applications utilize GEMM, albeit at different levels of precision. Accelerators supporting multi-precision GEMM computation have become a mainstream technology. To accelerate AI model training using high-performance HPC, AI accelerators need to be integrated into HPC chips. Therefore, how to process data of varying precision to maximize hardware resource utilization has become a problem that needs to be solved.

[0003] DMA (Direct Memory Access) can refer to both the computer technology of direct data access between memory and peripherals, and the hardware module that implements this technology. DMA transfers copy data from one address space to another, providing high-speed data transfer between peripherals and memory, or between memory devices. It is one of the core components of an accelerator. Its structure is as follows: Figure 5As shown. To support high-precision scientific computing, HPC applications typically use double-precision data, while AI applications often use single-precision, half-precision, and mixed-precision data. Both have a maximum data granularity of 64 bits. To support both HPC and AI applications simultaneously, DMA, as a core component, needs to support a minimum data width of 64 bits. However, for single-precision, half-precision, and mixed-precision data, the data width is 32 or 16 bits, resulting in an effective data width less than 64 bits. If this data is to be stored in a 64-bit address, the remaining bits must be filled with 0s, leading to underutilization of storage space and insufficient utilization of subsequent computing resources.

[0004] In the HPC field, there are numerous GEMM operations. In deep learning, convolution and GEMM are the most common operations. Convolution operations can be converted into GEMM operations using `img2col`, making GEMM the most fundamental operation in deep learning. In some layers of deep learning, in addition to the basic Y = AB + C operation, it is often necessary to perform the operation Y = AB. T +C or Y = A T B+C requires matrix transposition. In AI training, single-precision or half-precision data is commonly used for model training. Therefore, a multi-functional data processing method is needed to handle data concatenation and transposition, fully utilizing storage space, reducing subsequent computations, increasing parallelism, and thus improving hardware computing resource utilization. Deep learning models contain many different data structures, such as matrices, feature maps, and weights. Matrices can be stored using two-dimensional arrays, with two dimensions representing matrix height (H) and matrix width (W). Graphs and weights can be stored using four-dimensional arrays, with four dimensions representing batch size (N), feature map height (H), feature map width (W), and feature map channels (C). Different data structures have different parameters, but the parameters available for DMA are limited. How to construct a multi-functional data processing method for DMA that can handle different data structures with only a few parameters becomes a problem to be solved. To accelerate AI model training using high-performance HPC, AI accelerators need to be integrated into HPC chips. For standalone AI accelerators or HPC accelerators that integrate AI accelerators, how to process data of different precisions and maximize the utilization of hardware resources has become a problem that needs to be solved. Summary of the Invention

[0005] The technical problem to be solved by this invention is to provide a multi-functional DMA for HPC / AI accelerators, addressing the aforementioned problems of the prior art. This invention integrates normal data transmission, transpose transmission, and splicing transmission functions into the DMA, and can achieve efficient data splicing based on DMA according to the different precision and data structures of the data to be transferred. While ensuring the correct splicing of different data structures and processing matrix transposes, it improves the effective data transmission efficiency in HPC / AI accelerators and increases the utilization rate of hardware computing resources.

[0006] Different data structures have different parameters, but the parameters available for DMA are limited. Therefore, constructing a multi-functional data processing method for DMA that can handle different data structures with only a few parameters is a problem that needs to be solved. HPC applications typically use double-precision data, while AI applications often use single-precision, half-precision, and mixed-precision data for computation. Both have a maximum data granularity of 64 bits. To support both HPC and AI applications simultaneously, DMA, as a core component, needs to support a maximum data width of 64 bits. However, for single-precision, half-precision, and mixed-precision data, the data width is 32 bits or 16 bits, resulting in an effective data width of less than 64 bits. If DMA does not perform any processing on the transmitted data, it will inevitably lead to insufficient utilization of hardware computing resources and reduced chip performance. Therefore, how to provide a multi-functional DMA for HPC / AI accelerators that supports the aforementioned data concatenation / transpose methods is a problem that needs to be solved.

[0007] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0008] A multi-functional DMA for HPC / AI accelerators includes a channel pipe, a read request distribution module, a write request distribution module, and a register array for transposing and concatenating data. The channel pipe is interconnected with the read request distribution module and the write request distribution module, respectively. The channel pipe is used to connect to external source memory, and the read request distribution module and the write request distribution module are used to connect to external destination memory. The channel pipe includes a normal transmission module, a transpose transmission module, and a concatenation transmission module. The normal transmission module is used to output the data from the source memory directly without processing. The transpose transmission module is used to transpose the data from the source memory through the register array before outputting it. The concatenation transmission module is used to concatenate the data from the source memory through the register array before outputting it.

[0009] Optionally, the multi-functional DMA selects the operating state of the normal transmission module, transpose transmission module, and splicing transmission module in the channel pipe according to the mode parameters carried by the instructions issued by the control logic: when the mode parameter is normal mode, the normal transmission module in the channel pipe is selected to output the data of the source memory directly without processing; when the mode parameter is transpose mode, the transpose transmission module in the channel pipe is selected to transpose the data of the source memory before outputting; when the mode parameter is splicing mode, the splicing transmission module in the channel pipe is selected to splice the data of the source memory before outputting.

[0010] Optionally, the transpose operation refers to reading the data into the multi-function DMA in column-first order, then storing any nth row of the original data block as the nth column through the multi-function DMA, and writing it out in row-first order during output.

[0011] Optionally, the splicing operation includes splicing single-precision data for general matrix multiplication, splicing half-precision data for general matrix multiplication, splicing single-precision data for deep learning models, and splicing half-precision data for deep learning models.

[0012] Optionally, the concatenation of single-precision data for general matrix multiplication includes concatenating single-precision data at the same position in two input single-precision matrices matrix0 and matrix1 of size H×W into 64-bit data. The data of single-precision matrix0 is stored in the lower 32 bits, and the data of single-precision matrix1 is stored in the higher 32 bits, where H and W are the height and width of the matrix, respectively.

[0013] Optionally, the concatenation of half-precision data for general matrix multiplication includes concatenating the half-precision data at the same position of the four input half-precision matrices matrix0, matrix1, matrix2, and matrix3 of size H×W into 64-bit data, wherein the data of half-precision matrix matrix0 is stored in the lower 16 bits, and then the data of half-precision matrix matrix1, half-precision matrix matrix2, and half-precision matrix matrix3 are stored in the higher 16 bits in ascending order, where H and W are the height and width of the matrix, respectively.

[0014] Optionally, the concatenation of single-precision data for deep learning models includes concatenating the single-precision data at the same positions of two input single-precision matrices data0 and data1 of size N×C×H×W into 64-bit data. The data of single-precision matrix data0 is stored in the lower 32 bits, and the data of single-precision matrix data1 is stored in the higher 32 bits, where N, C, H, and W are the batch size, number of channels, height, and width of the single-precision matrix, respectively.

[0015] Optionally, the concatenation of half-precision data for the deep learning model includes concatenating the half-precision data at the same position of the four input half-precision matrices data0, data1, data2, and data3 of size N×C×H×W into 64-bit data. The data of half-precision matrix data0 is stored in the lower 16 bits, and then the data of half-precision matrices data1, data2, and data3 are stored in the upper 16 bits in ascending order. N, C, H, and W are the batch size, number of channels, height, and width of the half-precision matrix, respectively.

[0016] Optionally, the external source-end memory connected to the channel pipe includes n source-end memory units, from source-end memory unit 1 to source-end memory unit n.

[0017] Optionally, the external destination memory connected to the read request distribution module and the write request distribution module includes n destination memory units from destination memory unit 1 to destination memory unit n.

[0018] Compared with existing technologies, the present invention has the following main advantages: The present invention includes a channel pipe, a read request distribution module, and a write request distribution module. The channel pipe is interconnected with the read request distribution module and the write request distribution module, respectively. The channel pipe is used to connect to an external source memory, and the read request distribution module and the write request distribution module are used to connect to an external destination memory. The channel pipe includes a normal transmission module, a transpose transmission module, and a splicing transmission module. The normal transmission module is used to output the data from the source memory directly without processing. The transpose transmission module is used to transpose the data from the source memory before outputting it. The splicing transmission module is used to splice the data from the source memory before outputting it. The present invention integrates normal data transmission, transpose transmission, and splicing transmission functions in DMA, enabling data splicing and transposition. This allows data of different precisions and structures to be efficiently spliced ​​or transposed during DMA transmission, ensuring the correct splicing of different data structures and processing of matrix transposes, while improving the effective data transmission efficiency in HPC / AI accelerators and increasing the utilization of hardware computing resources. Attached Figure Description

[0019] Figure 1 This is a schematic diagram of the structure of the multi-functional DMA for HPC / AI accelerators in Embodiment 1 of the present invention.

[0020] Figure 2 This is a schematic diagram of the channel pipe working mode selection based on mode parameters in Embodiment 1 of the present invention.

[0021] Figure 3 This is an example of step one of the splicing in Embodiment 1 of the present invention.

[0022] Figure 4 This is an example of step two of splicing in Embodiment 1 of the present invention.

[0023] Figure 5 This is an example of step three of splicing in Embodiment 1 of the present invention.

[0024] Figure 6 This is an example of transposition in Embodiment 1 of the present invention. Detailed Implementation

[0025] Example 1:

[0026] like Figure 1 As shown, this embodiment of the multi-functional DMA for HPC / AI accelerators includes a channel pipe, a read request distribution module, a write request distribution module, and a register array for transposing and concatenating data. The channel pipe is interconnected with both the read request distribution module and the write request distribution module, connecting to external source memory. The read request distribution module and the write request distribution module connect to external destination memory. The channel pipe includes a normal transmission module, a transpose transmission module, and a concatenation transmission module. The normal transmission module outputs data from the source memory directly without processing. The transpose transmission module transposes the data from the source memory using the register array before outputting it. The concatenation transmission module concatenates the data from the source memory using the register array before outputting it. The read / write request distribution module distributes read / write requests to the respective destination memory banks.

[0027] like Figure 2As shown, in this embodiment, the multi-functional DMA selects the operating state of the normal transmission module, transpose transmission module, and splicing transmission module in the channel pipe according to the mode parameters carried by the instructions issued by the control logic: When the mode parameter is normal mode, the normal transmission module in the channel pipe outputs the data from the source memory directly without processing; when the mode parameter is transpose mode, the transpose transmission module in the channel pipe transposes the data from the source memory before outputting it; when the mode parameter is splicing mode, the splicing transmission module in the channel pipe splices the data from the source memory before outputting it. The control logic is the controller of the multi-functional DMA, used to control the operating state of the multi-functional DMA. The parameters it carries can be specified according to actual needs. For example, as an optional implementation, the parameters in this embodiment may include: mode, source address, destination address, number of matrix columns, number of matrix rows, source row offset, input channel, number of input images, adjacent channel offset, adjacent image offset, and floating-point type. The modes are divided into normal mode, transpose mode, and splicing mode.

[0028] The transpose operation refers to dividing the data into blocks, reading them into a multi-functional DMA in column-major order, storing any nth row of the original data block as the nth column via the multi-functional DMA, and then writing it out in row-major order during output. For example... Figure 6 As shown, data from the source memory is written to the register array of the multi-function DMA in rows with a 64-bit granularity. The first row ABCD is stored as the first column (destination memory) in the multi-function DMA, the second row EFGH is stored as the second column in the multi-function DMA, and so on.

[0029] In this embodiment, the splicing operation includes splicing single-precision data for general matrix multiplication, splicing half-precision data for general matrix multiplication, splicing single-precision data for deep learning models, and splicing half-precision data for deep learning models. As an optional implementation, this embodiment further subdivides the splicing mode into AI splicing mode and non-AI splicing mode. The AI ​​splicing mode is used to perform splicing of single-precision data for deep learning models and splicing of half-precision data for deep learning models, while the non-AI splicing mode is used to perform splicing of single-precision data for general matrix multiplication and splicing of half-precision data for general matrix multiplication. The splicing method within the AI ​​splicing mode and the non-AI splicing mode can be automatically selected based on the data type being spliced.

[0030] For general matrix multiplication (GEMM) concatenation in non-AI concatenation mode, since the computation of elements from different matrices has no data dependency, elements from different matrices can be concatenated to meet the needs of multi-precision parallel processing. For single-precision and half-precision data, we propose different concatenation methods:

[0031] A. The concatenation of single-precision data for general matrix multiplication involves taking two input single-precision matrices matrix0 and matrix1 of size H×W, concatenating the single-precision data at the same position in matrix0 and matrix1 into 64-bit data, with the data of single-precision matrix0 stored in the lower 32 bits and the data of single-precision matrix1 stored in the higher 32 bits, where H and W are the height and width of the matrix, respectively.

[0032] B. The concatenation of half-precision data for general matrix multiplication involves concatenating the half-precision data at the same positions of four input half-precision matrices matrix0, matrix1, matrix2, and matrix3 of size H×W into 64 bits of data. The data of half-precision matrix matrix0 is stored in the lower 16 bits. Then, the data of half-precision matrix matrix1, half-precision matrix matrix2, and half-precision matrix matrix3 are stored in the upper 16 bits in ascending order, where H and W are the height and width of the matrix, respectively.

[0033] For AI-based data stitching in deep learning models, in convolutional neural networks (CNNs), data is divided into graphs and weights. Since graphs and weights have the same structure, we collectively refer to this as data stitching. In AI applications, the most common data formats are NCHW and NHWC. This embodiment uses the NCHW format, where NCHW corresponds to the number of convolutional kernels (batch size), the number of graph channels, the graph height, and the graph width. CNNs are multi-layered network structures. For graph data, only the first layer needs to be stitched; for weight data, the weights of each layer need to be stitched. This embodiment proposes different stitching methods for single-precision and half-precision data:

[0034] C. The concatenation of single-precision data for deep learning models involves concatenating the single-precision data at the same positions of two input single-precision matrices, data0 and data1, of size N×C×H×W, into 64-bit data. The data of single-precision matrix data0 is stored in the lower 32 bits, and the data of single-precision matrix data1 is stored in the higher 32 bits. Here, N, C, H, and W are the batch size, number of channels, height, and width of the single-precision matrix, respectively.

[0035] D. The concatenation of half-precision data for deep learning models involves taking four input half-precision matrices data0, data1, data2, and data3 of size N×C×H×W, concatenating the half-precision data at the same positions of the matrices data0, data1, data2, and data3 into 64-bit data. The data of half-precision matrix data0 is stored in the lower 16 bits. Then, the data of half-precision matrices data1, data2, and data3 are stored in the higher 16 bits in ascending order. Here, N, C, H, and W are the batch size, number of channels, height, and width of the half-precision matrix, respectively.

[0036] Furthermore, in AI applications, the training of artificial intelligence learning models involves forward propagation and backward propagation stages (referred to as AI forward and AI backward propagation, respectively). Forward propagation starts with the input, using the result of the previous layer as the input of the next layer, ultimately obtaining the loss value. Backward propagation proceeds from the back to the front, calculating derivatives layer by layer to update the model weights and achieve the learning effect. Both forward and backward propagation involve a distinction between the first layer and intermediate layers. The first layer receives the input, while the intermediate layers process the information and calculate the results. Therefore, the hierarchical structure and data types of the multi-functional DMA support for HPC / AI accelerators provided in this embodiment are shown in Table 1.

[0037] Table 1: Data types supported at different levels.

[0038]

[0039]

[0040] BF16 is a relatively new floating-point number type proposed by Google, and its bit width is the same as FP16.

[0041] See Figure 1 In this embodiment, the external source-end memory devices connected to the channel pipe include source-end memory 1 to source-end memory n, totaling n source-end memory devices. See also Figure 1 In this embodiment, the external destination memory connected to the read request distribution module and the write request distribution module includes n destination memory units, from destination memory unit 1 to destination memory unit n. It should be noted that the number of destination memory units in the destination memory can be configured according to actual needs; it can be the same as or different from the number of source memory units in the source memory unit.

[0042] For ease of description, this embodiment assumes a transmission bit width of 256 bits and a register array size of 4×4, and provides a schematic diagram of transpose and splicing transmission. Figures 3-5As shown, assuming the matrices to be concatenated are single-precision matrices A and B, and the concatenated matrix is ​​C, both of which are 4×4 in size, the multi-functional DMA in this embodiment, when performing 32-bit single-precision data concatenation, first reads 8 elements of single-precision matrix A from the source end and stores them in the low-order bits of the first two rows of the register array, such as... Figure 3 As shown; then read 8 elements of the single-precision matrix B from the source and store them in the high-order bits of the first two rows of the register array, as follows. Figure 4 As shown; finally, the output is sent to the destination to complete the splicing, as follows. Figure 5 As shown. The concatenation of half-precision data is analogous to the concatenation of single-precision data, concatenating from the least significant bit to the most significant bit. For example... Figure 6 As shown, during transpose transmission, assuming the matrix to be transposed is the matrix C mentioned above, its size is 4×4 and its data width is 64 bits, the multi-functional DMA in this embodiment first reads the data into the register array row by row from the source end, and when outputting it to the destination end, it reads it out column by column to complete the transpose.

[0043] To verify the multi-functional DMA for HPC / AI accelerators in this embodiment, we first performed data concatenation and transposition on an Intel Core i5 10th generation CPU platform, and then performed data transfer using the normal DMA transfer mode. The time of the two processes was added together as the time base, and the time optimized by the multi-functional DMA for HPC / AI accelerators in this embodiment was compared. The results are shown in Tables 2 and 4.

[0044] Table 2: Test results of the Intel Core i5 10th Gen CPU platform:

[0045] 1024*768 3987 4982 768*768 2989 3976 768*512 2015 2984 512*512 1997 1990 512*256 1006 995 256*256 511 497

[0046] Table 3: Multifunctional DMA Test Results:

[0047] 1024*768 112.76 109.11 768*768 84.5 81.77 768*512 56.34 54.38 512*512 37.56 36.20 512*256 18.78 18.07 256*256 9.37 9.00

[0048] Table 4: Performance comparison of the two under different matrix sizes ((DMA time ÷ CPU time) × 100%)

[0049] 1024*768 2.8282％ 2.1901％ 768*768 2.827％ 2.0566％ 768*512 2.796％ 1.8224％ 512*512 1.8808％ 1.8191％ 512*256 1.8668％ 1.8161％ 256*256 1.8337％ 1.8109％

[0050] Referring to Tables 2 to 4, it can be seen that the multi-functional DMA for HPC / AI accelerators proposed in this embodiment can effectively reduce the time overhead of data transposition and splicing, while eliminating the need for data preprocessing, thus providing guidance for DMA design. In summary, the purpose of this multi-functional DMA for HPC / AI accelerators is to optimize data transfer efficiency for HPC chips supporting AI model computation, maximizing transfer efficiency and hardware resource utilization while ensuring computational accuracy. The following section will provide a more detailed description of this multi-functional DMA for HPC / AI accelerators using examples.

[0051] Example 2:

[0052] This embodiment is basically the same as Embodiment 1, with the main difference being that the transmission bit width of the multi-functional DMA in this embodiment is 512 bits, meaning it can transmit 8 64-bit data packets or 16 32-bit data packets at a time. To maximize the utilization of the multi-functional DMA's transmission bit width while maintaining high flexibility, an 8×8 register array with a bit width of 64 bits is used for data transposition and concatenation. It should be noted that the transposition transmission of the multi-functional DMA in this embodiment only supports 64-bit word granularity. The basic matrix supported by 64-bit word granularity transmission is 8×8 (8 rows and 8 columns). This means that the rows and columns of the matrix supported by matrix transposition transmission must both be integer multiples of 8. The matrix size at 64-bit word granularity is 8n×8k (k and n are integers greater than zero).

[0053] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0054] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multi-functional DMA for HPC / AI accelerators, characterized in that, The system includes a channel pipe, a read request distribution module, a write request distribution module, and a register array for transposing and concatenating data. The channel pipe is interconnected with the read request distribution module and the write request distribution module, respectively. The channel pipe is used to connect to external source memory, and the read request distribution module and the write request distribution module are used to connect to external destination memory. The channel pipe contains a normal transmission module, a transpose transmission module, and a concatenation transmission module. The normal transmission module outputs data from the source memory directly without processing, and the transpose transmission module transposes the data from the source memory using the register array before outputting it. The splicing and transmission module is used to splice data from the source memory through a register array and then output it. The splicing operation includes splicing single-precision data for general matrix multiplication, splicing half-precision data for general matrix multiplication, splicing single-precision data for deep learning models, and splicing half-precision data for deep learning models. The splicing of single-precision data for general matrix multiplication involves taking two input single-precision matrices matrix0 and matrix1 of size H×W, and splicing the single-precision data at the same positions in matrix0 and matrix1 into 64-bit data. Data is stored in the lower 32 bits, and the data of the single-precision matrix matrix1 is stored in the higher 32 bits, where H and W are the height and width of the matrix, respectively. The concatenation of half-precision data for general matrix multiplication involves concatenating the half-precision data at the same positions of the four input half-precision matrices matrix0, matrix1, matrix2, and matrix3 of size H×W into 64 bits of data, with the data of half-precision matrix matrix0 stored in the lower 16 bits. Then, the data of half-precision matrix matrix1, matrix2, and matrix3 are concatenated into the higher 32 bits of the matrix matrix1. The data of trix2 and half-precision matrix matrix3 are stored in the high 16 bits in ascending order, where H and W are the height and width of the matrix, respectively. The concatenation of single-precision data for deep learning models includes concatenating the single-precision data at the same position of the two input single-precision matrices data0 and data1 of size N×C×H×W into 64 bits of data. The data of single-precision matrix data0 is stored in the low 32 bits, and the data of single-precision matrix data1 is stored in the high 32 bits, where N, C, H, and W are the batch size, number of channels, height, and width of the single-precision matrix, respectively.The concatenation of half-precision data for deep learning models involves taking four input half-precision matrices data0, data1, data2, and data3 of size N×C×H×W, concatenating the half-precision data at the same positions of these matrices into 64 bits of data. The data of half-precision matrix data0 is stored in the lower 16 bits, and then the data of half-precision matrices data1, data2, and data3 are stored in the higher 16 bits in ascending order. Here, N, C, H, and W represent the batch size, number of channels, height, and width of the half-precision matrix, respectively.

2. The multi-functional DMA for HPC / AI accelerators according to claim 1, characterized in that, The multi-functional DMA selects the operating state of the normal transmission module, transpose transmission module, and splicing transmission module in the channel pipe according to the mode parameters carried by the instructions issued by the control logic: when the mode parameter is normal mode, the normal transmission module in the channel pipe outputs the data from the source memory directly without processing; when the mode parameter is transpose mode, the transpose transmission module in the channel pipe transposes the data from the source memory before outputting it; when the mode parameter is splicing mode, the splicing transmission module in the channel pipe splices the data from the source memory before outputting it.

3. The multi-functional DMA for HPC / AI accelerators according to claim 1, characterized in that, The transpose operation refers to reading the data into the multi-function DMA in column-first order, then storing any nth row of the original data block as the nth column through the multi-function DMA, and writing it out in row-first order during output.

4. The multi-functional DMA for HPC / AI accelerators according to claim 1, characterized in that, The external source-end memory devices connected to the channel pipe include source-end memory 1 to source-end memory n, totaling n source-end memory devices.

5. The multi-functional DMA for HPC / AI accelerators according to claim 1, characterized in that, The external destination memory connected to the read request distribution module and the write request distribution module includes destination memory bank 1 to destination memory bank n, totaling n destination memory banks.

Images