Matrix transposition method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention discloses a matrix transposition method and device, electronic equipment and a storage medium, the method and device are applied to a tensor processor TPU, the TPU comprises two tensor calculation cores, each tensor calculation core comprises two transposition processing units and two buffer areas, and the method comprises the steps that when it is determined that a to-be-transposed matrix meets a preset splitting condition, the to-be-transposed matrix is divided into two transposition processing units and two buffer areas; splitting the to-be-transposed matrix into two to-be-transposed sub-matrixes, and determining a tensor calculation core and a plurality of original matrix blocks respectively corresponding to each to-be-transposed sub-matrix; alternately loading the original matrix blocks into the double buffer regions of the corresponding tensor calculation core, and transposing the loaded original matrix blocks in the corresponding double buffer regions into transposed matrix blocks through a double transposed processing unit in the tensor calculation core in the loading process of the original matrix blocks; according to each transpose matrix block, the transpose completion matrix corresponding to the matrix to be transposed is determined, hardware resources in the TPU are fully utilized, and the matrix transpose efficiency is improved.

Description

Technical Field

[0001] This invention relates to the field of electronic digital data processing technology, and in particular to a matrix transposition method, apparatus, electronic device, and storage medium. Background Technology

[0002] Matrix transpose, as an operation that interchanges the rows and columns of a matrix, is a key step in matrix multiplication, convolution kernel rearrangement, and feature map format conversion in deep learning. Its computational performance directly affects the training and inference efficiency of upper-layer models such as Residual Networks (ResNet).

[0003] In existing technologies, matrix transpose methods on central processing units (CPUs) or graphics processing units (GPUs) are typically ported directly to tensor processing units (TPUs) for matrix transpose.

[0004] Specifically, the matrix transpose method ported to the TPU works as follows: a transpose processing unit in a tensor computation core initiates transpose only after a block of original matrix is ​​transferred from high-bandwidth memory (HBM) to a temporary register, and loads the next block of original matrix only after the transposed block of original matrix is ​​returned to HBM.

[0005] Therefore, although matrix transpose technology is relatively mature on CPUs and GPUs, porting it to TPUs not only causes one tensor computation core in the TPU to be completely idle, but also causes the only activated transpose processing unit to be completely idle during the data transfer phase, thereby reducing matrix transpose efficiency. Summary of the Invention

[0006] This invention provides a matrix transposition method, apparatus, electronic device, and storage medium, which solves the problem that the existing technology can only use a single tensor calculation core and a single transposition processing unit on the TPU for matrix transposition, resulting in low matrix transposition efficiency. It makes full use of the hardware resources in the TPU and improves the matrix transposition efficiency.

[0007] In a first aspect, embodiments of the present invention provide a matrix transposition method applied to a TPU. The TPU includes two tensor computation cores, each of which includes two transposition processing units and a temporary register. Each temporary register includes two buffers. The matrix transposition method includes: obtaining a matrix to be transposed and its original matrix size, and determining whether the matrix to be transposed meets a preset splitting condition based on the original matrix size; when the matrix to be transposed meets the preset splitting condition, splitting the matrix to be transposed into a first sub-matrix to be transposed and a second sub-matrix to be transposed, and determining the tensor computation core corresponding to each sub-matrix to be transposed; determining multiple original matrix blocks corresponding to each sub-matrix to be transposed based on a predefined matrix block size; alternately loading each original matrix block into the dual buffers of the corresponding tensor computation core, and during the loading of the original matrix blocks, transposing the original matrix blocks loaded in the corresponding dual buffers into transposed matrix blocks through the dual transposition processing units in the tensor computation core; and determining the transposed matrix corresponding to the matrix to be transposed based on each transposed matrix block.

[0008] Secondly, embodiments of the present invention also provide a matrix transposition device applied to a TPU. The TPU includes two tensor computation cores, each tensor computation core including two transposition processing units and a temporary register. Each temporary register includes two buffers. The matrix transposition device includes: a splitting condition judgment module, used to obtain the matrix to be transposed and the original matrix size of the matrix to be transposed, and determine whether the matrix to be transposed meets a preset splitting condition based on the original matrix size; and a computation core determination module, used to split the matrix to be transposed into a first submatrix to be transposed and a second submatrix to be transposed when it is determined that the matrix to be transposed meets the preset splitting condition. The system comprises the following modules: a transpose submatrix and a tensor computation core corresponding to each transpose submatrix; a matrix block determination module, which determines multiple original matrix blocks corresponding to each transpose submatrix based on a predefined matrix block size; a matrix block transpose module, which alternately loads each original matrix block into the double buffer of the corresponding tensor computation core, and transposes the loaded original matrix blocks in the corresponding double buffer into transpose matrix blocks through the double transpose processing unit in the tensor computation core during the loading of the original matrix blocks; and a transpose matrix splicing module, which determines the transpose matrix corresponding to the transpose matrix based on each transpose matrix block.

[0009] Thirdly, embodiments of the present invention also provide an electronic device, the electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the matrix transpose method provided in any embodiment of the present invention.

[0010] Fourthly, embodiments of the present invention also provide a computer-readable storage medium storing computer instructions that are used to cause a processor to execute the matrix transpose method of any embodiment of the present invention.

[0011] The technical solution of this invention solves the problem of low matrix transposition efficiency caused by the prior art, which can only utilize a single tensor computing core and a single transposition processing unit on the TPU for matrix transposition. This is achieved by: splitting the matrix to be transposed into two sub-matrices when a preset splitting condition is met; determining the tensor computation core corresponding to each sub-matrice; determining multiple original matrix blocks corresponding to each sub-matrice based on a predefined matrix block size; alternately loading each original matrix block into the dual buffer of the corresponding tensor computation core; and transposing the loaded original matrix blocks in the corresponding dual buffer into transposed matrix blocks using the dual transposition processing unit in the tensor computation core. Based on each transposed matrix block, the transposed matrix corresponding to the matrix to be transposed is determined. This fully utilizes the hardware resources of the TPU and improves the matrix transposition efficiency.

[0012] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the structure of a TPU provided according to an embodiment of the present invention.

[0015] Figure 2 This is a flowchart of a matrix transpose method provided according to Embodiment 1 of the present invention.

[0016] Figure 3 This is a flowchart of another matrix transpose method provided according to Embodiment 2 of the present invention.

[0017] Figure 4 This is a schematic diagram of a matrix transpose device according to Embodiment 3 of the present invention.

[0018] Figure 5This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention.

[0019] Reference numerals: Electronic device 10; Processor 11; Read-Only Memory (ROM) 12; Random Access Memory (RAM) 13; Bus 14; Input / Output (I / O) interface 15; Input unit 16; Output unit 17; Storage unit 18; Communication unit 19. Detailed Implementation

[0020] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0021] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0022] Example 1

[0023] This embodiment is applicable to the transposition of matrices. The method can be executed by a matrix transposition device, which can be implemented in hardware and / or software and can be configured in an electronic device such as a TPU.

[0024] To explain the matrix transpose method in this invention in detail, we will first introduce the hardware foundation supporting the implementation of the matrix transpose method, namely the TPU. For example... Figure 1As shown, the TPU includes two tensor computation cores. Each tensor computation core includes two transpose processing units and a temporary register. Each temporary register includes two buffers. Specifically, the buffers can be used to temporarily store the original matrix blocks corresponding to the matrix to be transposed. The transpose processing units can be used to transpose the original data blocks temporarily stored in the buffers.

[0025] by Figure 1 Based on the TPU shown, this embodiment discloses a matrix transpose method, such as... Figure 2 As shown, it includes S110-S150.

[0026] S110. Obtain the matrix to be transposed and its original matrix size, and determine whether the matrix to be transposed satisfies the preset splitting conditions based on its original matrix size.

[0027] In this embodiment, the original matrix size may include the original number of rows and the original number of columns. The preset splitting condition can be that the original number of rows of the matrix to be transposed exceeds a preset maximum number of rows, or the original number of columns of the matrix to be transposed exceeds a preset maximum number of columns. The preset maximum number of rows and columns can be set according to user needs and historical experience; for example, both the preset maximum number of rows and columns can be set to 128.

[0028] Specifically, in this step, if the number of rows in the original matrix of the matrix to be transposed exceeds a preset maximum number of rows, or the number of columns in the original matrix of the matrix to be transposed exceeds a preset maximum number of columns, then the matrix to be transposed is determined to meet a preset splitting condition. If the number of rows in the original matrix of the matrix to be transposed is less than or equal to the preset maximum number of rows, and the number of columns in the original matrix of the matrix to be transposed is also less than or equal to the preset maximum number of columns, then the matrix to be transposed is determined to meet a non-preset splitting condition.

[0029] S120. When it is determined that the matrix to be transposed meets the preset splitting conditions, the matrix to be transposed is split into a first submatrix to be transposed and a second submatrix to be transposed, and the tensor calculation core corresponding to each submatrix to be transposed is determined.

[0030] In this step, specifically, when it is determined that the matrix to be transposed meets the preset splitting conditions, the matrix to be transposed can be evenly split into a first submatrix and a second submatrix, either row-wise or column-wise. Then, since the computational capabilities of the two tensor computation cores are exactly the same, different tensor computation cores can be randomly assigned to the two submatrixes to be transposed.

[0031] For example, if the number of rows in the original matrix to be transposed is greater than or equal to the number of columns, the matrix to be transposed can be evenly divided into a first submatrix and a second submatrix by rows. If the number of rows in the original matrix to be transposed is less than the number of columns, the matrix to be transposed can be evenly divided into a first submatrix and a second submatrix by columns.

[0032] S130. Based on the predefined matrix block size, determine multiple original matrix blocks corresponding to each submatrix to be transposed.

[0033] In one implementation, each submatrix to be transposed can be split according to a predefined matrix block size to obtain multiple original matrix blocks corresponding to each submatrix to be transposed.

[0034] In another embodiment, the matrix to be transposed can be split according to a predefined matrix block size to obtain multiple original matrix blocks. Then, based on the matrix size of each submatrix to be transposed, multiple original matrix blocks corresponding to each submatrix to be transposed can be determined. It is worth noting that since the number of original matrix blocks corresponding to the matrix to be transposed may be odd, excess original matrix blocks can be randomly assigned to the first or second submatrix to be transposed; this embodiment does not impose any restrictions on this.

[0035] S140. Each original matrix block is alternately loaded into the double buffer of the corresponding tensor calculation core. During the loading of the original matrix block, the original matrix block that has been loaded in the corresponding double buffer is transposed into a transposed matrix block by the double transpose processing unit in the tensor calculation core.

[0036] In some implementations, to avoid computational errors caused by unprepared original matrix blocks and the temporary storage of two matrix blocks in the same buffer, the original matrix block in one buffer can be processed by the double transpose processing unit in the tensor computation core after it has been loaded into that buffer. While the double transpose processing unit is being utilized, to fully utilize the TPU's hardware resources and improve matrix transpose efficiency, the next original matrix block can be loaded into another buffer. This allows the transpose processing of the next original matrix block to continue when the double transpose processing unit is idle, thereby reducing the idle time of the double transpose processing unit.

[0037] It is worth noting that when the original matrix block that has been loaded in the corresponding dual buffer is transposed into a transposed matrix block by the dual transpose processing unit in the tensor computation core, the transpose processing time of the two transpose processing units on the same original matrix block can be completely overlapped by synchronously sending the same control command to the dual transpose processing unit, or the transpose processing time of the two transpose processing units on the same original matrix block can be partially overlapped by asynchronously sending the same control command to the dual transpose processing unit.

[0038] In other implementations, to reduce the number of times the double transpose processing unit in the tensor computation core switches between different buffers, multiple original matrix blocks can be loaded into one buffer first, and then the original matrix blocks in that buffer can be processed by the double transpose processing unit in the tensor computation core. After the double transpose processing unit finishes processing the multiple original matrix blocks in that buffer, it continues to process multiple original matrix blocks in another buffer, while simultaneously loading new, untransposed original matrix blocks into the empty buffer, thereby completing the transpose processing of all original matrix blocks.

[0039] Optionally, before the dual transpose processing unit in the tensor computation core transposes the original matrix block that has been loaded in the corresponding dual buffer into a transpose matrix block, the method further includes: obtaining the maximum data processing capacity of the transpose processing unit and determining the lower limit of the number of matrix block columns based on the maximum data processing capacity; obtaining the number of matrix block columns of the original matrix block that has been loaded, and when the number of matrix block columns is less than the lower limit of the number of matrix block columns, padding the number of matrix block columns of the original matrix block that has been loaded to the lower limit of the number of matrix block columns to adapt to the data processing capacity of the transpose processing unit.

[0040] Specifically, when the number of columns in a matrix block is less than the lower limit for the number of columns in a matrix block, the formula can be used to... Determine the supplementary values ​​for the column numbers. Among them, To supplement the column number with values, The column count of the matrix block is then determined. A column count supplement can then be added to the left and right sides of the original matrix block to obtain an original matrix block with a column count equal to the lower limit of the matrix block. This adapts the matrix block to the data processing capabilities of the transpose processing unit, eliminating the performance overhead of non-aligned access.

[0041] Based on the maximum data processing capacity of the transpose processing unit Taking a matrix block with 130 columns as an example, it can be solved using the formula... First, determine that the column fill value is 63. Then, fill the left and right sides of the original matrix block with 63 each to obtain an original matrix block with 256 columns.

[0042] S150. Based on each transpose matrix block, determine the transpose completion matrix corresponding to the matrix to be transposed.

[0043] In this step, specifically, the relative positions of the original matrix blocks corresponding to each transpose matrix block within the matrix to be transposed can be obtained. Then, the transpose matrix blocks can be concatenated according to the aforementioned relative positions to obtain the transposed matrix corresponding to the matrix to be transposed.

[0044] The technical solution of this embodiment obtains the matrix to be transposed and its original matrix size, and determines whether the matrix to be transposed meets a preset splitting condition based on the original matrix size. When it is determined that the matrix to be transposed meets the preset splitting condition, it splits the matrix to be transposed into a first sub-matrix to be transposed and a second sub-matrix to be transposed, and determines the tensor computation core corresponding to each sub-matrix to be transposed. Based on the predefined matrix block size, it determines multiple original matrix blocks corresponding to each sub-matrix to be transposed. It alternately loads each original matrix block into the double buffer of the corresponding tensor computation core, and during the loading of the original matrix block, the double transpose processing unit in the tensor computation core transposes the original matrix block that has been loaded in the corresponding double buffer into a transposed matrix block. Based on each transposed matrix block, it determines the transposed matrix corresponding to the matrix to be transposed. This technical means solves the problem that the existing technology can only use a single tensor computation core and a single transpose processing unit on the TPU for matrix transposition, resulting in low matrix transposition efficiency. It makes full use of the hardware resources in the TPU and improves the matrix transposition efficiency.

[0045] Example 2

[0046] This embodiment is a further optimization and extension based on the above embodiments, and can be combined with various optional technical solutions in the above embodiments.

[0047] This embodiment discloses a matrix transpose method applied to TPU, such as... Figure 1 As shown, the TPU includes a first tensor computation core and a second tensor computation core. The first tensor computation core includes a first transpose processing unit, a second transpose processing unit, and a first temporary register, which includes a first buffer and a second buffer. The second tensor computation core includes a third transpose processing unit, a fourth transpose processing unit, and a second temporary register, which includes a third buffer and a fourth buffer.

[0048] by Figure 1 Based on the TPU shown, this embodiment discloses a matrix transpose method, such as... Figure 3 As shown, it includes S210-S280.

[0049] S210. Obtain the matrix to be transposed, and the original matrix size of the matrix to be transposed.

[0050] S220. Based on the original matrix size of the matrix to be transposed, determine whether the matrix to be transposed meets the preset splitting conditions. If yes, execute S230; otherwise, execute S280.

[0051] S230. The matrix to be transposed is split into a first submatrix to be transposed and a second submatrix to be transposed, and the tensor computation kernel corresponding to each submatrix to be transposed is determined.

[0052] S240. Based on the predefined matrix block size, determine multiple original matrix blocks corresponding to each submatrix to be transposed.

[0053] Optionally, before determining the multiple original matrix blocks corresponding to each transpose submatrix based on the predefined matrix block size, the method further includes: determining the matrix block size corresponding to the transpose matrix based on the original matrix size of the transpose matrix, the element format of each matrix element in the transpose matrix, and the temporary register capacity, so that the memory capacity occupied by each original matrix block obtained by dividing the transpose matrix is ​​less than or equal to the temporary register capacity.

[0054] Specifically, the actual memory capacity of each matrix element in the transpose matrix can be determined based on its element format. For example, when the element format is 32-bit floating-point (FP32), the memory capacity is 4 bytes; when the element format is 16-bit brain floating-point (BF16), the memory capacity is 2 bytes. Then, a binary search method can be used to determine the corresponding matrix block size based on the original matrix size, the actual content capacity of each matrix element, and the temporary register capacity.

[0055] Furthermore, a binary search method is employed to determine the matrix block size corresponding to the transpose matrix based on the original matrix size, the actual content capacity occupied by each matrix element, and the temporary storage capacity. This may include: obtaining the maximum data processing volume of the transpose processing unit and determining the dimension boundary value of each original matrix block based on the maximum data processing volume. Then, it can be determined whether the number of rows and columns of the original matrix are less than or equal to the dimension boundary value. If the number of rows of the original matrix is ​​less than or equal to the dimension boundary value, but the number of columns of the original matrix is ​​greater than the dimension boundary value, then a binary search method is used to gradually reduce the number of columns of the original matrix based on the number of rows, the number of columns, and the actual memory capacity occupied by each matrix element, until the maximum number of columns is obtained that makes the capacity occupied by the original matrix block less than the temporary storage capacity. The original matrix number of rows and the maximum number of columns are then used together as the matrix block size. If the number of columns in the original matrix is ​​less than or equal to the dimension boundary value, but the number of rows in the original matrix is ​​greater than the dimension boundary value, then a binary search method is used. Based on the number of rows in the original matrix, the number of columns in the original matrix, and the actual memory capacity occupied by each matrix element, the maximum number of matrix rows that makes the capacity occupied by the original matrix block less than the capacity of the temporary register is determined step by step, and the number of matrix rows and the number of columns in the original matrix are used together as the matrix block size.

[0056] If both the number of rows and columns of the original matrix are greater than the dimensionality boundary value, then after setting the number of rows of the original matrix as the dimensionality boundary value, a binary search method is used to determine the maximum number of columns in the block that makes the capacity of the original matrix block less than the capacity of the temporary register. Then, using the binary search method, based on the maximum number of columns in the block, the number of rows in the original matrix, and the actual memory capacity occupied by each matrix element, the maximum number of rows in the block that makes the capacity of the original matrix block less than the capacity of the temporary register is determined. Finally, the maximum number of rows and columns in the block can be used together as the matrix block size.

[0057] Taking an example where the original matrix to be transposed has 4096 rows and columns, each element is in FP32 format, the temporary register capacity is 16MB, and the maximum data processing capacity of the transpose processing unit is 128×128, the dimension boundary value can be set to 128 based on the maximum data processing capacity. Then, since the original matrix has more than 128 rows and columns, after temporarily fixing the original matrix row count to 128, the maximum number of columns in the block matrix that makes the original matrix block's capacity less than 16MB is determined to be 4096. Afterwards, a binary search method can be used to determine the maximum number of rows in the block matrix. Since the second block matrix has 384 rows, it satisfies... Therefore, the maximum number of rows in the block matrix can be determined to be 384. Finally, the maximum number of rows in the block matrix (384) and the maximum number of columns in the block matrix (4096) can be used together as the matrix block size.

[0058] The advantage of this setup is that by determining the matrix block size with the goal of ensuring that the capacity occupied by the original matrix block is less than the capacity of the temporary register, it avoids the situation where the original matrix size exceeds the capacity of the temporary register, leading to memory overflow. Secondly, by selecting the maximum value among the matrix block sizes that meet the target, it avoids the situation where the scheduling overhead increases dramatically due to the blocks being too small, thus ensuring the efficiency of transpose processing for large-scale matrices.

[0059] S250. The original matrix blocks corresponding to the first submatrix to be transposed are alternately loaded into the first buffer and the second buffer of the first tensor calculation core corresponding to the first submatrix to be transposed. During the process of alternately loading the original matrix blocks into the first buffer and the second buffer, the original matrix blocks that have been loaded in the first buffer and the second buffer are transposed into transposed matrix blocks by the double transpose processing unit in the first tensor calculation core.

[0060] In this embodiment, it can be achieved through the formula Determine the size of the buffer. Indicates the size of the buffer. This refers to memory alignment instructions. This represents the number of rows in the original matrix block. This represents the number of columns in the original matrix block.

[0061] In this step, specifically, the first original matrix block in the first transpose submatrix can be determined based on the index values ​​of each original matrix block corresponding to the first transpose submatrix. This first original matrix block is used as the current matrix block and loaded into the first buffer. The double transpose processing unit in the first tensor computation core transposes the current matrix block in the first buffer, and simultaneously loads the current adjacent matrix block (with an index value greater than the current matrix block) into the second buffer. After obtaining the transpose matrix block corresponding to the current matrix block, the double transpose processing unit in the first tensor computation core transposes the current adjacent matrix block in the second buffer. It is then determined whether the processing of all original matrix blocks corresponding to the first transpose submatrix is ​​complete. If not, during the transpose processing of the current adjacent matrix block in the second buffer, the next adjacent matrix block (with an index value greater than the current adjacent matrix block) is obtained and loaded into the first buffer as the current matrix block. Return to the execution and perform the transpose operation on the current matrix block in the first buffer through the double transpose processing unit in the first tensor computation core, until the processing of all original matrix blocks corresponding to the first submatrix to be transposed is completed.

[0062] In some implementations, the index value of each original matrix block can be determined based on the starting row and column numbers of each original matrix block corresponding to the first transposed submatrix, and the original matrix block with the smallest index value can be used as the first matrix block. Then, during the process of loading the first original matrix block as the current matrix block into the first buffer, the computation process can be blocked until the current matrix block is fully loaded into the first buffer. Only then, the double transpose processing unit in the first tensor computation core performs transpose processing on the current matrix block in the first buffer. The transpose processing of the current matrix block in the first buffer by the double transpose processing unit in the first tensor computation core can be performed as follows: the current matrix block is divided into a left matrix block and a right matrix block; the left matrix block is transposed by the first transpose processing unit in the first tensor computation core to obtain the transpose result of the left matrix; the right matrix block is transposed by the second transpose processing unit in the first tensor computation core to obtain the transpose result of the right matrix; the transpose matrix block corresponding to the current matrix block is obtained by combining the transpose results of the left and right matrices.

[0063] For example, assuming there are four original matrix blocks corresponding to the first submatrix to be transposed, and the relative positions of the four original matrix blocks are top left, bottom left, top right, and bottom right, the index values ​​of the original matrix blocks located at the top left, bottom left, top right, and bottom right of the first submatrix to be transposed can be set to 1, 2, 3, and 4, respectively. Then, the original matrix block with index value 1 can be loaded into the first buffer as the current matrix block, and when the current matrix block is loaded, the original matrix block with index value 1 is transposed by the double transpose processing unit in the first tensor computation core.

[0064] During the transpose of the original matrix block with index 1, the original matrix block with index 2 can be loaded into the second buffer. After transposing the original matrix block with index 1, the original matrix block with index 2 can be processed by the double transpose processing unit in the first tensor computation core, while the original matrix block with index 3 is loaded into the first buffer as the current matrix block. This matrix block transpose and loading process is repeated until all original matrix blocks in the first submatrix to be transposed are processed.

[0065] The advantage of this setup is that by transposing one original matrix block while loading another original matrix block, parallel transmission and computation can be achieved, thereby reducing the idle time of the transpose processing unit and improving matrix transpose efficiency.

[0066] Optionally, the double transpose processing unit in the first tensor computation core transposes the loaded original matrix blocks in the first and second buffers into transposed matrix blocks, including: dividing a loaded original matrix block into a left matrix block and a right matrix block; transposing the left matrix block using the first transpose processing unit in the first tensor computation core to obtain a left matrix transpose result; transposing the right matrix block using the second transpose processing unit in the first tensor computation core to obtain a right matrix transpose result; and obtaining a transposed matrix block corresponding to a loaded original matrix block from the left and right matrix transpose results. Alternatively, a loaded original matrix block can be divided into an upper matrix block and a lower matrix block; the upper matrix block can be transposed using the first transpose processing unit in the first tensor computation core to obtain the transpose result of the upper matrix; the lower matrix block can be transposed using the second transpose processing unit in the first tensor computation core to obtain the transpose result of the lower matrix; the transpose results of the upper and lower matrices can be combined to obtain the transpose matrix block corresponding to a loaded original matrix block.

[0067] S260. The original matrix blocks corresponding to the second submatrix to be transposed are alternately loaded into the third and fourth buffers of the second tensor calculation core corresponding to the second submatrix to be transposed. During the process of alternately loading the original matrix blocks into the third and fourth buffers, the original matrix blocks that have been loaded in the third and fourth buffers are transposed into transposed matrix blocks by the double transpose processing unit in the second tensor calculation core.

[0068] In this step, specifically, the first original matrix block in the second transpose submatrix is ​​determined based on the index values ​​of each original matrix block corresponding to the second transpose submatrix. This first original matrix block is then loaded into the third buffer as the first matrix block to be processed. The first matrix block in the third buffer is transposed using the double transpose processing unit in the second tensor computation core. Simultaneously, the second matrix block to be processed, whose index value is greater than that of the first matrix block and which is adjacent to the first matrix block, is loaded into the fourth buffer. After obtaining the transpose matrix block corresponding to the first matrix block, the second matrix block to be processed in the fourth buffer is transposed using the double transpose processing unit in the second tensor computation core. It is then determined whether the processing of all original matrix blocks corresponding to the second transpose submatrix is ​​complete. If not, during the transpose processing of the second matrix block in the third buffer, the third matrix block to be processed, whose index value is greater than that of the second matrix block and which is adjacent to the second matrix block, is obtained and loaded into the third buffer as the first matrix block to be processed. The process returns to the double transpose processing unit in the second tensor computation core, which transposes the first matrix block to be processed in the third buffer until all original matrix blocks corresponding to the second transpose submatrix are processed.

[0069] Further, the first matrix block to be processed in the third buffer is transposed using the double transpose processing unit in the second tensor computation core. This includes: dividing the first matrix block to be processed into a left matrix block and a right matrix block; transposing the left matrix block using the third transpose processing unit in the second tensor computation core to obtain the transposed left matrix result; transposing the right matrix block using the fourth transpose processing unit in the second tensor computation core to obtain the transposed right matrix result; and obtaining the transposed left matrix result and the transposed right matrix result to obtain the transposed matrix block corresponding to the first matrix block to be processed. Alternatively, the first matrix block to be processed is divided into an upper matrix block and a lower matrix block; transposing the upper matrix block using the third transpose processing unit in the second tensor computation core to obtain the upper matrix transpose result; transposing the lower matrix block using the fourth transpose processing unit in the second tensor computation core to obtain the lower matrix transpose result; and obtaining the transposed upper matrix result and the transposed lower matrix result to obtain the transposed matrix block corresponding to the first matrix block to be processed.

[0070] Optionally, to save storage space within HBM, the element format of each matrix element in each original matrix block can be set to BF16 format, and the BF16 formatted original matrix blocks can be packaged and stored in HBM. The packaging rule for the original matrix blocks is: the high and low parts of a 32-byte storage unit are used to store two 16-byte data blocks respectively. Then, to speed up the transmission efficiency of each original matrix block, the packaged original matrix blocks can be directly sent to a temporary register, where they are unpacked. Finally, since the transpose processing unit only supports 32-byte input, each matrix element in the unpacked original matrix block can be expanded to 32 bytes to adapt to the transpose processing unit.

[0071] S270. Based on each transpose matrix block, determine the transpose matrix corresponding to the matrix to be transposed.

[0072] In this step, specifically, the transposed matrix blocks corresponding to each original matrix block can be written back to HBM in the order they were transferred to the temporary register, to ensure the integrity of the transposed matrix. The transposed matrix blocks can be written back to HBM using asynchronous direct memory access (DMA). Then, the transposed matrix blocks can be concatenated in HBM to obtain the transposed matrix corresponding to the matrix to be transposed.

[0073] Optionally, the next round of matrix transposition process can be started or the matrix transposition process can be ended only after ensuring that each transpose matrix block corresponding to the matrix to be transposed has been transmitted to HBM, in order to avoid transposition failure caused by missing transpose matrix blocks received by HBM.

[0074] S280. Load the matrix to be transposed into the arbitrary tensor computation kernel for transposition processing to obtain the transposed matrix corresponding to the matrix to be transposed.

[0075] The technical solution of this embodiment splits the matrix to be transposed into two sub-matrices when it is determined that the matrix to be transposed meets the preset splitting conditions, and processes the two sub-matrices in parallel through a dual tensor computation core to obtain the transposed matrix corresponding to the matrix to be transposed. When it is determined that the matrix to be transposed does not meet the preset splitting conditions, it is loaded into an arbitrary tensor computation core for transposition processing to obtain the transposed matrix corresponding to the matrix to be transposed. Different transposition processing strategies can be configured for matrices of different sizes. While improving the transposition efficiency of large-scale matrices, it omits the splitting step of small-scale matrices, avoiding the situation where the matrix splitting and merging process slows down the transposition efficiency of small-scale matrices.

[0076] Example 3

[0077] Figure 4This is a schematic diagram of a matrix transpose device according to Embodiment 3 of the present invention. This embodiment is applicable to the case of transposing a matrix. The matrix transpose device can be implemented in hardware and / or software and can be configured in an electronic device such as a TPU.

[0078] The matrix transpose device disclosed in this embodiment is applied to a TPU. The TPU includes two tensor computation cores, each of which includes two transpose processing units and a temporary register. Each temporary register includes two buffers.

[0079] like Figure 4 As shown, the matrix transpose device includes: a splitting condition judgment module 41, used to obtain the matrix to be transposed and its original matrix size, and determine whether the matrix to be transposed meets the preset splitting condition based on the original matrix size; a computation core determination module 42, used to split the matrix to be transposed into a first sub-matrix and a second sub-matrix when it is determined that the matrix to be transposed meets the preset splitting condition, and determine the tensor computation core corresponding to each sub-matrix; a matrix block determination module 43, used to determine multiple original matrix blocks corresponding to each sub-matrix based on a predefined matrix block size; a matrix block transpose module 44, used to alternately load each original matrix block into the double buffer of the corresponding tensor computation core, and during the loading of the original matrix block, transpose the original matrix block that has been loaded in the corresponding double buffer into a transpose matrix block through the double transpose processing unit in the tensor computation core; and a transpose matrix splicing module 45, used to determine the transpose completed matrix corresponding to the matrix to be transposed based on each transpose matrix block.

[0080] The technical solution in this embodiment, through the cooperation of the split condition judgment module 41, the calculation core determination module 42, the matrix block determination module 43, the matrix block transpose module 44, and the transpose matrix splicing module 45, solves the problem that the existing technology can only use a single tensor calculation core and a single transpose processing unit on the TPU to perform matrix transpose, resulting in low matrix transpose efficiency. It makes full use of the hardware resources in the TPU and improves the matrix transpose efficiency.

[0081] Optionally, the device further includes a block size determination module, which is used to: determine the matrix block size corresponding to the transpose matrix based on the original matrix size of the transpose matrix, the element format of each matrix element in the transpose matrix, and the temporary register capacity, so that the memory capacity occupied by each original matrix block obtained by dividing the transpose matrix is ​​less than or equal to the temporary register capacity.

[0082] Optionally, the matrix block transpose module 44 includes: a first transpose processing unit, used to alternately load each original matrix block corresponding to the first submatrix to be transpose into the first buffer and the second buffer of the first tensor calculation core corresponding to the first submatrix to be transpose, and during the process of alternately loading each original matrix block into the first buffer and the second buffer, the original matrix blocks loaded in the first buffer and the second buffer are transposed into transpose matrix blocks through the double transpose processing unit in the first tensor calculation core; and a second transpose processing unit, used to alternately load each original matrix block corresponding to the second submatrix to be transpose into the third buffer and the fourth buffer of the second tensor calculation core corresponding to the second submatrix to be transpose, and during the process of alternately loading each original matrix block into the third buffer and the fourth buffer, the original matrix blocks loaded in the third buffer and the fourth buffer are transposed into transpose matrix blocks through the double transpose processing unit in the second tensor calculation core.

[0083] Optionally, the first transpose processing unit is specifically used to: divide the loaded original matrix block into a left matrix block and a right matrix block; transpose the left matrix block using the first transpose processing unit in the first tensor computation core to obtain the left matrix transpose result; transpose the right matrix block using the second transpose processing unit in the first tensor computation core to obtain the right matrix transpose result; and obtain the transpose matrix block corresponding to the loaded original matrix block from the left matrix transpose result and the right matrix transpose result.

[0084] Optionally, the device further includes a matrix alignment module, which is used to: obtain the maximum data processing volume of the transpose processing unit and determine the lower limit of the number of matrix block columns based on the maximum data processing volume; obtain the number of matrix block columns of the original matrix block that has been loaded, and when the number of matrix block columns is less than the lower limit of the number of matrix block columns, pad the number of matrix block columns of the original matrix block that has been loaded to the lower limit of the number of matrix block columns, so as to adapt to the data processing capability of the transpose processing unit.

[0085] Optionally, the device also includes a matrix transpose module, which is used to: when it is determined that the matrix to be transposed does not meet the preset splitting conditions, load the matrix to be transposed into an arbitrary tensor calculation core for transpose processing to obtain a transposed matrix corresponding to the matrix to be transposed.

[0086] The matrix transpose apparatus provided in this embodiment of the invention can execute the matrix transpose method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the method execution. Content not described in detail in this embodiment can be referred to the description in any method embodiment of this application.

[0087] Example 4

[0088] Figure 5A schematic diagram of the structure of an electronic device 10 that can be used to implement embodiments of the present invention is shown. For example... Figure 5 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as ROM 12 or RAM 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded into the RAM 13 from the storage unit 18. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An I / O interface 15 is also connected to the bus 14.

[0089] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0090] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, CPUs, GPUs, various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the matrix transpose method.

[0091] In some embodiments, the matrix transpose method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the matrix transpose method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the matrix transpose method by any other suitable means (e.g., by means of firmware).

[0092] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0093] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0094] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0095] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0096] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0097] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0098] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0099] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A matrix transpose method, characterized in that, It is applied to a tensor processor (TPU), which includes two tensor computation cores, each of which includes two transpose processing units and a temporary register, and each temporary register includes two buffers. The matrix transpose method includes: Obtain the matrix to be transposed and its original matrix size, and determine whether the matrix to be transposed satisfies a preset splitting condition based on its original matrix size. When it is determined that the matrix to be transposed meets the preset splitting conditions, the matrix to be transposed is split into a first submatrix to be transposed and a second submatrix to be transposed, and the tensor calculation core corresponding to each submatrix to be transposed is determined. Based on a predefined matrix block size, determine multiple original matrix blocks corresponding to each transpose submatrix; Each of the original matrix blocks is alternately loaded into the double buffer of the corresponding tensor computation core. During the loading of the original matrix blocks, the double transpose processing unit in the tensor computation core transposes the original matrix blocks that have been loaded in the corresponding double buffer into transpose matrix blocks. Based on each of the transpose matrix blocks, determine the transpose completed matrix corresponding to the matrix to be transposed.

2. The matrix transpose method according to claim 1, characterized in that, Before determining the multiple original matrix blocks corresponding to each transpose submatrix based on a predefined matrix block size, the process also includes: Based on the original matrix size of the matrix to be transposed, the element format of each matrix element in the matrix to be transposed, and the temporary register capacity, the matrix block size corresponding to the matrix to be transposed is determined so that the memory capacity occupied by each original matrix block obtained by dividing the matrix to be transposed is less than or equal to the temporary register capacity.

3. The matrix transpose method according to claim 1, characterized in that, Each of the original matrix blocks is alternately loaded into the double buffer of the corresponding tensor computation core. During the loading of the original matrix blocks, the double transpose processing unit in the tensor computation core transposes the original matrix blocks that have been loaded in the corresponding double buffer into transposed matrix blocks, including: Each original matrix block corresponding to the first submatrix to be transposed is alternately loaded into the first buffer and the second buffer of the first tensor calculation core corresponding to the first submatrix to be transposed. During the process of alternately loading each original matrix block into the first buffer and the second buffer, the original matrix blocks that have been loaded in the first buffer and the second buffer are transposed into transposed matrix blocks by the double transpose processing unit in the first tensor calculation core. Each original matrix block corresponding to the second submatrix to be transposed is alternately loaded into the third and fourth buffers of the second tensor computation core corresponding to the second submatrix to be transposed. During the process of alternately loading each original matrix block into the third and fourth buffers, the double transpose processing unit in the second tensor computation core transposes the original matrix blocks that have been loaded in the third and fourth buffers into transposed matrix blocks.

4. The matrix transpose method according to claim 3, characterized in that, The double transpose processing unit in the first tensor computation core transposes the original matrix blocks loaded in the first and second buffers into transposed matrix blocks, including: The loaded original matrix block is divided into a left matrix block and a right matrix block; The left matrix block is transposed by the first transpose processing unit in the first tensor calculation core to obtain the left matrix transpose result. The right matrix block is transposed by the second transpose processing unit in the first tensor calculation core to obtain the right matrix transpose result; The transpose of the left matrix and the transpose of the right matrix are used to obtain the transpose matrix block corresponding to the original matrix block that has been loaded.

5. The matrix transpose method according to claim 1, characterized in that, Before the original matrix block that has been loaded in the corresponding dual buffer is transposed into a transposed matrix block by the double transpose processing unit in the tensor computation core, the following steps are also included: Obtain the maximum data processing volume of the transpose processing unit, and determine the lower limit of the number of columns in the matrix block based on the maximum data processing volume; The number of columns in the loaded original matrix block is obtained, and if the number of columns in the matrix block is less than the lower limit of the number of columns in the matrix block, the number of columns in the loaded original matrix block is padded to the lower limit of the number of columns in the matrix block to adapt to the data processing capability of the transpose processing unit.

6. The matrix transpose method according to claim 1, characterized in that, After determining whether the matrix to be transposed satisfies the preset splitting condition, the process further includes: When it is determined that the matrix to be transposed does not meet the preset splitting conditions, the matrix to be transposed is loaded into an arbitrary tensor computation kernel for transposition processing to obtain the transposed matrix corresponding to the matrix to be transposed.

7. A matrix transpose device, characterized in that, An application is provided in a tensor processor (TPU), wherein the TPU includes two tensor computation cores, each tensor computation core includes two transpose processing units and a temporary register, and each temporary register includes two buffers. The matrix transpose device includes: The splitting condition judgment module is used to obtain the matrix to be transposed and the original matrix size of the matrix to be transposed, and to determine whether the matrix to be transposed satisfies the preset splitting condition based on the original matrix size of the matrix to be transposed. The computation core determination module is used to split the matrix to be transposed into a first sub-matrix to be transposed and a second sub-matrix to be transposed when it is determined that the matrix to be transposed meets the preset splitting conditions, and to determine the tensor computation core corresponding to each sub-matrix to be transposed. The matrix block determination module is used to determine multiple original matrix blocks corresponding to each transpose submatrix based on a predefined matrix block size. The matrix block transpose module is used to alternately load each of the original matrix blocks into the double buffer of the corresponding tensor calculation core. During the loading of the original matrix blocks, the double transpose processing unit in the tensor calculation core transposes the original matrix blocks that have been loaded in the corresponding double buffer into transpose matrix blocks. The transpose matrix splicing module is used to determine the transpose completed matrix corresponding to the matrix to be transposed based on each of the transpose matrix blocks.

8. The matrix transpose device according to claim 7, characterized in that, The matrix transpose device further includes a block size determination module; the block size determination module is specifically used for: Based on the original matrix size of the matrix to be transposed, the element format of each matrix element in the matrix to be transposed, and the temporary register capacity, the matrix block size corresponding to the matrix to be transposed is determined so that the memory capacity occupied by each original matrix block obtained by dividing the matrix to be transposed is less than or equal to the temporary register capacity.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the matrix transpose method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the matrix transpose method according to any one of claims 1-6.

Images