Multiplication acceleration method and device of double sparse matrixes, equipment, medium and product

Abstract

The invention discloses a multiplication acceleration method and device for double sparse matrixes, equipment, a medium and a product. The method comprises the following steps: positioning a compression matrix set matched with a first sparse matrix and a second sparse matrix in a global memory of a computing chip according to a sparse matrix multiplication demand; carrying the compression matrix set and the second sparse matrix step by step from the global memory to a hardware register of the computing chip in a data block form; and gradually calculating a multiplication result of the first sparse matrix and the second sparse matrix through a sparse calculation unit of the calculation chip according to the data loaded in the hardware register by several times. According to the technical scheme of the embodiment of the invention, the hardware acceleration performance of the sparse computing unit in the computing chip can be brought into full play, so that the overhead of computing, bandwidth and storage resources in the double sparse matrix multiplication process is greatly optimized, and the method is particularly suitable for a model computing scene of a hybrid expert model.

Description

Technical Field

[0001] The present invention relates to the field of computer hardware technology, and in particular to a method, device, equipment, medium and product for accelerating the multiplication of double sparse matrices. Background Art

[0002] Operator data flow optimization and calculation and orchestration methods are a technical solution that can effectively optimize the utilization of computing, bandwidth and storage resources in the hardware computing process.

[0003] Existing data flow optimization, calculation and arrangement methods do not implement matrix multiplication operators for double-sparse pattern data (i.e., both the left and right operands of the operator are sparse data). Therefore, directly using existing solutions cannot perform efficient matrix multiplication calculations on double-sparse pattern data. Summary of the invention

[0004] Embodiments of the present invention provide a method, device, equipment, medium and product for accelerating the multiplication of double sparse matrices, so as to give full play to the acceleration performance of sparse computing hardware when calculating the multiplication of double sparse matrices.

[0005] According to one aspect of an embodiment of the present invention, a method for accelerating multiplication of double sparse matrices is provided, comprising:

[0006] According to the sparse matrix multiplication requirement, a compressed matrix set and a second sparse matrix matching the first sparse matrix are located in the global memory of the computing chip;

[0007] The first sparse matrix includes a plurality of compression units of a set size, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit;

[0008] The compressed matrix set and the second sparse matrix are moved from the global memory to the hardware registers of the computing chip in the form of data blocks;

[0009] The multiplication results of the first sparse matrix and the second sparse matrix are gradually calculated by the sparse computing unit of the computing chip according to the data loaded in batches in the hardware register.

[0010] According to another aspect of an embodiment of the present invention, a dual sparse matrix multiplication acceleration device is provided, comprising:

[0011] A multiplication element locating module, used for locating a set of compressed matrices and a second sparse matrix matching the first sparse matrix in a global memory of a computing chip according to a sparse matrix multiplication requirement;

[0012] The first sparse matrix includes a plurality of compression units of a set size, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit;

[0013] A step-by-step transfer module is used to transfer the compressed matrix set and the second sparse matrix from the global memory to the hardware register of the computing chip step by step in the form of data blocks;

[0014] The multiplication calculation module is used to gradually calculate the multiplication results of the first sparse matrix and the second sparse matrix according to the data loaded in batches in the hardware register through the sparse calculation unit of the calculation chip.

[0015] According to another aspect of an embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0016] at least one computing chip; and

[0017] A memory communicatively connected to the at least one computing chip; wherein,

[0018] The memory stores a computer program that can be executed by the at least one computing chip, and the computer program is executed by the at least one computing chip so that the at least one computing chip can execute the multiplication acceleration method of double sparse matrices as described in any embodiment of the present invention.

[0019] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the double sparse matrix multiplication acceleration method described in any embodiment of the present invention when a computing chip executes the computer instructions.

[0020] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a computer program, which, when executed by a computing chip, implements the steps of the double sparse matrix multiplication acceleration method as described in any embodiment of the present invention.

[0021] The technical solution of the embodiment of the present invention is to compress and store the double sparse matrix that needs to be multiplied in a specific compression format, and then move the double sparse matrix in the above specific compression format from the global memory to the hardware register of the computing chip in the form of data blocks step by step, and use the sparse computing unit of the computing chip to gradually calculate the multiplication result of the double sparse matrix according to the data loaded in the hardware register in batches. The implementation method fully considers the sparse characteristics when performing matrix multiplication calculations based on the double sparse matrix. By adopting data blocking and data moving technology that are compatible with the sparse characteristics, the hardware acceleration performance of the sparse computing unit in the computing chip can be fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the double sparse matrix multiplication operation process, and is particularly suitable for model calculation scenarios of hybrid expert models.

[0022] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a flow chart of a method for accelerating the multiplication of double sparse matrices provided according to an embodiment of the present invention;

[0025] Figure 2 is a flowchart of another method for accelerating the multiplication of double sparse matrices provided according to an embodiment of the present invention;

[0026] Figure 3 It is a schematic diagram of a comparison structure of a first sparse matrix and a matching set of compressed matrices applicable to an embodiment of the present invention;

[0027] Figure 4 It is a block diagram of a data segmentation and multi-stage transfer process applicable to an embodiment of the present invention;

[0028] Figure 5 is a schematic diagram of a register mapping relationship applicable to an embodiment of the present invention;

[0029] Figure 6 It is a structural diagram of a sparse computing unit and an intermediate register used in an embodiment of the present invention to cooperate in computing;

[0030] Figure 7 is a schematic structural diagram of a double sparse matrix multiplication acceleration device provided according to an embodiment of the present invention;

[0031] Figure 8 It is a structural schematic diagram of an electronic device for implementing the dual sparse matrix multiplication acceleration method of an embodiment of the present invention. DETAILED DESCRIPTION

[0032] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0033] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0034] Figure 1 The flowchart of a method for accelerating the multiplication of double sparse matrices provided by an embodiment of the present invention is applicable to the case where the multiplication of double sparse matrices is accelerated by using a specific sparse computing unit in a computing chip. The method can be executed by a device for accelerating the multiplication of double sparse matrices. The device can be implemented in the form of hardware and / or software and can generally be configured in an electronic device including the computing chip. Figure 1 As shown, the method includes:

[0035] S110. According to the sparse matrix multiplication requirement, locate a compressed matrix set and a second sparse matrix matching the first sparse matrix in the global memory of the computing chip.

[0036] Optionally, the computing chip can be understood as an integrated circuit used to implement a set computing task (for example, Internet of Things control, high-performance computing or mobile computing, etc.). The computing chip can be a general computing chip, such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), or a dedicated computing chip, such as an NPU (Neural Processing Unit), which is not limited in this embodiment.

[0037] In this embodiment, the above-mentioned dual sparse matrix multiplication acceleration method can be encapsulated in the form of an operator interface, for example, a sparse multiplication operator is specially constructed for implementing dual sparse matrix multiplication. Furthermore, the multiplication acceleration method can be triggered to execute by calling the operator interface of the sparse multiplication operator.

[0038] Specifically, double sparse matrix multiplication can be understood as the left operand and the right operand of the matrix multiplication are both sparse matrices. Accordingly, when an interface call request for the sparse multiplication operator is detected, it can be determined that a sparse matrix multiplication requirement is detected. Furthermore, the identification information of the left operand (hereinafter referred to as the first sparse matrix) and the right operand (hereinafter referred to as the second sparse matrix) that need to perform double sparse matrix multiplication can be obtained from the interface call request. Based on the above identification information, a compressed matrix set and a second sparse matrix that match the first sparse matrix can be located in the global memory of the computing chip.

[0039] It is understandable that the double sparse matrix is ​​multiplied in the computing chip. Furthermore, the first sparse matrix and the second sparse matrix to be calculated need to be preloaded into the computing chip. In order to improve the computing speed of the computing chip, the calculation data is initially stored in the global memory of the computing chip. Subsequently, the calculation data can be moved in blocks to the hardware registers adapted to the hardware computing unit in a step-by-step manner, and the hardware computing unit calculates the final calculation result based on the data blocks stored in the hardware registers.

[0040] In this embodiment, in order to achieve the final multiplication acceleration, the sparse format of the first sparse matrix as the left operand is specially limited. At the same time, the storage method of the first sparse matrix in the global memory of the computing chip is also optimized accordingly.

[0041] Among them, the first sparse matrix includes multiple compression units of set sizes, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the positions of valid rows in the first sparse matrix in the corresponding compression units, and a metadata matrix for storing the positions of non-zero data in the corresponding structured sparse computing storage units.

[0042] Specifically, the first sparse matrix contains an integer number (one or more) of compression units, that is, the size of the first sparse matrix (number of rows * number of columns) can divide the size of the compression unit. The compression unit can be understood as a small matrix of a specific size, which has at least two matrix rows. The at least two matrix rows have at least one valid row and at least one sparse row. A valid row can be understood as a matrix row containing at least one valid data (non-0), and a sparse row refers to a matrix row in which all data is 0.

[0043] Further, each valid row in the compression unit contains an integer number (one or more) of structured sparse computing storage units. Wherein, each structured sparse computing storage unit contained in the first sparse matrix is ​​in the same sparse mode. Generally speaking, in order to facilitate calculation, the size of the structured sparse computing storage unit can be adapted to the calculation scale of the hardware computing unit in the computing chip.

[0044] The sparse pattern can be understood as the ratio of valid data (non-zero value data) to all data. For example, the sparse pattern can be 2:4 or 4:8. That is, for a 2:4 sparse pattern, the structured sparse computing storage unit contains a total of 4 data, 2 of which are non-zero data. The arrangement positions of the above two non-zero data in the structured sparse computing storage unit are not restricted.

[0045] It is understandable that the first sparse matrix contains a large amount of zero-valued data. If the first sparse matrix is ​​directly stored in the global memory of the computing chip according to the original size of the first sparse matrix, a large amount of storage units will be wasted. In addition, the subsequent double sparse matrix multiplication is directly implemented based on the first sparse matrix of the original size, which has low computational efficiency and cannot efficiently use the hardware computing units in the computing chip, that is, the sparse computing units.

[0046] In view of this, in this embodiment, a new type of efficient data compression method is creatively proposed for the left operand (first sparse matrix) of the double sparse multiplication calculation of the above special structure to solve the above technical problems.

[0047] In this embodiment, the first sparse matrix can be compactly stored in the form of a compressed matrix set by a specific data compression method. Specifically, three compressed matrices corresponding to the first sparse matrix are stored in the compressed matrix set, namely, a compressed data matrix, an index matrix, and a metadata matrix.

[0048] Among them, the compressed data matrix is ​​used to store the compressed data matrix of non-zero data in the first sparse matrix, the index matrix is ​​used to store the position of the valid rows in the first sparse matrix in the corresponding compression unit, and the metadata matrix is ​​used to store the position of the non-zero data in the corresponding structured sparse computing storage unit.

[0049] Obviously, through the above-mentioned data compression method, the specific position of each non-zero data in the first sparse matrix in the original first sparse matrix can be clearly determined through three small matrices. This data compression method can also effectively reduce the consumption of storage resources in the computing chip, and can also facilitate the implementation of calculations based on the first sparse matrix.

[0050] In this embodiment, the second sparse matrix may not be stored in compression. However, considering the actual application requirements of sparse matrix multiplication, for example, in a hybrid expert model, the second sparse matrix generally also has a specific data structure. Typically, one or more matrix rows in the second sparse matrix are sparse rows (all 0).

[0051] Based on this, in addition to storing the second sparse matrix completely, the global memory of the computing chip further stores a selection array matching the second sparse matrix, which is used to describe the valid row positions in the second sparse matrix. In a specific example, assuming that in the second sparse matrix, except for the 0th row, the 3rd row and the 5th row, all are sparse rows, then a selection array of the form {0,3,5} can be constructed and associated with the second sparse matrix for storage.

[0052] Correspondingly, while locating the second sparse matrix in the global memory of the computing chip, a selection array matching the second sparse matrix can be synchronously located.

[0053] S120, moving the compressed matrix set and the second sparse matrix from the global memory to the hardware registers of the computing chip step by step in the form of data blocks.

[0054] In an optional implementation of this embodiment, the computing chip includes a three-level storage architecture of global memory->shared memory->hardware registers. The three-level storage is gradually approached to the hardware computing units used to perform calculations in the computing chip in terms of physical location, especially the hardware registers are arranged close to the hardware computing units.

[0055] Generally speaking, the closer the storage device is to the computing unit, the faster its data read and write speed is, but the smaller its storage capacity is. Therefore, for a matrix multiplication of C(M×N)=A(M×K)×B(K×N), the complete data of A and B cannot be fully stored in the shared memory or hardware registers, and often need to be moved in blocks to the hardware registers through multi-level transfer, so that the hardware computing unit in the computing chip can calculate the final complete multiplication result in batches.

[0056] In an optional implementation of the present embodiment, the special data structure of the compressed matrix set can be combined to perform data block processing on each matrix in the compressed matrix set. At the same time, in combination with a selection array matching the second sparse matrix, the second sparse matrix can be processed in data block to improve the efficiency of data handling and subsequent multiplication calculations.

[0057] S130. By using the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second sparse matrix are gradually calculated according to the data loaded in batches in the hardware register.

[0058] In this embodiment, the sparse computing unit can be understood as a special hardware circuit in the computing chip for performing matrix multiplication calculations on double sparse matrices. Such special hardware circuits can accelerate double sparse matrix multiplication. The computing chip generally includes multiple sparse computing units, and within each sparse computing unit, one or more threads can be opened to implement corresponding sparse computing tasks.

[0059] By using the sparse computing units in the computing chip in coordination, partial data in the first sparse matrix and the second sparse matrix can be loaded from each hardware register in batches for local calculation, and the local calculation results can be combined or accumulated to finally obtain the multiplication result of the first sparse matrix and the second sparse matrix.

[0060] The technical solution of the embodiment of the present invention is to compress and store the double sparse matrix that needs to be multiplied in a specific compression format, and then move the double sparse matrix in the above specific compression format from the global memory to the hardware register of the computing chip in the form of data blocks step by step, and use the sparse computing unit of the computing chip to gradually calculate the multiplication result of the double sparse matrix according to the data loaded in the hardware register in batches. The implementation method fully considers the sparse characteristics when performing matrix multiplication calculations based on the double sparse matrix. By adopting data blocking and data moving technology that are compatible with the sparse characteristics, the hardware acceleration performance of the sparse computing unit in the computing chip can be fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the double sparse matrix multiplication operation process, and is particularly suitable for model calculation scenarios of hybrid expert models.

[0061] Figure 2 A flowchart of another method for accelerating the multiplication of double sparse matrices provided by an embodiment of the present invention is provided. This embodiment is optimized based on the above embodiments. In this embodiment, the operation of "moving the compressed matrix set and the second sparse matrix from the global memory to the hardware register of the computing chip in the form of data blocks" is specifically concretized.

[0062] Correspondingly, such as Figure 2 As shown, the method may include:

[0063] S210. According to the sparse matrix multiplication requirement, locate a compressed matrix set and a second sparse matrix matching the first sparse matrix in the global memory of the computing chip.

[0064] The inventors have found through research that when a hybrid expert model is used to calculate a task (typically, a model reasoning task), the matrix multiplication calculation between the model weight matrix of each model layer and the intermediate activation value sparse matrix input to the hybrid model layer in the hybrid expert model belongs to double sparse matrix multiplication. Furthermore, the methods of the embodiments of the present invention can be applied in the model calculation scenario of the hybrid expert model.

[0065] In this model calculation scenario, the dual sparse matrix multiplication described in each embodiment of the present invention can be implemented by calling a pre-packaged sparse multiplication operator based on a specific model weight matrix and a matching intermediate activation value sparse matrix.

[0066] Accordingly, in an optional implementation of the embodiment of the present invention, the first sparse matrix is ​​a model weight matrix in the hybrid expert model, and the second sparse matrix is ​​an intermediate activation value sparse matrix input to the hybrid model layer in the hybrid expert model.

[0067] Furthermore, the first sparse matrix may specifically include a plurality of compression units of size M*V;

[0068] Specifically, each compression unit includes d valid rows, where 1≤d<M; and

[0069] Each structured sparse computing storage unit is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

[0070] Correspondingly, for the first sparse matrix of size m*k, a compressed data matrix of size (m / M*d)*(k / (L / N)), an index matrix of size (m / M*d)*(k / V), and a metadata matrix of size (m / M*d)*(k / (L / N)) can be obtained.

[0071] For ease of explanation, Figure 3A schematic diagram of the comparison structure of a first sparse matrix and a matching set of compressed matrices applicable to various embodiments of the present invention is shown in FIG.

[0072] Specifically, Figure 3 The diagram shown is a schematic diagram of a set of compressed matrices matching a first sparse matrix with m=4, k=16. In the first sparse matrix, there are 4 compression units with M=2, V=8, each of which contains d=1 valid rows, each of which contains 2 structured sparse computing storage units, and each structured sparse computing storage unit is in a sparse mode with N:L of 2:4. That is, A, B, ..., L filled in each matrix position of the first sparse matrix represent non-zero data, and the blank position represents the data filled with zero value.

[0073] Correspondingly, the size of the compressed data matrix corresponding to the first sparse matrix is ​​(m / M*d)*(k / (L / N))=2*8. The compressed data matrix stores the non-zero data A, B, ..., L in the first sparse matrix in sequence.

[0074] Furthermore, the size of the index matrix corresponding to the first sparse matrix is ​​(m / M*d)*(k / V)=2*2, and the positions of each valid row in the first sparse matrix in the corresponding compression unit are stored in different positions of the index matrix. Figure 3 As shown, the first sparse matrix includes a 2*8 compression unit 1 where A, B, C, and D are located. Since d=1, and the row position where the valid row in the compression unit 1 is located, that is, the row where A, B, C, and D are located, is the 0th row, and further, the above positional relationship can be identified by the 0 stored at the matrix position in the upper left corner of the index matrix.

[0075] Furthermore, the size of the metadata matrix corresponding to the first sparse matrix is ​​(m / M*d)*(k / (L / N))=2*8, and the positions of each non-zero data in the corresponding structured sparse computing storage unit are stored in different positions of the metadata matrix. Figure 3 As shown, the structured sparse computing storage unit 1 containing non-zero elements A and B is located in the upper left corner of the first sparse matrix. Furthermore, the 0 and 2 at the first two column positions of the first row of the metadata matrix represent the column positions of A and B in the structured sparse computing storage unit 1, respectively, with A being located in the 0th column and B being located in the 2nd column.

[0076] It is understandable that by constructing the above-mentioned compressed data matrix, index matrix and metadata data, the first sparse matrix of the above-mentioned specific structure can be uniquely determined. Furthermore, the sparse model weight matrices contained in the hybrid expert model can be efficiently compressed and stored, thereby effectively reducing the storage overhead of the hybrid expert model on the configured computing chip.

[0077] S220, dividing the compressed data matrix in the global memory into data blocks, and dividing the index matrix and the metadata matrix into data blocks that match each other according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix.

[0078] In this embodiment, since the compressed data matrix stores all non-zero data in the first sparse matrix, data can be partitioned according to the data size of the compressed data matrix. In the process of partitioning the compressed data matrix, in order to further consider the efficiency of subsequent data handling and multiplication calculations, it can be set that in the process of partitioning the compressed data matrix in the global memory, V (the number of columns contained in each compression unit) is an integer multiple of the partition size in the column partition direction.

[0079] For example, suppose that in global memory, the compressed data matrix is ​​split into multiple m b *k b In the process of data segmentation, V needs to be k. b An integer multiple of, for example, k b It is V / 2 or V / 4, etc.

[0080] As shown above, since there is a data or position correspondence between the compressed data matrix, the index matrix and the metadata matrix, after determining the compressed data matrix blocks cut out from the compressed data matrix, the index matrix blocks and the metadata matrix blocks that match the compressed data matrix blocks can be uniquely determined. That is, according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix, the data blocks that match the index matrix and the metadata matrix are matched.

[0081] S230 . Divide the second sparse matrix in the global memory into blocks according to the selection array.

[0082] As shown above, the second sparse matrix may contain a large number of sparse rows. If these sparse rows are also divided into data blocks and sent to the hardware computing unit for matrix multiplication calculation, this will result in the loading and calculation of redundant data, reducing the computing efficiency.

[0083] Based on this, in this embodiment, in the global memory, only the valid data rows in the second sparse matrix can be partitioned based on the valid row positions defined by the selection array. Optionally, one or more valid data rows can be taken as a data block each time, or a set amount of data in a valid data row can be taken as a data block each time.

[0084] S240, loading each compressed data matrix block, each index matrix block, and each metadata matrix block from the global memory into the shared memory one by one.

[0085] After completing the data partitioning for the compressed matrix set in the global memory, each compressed data matrix block, each index matrix block and each metadata matrix block may be loaded from the global memory to the shared memory one by one.

[0086] On the basis of the above embodiments, in order to prevent the data in the shared memory from being restricted by a bank conflict (bank conflict) in which the same storage body (also referred to as a bank) in the computing chip cannot be accessed by multiple threads at the same time during the access (read / write) process, in the embodiments of the present invention, it is further considered to rearrange the storage of data in the shared memory based on the data offset using a specific data rearrangement function.

[0087] Accordingly, in an optional implementation of this embodiment, after each compressed data matrix block, each index matrix block, and each metadata matrix block are successively moved from the global memory to the shared memory, the following may also be included:

[0088] Using a preset data rearrangement function, each compressed data matrix block, each index matrix block, and each metadata matrix block are rearranged in a shared memory to avoid storage conflicts.

[0089] Specifically, according to the specific type and model of the computing chip, a matching data rearrangement function can be selected in the adapted function library to rearrange the data in the shared memory, and this embodiment does not limit this.

[0090] S250, dividing each second sparse matrix into blocks, and moving them from the global memory to the shared memory one by one.

[0091] Similarly, after the data of the second sparse matrix is ​​divided into blocks in the global memory, the blocks of the second sparse matrix can be correspondingly moved from the global memory to the shared memory one by one.

[0092] Meanwhile, in order to avoid the problem of memory bank conflict, in an optional implementation of this embodiment, after each second sparse matrix is ​​divided into blocks and moved from the global memory to the shared memory one by one, the following steps may also be included:

[0093] Using a preset data rearrangement function, each second sparse matrix block is rearranged in a shared memory to avoid storage bank conflicts.

[0094] Of course, it is understandable that in addition to using the data rearrangement function to rearrange the compressed data matrix blocks, the index matrix blocks, the metadata matrix blocks and the second sparse matrix blocks in the shared memory, it is also possible to add data padding to the compressed data matrix blocks, the index matrix blocks, the metadata matrix blocks and the second sparse matrix blocks in the shared memory to avoid storage conflicts.

[0095] S260, performing secondary block division on each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory.

[0096] In this embodiment, in order to adapt to the storage limitation of the hardware register, each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory may be divided into two blocks.

[0097] Furthermore, when performing secondary partitioning on each compressed data matrix block, each index matrix block, and each metadata matrix block in the shared memory, the compressed data matrix block can also be firstly secondary partitioned. In the process of secondary partitioning of the compressed data matrix, in order to further consider the efficiency of subsequent data handling and multiplication calculation, it can be set that in the process of secondary partitioning of the compressed data matrix block in the shared memory, V is also an integer multiple of the partition size in the column partitioning direction. For example, secondary partitioning can be selected in the column partitioning direction or no secondary partitioning can be selected (partitioning is performed only in the row direction).

[0098] For example, suppose that in global memory, the compressed data matrix is ​​divided into multiple m blocks b1 *k b1 In the process of new data block size, V needs to be k b1 An integer multiple of, for example, k b1 It is V / 4 or V / 8 ​​etc.

[0099] Similarly, since there is a data or position correspondence between the compressed data matrix, the index matrix and the metadata matrix, after completing the secondary blocking of the compressed data matrix, the secondary blocking of the index matrix and the metadata matrix can be performed accordingly.

[0100] S270, performing secondary block division on each second sparse matrix block in the shared memory.

[0101] S280. According to the register mapping relationship matching the sparse computing unit, at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block is moved from the shared memory to the hardware register one by one.

[0102] For a more intuitive understanding, Figure 4 FIG. 2 shows a block diagram of a data segmentation and multi-stage transfer process applicable to an embodiment of the present invention. Figure 4 As shown, it is described that the compressed data matrix A in the first sparse matrix and the second sparse matrix B are divided into data blocks in the global memory and the shared memory respectively, and are moved step by step in the order of global memory->shared memory->register. After the calculation is completed by the sparse computing unit and written back step by step, the corresponding multiplication result matrix C is obtained in the global memory.

[0103] In this example, an example of data partitioning and data movement for matrix multiplication of any size C(M×N)=A(M×K)×B(K×N) is given. In this embodiment, in order to implement matrix multiplication calculation, the second sparse matrix B is swapped in rows and columns, that is, an array is selected to describe the position of the valid data column in the second sparse matrix B.

[0104] Specifically, in global memory, since the selection array only contains the locations of valid data columns (rows), the total amount of data in the selection array is len d Then the number of valid values ​​of n dimension in the second sparse matrix B is len d The compressed data matrix A is in m dimensions with m b The second sparse matrix B is divided into several blocks of uniform size. The number of valid column vectors in each block of data is the same as the number of n in the selection array. b Considering the storage size limit of the computing device, the compressed data matrix A and the second sparse matrix B will be further divided into k blocks along the k direction. b To evenly divide the data into different sizes, multiple sub-blocks in the k direction are multiplied and accumulated to obtain the final multiplication result matrix C block result. Based on this data division strategy, the size of the multiplication result matrix C calculated for each block data is m b ×n b , and the block calculation is a part of the final result of the multiplication result matrix C, which corresponds to the same row offset of the data block of the compressed data matrix A in the compressed data matrix A, and the same column offset of the data block of the second sparse matrix B in the selection array. The parallelism of the double sparse matrix multiplication operator is improved by each thread group being responsible for one block.

[0105] Similarly, in the shared memory, it also involves a process of performing a secondary block division on each data block transferred from the global memory, and after the block is moved to the register for storage, the adapted sparse computing unit performs a matrix multiplication calculation, which will not be described in detail here. In this embodiment, the matrix specification of the compressed data matrix A processed by the sparse computing unit once is mi*ki, and the matrix specification of the second sparse matrix B is ki*ni. In order to facilitate subsequent calculations, the aforementioned V is an integer multiple of ki.

[0106] In an optional implementation of the present embodiment, each compressed data matrix secondary block, each index matrix secondary block, and each metadata matrix secondary block may be moved to the register to implement calculations. Alternatively, considering that register resources are very valuable, only each compressed data matrix secondary block and each metadata matrix secondary block may be moved to the register to implement calculations, while the index matrix may be retained in the shared memory and retrieved from the shared memory only when needed.

[0107] It should be emphasized that, unlike the computing logic of general hardware computing units, when using sparse computing units to accelerate multiplication calculations, the data to be calculated needs to be placed in an adapted instruction register in accordance with the requirements of the sparse instruction set. Therefore, each embodiment of the present invention needs to move the data in the shared memory to the matching hardware register based on the register mapping relationship that matches the sparse computing unit.

[0108] Optionally, the register mapping relationship can be read from a specification file of the factory configuration of the computing chip, and the register hardware file describes the mapping relationship between the computing data at different locations and the hardware registers.

[0109] Specifically, in Figure 5 , a schematic diagram of a register mapping relationship applicable to an embodiment of the present invention is shown in FIG. , and the register mapping relationship is adapted to the secondary block division of the compressed data matrix. Figure 5 As shown, the T0{a0, a1} in the upper left corner of the register mapping relationship represents: the data in the 0th column and the 1st column of the 0th row in the secondary block of the compressed data matrix are allocated to registers a0 and a1 matching thread T0. Similarly, the T0 in the last position of row 0 in the register mapping relationship 0…3 {a4, a5} represents: the data of the 8th to 15th columns of the 0th row in the secondary block of the compressed data matrix are respectively allocated to register a4 and register a5 matching thread T0, thread T1, thread T2 and thread T3. Different threads T are allocated to different sparse computing units in advance.

[0110] Similarly, the sparse computing unit also has a corresponding register mapping relationship for the metadata matrix secondary blocking and the index matrix secondary blocking. Based on the above register mapping relationship, it is possible to move each calculation data from the shared memory to the hardware register one by one.

[0111] In an optional implementation of the present embodiment, a preset hardware instruction can be called according to a register mapping relationship matching the sparse computing unit to move each compressed data matrix secondary block and each metadata matrix secondary block, or each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from a shared memory to a hardware register one by one, so as to realize hardware acceleration of data loading.

[0112] The hardware instruction is associated with the specification of the computing chip and can be queried and obtained from the instruction library of the computing chip. For example, it can be a 1dmatrix instruction.

[0113] S290 . Divide each second sparse matrix into blocks according to a register mapping relationship matched with the sparse computing unit, and load the blocks from the shared memory to the hardware registers one by one.

[0114] As shown above, by querying and obtaining the register mapping relationship set by the sparse computing unit for the second sparse matrix secondary block, each second sparse matrix secondary block can be loaded from the shared memory to the hardware register one by one.

[0115] Similarly, in an optional implementation of this embodiment, the preset hardware instructions can be called according to the register mapping relationship matching the sparse computing unit to divide each second sparse matrix into secondary blocks and load them one by one from the shared memory to the hardware register to achieve hardware acceleration of data loading.

[0116] S2100. By using the sparse computing unit of the computing chip, the multiplication results of the first sparse matrix and the second sparse matrix are gradually calculated according to the data loaded in batches in the hardware register.

[0117] In an optional implementation of this embodiment, by using a sparse computing unit of a computing chip, gradually calculating the multiplication result of the first sparse matrix and the second sparse matrix according to the data loaded in batches in the hardware register may include:

[0118] Obtaining, through the sparse computing unit, the current compressed data matrix secondary block, the current metadata matrix secondary block, and the current second sparse matrix secondary block currently loaded in the hardware register;

[0119] Generate a primary sparse block matrix according to the secondary block of the current metadata matrix and the secondary block of the current compressed data matrix through a sparse computing unit;

[0120] The multiplication calculation is performed by the sparse computing unit according to the primary sparse block matrix and the secondary block of the current second sparse matrix.

[0121] As mentioned above, when compressing and storing the first sparse matrix, the compressed data matrix only stores the non-zero data in each valid row, while in the actual first sparse matrix, each valid row contains at least one structured sparse computing storage unit in a set sparse mode. In order to ensure the accuracy of the multiplication calculation, it is necessary to first restore the current compressed data matrix secondary block to the form of the aforementioned structured sparse computing storage unit. Since the metadata matrix stores the position of the non-zero data in the structured sparse computing storage unit, a sparse block matrix can be generated based on the current metadata matrix secondary block and the current compressed data matrix secondary block.

[0122] That is, the primary sparse block matrix can be understood as a matrix after restoring the structured sparse computing storage unit in the secondary block of the current compressed data matrix. In a specific example, if the secondary block of the current compressed data matrix is ​​{A, B}, the corresponding secondary block of the current metadata matrix is ​​{0, 2}, and then, the primary sparse block matrix of the form {A, 0, B, 0} can be restored.

[0123] Among them, by the sparse computing unit, according to the primary sparse block matrix and the current second sparse matrix secondary block, performing multiplication calculation, which may specifically include:

[0124] The intermediate calculation amount generated in the multiplication calculation process is stored in the intermediate register of the computing chip through the sparse computing unit; and the matching multiplication calculation is performed by obtaining the intermediate calculation amount from the intermediate register through the sparse computing unit.

[0125] When the prior art uses a general computing unit to perform multiplication calculations, the results of the matrix multiplication calculations can be directly accumulated, but this essentially adds a lot of redundant data calculations. In contrast, when the sparse computing units of the various embodiments of the present invention perform sparse multiplication calculations, when the multiplication calculations are iterated along the K direction, since the sparse data comes from different rows, in order to ensure correctness, when different sub-blocks move along the K direction, the output results need to be mapped to different rows. Traditionally, passing the output results to a specific register based on the index may cause the output C matrix to be reloaded into the local memory, which has a significant impact on the performance of the operator.

[0126] In order to avoid the above problems, each embodiment of the present invention introduces an additional intermediate register C IR The frequent memory transfer between global memory and registers is reduced by Figure 6FIG. 2 shows a schematic diagram of a structure in which a sparse computing unit and an intermediate register cooperate to implement a calculation in accordance with an embodiment of the present invention. Figure 6 As shown, in the sparse computing unit, the current compressed data matrix is ​​divided into two blocks ( Figure 6 A in), the current metadata matrix is ​​divided into two blocks ( Figure 6 metadata in ) and the first second sparse matrix secondary block ( Figure 6 B) When performing sparse matrix multiplication calculations, for multiple result data, the required intermediate results are loaded into C during the calculation process. IR and use the intermediate results for calculation, and store the intermediate results back to the corresponding result data location C after the calculation is completed.

[0127] By adding a new intermediate register to the computing chip, which is a simple hardware improvement, the computing performance and efficiency can be greatly improved during the double sparse matrix multiplication calculation in the sparse computing unit.

[0128] On the basis of the above embodiments, after performing multiplication calculation according to the primary sparse block matrix and the secondary block of the current second sparse matrix by the sparse computing unit, the method may further include:

[0129] After determining that the complete multiplication calculation result is stored in the intermediate register, the complete multiplication calculation result is converted into a secondary sparse block matrix according to the secondary block of the current index matrix that matches the secondary block of the current compressed data matrix; the secondary sparse block matrix is ​​written back to the global memory of the computing chip level by level.

[0130] Furthermore, since the sparse rows in the first sparse matrix are not considered when the complete multiplication result is calculated, it is necessary to obtain the complete multiplication result, combine the current index matrix secondary block that matches the current compressed data matrix secondary block, and add the matching sparse rows to the complete multiplication result to obtain the secondary sparse block matrix as the real block calculation result. As mentioned above, the current index matrix secondary block can also be transferred to the hardware register, or it can be stored only in the shared memory, and this embodiment does not limit this.

[0131] The technical solution of the embodiment of the present invention fully considers the sparse characteristics when performing matrix multiplication calculations based on double sparse matrices. By adopting data partitioning and data handling technologies that are compatible with the sparse characteristics, the hardware acceleration performance of the sparse computing unit in the computing chip can be fully utilized, thereby greatly optimizing the computing, bandwidth and storage resource overheads in the double sparse matrix multiplication operation process, and is particularly suitable for model calculation scenarios of hybrid expert models.

[0132] Figure 7The structure diagram of a dual sparse matrix multiplication acceleration device provided by an embodiment of the present invention is as follows. Figure 7 As shown, the device includes: a multiplication element positioning module 710, a step-by-step transport module 720 and a multiplication calculation module 730, wherein:

[0133] A multiplication element locating module 710, configured to locate a set of compressed matrices and a second sparse matrix matching the first sparse matrix in a global memory of a computing chip according to a sparse matrix multiplication requirement;

[0134] The first sparse matrix includes a plurality of compression units of a set size, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit;

[0135] A step-by-step transfer module 720 is used to transfer the compressed matrix set and the second sparse matrix from the global memory to the hardware register of the computing chip step by step in the form of data blocks;

[0136] The multiplication calculation module 730 is used to gradually calculate the multiplication results of the first sparse matrix and the second sparse matrix according to the data loaded in batches in the hardware register through the sparse calculation unit of the computing chip.

[0137] The technical solution of the embodiment of the present invention is to compress and store the double sparse matrix that needs to be multiplied in a specific compression format, and then move the double sparse matrix in the above specific compression format from the global memory to the hardware register of the computing chip in the form of data blocks step by step, and use the sparse computing unit of the computing chip to gradually calculate the multiplication result of the double sparse matrix according to the data loaded in the hardware register in batches. The implementation method fully considers the sparse characteristics when performing matrix multiplication calculations based on the double sparse matrix. By adopting data blocking and data moving technology that are compatible with the sparse characteristics, the hardware acceleration performance of the sparse computing unit in the computing chip can be fully utilized, thereby greatly optimizing the calculation, bandwidth and storage resource overhead in the double sparse matrix multiplication operation process, and is particularly suitable for model calculation scenarios of hybrid expert models.

[0138] Based on the above embodiments, the step-by-step transport module 720 may specifically include:

[0139] The first type of primary blocking unit is used to block the compressed data matrix in the global memory, and block the index matrix and the metadata matrix according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix;

[0140] The first type of primary transfer unit is used to load each compressed data matrix block, each index matrix block and each metadata matrix block from the global memory to the shared memory one by one;

[0141] The first type of secondary blocking unit is used to perform secondary blocking on each compressed data matrix block, each index matrix block and each metadata matrix block in the shared memory;

[0142] The first type of secondary transfer unit is used to move at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from the shared memory to the hardware register one by one according to the register mapping relationship matching the sparse computing unit.

[0143] On the basis of the above embodiments, the following may also be included:

[0144] A selection array positioning module is used to locate a set of compressed matrices matching the first sparse matrix and a second sparse matrix in the global memory of the computing chip, and at the same time locate a selection array matching the second sparse matrix in the computing chip, wherein the selection array is used to describe valid row positions in the second sparse matrix;

[0145] Accordingly, the step-by-step transport module 720 may further include:

[0146] The second type of first-level blocking unit is used to block the second sparse matrix in the global memory according to the selection array;

[0147] The second type of first-level transfer unit is used to transfer each second sparse matrix block from the global memory to the shared memory one by one;

[0148] The second type of secondary blocking unit is used to perform secondary blocking on each second sparse matrix block in the shared memory;

[0149] The second type of secondary transfer unit is used to divide each second sparse matrix into blocks according to the register mapping relationship matched with the sparse computing unit, and load them from the shared memory to the hardware register one by one.

[0150] On the basis of the above embodiments, the present invention may further include a first type of rearrangement unit, which is used to:

[0151] After moving each compressed data matrix block, each index matrix block, and each metadata matrix block from the global memory to the shared memory one by one, using a preset data rearrangement function, rearrange each compressed data matrix block, each index matrix block, and each metadata matrix block in the shared memory to avoid storage bank conflicts; and

[0152] The invention may further include a second type of rearrangement unit, which is used for:

[0153] After each second sparse matrix block is moved from the global memory to the shared memory one by one, a preset data rearrangement function is used to rearrange each second sparse matrix block in the shared memory to avoid storage conflicts.

[0154] Based on the above embodiments, the first type of secondary moving unit can be specifically used for:

[0155] According to the register mapping relationship matched with the sparse computing unit, the preset hardware instructions are called to move each compressed data matrix secondary block and each metadata matrix secondary block, or each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from the shared memory to the hardware register one by one, so as to realize the hardware acceleration of data loading; and

[0156] The second type of secondary transfer unit can be specifically used for:

[0157] According to the register mapping relationship matching the sparse computing unit, the preset hardware instructions are called to divide each second sparse matrix into blocks and load them from the shared memory to the hardware registers one by one to achieve hardware acceleration of data loading.

[0158] Based on the above embodiments, the multiplication calculation module 730 may specifically include:

[0159] A multiplication calculation element acquisition unit, used for acquiring the current compressed data matrix secondary block, the current metadata matrix secondary block and the current second sparse matrix secondary block currently loaded in the hardware register through the sparse calculation unit;

[0160] A primary sparse block matrix generating unit is used to generate a primary sparse block matrix according to the secondary block of the current metadata matrix and the secondary block of the current compressed data matrix through the sparse computing unit;

[0161] The multiplication calculation execution unit is used to perform multiplication calculation according to the primary sparse block matrix and the current second sparse matrix secondary block through the sparse calculation unit.

[0162] Based on the above embodiments, the multiplication calculation execution unit can be specifically used for:

[0163] The intermediate calculation amount generated in the multiplication calculation process is stored in the intermediate register of the computing chip through the sparse computing unit; and

[0164] The sparse computing unit performs the matching multiplication calculation by acquiring the intermediate calculation amount from the intermediate register.

[0165] Based on the above embodiments, a data write-back module may be further included, which is used to:

[0166] After performing multiplication calculations according to the primary sparse block matrix and the current second sparse matrix secondary block by the sparse computing unit, after determining that the intermediate register stores the complete multiplication calculation result, converting the complete multiplication calculation result into a secondary sparse block matrix according to the current index matrix secondary block matching the current compressed data matrix secondary block;

[0167] The secondary sparse block matrix is ​​written back to the global memory of the computing chip level by level.

[0168] Based on the above embodiments, the first sparse matrix may specifically include a plurality of compression units of size M*V;

[0169] Specifically, each compression unit includes d valid rows, where 1≤d<M; and

[0170] Each structured sparse computing storage unit is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

[0171] On the basis of the above embodiments, in each data segmentation process, V is an integer multiple of the segmentation size in the column segmentation direction; and

[0172] The matrix size of the first sparse matrix processed once in each sparse computing unit is mi*ki, where V is an integer multiple of ki.

[0173] Based on the above embodiments, the first sparse matrix is ​​a model weight matrix in the hybrid expert model, and the second sparse matrix is ​​an intermediate activation value sparse matrix input to the hybrid model layer in the hybrid expert model.

[0174] The double sparse matrix multiplication acceleration device provided in the embodiment of the present invention can execute the double sparse matrix multiplication acceleration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0175] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0176] Figure 8 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0177] like Figure 8 As shown, the electronic device 10 includes at least one computing chip 11, and a memory connected to the at least one computing chip 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one computing chip, and the computing chip 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The computing chip 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. The input / output (I / O) interface 15 is also connected to the bus 14.

[0178] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0179] The computing chip 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing chip 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various computing chips running machine learning model algorithms, digital signal computing chips (DSPs), and any appropriate computing chips, controllers, microcontrollers, etc. The computing chip 11 executes the various methods and processes described above, such as executing the multiplication acceleration method of the double sparse matrix as described in any one of the embodiments of the present invention.

[0180] That is: according to the sparse matrix multiplication requirement, a compressed matrix set matching the first sparse matrix and a second sparse matrix are located in the global memory of the computing chip;

[0181] The first sparse matrix includes a plurality of compression units of a set size, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit;

[0182] The compressed matrix set and the second sparse matrix are moved from the global memory to the hardware registers of the computing chip in the form of data blocks;

[0183] The multiplication results of the first sparse matrix and the second sparse matrix are gradually calculated by the sparse computing unit of the computing chip according to the data loaded in batches in the hardware register.

[0184] In some embodiments, the method for accelerating the multiplication of double sparse matrices as described in any one of the embodiments of the present invention may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the computing chip 11, one or more steps of the method for accelerating the multiplication of double sparse matrices as described in any one of the embodiments of the present invention described above may be executed. Alternatively, in other embodiments, the computing chip 11 may be configured to execute the method for accelerating the multiplication of double sparse matrices as described in any one of the embodiments of the present invention by any other appropriate means (e.g., by means of firmware).

[0185] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable computing chip, which can be a dedicated or general-purpose programmable computing chip that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0186] The computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a computing chip of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that when the computer program is executed by the computing chip, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0187] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0188] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0189] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0190] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0191] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0192] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for accelerating the multiplication of double sparse matrices, characterized in that: include: According to the sparse matrix multiplication requirement, a compressed matrix set and a second sparse matrix matching the first sparse matrix are located in the global memory of the computing chip; The first sparse matrix includes a plurality of compression units of a set size, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit; The compressed matrix set and the second sparse matrix are moved from the global memory to the hardware registers of the computing chip in the form of data blocks; The multiplication results of the first sparse matrix and the second sparse matrix are gradually calculated by the sparse computing unit of the computing chip according to the data loaded in batches in the hardware register.

2. The method according to claim 1, characterized in that The compressed matrix set is moved from the global memory to the hardware registers of the computing chip in the form of data blocks, including: The compressed data matrix in the global memory is divided into data blocks, and according to the data correspondence between the compressed data matrix, the index matrix and the metadata matrix, the index matrix and the metadata matrix are matched into data blocks; Load each compressed data matrix block, each index matrix block, and each metadata matrix block from the global memory to the shared memory one by one; Secondarily partition each compressed data matrix block, each index matrix block, and each metadata matrix block in the shared memory; According to the register mapping relationship matched with the sparse computing unit, at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block is moved from the shared memory to the hardware register one by one.

3. The method according to claim 2, characterized in that While locating the compressed matrix set and the second sparse matrix matching the first sparse matrix in the global memory of the computing chip, the method further includes: Locating a selection array matching the second sparse matrix in the computing chip, wherein the selection array is used to describe valid row positions in the second sparse matrix; Accordingly, the second sparse matrix is ​​moved from the global memory to the hardware register of the computing chip in the form of data blocks, step by step, including: According to the selection array, the second sparse matrix in the global memory is divided into blocks; Divide each second sparse matrix into blocks and move them from the global memory to the shared memory one by one; Perform secondary block division on each second sparse matrix block in the shared memory; According to the register mapping relationship matched with the sparse computing unit, each second sparse matrix is ​​divided into blocks twice and loaded from the shared memory into the hardware register one by one.

4. The method according to claim 3, characterized in that After each compressed data matrix block, each index matrix block, and each metadata matrix block are successively moved from the global memory to the shared memory, the following further includes: Using a preset data rearrangement function, rearrange each compressed data matrix block, each index matrix block, and each metadata matrix block in a shared memory to avoid storage bank conflicts; and After each second sparse matrix is ​​divided into blocks and moved from the global memory to the shared memory one by one, the method further includes: Using a preset data rearrangement function, each second sparse matrix block is rearranged in a shared memory to avoid storage bank conflicts.

5. The method according to claim 2, characterized in that: According to the register mapping relationship matched with the sparse computing unit, at least one item of each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block is moved from the shared memory to the hardware register one by one, including: According to the register mapping relationship matched with the sparse computing unit, the preset hardware instructions are called to move each compressed data matrix secondary block and each metadata matrix secondary block, or each compressed data matrix secondary block, each index matrix secondary block and each metadata matrix secondary block from the shared memory to the hardware register one by one, so as to realize the hardware acceleration of data loading; and According to the register mapping relationship matching the sparse computing unit, each second sparse matrix is ​​divided into blocks and loaded from the shared memory into the hardware register one by one, including: According to the register mapping relationship matching the sparse computing unit, the preset hardware instructions are called to divide each second sparse matrix into blocks and load them from the shared memory to the hardware registers one by one to achieve hardware acceleration of data loading.

6. The method according to claim 3, characterized in that The multiplication results of the first sparse matrix and the second sparse matrix are calculated step by step according to the data loaded in the hardware register in batches by the sparse computing unit of the computing chip, including: Obtaining, through the sparse computing unit, the current compressed data matrix secondary block, the current metadata matrix secondary block, and the current second sparse matrix secondary block currently loaded in the hardware register; Generate a primary sparse block matrix according to the secondary block of the current metadata matrix and the secondary block of the current compressed data matrix through a sparse computing unit; The multiplication calculation is performed by the sparse computing unit according to the primary sparse block matrix and the secondary block of the current second sparse matrix.

7. The method according to claim 6, characterized in that The sparse computing unit performs a multiplication calculation according to the primary sparse block matrix and the secondary block of the current second sparse matrix, specifically including: The intermediate calculation amount generated in the multiplication calculation process is stored in the intermediate register of the computing chip through the sparse computing unit; and The sparse computing unit performs the matching multiplication calculation by acquiring the intermediate calculation amount from the intermediate register.

8. The method according to claim 7, characterized in that After performing multiplication calculation according to the primary sparse block matrix and the current second sparse matrix secondary block through the sparse computing unit, the method further includes: After determining that the intermediate register stores the complete multiplication calculation result, converting the complete multiplication calculation result into a secondary sparse block matrix according to the current index matrix secondary block matching the current compressed data matrix secondary block; The secondary sparse block matrix is ​​written back to the global memory of the computing chip level by level.

9. The method according to any one of claims 1 to 8, characterized in that: The first sparse matrix specifically includes a plurality of compression units of size M*V; Specifically, each compression unit includes d valid rows, where 1≤d<M; and Each structured sparse computing storage unit is in a sparse mode of N:L, where L is the total amount of data contained in the structured sparse computing storage unit, and N is the amount of non-zero data contained in the structured sparse computing storage unit.

10. The method according to claim 9, characterized in that In each data segmentation process, V is an integer multiple of the segmentation size in the column segmentation direction; and The matrix size of the first sparse matrix processed once in each sparse computing unit is mi*ki, where V is an integer multiple of ki.

11. The method according to any one of claims 1 to 8, characterized in that: The first sparse matrix is ​​a model weight matrix in the hybrid expert model, and the second sparse matrix is ​​an intermediate activation value sparse matrix input to the hybrid model layer in the hybrid expert model.

12. A dual sparse matrix multiplication acceleration device, characterized in that: include: A multiplication element locating module, used for locating a set of compressed matrices and a second sparse matrix matching the first sparse matrix in a global memory of a computing chip according to a sparse matrix multiplication requirement; The first sparse matrix includes a plurality of compression units of a set size, each compression unit includes at least one valid row and at least one sparse row, each valid row includes at least one structured sparse computing storage unit; each structured sparse computing storage unit is in the same sparse mode; the compression matrix set includes a compression data matrix for storing non-zero data in the first sparse matrix, an index matrix for storing the position of the valid row in the first sparse matrix in the corresponding compression unit, and a metadata matrix for storing the position of the non-zero data in the corresponding structured sparse computing storage unit; A step-by-step transfer module is used to transfer the compressed matrix set and the second sparse matrix from the global memory to the hardware register of the computing chip step by step in the form of data blocks; The multiplication calculation module is used to gradually calculate the multiplication results of the first sparse matrix and the second sparse matrix according to the data loaded in batches in the hardware register through the sparse calculation unit of the calculation chip.

13. An electronic device, characterized in that: The electronic device comprises: at least one computing chip; and A memory communicatively connected to the at least one computing chip; wherein, The memory stores a computer program that can be executed by the at least one computing chip, and the computer program is executed by the at least one computing chip so that the at least one computing chip can execute the multiplication acceleration method of double sparse matrices described in any one of claims 1-11.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the multiplication acceleration method of double sparse matrices described in any one of claims 1-11 when executed by a computing chip.

15. A computer program product, characterized in that The computer program product comprises a computer program which, when executed by a computing chip, implements the method for accelerating the multiplication of double sparse matrices according to any one of claims 1 to 11.

Images