Matrix calculation device, method, system, circuit, chip and equipment

Abstract

The invention discloses a matrix calculation device, method, system, circuit, chip and equipment, which can be used for directly calculating a matrix in a compression format so as to improve the calculation efficiency of the matrix in the compression format. The matrix calculation device comprises a vector outer product processing engine and an accumulator, and the matrix calculation device calculates a first matrix and a second matrix in a compression format based on a vector outer product. The vector outer product processing engine performs vector outer product calculation on a first column vector with row coordinates reserved and a second column vector with column coordinates reserved, and then the accumulator accumulates third element values with the same position coordinates based on the index of the position coordinates, so that a result matrix obtained by calculating two matrixes in compressed formats is obtained; compared with a traditional method that a matrix in a compression format needs to be decompressed firstly and then matrix calculation is conducted on the decompressed matrix, the matrix calculation device provided by the embodiment of the invention can effectively improve the calculation efficiency of the matrix in the compression format.

Description

[0001] This application is a divisional application. The application number of the original application is 202110181498.6, and the original application date is February 8, 2021. The entire contents of the original application are incorporated into this application by reference. Technical Field

[0002] The present application relates to the field of computers, and in particular to a matrix computing device, method, system, circuit, chip and equipment. Background Art

[0003] Matrix computing is an important type of computing in different application scenarios such as artificial intelligence, scientific computing, and graphics computing. A matrix is ​​a collection of element values ​​arranged in a rectangular array. The element values ​​in the matrix may contain two types of values: zero element values ​​and non-zero element values. When there are a large number of zero element values ​​in the matrix, in order to save storage space, only the non-zero element values ​​in the matrix can be stored, that is, the matrix can be compressed and stored in a compressed format.

[0004] In current technology, the common method for calculating matrices in compressed format is to first decompress the compressed matrix, that is, convert the compressed matrix into an uncompressed matrix, and then perform matrix calculations on the uncompressed matrix. During the matrix calculation process, since the compressed matrix needs to be decompressed, and the decompressed data will occupy a very large memory space, the calculation speed of the matrix will be limited by the memory access bandwidth. When the memory access bandwidth is fixed, the calculation speed of the matrix cannot be improved, resulting in low computing efficiency. Summary of the invention

[0005] The present application provides a matrix calculation device, method, system, circuit, chip and equipment, which can directly calculate the matrix in compressed format without decompressing the matrix in compressed format, thereby improving the calculation performance of the matrix in compressed format.

[0006] In a first aspect, an embodiment of the present application provides a matrix calculation device, which includes: a vector outer product processing engine and an accumulator; the vector outer product processing engine is used to calculate the vector outer products of N first column vectors and N first row vectors to obtain N intermediate result matrices, the first column vector includes a first element value and a row coordinate of the first element value, the first row vector includes a second element value and a column coordinate of the second element value, the intermediate result matrix includes a third element value and a position coordinate of the third element value, the position coordinate includes the row coordinate of the first element value and the column coordinate of the second element value; wherein the N first column vectors are obtained by converting a first matrix in a compressed format, the N first row vectors are obtained by converting a second matrix in a compressed format, and N is an integer greater than or equal to 1; the accumulator is used to accumulate the third element values ​​with the same position coordinates in the N intermediate result matrices according to the index of the position coordinate of the third element value to obtain a result matrix. In an embodiment of the present application, a matrix calculation device calculates a first matrix and a second matrix in a compressed format based on a vector outer product. During the calculation process, the row coordinates of the element values ​​in the first column vector are retained, and the column coordinates of the element values ​​in the second column vector are retained. Then, based on the index of the position coordinates, the third element values ​​with the same position coordinates are accumulated to obtain a result matrix of the calculation of the two matrices in the compressed format. Compared with the traditional method of first decompressing the matrix in the compressed format and then performing matrix calculation on the decompressed matrix, the matrix calculation device provided in the embodiment of the present application can effectively improve the computing efficiency of the matrix in the compressed format.

[0007] In an optional implementation, the N intermediate result matrices include at least a first intermediate result matrix and a second intermediate result matrix, the position coordinates of the third element value in the first intermediate result matrix are the first position coordinates, and the position coordinates of the third element value in the second intermediate result matrix are the second position coordinates; the accumulator is used to write the third element value in the first intermediate result matrix into the corresponding position in the buffer according to the first position coordinates in the generation order of the N intermediate result matrices; and then read the cache value of the second position coordinate in the corresponding position in the buffer according to the second position coordinate of the third element value in the second intermediate result matrix, and accumulate the third element value and the cache value in the second intermediate result matrix to obtain a result matrix in an uncompressed format. In the above optional implementation, the matrix calculation device can output a matrix in an uncompressed format according to different application scenarios, which increases the applicable scenarios of the matrix calculation device.

[0008] In an optional implementation, the matrix calculation device further includes a matrix compression unit; the matrix compression unit is used to compress the result matrix in an uncompressed format to obtain a result matrix in a compressed format. In the above optional implementation, the matrix calculation device can output a matrix in a compressed format according to different application scenarios, which increases the applicable scenarios of the matrix calculation device. And the matrix calculation device compresses the result matrix and outputs a matrix in a compressed format, thereby saving transmission resources or facilitating the next calculation operation.

[0009] In an optional implementation, the accumulator is also specifically used to: sort the third element values ​​in the N intermediate result matrices according to the position coordinates of the third element values, for example, sort them according to the row coordinates of the third element values, or sort them according to the column coordinates of the third element values; then compare the position coordinates in the sorted N intermediate result matrices, add the third element values ​​with the same position coordinates, and delete the position coordinates of the zero element value to obtain a result matrix in a compressed format. In the above optional implementation, a matrix in a compressed format can be directly obtained, and the matrix in a compressed format output by the matrix calculation device can be applied to certain application scenarios where a matrix in a compressed format is required for subsequent calculations. And because the matrix calculation device outputs a matrix in a compressed format, the transmission resources of the subsequent transmission matrix can be reduced.

[0010] In an optional implementation, the matrix calculation device also includes a format conversion unit; the format conversion unit is used to obtain the first matrix and the second matrix, convert the first matrix into N first column vectors, and retain the row coordinates of the first element value in the first column vector, and convert the second matrix into N first row vectors, and retain the column coordinates of the first element value in the first row vector, so that the matrix calculation device can calculate the two compressed format matrices based on the vector outer product.

[0011] In an optional implementation, the matrix calculation device further includes a format conversion unit; the format conversion unit is used to obtain a fifth matrix and a sixth matrix, perform format conversion on the fifth matrix to obtain a first matrix, and perform format conversion on the sixth matrix to obtain a second matrix, wherein at least one of the fifth matrix and the sixth matrix is ​​a matrix in an uncompressed format. In the above optional implementation, the matrix calculation device can receive a matrix in an uncompressed format, and then convert the matrix in the uncompressed format into a matrix in a compressed format, so that the matrix calculation device can support matrix calculations in multiple formats.

[0012] In an optional implementation, the matrix calculation device further includes a format conversion unit, which is further used to split the first column vector into X second column vectors and split the first row vector into X second row vectors, wherein the precision of the element values ​​included in the second column vector and the second row vector is the second precision, the precision of the element values ​​included in the first column vector and the first row vector is the first precision, and the first precision is higher than the second precision, and X is an integer greater than or equal to 2; the vector outer product processing engine is further used to calculate the vector outer product of the X second column vectors and the X second row vectors to obtain X 2 A fourth matrix is ​​formed, the fourth matrix includes a fourth element value and a position coordinate of the fourth element value, the position coordinate of the fourth element value includes a row coordinate of the first element value and a column coordinate of the second element value, and the precision of the fourth element value is the first precision; then the accumulator is further used to convert X according to the index of the position coordinate of the fourth element value 2 The fourth element values ​​with the same position coordinates in the fourth matrices are accumulated to obtain an intermediate result matrix, and the precision of the third element value in the intermediate result matrix is ​​the first precision. In the above optional implementation, the matrix calculation device can realize high-precision matrix calculation based on the low-precision matrix calculation device, thereby improving the applicability of the matrix calculation unit.

[0013] In a second aspect, an embodiment of the present application provides a matrix calculation method, which is applied to a matrix calculation device, and the method includes: first, obtaining a first calculation instruction, the first calculation instruction includes N first column vectors and N first row vectors; then, calculating the vector outer product of the N first column vectors and the N first row vectors to obtain N intermediate result matrices, wherein the first column vector includes a first element value and a row coordinate of the first element value, the first row vector includes a second element value and a column coordinate of the second element value, the intermediate result matrix includes a third element value and a position coordinate of the third element value, and the position coordinate includes the row coordinate of the first element value and the column coordinate of the second element value; the N first column vectors are obtained by converting a first matrix in a compressed format, and the N first row vectors are obtained by converting a second matrix in a compressed format, and N is an integer greater than or equal to 1; finally, according to the index of the position coordinate of the third element value, the third element values ​​with the same position coordinates in the N intermediate result matrices are added to obtain a result matrix. In an embodiment of the present application, a first matrix and a second matrix are calculated based on the vector outer product of N first column vectors and N first row vectors. During the calculation process, the row coordinates of the element values ​​in the first column vector are retained, and the column coordinates of the element values ​​in the second column vector are retained. Then, based on the index of the position coordinates, the third element values ​​with the same position coordinates are accumulated to obtain a result matrix of the calculation of two matrices in compressed formats. Compared with the traditional method of first decompressing the matrix in compressed format and then performing matrix calculation on the decompressed matrix, the matrix calculation device provided in the embodiment of the present application can effectively improve the computing efficiency of the matrix in compressed format.

[0014] In an optional implementation, the N intermediate result matrices include at least a first intermediate result matrix and a second intermediate result matrix, the position coordinates of the third element value in the first intermediate result matrix are the first position coordinates, and the position coordinates of the third element value in the second intermediate result matrix are the second position coordinates; in this method, the third element values ​​with the same position coordinates in the N intermediate result matrices are accumulated according to the index of the position coordinates of the third element value, and the result matrix can be obtained by specifically including: first, according to the generation order of the N intermediate result matrices, the third element value in the first intermediate result matrix is ​​written into the corresponding position in the buffer according to the first position coordinate; then, further according to the second position coordinate of the third element value in the second intermediate result matrix, the cache value of the corresponding position in the buffer of the second position coordinate is read, and the third element value and the cache value in the second intermediate result matrix are accumulated to obtain the result matrix in an uncompressed format. In the above optional implementation, the matrix calculation device can output a matrix in an uncompressed format according to different application scenarios, which increases the applicable scenarios of the matrix calculation device.

[0015] In an optional implementation, the method further includes: compressing the result matrix in an uncompressed format to obtain a result matrix in a compressed format. In the above optional implementation, the matrix calculation device can output a matrix in a compressed format according to different application scenarios, thereby increasing the applicable scenarios of the matrix calculation device. In addition, the matrix calculation device compresses the result matrix and outputs a matrix in a compressed format, thereby saving transmission resources or facilitating the next calculation operation.

[0016] In an optional implementation, the method accumulates the third element values ​​with the same position coordinates in N intermediate result matrices according to the index of the position coordinates of the third element value to obtain a result matrix, which can specifically include: first, sorting the third element values ​​in the N intermediate result matrices according to the position coordinates of the third element values, for example, sorting according to the row coordinates of the third element values, or sorting according to the column coordinates of the third element values; comparing the position coordinates in the sorted N intermediate result matrices, adding the third element values ​​with the same position coordinates, and deleting the position coordinates of the zero element value to obtain a result matrix in a compressed format. In the above optional implementation, a matrix in a compressed format can be directly obtained, and the matrix in a compressed format output by the matrix calculation device can be applied to certain application scenarios where a matrix in a compressed format is required for subsequent calculations. And because the matrix calculation device outputs a matrix in a compressed format, the transmission resources of the subsequent transmission matrix can be reduced.

[0017] In an optional implementation, before obtaining the first calculation instruction, the method also includes: obtaining a second calculation instruction, the second calculation instruction includes a first matrix and a second matrix; converting the first matrix into N first column vectors and retaining the row coordinates of the first element value in the first column vector, converting the second matrix into N first row vectors and retaining the column coordinates of the first element value in the first row vector, so that the matrix calculation device can calculate two compressed format matrices based on the vector outer product.

[0018] In an optional implementation, before obtaining the second calculation instruction, the method further includes: obtaining a third calculation instruction, the third calculation instruction includes a fifth matrix and a sixth matrix, wherein at least one of the fifth matrix and the sixth matrix is ​​a matrix in an uncompressed format; then performing format conversion on the fifth matrix to obtain a first matrix in a compressed format, and performing format conversion on the sixth matrix to obtain a second matrix. In the above optional implementation, the matrix calculation device can receive a matrix in an uncompressed format, and then convert the matrix in an uncompressed format into a matrix in a compressed format, so that the matrix calculation device can support matrix calculations in multiple formats.

[0019] In an optional implementation, the first column vector may be first split into X second column vectors, and the first row vector may be split into X second row vectors, wherein the precision of the element values ​​included in the second column vector and the second row vector is the second precision, and the precision of the element values ​​included in the first column vector and the first row vector is the first precision; the first precision is higher than the second precision, and X is an integer greater than or equal to 2; in this method, calculating the vector outer products of the N first column vectors and the N first row vectors to obtain N intermediate result matrices may include: calculating the vector outer products of the X second column vectors and the X second row vectors to obtain X 2 A fourth matrix is ​​formed, the fourth matrix includes a fourth element value and a position coordinate of the fourth element value, the position coordinate of the fourth element value includes a row coordinate of the first element value and a column coordinate of the second element value, and the precision of the fourth element value is the first precision; further, according to the index of the position coordinate of the fourth element value, X 2 The fourth element values ​​with the same position coordinates in the fourth matrices are accumulated to obtain an intermediate result matrix, and the precision of the third element value in the intermediate result matrix is ​​the first precision. In the above optional implementation, the matrix calculation device can realize high-precision matrix calculation based on the low-precision matrix calculation device, thereby improving the applicability of the matrix calculation unit.

[0020] In a third aspect, a matrix calculation circuit is provided, wherein the matrix calculation circuit is used to execute the operation steps of the matrix calculation method provided by the second aspect or any possible implementation of the second aspect.

[0021] In a fourth aspect, a matrix computing system is provided, which includes a processor and a matrix computing device, wherein the processor is used to send computing instructions to the matrix computing device, and the matrix computing device is used to execute the operating steps of the matrix computing method provided by the second aspect or any possible implementation method of the second aspect.

[0022] In a fifth aspect, a chip is provided, comprising a processor, wherein a matrix calculation device is integrated in the processor, and the matrix calculation device is used to execute the operation steps of the matrix calculation method provided by the second aspect or any possible implementation method of the second aspect.

[0023] In a sixth aspect, a matrix computing device is provided, which includes the matrix computing system provided in the fourth aspect or the chip provided in the fifth aspect.

[0024] In a seventh aspect, a readable storage medium is provided, in which instructions are stored. When the readable storage medium is run on a device, the device executes the operation steps of the matrix calculation method provided by the second aspect or any possible implementation of the second aspect.

[0025] In an eighth aspect, a computer program product is provided. When the computer program product runs on a computer, the computer is caused to execute the operation steps of the matrix calculation method provided by the second aspect or any possible implementation of the second aspect.

[0026] It can be understood that the device, computer storage medium or computer program product of any of the matrix calculation methods provided above is used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding method provided above, and will not be repeated here.

[0027] Based on the implementations provided in the above aspects, the present application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1A A schematic diagram of a matrix in COO compression format in an embodiment of the present application;

[0029] Figure 1B A schematic diagram of a matrix in CSR compression format in an embodiment of the present application;

[0030] Figure 1C Schematic diagram of a matrix in CSC compression format in an embodiment of the present application;

[0031] Figure 2 A schematic diagram of the structure of a computing device provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of the structure of a processor provided in an embodiment of the present application;

[0033] Figure 4A A schematic diagram of the structure of a matrix computing device provided in an embodiment of the present application;

[0034] Figure 4B A schematic diagram of the structure of a vector outer product processing engine provided in an embodiment of the present application;

[0035] Figure 4C A schematic diagram of the structure of a MAC operation subunit provided in an embodiment of the present application;

[0036] Figure 4D A schematic diagram of the structure of an accumulator provided in an embodiment of the present application;

[0037] Figure 4E A schematic diagram of the structure of an adder provided in an embodiment of the present application;

[0038] Figure 5A It is a schematic diagram of a matrix calculation device in an embodiment of the present application converting a first matrix into N first column vectors and a second matrix into a plurality of first row vectors;

[0039] Figure 5B A schematic diagram of a matrix calculation device in an embodiment of the present application performing outer product calculations on N first column vectors and N first row vectors to obtain N intermediate result matrices;

[0040] Figure 5C Schematic diagram of element values ​​and position coordinates in the first intermediate result matrix in an embodiment of the present application;

[0041] Figure 5D Schematic diagram of element values ​​and position coordinates in the second intermediate result matrix in an embodiment of the present application;

[0042] Figure 5E It is a schematic diagram of a matrix calculation device in an embodiment of the present application accumulating the third element values ​​with the same position coordinates in N intermediate result matrices to obtain a result matrix;

[0043] Figure 6 A schematic diagram of an implementation method of accumulating third element values ​​with the same position coordinates in N intermediate result matrices to obtain a result matrix in an embodiment of the present application;

[0044] Figure 7 A schematic diagram of the structure of another matrix computing device provided in an embodiment of the present application;

[0045] Figure 8It is a schematic diagram of another implementation method of accumulating the third element values ​​with the same position coordinates in N intermediate result matrices to obtain a result matrix in an embodiment of the present application;

[0046] Fig. 9 It is a schematic diagram of a format conversion unit converting a matrix in a non-compressed format into a matrix in a compressed format in an embodiment of the present application;

[0047] Fig.10 A schematic diagram of decomposing an integer value of a first precision into multiple integer values ​​of a second precision in an embodiment of the present application;

[0048] Fig.11 It is a schematic diagram of decomposing a first column vector of first precision into a plurality of second column vectors of second precision and decomposing a first row vector of first precision into a plurality of second row vectors of second precision in an embodiment of the present application;

[0049] Fig.12 This is a schematic diagram of a matrix calculation device in an embodiment of the present application performing vector outer product calculation on a second row vector and a second column vector to obtain a fourth matrix;

[0050] Fig.13 A schematic diagram of a step flow chart of an embodiment of a matrix calculation method in an embodiment of the present application;

[0051] Fig.14 This is a schematic diagram of the step flow of another embodiment of a matrix calculation method in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. The terms "first", "second", etc. in the specification and claims of the present application and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0053] In order to better understand the present application, the relevant terms involved in the present application are first explained.

[0054] Matrix: An m×n matrix of dimension is a rectangular array of elements arranged in m rows and n columns. For example, matrix A is shown in equation (1) below, and matrix B is shown in equation (2) below.

[0055]

[0056] Matrix addition and subtraction: Matrices of the same dimension can be added and subtracted with each other, specifically, the elements at each position are added and subtracted. For example, matrix A and matrix B are both m×n matrices. The matrix C is obtained by adding matrix A and matrix B as shown in the following formula (3).

[0057]

[0058] Matrix multiplication: Two matrices can be multiplied only when the number of columns of the first matrix A is equal to the number of rows of the other matrix. For example, if matrix A is an m×n matrix and matrix B is an n×p matrix, the product of matrix A and matrix B is an m×p matrix, and an element of the m×p matrix is ​​shown in equation (4):

[0059] Among them, 1≤i≤m, 1≤j≤p.

[0060] A row vector is a matrix with a dimension of 1×m, where m is a positive integer. For example, a row vector is as shown in equation (5):

[0061] X=[x1 x2...x m ], formula (5).

[0062] A column vector is a matrix with a dimension of m×1, where m is a positive integer. For example, a column vector is as shown in the following formula (6):

[0063]

[0064] Vector outer product: The tensor product of two vectors. The tensor product is a matrix. For example, given a column vector U with dimension m×1 and a row vector V with dimension 1×n, the outer product of vector U and vector V, U×V, is defined as a matrix D with dimension m×n. Matrix D is shown in equation (7):

[0065]

[0066] Compressed Matrix: When a matrix contains both zero and non-zero values, in order to save storage space, the non-zero values ​​can be stored in a certain format instead of zero values. This process is called compressing the matrix, and the compressed matrix is ​​called a compressed matrix. Methods for compressing a matrix include but are not limited to coordinate representation (COO), compressed sparse row (CSR), compressed sparse column (CSC), etc.

[0067] The following is an exemplary description of the three compression methods, namely COO, CSR and CSC.

[0068] COO: A matrix is ​​represented by a triple. A triple contains three values: row number, column number, and element value. The row number and column number are used to identify the position of the element value. For example, the triple is (row number, column number, element value), or the triple is (element value, row number, column number), etc. The specific order of the three values ​​in the triple is not limited. For example, see Figure 1A , Figure 1A A matrix Y with a dimension of 4×4 is shown in the figure, which includes zero element values ​​and non-zero element values, wherein the non-zero element values ​​are: 1, 2, 3, 4, 5, 6, 7, 8, 9. For example, the position of the non-zero element value "1" is the 0th row and the 0th column, and the triple is represented as (0, 0, 1). The position of the non-zero element value "2" is the 0th row and the 1st column, and the triple is represented as (0, 1, 2). The position of the non-zero element value "3" is the 1st row and the 1st column, and the triple is represented as (1, 1, 3). Each element value is not described one by one here. For example, the triple form of the matrix in the compressed format is: (0, 0, 1), (0, 1, 2), (1, 1, 3), (1, 2, 4), (2, 0, 5), (2, 2, 6), (2, 3, 7), (3, 1, 8), (3, 3, 9).

[0069] Optionally, the compressed matrix Y can be expressed as shown in the following equation (8).

[0070] Row coordinates = ([0,0,1,1,2,2,2,3,3])

[0071] Column coordinates = ([0,1,1,2,0,2,3,1,3])

[0072] Element value = ([1,2,3,4,5,6,7,8,9]), formula (8).

[0073] CSR: The matrix is ​​represented by three types of data: element value, column number, and row offset value. The element value and column number in CSR are similar to those in the COO method. The difference between CSR and COO is that the row offset indicates the starting offset position of the first element of a row among all element values. See Figure 1B As shown, first, Figure 1BThe non-zero element values ​​in the matrix Y shown in are arranged in rows, and all the element values ​​are: 1, 2, 3, 4, 5, 6, 7, 8, 9. The first non-zero element value in the first row is "1", and the offset of element value "1" in all element values ​​is "0". Similarly, the first non-zero element value in the second row is "3", and the offset of element value "3" in all element values ​​is "2". The first non-zero element value in the third row is "5", and the offset of element value "5" in all element values ​​is "4". The first non-zero element value in the fourth row is "8", and the offset of element value "8" in all element values ​​is "7". Finally, the total number of non-zero element values ​​in the matrix (such as "9") is added to the last position of the row where the row offset is located.

[0074] The compressed matrix Y can be expressed as shown in the following equation (9).

[0075] Row offset = ([0,2,4,7,9])

[0076] Column coordinates = ([0,1,1,2,0,2,3,1,3])

[0077] Element value = ([1,2,3,4,5,6,7,8,9]), formula (9).

[0078] CSC: The matrix is ​​represented by three types of data: element value, row number, and column offset value. The element value and row number in CSR are similar to those in the COO method. The difference between CSC and COO is that the column offset indicates the starting offset position of the first element of a column among all element values. See Figure 1C As shown, first, Figure 1C The non-zero element values ​​in the matrix Y shown in are arranged by columns, and all the element values ​​are: 1, 5, 2, 3, 8, 4, 6, 7, 9. The first non-zero element value in the first column is "1", and the offset of element value "1" in all element values ​​is "0". Similarly, the first non-zero element value in the second column is "5", and the offset of element value "5" in all element values ​​is "2". The first non-zero element value in the third column is "2", and the offset of element value "2" in all element values ​​is "5". The first non-zero element value in the fourth column is "3", and the offset of element value "3" in all element values ​​is "7". Finally, the total number of non-zero element values ​​in the matrix (such as "9") is added to the last position of the row where the column offset is located.

[0079] The compressed matrix Y can be expressed as shown in the following equation (10).

[0080] Column offset = ([0,2,5,7,9])

[0081] Row coordinates = ([0,0,1,1,2,2,2,3,3])

[0082] Element value = ([1,5,2,3,8,4,6,7,9]), formula (10).

[0083] Through the description of the above three matrix compression methods, it can be seen that each element value in the matrix of COO compression format has corresponding row coordinates (row number) and column coordinates (column number). Each element value in the matrix of CSR compression format will have corresponding column coordinates. Each element value in the matrix of CSC compression format will have corresponding row coordinates.

[0084] Matrices in uncompressed format: see Figure 1A The matrix Y shown in FIG. 1 includes zero element values ​​and non-zero element values ​​in the non-compressed format. It should be noted that, in general, a matrix in a compressed format is also referred to as a sparse matrix, and a matrix in an non-compressed format is also referred to as a dense matrix.

[0085] Numeric type: In the computer field, numeric types include integer and floating-point types. Integer is mainly used to represent whole numbers, and floating-point is mainly used to represent decimals. Among them, the precision of integer includes int2, int4, int8, int16, int32 and other precisions. Among them, int is used to represent integer functions. The number after int represents the number of bits (bits) in the binary value range. 1 bit (bit) is 0 or 1. For example, the binary value range of int4 is 4 bits (0000-1111), which is converted to decimal as (-8, 7). Similarly, the binary value range of int8 is (-2 7 , 2 7 -1). The computer storage capacity unit 1 byte is 8 bits, so the corresponding int8 has 1 byte and int16 has 2 bytes. The binary value range of int16 occupies 2 bytes, which is converted to decimal as (-32768, 32767). The binary value range of int32 occupies 4 bytes, which is converted to decimal as (-2147483648, 2147483647).

[0086] An integer matrix refers to a matrix with integer values ​​as elements. For example, an integer matrix with m rows and n columns includes m×n elements, and these m×n elements are integer values. The integer values ​​can be of precision such as int2, int4, int8, int16 or int32. For example, an integer matrix can also include matrices of different integer formats, such as a matrix including integer values ​​in int8 format, a matrix including integer values ​​in int16 format, and a matrix including integer values ​​in int32 format.

[0087] Floating point numbers (FP) are mainly used to represent decimals and usually consist of three parts, namely the sign bit, the exponent bit and the mantissa bit. The exponent bit can also be called the exponent bit. Among them, the sign bit can be 1 bit, and the exponent bit and the mantissa bit can be multiple bits. Floating point numbers can usually include a variety of formats, such as half-precision floating point numbers, single-precision floating point numbers and double-precision floating point numbers in the IEEE754 standard. Among them, half-precision floating-point numbers occupy 16 bits (that is, 2 bytes) in the computer memory, and can also be referred to as FP16. The absolute value range of the numerical value that can be represented by half-precision floating-point numbers is approximately [6.10×10 -5 ,6.55×10 4 ] A single-precision floating-point number occupies 32 bits (4 bytes) in computer memory and can be referred to as FP32. The absolute value range of the value that can be represented by a single-precision floating-point number is approximately [1.18×10 -38 , 3.40×10 38 ]. Double precision floating point numbers occupy 64 bits (8 bytes) in computer memory, and can also be referred to as FP64. Double precision floating point numbers can represent 15 or 16 decimal digits, and the absolute value range of the representable value is approximately [2.23×10 -308 , 1.80×10 308 ].

[0088] Table 1 below shows a storage format for the above three floating-point numbers. Among them, in the 16 bits occupied by FP16, the sign bit occupies 1 bit, the exponent occupies 5 bits, and the mantissa occupies 10 bits. In the 32 bits occupied by FP32, the sign bit occupies 1 bit, the exponent occupies 8 bits, and the mantissa occupies 23 bits. In the 64 bits occupied by FP64, the sign bit occupies 1 bit, the exponent occupies 11 bits, and the mantissa occupies 52 bits.

[0089] Table 1

[0090] Sign bit Exponent bit (exponent) Mantissa FP16 1bit 5bits 10bits FP32 1bit 8bits 23bits FP64 1bit 11bits 52 bits

[0091] A floating point matrix may refer to a matrix with floating point numbers as elements. For example, a floating point matrix with m rows and n columns includes m×n elements, and these m×n elements may be floating point numbers. Similar to floating point numbers, a floating point matrix may also include matrices of different floating point formats, such as a matrix including floating point numbers in FP16 format, a matrix including floating point numbers in FP32 format, and a matrix including floating point numbers in FP64 format.

[0092] Figure 2 This is a schematic diagram of the structure of a computing device provided in this embodiment. The computing device may be a device with computing capabilities such as a terminal, a network device or a server. Figure 2 The computing device may include a memory 201 , a processor 202 , a communication interface 203 and a bus 204 , and the memory 201 , the processor 202 and the communication interface 203 are interconnected via the bus 204 .

[0093] Among them, the memory 201 can be used to store data, software programs and modules, mainly including a program storage area and a data storage area. The program storage area can store an operating system, software applications and intermediate layer software required for at least one function, etc., and the data storage area can store data created when the device is used. For example, the operating system can include a Linux operating system, a Unix operating system, or a Window operating system, etc.; the software application required for at least one function can include applications related to artificial intelligence, applications related to high performance computing (HPC), applications related to deep learning, or applications related to scientific computing, etc., and the intermediate layer software can include linear algebra library functions, etc. In a possible exemplary embodiment, the memory 201 includes but is not limited to static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM) or high-speed random access memory, etc. Further, the memory 201 can also include other non-volatile memories, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0094] In addition, the processor 202 is used to control and manage the operation of the computing device, such as by running or executing software programs and / or modules stored in the memory 201, and calling the data stored in the memory 201, to perform various functions of the computing device and process data. In a possible exemplary embodiment, the processor 202 includes but is not limited to a central processing unit (CPU), a network processing unit (NPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, transistor logic devices, logic circuits or any combination thereof. It can implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. The processor 202 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc.

[0095] The communication interface 203 is used to enable the computing device to communicate with an external device. The communication interface 203 may include an input interface and an output interface. The input interface may be used to obtain the first matrix and the second matrix in the compressed format in the following embodiments. In some feasible embodiments, the input interface may have only one input interface or multiple input interfaces. The output interface may be used to output the result matrix in the following embodiments. In some feasible embodiments, the result matrix may be directly output by the processor, or may be first stored in a memory and then output via the memory. In other feasible embodiments, there may be only one output interface or multiple output interfaces.

[0096] The bus 204 may be a Peripheral Component Interconnect Express (PCIe) bus or an extended industry standard architecture (EISA) bus. The bus 204 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0097] In this embodiment, the processor 202 may include a matrix computing device, which may be an ASIC, an FPGA, or a logic circuit, etc. Of course, the device may also be implemented by software, and this embodiment of the present application does not impose specific restrictions on this. The matrix computing device may be used to calculate matrix calculations involved in artificial intelligence, scientific computing, graphic computing, etc.

[0098] Furthermore, the processor 202 may also include one or more other processing units such as a CPU, a GPU or an NPU. Figure 3 As shown, taking the processor 202 including the CPU 1 and the matrix computing device 2 as an example, the matrix computing device 2 can be integrated with the CPU 1 (for example, the matrix computing device 2 is integrated inside the SoC where the CPU 1 is located), or can be separately arranged in parallel with the CPU 1 (for example, the matrix computing device 2 is arranged in the form of a PCIe card). Figure 3 (a) and Figure 3 As shown in (b) in FIG. 1 , the CPU 1 may further include a controller 11 , one or more arithmetic logic units (ALUs) 12 , a cache 13 , and a memory management unit (MMU) 14 . Figure 3 In the description, the memory 201 is taken as a dynamic random access memory DRAM as an example.

[0099] In an embodiment of the present application, the matrix calculation device can calculate a matrix in a compressed format. When calculating the multiplication of two matrices in a compressed format, the matrix calculation device first obtains N first column vectors converted from one of the matrices, and the first column vector will contain the first element value and the row coordinates of the first element value, and obtains N first row vectors converted from another matrix, and the first row vector will contain the second element value and the column coordinates of the second element value. The matrix calculation device calculates the vector outer product of the N first column vectors and the N first row vectors, thereby obtaining N intermediate result matrices, and the intermediate result matrix will include the element value and the position coordinates corresponding to the element value, and the position coordinates include the row coordinates of the first element value and the column coordinates of the second element value. The matrix calculation device can accumulate the third element values ​​with the same position coordinates in the N intermediate result matrices according to the index of the position coordinate, and then obtain the result matrix. In an embodiment of the present application, a matrix calculation device calculates a first matrix and a second matrix based on a vector outer product. During the calculation process, the row coordinates of the element values ​​in the first column vector are retained, and the column coordinates of the element values ​​in the second column vector are retained. Then, based on the index of the position coordinates, the third element values ​​with the same position coordinates are accumulated to obtain a result matrix of the calculation of two matrices in compressed formats. Compared with the traditional method of first decompressing the matrix in compressed format and then performing matrix calculation on the decompressed matrix, the matrix calculation device provided in the embodiment of the present application can effectively improve the computing performance of the matrix in compressed format.

[0100] The present application embodiment provides a matrix computing device. Figure 4A As shown, the matrix computing device includes a vector outer product processing engine 401 and an accumulator 402. Optionally, the matrix computing device includes a first buffer 403, and the vector outer product processing engine 401, the accumulator 402 and the first buffer 403 are connected in sequence. The first buffer 403 can be a first buffer 403 (such as a register) in the matrix computing device, or the first buffer 403 can also be Figure 3 The buffer 13 in the central processing unit 1 shown in FIG. 1 is a buffer 13 in the central processing unit 1 shown in FIG. 1. Optionally, the matrix calculation device further includes a format conversion unit 405 and a second buffer 406. The function of the format conversion unit 405 can be Figure 3 The second buffer 406 may be a register in the matrix computing device, or the second buffer 406 may be a Figure 3 The buffer 13 in the central processing unit 1 is shown in FIG.

[0101] The format conversion unit 405 is used to convert the first matrix in a compressed format into N first column vectors, and convert the second matrix in a compressed format into N first row vectors, where the first column vector includes the first element value and the row coordinate of the first element value, and the first row vector includes the second element value and the column coordinate of the second element value.

[0102] Optionally, the structure of the vector outer product processing engine 401 is described below. Figure 4B As shown, Figure 4B Schematic diagram of the structure of the vector outer product processing engine 401. The vector outer product processing engine 401 has multiple processing elements (PE) 4011, and the multiple PEs 4011 are a two-dimensional array. Each PE 4011 includes a multiply accumulate (MAC) operator 40110 and a coordinate merging subunit 40111. Each PE receives two sets of input data and outputs a set of data. Each set of input data includes a value and a coordinate. One set of input data includes a first element value and a row coordinate corresponding to the first element value, and the other set of input data includes a second element value and a column coordinate corresponding to the second element value.

[0103] Each PE has two functions, one of which is that each PE4011 receives two sets of input data and outputs a set of output data according to the set of input data. The output data includes a third element value and a position coordinate of the third element value, wherein the third element value is obtained by multiplying the first element value and the second element value. The position coordinate is obtained by merging the first row coordinate and the first column coordinate. For example, taking the PE at the 0th row and the 0th column in the PE array as an example, a set of input data is the first element value (such as a0) in the first row vector and the row coordinate corresponding to the first element value (such as i0). Another set of input data is the second element value (such as b0) in the first column vector and the column coordinate corresponding to the second element value (such as j0). The MAC operation subunit 40110 is used to receive the first element value and the second element value, multiply the first element value and the second element value, and then output the product of the first element value and the second element value (i.e., the third element value). The coordinate merging subunit 40111 is used to receive the row coordinate of the first element value and the column coordinate of the second element value, merge the row coordinate and the column coordinate to obtain the position coordinate. The data output by a PE includes a third coordinate value and a position coordinate corresponding to the third coordinate value.

[0104] Another function of PE is that a PE needs to transmit the first element value and the column coordinate corresponding to the first element value to the next PE in the row direction, and the PE needs to transmit the second element value and the row coordinate corresponding to the second element value to the next PE in the column direction. For example, after the first clock cycle, the first PE in the PE array (i.e., the PE in the 0th row and the 0th column) transmits a0 and i0 to the next PE in the row direction (such as the PE in the 0th row and the 1st column), and transmits b0 and j0 to the next PE in the column direction (i.e., the PE in the 1st row and the 0th column). Optionally, the data transmission method of each PE can be to transmit data to the PE of the next level according to each clock cycle, or, within a clock cycle, the first PE transmits a0 and i0 to the next PE in the row direction (i.e., the PE in the 0th row and the 1st column) until it is transmitted to the last PE in the row (i.e., the 0th row), and transmits b0 and j0 to the last PE in the column (the 0th column). In this example, the number of levels of PE transmission in each clock unit can be designed according to actual needs, and is not specifically limited.

[0105] Optionally, the structure of the MAC operation subunit in the PE is described below. Figure 4C As shown, Figure 4C : is a schematic diagram of the structure of the MAC operator unit. The MAC operator unit in each PE includes a sign subunit 40112, an exponent subunit 40113, an integer subunit 40114 and a precision format conversion subunit 40115. The sign subunit 40112, the exponent subunit 40113 and the integer subunit 40114 are all connected to the precision format conversion subunit 40115. Among them, the sign subunit 40112 is used to process the positive and negative signs of the input value. The exponent subunit 40113 is used to process the decimal point displacement calculation when two floating point numbers are multiplied. The integer subunit 40114 is used to calculate the multiplication of two integers. The precision format conversion subunit 40115 is used to output a numerical format that meets the specification (such as FP16, FP32, FP64 and other floating point formats or integer formats such as int32, int16, int8, etc.). In this example, the MAC operator unit is a general unit that supports floating point formats and integer formats, which enhances the applicability of the matrix computing device.

[0106] Optionally, the structure of the accumulator 402 in the matrix calculation device is described below. Figure 4D As shown, Figure 4D4 is a schematic diagram of the structure of the accumulator 402. The accumulator 402 includes a plurality of accumulator (ACC) processing units 4021, that is, the accumulator 402 includes an ACC processing unit array composed of a plurality of ACC processing units 4021. Each ACC processing unit 4021 receives a set of data output by the vector outer product processing engine 401, and the set of data includes a third element value and the position coordinate corresponding to the third element value. Each ACC processing unit 4021 includes an adder 40210 and a data acquisition subunit 40211. Among them, the data acquisition subunit 40211 is used to receive the position coordinate and output the position coordinate to the first buffer 403, so that the adder 40210 obtains the cache value corresponding to the position coordinate from the first buffer 403. The adder 40210 is used to accumulate the cache value output by the first buffer 403 with the third element value. For example, an ACC processing unit 4021 receives a “third element value 0” (such as c 00 ) and the c 00 The ACC processing unit 4021 first writes the “third element value 0” into the first buffer 403 according to the position coordinates (such as 1, 1), and then receives the “third element value 1” (such as c1) in the intermediate result matrix C1 output by the vector outer product processing engine 401. 10 ) and the c 10 When the corresponding position coordinate (such as 1, 1) is outputted by the first buffer 403, the ACC processing unit 4021 receives the cache value (such as c 00 ), then, adder 40210 is used to convert c 00 and c 10 Add and c 00 and c 10 The accumulated value is output to the first buffer 403.

[0107] Optionally, see Figure 4EAs shown, the adder 40210 also specifically includes a sign unit 40215, an exponent unit 40216, an integer unit 40217 and a precision format conversion unit 40218. The sign unit 40215, the exponent unit 40216, and the integer unit 40217 are all connected to the precision format conversion unit 40218. Among them, the sign unit 40215 is used to process the positive and negative signs of the input value. The exponent unit 40216 is used to process the decimal point displacement calculation when two floating point numbers are multiplied. The integer unit 40217 is used to calculate the multiplication operation of two integers. The precision format conversion unit 40218 is used to output a numerical format that meets the specification (such as floating point formats such as FP16, FP32, FP64 or integer formats such as int32, int16, int8). In this example, the adder 40210 is a general unit that supports floating point formats and integer formats, which enhances the applicability of the matrix calculation device.

[0108] The specific role of the format conversion unit 405 is described below. The format conversion unit 405 is used to convert the first matrix in the compressed format into N first column vectors, and convert the second matrix in the compressed format into N first row vectors, the first column vector includes the first element value and the row coordinate of the first element value, and the first row vector includes the second element value and the column coordinate of the second element value. The dimension of the first matrix is ​​M×N, and the dimension of the second matrix is ​​N×K, where M, N, and K are integers greater than or equal to 1.

[0109] See also Figure 5A As shown, for the convenience of explanation, in this example, M, N, and K are all described by taking 4 as an example, that is, the dimensions of the first matrix and the second matrix are both 4×4. The first matrix is ​​described by taking matrix A as an example, and the second matrix is ​​described by taking matrix B as an example. The compression format of the first matrix and the second matrix is ​​described by taking COO as an example.

[0110] The format conversion unit 405 is used to split the matrix A by columns, and split the matrix A into 4 first column vectors, and the 4 first column vectors are A0, A1, A2 and A3 respectively. The format conversion unit 405 splits the matrix B by rows, and splits the matrix B into 4 first row vectors, and the 4 first row vectors are B0, B1, B2 and B3 respectively. Exemplarily, the first column vector is described by taking A0 as an example, and A0 is [a0, a1, a2, a3] T, where a0, a1, a2, and a3 in A0 are all element values. Each element value has a corresponding row coordinate. For example, the row coordinate of a0 is i0, the row coordinate of a1 is i1, the row coordinate of a2 is i2, and the row coordinate of a3 is i3. The first row vector is illustrated by taking B0 as an example. B0 is [b0, b1, b2, b3], where b0, b1, b2, and b3 in B0 are all element values. Each element value in B0 has a corresponding column coordinate. For example, the column coordinate of b0 is j0, the column coordinate of b1 is j1, the column coordinate of b2 is j2, and the column coordinate of b3 is j3. It should be understood that matrix A is split according to N columns, and each element value in the first column vector after splitting only retains the row coordinate of the element value. Matrix B is split according to N rows, and each element value in the first row vector after splitting only retains the column coordinate of the element value. Similarly, A1 is [c0,c1,c2,c3] T , where c0, c1, c2, c3 in A1 are all element values. Each element value has a corresponding row coordinate. For example, the row coordinate of c0 is k0, the row coordinate of c1 is k1, the row coordinate of c2 is k2, and the row coordinate of c3 is k3. B1 is [d0, d1, d2, d3], where d0, d1, d2, d3 in B1 are all element values. Each element value in B1 has a corresponding column coordinate. For example, the column coordinate of d0 is l0, the column coordinate of d1 is l1, the column coordinate of d2 is l2, and the column coordinate of d3 is l3.

[0111] For example, the first matrix in compressed format is represented in COO format as: (0, 0, 1), (1, 0, 2), (1, 1, 3), (1, 2, 4), (2, 0, 5), (2, 2, 6), (2, 3, 7), (3, 1, 8), (3, 3, 9), (3, 0, 6). The format conversion unit 405 splits the first matrix in compressed format by columns, wherein the element values ​​in (0, 0, 1), (1, 0, 2), (2, 0, 5) and (3, 0, 6) are element values ​​of the same column, that is, they are all element values ​​of the 0th column. It should be understood that when the format conversion unit 405 splits the matrix A by columns, the element values ​​"1", "2", "5" and "6" are used as element values ​​in the A0 vector, and the element values ​​"1", "2", "5" and "6" are all element values ​​in the same column, so only the row coordinates of each element value are retained. In this example, vector A0 includes 4 element values, which are "1", "2", "5" and "6" respectively, and the row coordinate of the element value "1" is "0". Similarly, the row coordinate of the element value "2" is "1", the row coordinate of the element value "5" is "2", and the row coordinate of the element value "6" is "3".

[0112] For another example, the second matrix in the compressed format is represented in the COO format as: (0, 0, 1), (1, 0, 2), (2, 1, 3), (1, 2, 4), (2, 0, 5), (2, 2, 6), (2, 3, 7), (3, 1, 8), (3, 3, 9), (3, 0, 6). The format conversion unit 405 splits the second matrix according to the row coordinates, that is, the element values ​​of the same row are used as the element values ​​in the same vector. For example, the row coordinates in the four triplets (2, 1, 3), (2, 0, 5), (2, 2, 6) and (2, 3, 7) are the same, then the element values ​​in these four triplets are split into a row vector (such as row vector B0), and the element values ​​"3", "5", "6" and "7" are used as the element values ​​in the first row vector B0, and each element value in the first row vector B0 has a corresponding column coordinate. For example, the column coordinate of the element value "3" is "1", the column coordinate of the element value "5" is "0", the column coordinate of the element value "6" is "2", and the column coordinate of the element value "7" is "3". It should be noted that the above specific values ​​in the COO format are only examples for the convenience of explanation and do not constitute a limiting description of the present application.

[0113] The vector outer product processing engine 401 is used to calculate the vector outer product of N first column vectors and N first row vectors to obtain N intermediate result matrices, the intermediate result matrix includes the third element value and the position coordinate of the third element value, and the position coordinate includes the row coordinate of the first element value and the column coordinate of the second element value. For example, please refer to Figure 5B As shown, the vector outer product processing engine 401 calculates the vector outer product of the first column vector and the first row vector. For example, the vector outer product processing engine 401 calculates the vector outer product of A0 and B0 to obtain an intermediate result matrix C0. Similarly, the vector outer product processing engine 401 calculates the vector outer product of A1 and B1 to obtain an intermediate result matrix C1. The vector outer product processing engine 401 calculates the vector outer product of A2 and B2 to obtain an intermediate result matrix C2. The vector outer product processing engine 401 calculates the vector outer product of A3 and B3 to obtain an intermediate result matrix C3. Taking C0 as an example, the intermediate result matrix calculated by the vector outer product processing engine 401 is shown in the following formula (11).

[0114]

[0115] The intermediate result matrix C0 includes third element values, each of which has a corresponding position coordinate. The position coordinate is the combined value of the row coordinate of the first element value and the column coordinate of the second element value. For example, see Figure 5CAs shown, the row coordinate of a0 is i0, and the column coordinate of b0 is j0, then the position coordinate of the third element value a0b0 is (i0, j0). Similarly, the row coordinate of a0 is i0, and the column coordinate of b1 is j1, then the position coordinate of the third element value a0b1 is (i0, j1). The row coordinate of a1 is i1, and the column coordinate of b0 is j0, then the position coordinate of the third element value a1b0 is (i1, j0), etc. The position coordinates of the third element values ​​in the intermediate result matrix C0 are not illustrated one by one.

[0116] For example, a0 is 1, a1 is 2, a2 ​​is 5, and a3 is 6. The row coordinate of a0 is "0". Similarly, the row coordinate of a1 is "1", the row coordinate of a2 is "2", and the row coordinate of a3 is "3". b0 is 3, b1 is 5, b2 is 6, and b3 is 7. The column coordinate of b0 is "1", the column coordinate of b1 is "0", the column coordinate of b2 is "2", and the column coordinate of b3 is "3". The third element value a0b0=1×3=3, the position coordinate of a0b0 includes the row coordinate of a0 and the column coordinate of b0, that is, the position coordinate of a0b0 is (0, 3). The third element value a0b1=1×5=5, the position coordinate of a0b1 includes the row coordinate of a0 and the column coordinate of b1, that is, the position coordinate of a0b1 is (0, 0). The third element value a1b0=2×3=6, and the position coordinates of a1b0 include the row coordinates of a1 and the column coordinates of b0, that is, the position coordinates of a1b0 are (1, 3).

[0117] See also Figure 5D As shown, the vector outer product processing engine 401 obtains an intermediate result matrix according to the vector outer product calculation formula in the above formula (7). Taking C1 as an example, the intermediate result matrix calculated by the vector outer product processing engine 401 is shown in the following formula (12).

[0118]

[0119] Similarly, the intermediate result matrix C1 includes the third element value, and each third element value in C1 has a corresponding position coordinate. For example, see Figure 5D As shown, the row coordinate of c0 is k0, and the column coordinate of d0 is l0, then the position coordinate of the third element value c0d0 is (k0, l0). Similarly, the row coordinate of c0 is k0, and the column coordinate of d1 is l1, then the position coordinate of the third element value c0d1 is (k0, l1). The row coordinate of c1 is k1, and the column coordinate of d0 is l0, then the position coordinate of the third element value c1d0 is (k1, l0), etc. The position coordinates of the third element values ​​in the intermediate result matrix C1 are not illustrated one by one.

[0120] The accumulator 402 is used to accumulate the third element values ​​with the same position coordinates in the N intermediate result matrices according to the index of the position coordinates of the third element value to obtain a result matrix. Figure 5B As shown, in this example, the number of intermediate result matrices is 4. The third element value in each of the intermediate result matrices C0, C1, C2 and C3 has a position coordinate, and the accumulator 402 accumulates the third element values ​​with the same position coordinate to obtain a result matrix. For example, see Figure 5E As shown, the accumulator 402 adds the 4 third element values ​​of C0, C1, C2 and C3 with the position coordinates (0, 0) to obtain a fourth element value in the result matrix. Similarly, the accumulator 402 adds the 4 third element values ​​of C0, C1, C2 and C3 with the position coordinates (1, 1) to obtain a fourth element value in the result matrix, etc. In this example, the specific values ​​of the matrix position coordinates are only examples for convenience of explanation and are not limited.

[0121] Optionally, the accumulator 402 accumulates the third element values ​​with the same position coordinates in the N intermediate result matrices according to the index of the position coordinates of the third element value, which implementation manner includes at least the following two implementation manners.

[0122] In a first possible implementation, when the vector outer product processing engine 401 performs vector outer product calculations on the first row vector and the first column vector, it generates an intermediate result matrix in a certain order. Figure 5B Take the calculation of the vector outer product of the first row vector and the first column vector shown in as an example. The vector outer product processing engine 401 calculates the vector outer product of A0 and B0 to obtain an intermediate result matrix C0. Then, the vector outer product processing engine 401 calculates the vector outer product of A1 and B1 to obtain an intermediate result matrix C1. After that, the vector outer product processing engine 401 calculates the vector outer product of A2 and B2 to obtain an intermediate result matrix C2. Finally, the vector outer product processing engine 401 calculates the vector outer product of A3 and B3 to obtain an intermediate result matrix C3. For example, the order of the four intermediate result matrices is C0, C1, C2, C3. The accumulator 402 receives the four intermediate result matrices from the vector outer product processing engine 401 according to the generation order of the four intermediate result matrices.

[0123] See also Figure 6 As shown, the four intermediate result matrices include at least a first intermediate result matrix (such as C0) and a second intermediate result matrix (such as C1). In order to distinguish the position coordinates in the first intermediate result matrix from the position coordinates in the second intermediate result matrix, the position coordinates of the third element value in the first intermediate result matrix are referred to as "first position coordinates", and the position coordinates of the third element value in the second result matrix are referred to as "second position coordinates".

[0124] Exemplarily, in the first buffer 403, the first buffer 403 is divided into a plurality of storage locations by row coordinate identification and ordinate identification. For example, in the first buffer 403, the storage space of the first buffer 403 is divided into q×p storage locations by p rows and q columns. After receiving C0, the accumulator 402 writes the third element value in C0 into the corresponding position in the first buffer 403 according to the first position coordinate. For example, the position coordinate (i0, j0) of the third element value a0b0 in C0, if (i0, j0) is (1, 1), then the accumulator 402 writes a0b0 into the position of the 1st row and the 1st column in the first buffer 403 according to the position coordinate (i0, j0). Similarly, the position coordinates (i0, j1) of the third element value a0b1 in C0 are (1, 2), and the accumulator 402 writes a0b1 to the position of the 1st row and 2nd column in the first buffer 403 according to the position coordinates (i0, j1). The position coordinates (i0, j2) of the third element value a0b2 in C0 are (1, 3), and the accumulator 402 writes a0b2 to the position of the 1st row and 3rd column in the first buffer 403 according to the position coordinates (i0, j2). The position coordinates (i0, j3) of the third element value a0b3 in C0 are (1, 4), and the accumulator 402 writes a0b3 to the position of the 1st row and 4th column in the first buffer 403 according to the position coordinates (i0, j3). The process of the accumulator 402 writing other third element values ​​in C0 into the first buffer 403 is not described in detail. The final result is that the accumulator 402 writes all third element values ​​in C0 into the first buffer 403 according to the first position coordinates corresponding to each third element value.

[0125] Then, when the accumulator 402 receives C1, the accumulator 402 searches the first buffer 403 for a cache value at the corresponding position of the second position coordinate according to the second position coordinate of the third element value in C1. If there is no cache value at the corresponding position of the second position coordinate, the accumulator 402 writes the third element value into the corresponding position of the second position coordinate in the first buffer 403. If there is a cache value at the corresponding position of the second position coordinate, the accumulator 402 reads the cache value from the first buffer 403 and accumulates the cache value with the third element value in C1. For example, the position coordinate (k0, l0) of the third element value c0d0 in C1 is (1, 0), if (k0, l0) is (1, 0), the accumulator 402 searches the first buffer 403 according to the position coordinate (k0, l0). If there is no cache value at the position of the 1st row and 0th column in the first buffer 403, the accumulator 402 writes c0d0 into the position of the 1st row and 0th column in the first buffer 403. The position coordinates of the third element value c0d1 in C1 are (k0, l1). If (k0, l1) is (1, 1), the accumulator 402 queries the first buffer 403 and finds that there is a cache value a0b0 at the position of the 1st row and 1st column. The accumulator 402 reads a0b0, adds c0d1 and a0b0, and then writes the result of the addition (c0d1+a0b0) into the corresponding position (1st row and 1st column) of the first buffer 403 according to the position coordinates of c0d1. Similarly, the position coordinates (k0, l2) of the third element value c0d2 in C1, such as (k0, l2) is (1, 2), the accumulator 402 queries the first buffer 403 for the position of the first row and second column with the cache value a0b1, the accumulator 402 reads the cache value a0b1 from the first buffer 403, and adds c0d2 and a0b1, and then writes the added accumulated value (c0d2+a0b1) to the corresponding position (1st row and 2nd column) of the first buffer 403 according to the position coordinates of c0d2. In this example, the third element value in C1 is not illustrated one by one. The final result is that the accumulator 402 adds the third element values ​​with the same position coordinates in C1 and C0, and writes the added result to the corresponding position in the first buffer 403 according to the position coordinates. Similarly, when the accumulator 402 receives the intermediate result matrix C2 transmitted by the vector outer product processing engine 401, the accumulator 402 reads the cache value (the accumulated value of the third element values ​​with the same position coordinates in C0 and C1) from the corresponding position of the first buffer 403 according to the position coordinates of each third element value in C2, and the accumulator 402 accumulates the third element values ​​with the same position coordinates in C0, C1 and C2, and then writes the accumulated result into the corresponding position of the position coordinates in the first buffer 403. It should be understood that the processing process of the accumulator 402 for C2 and C3 is similar to the processing process for C1, and will not be repeated here.The final processing result of the accumulator 402 is to accumulate the third element values ​​with the same position coordinates in C0, C1, C2 and C3 to obtain the fifth element value, and the result matrix includes multiple fifth element values. Finally, the first buffer 403 outputs the result matrix.

[0126] It should be noted that the fifth element value is the accumulated value of the third element value in at least one of the intermediate result matrices C0, C1, C2 and C3. For example, please refer again to Figure 6 As shown, the position coordinates of a0b0 in C0 are (1, 0), while the position coordinates of all third element values ​​in the other three intermediate result matrices (C1, C2, and C3) do not have (1, 0), then the value written at the position of the 1st row and 0th column of the first buffer 403 is a0b0, and this a0b0 will not be accumulated with the third element values ​​in other intermediate result matrices. Similarly, the position coordinates of a0b1 in C0 and c0d1 in C1 are both (1, 1), while the position coordinates of the third element values ​​in the other two intermediate result matrices (C2 and C3) do not have (1, 1), then the value written at the position of the 1st row and 0th column of the first buffer 403 is the accumulated value of a0b1 and c0d1. In summary, it should be understood that a fifth element value may be a third element value, or it may be the accumulated value of two third element values, or it may be the accumulated value of three third element values, or it may be the accumulated value of four third element values, etc. In actual operations, a fifth element value in the result matrix is ​​obtained by summing up several third element values ​​and is determined by the number of corresponding third element values ​​at the same position coordinates, and is not specifically limited.

[0127] Furthermore, the fifth element value is the accumulated value of the third element value, so the fifth element value may be a zero element value. In this example, the result matrix is ​​a matrix in an uncompressed format. Optionally, in order to save transmission resources or facilitate the next calculation operation, the matrix calculation device may compress the result matrix to output a matrix in a compressed format. Figure 7 As shown, Figure 7 4 is another structural diagram of a matrix computing device. The matrix computing device and the matrix compression unit 404, the matrix compression unit 404 is connected to the first buffer 403, the first buffer 403 outputs the result matrix, the matrix compression unit 404 receives the result matrix, and converts the result matrix into a matrix in a compressed format according to the row coordinate identifier and the column coordinate identifier in the first buffer 403. The result matrix can be in COO format, CSR or CSC format, etc., and is not specifically limited.

[0128] It should be understood that, during the accumulation calculation process of the accumulator 402, the first buffer 403 is used to store the third element value and the result after accumulating the third element value. The storage space of the first buffer 403 needs to be greater than or equal to a certain value. For example, if the dimension of each intermediate result matrix is ​​M×P, that is, each intermediate result matrix includes M×P third element values. The number of cache locations included in the first buffer 403 is greater than or equal to M×P. Then the storage space of the first buffer 403 needs to be able to store at least M×P values.

[0129] In a first possible implementation, the matrix calculation device can output a matrix in an uncompressed format or a matrix in a compressed format according to different application scenarios, thereby increasing the applicable scenarios of the matrix calculation device.

[0130] In the second possible implementation, see Figure 8 As shown, the accumulator 402 sorts the third element values ​​in the N intermediate result matrices according to the position coordinates of the third element values, and then compares the position coordinates in the sorted N intermediate result matrices, adds the third element values ​​with the same position coordinates, and deletes the position coordinates of the zero element value to obtain a result matrix in a compressed format. For example, when the accumulator 402 receives C0, the accumulator 402 first sorts the third element values ​​in C0 according to the position coordinates of the third element values ​​and then writes them into the first buffer 403. The accumulator 402 can sort according to the row coordinates of the third element values. Optionally, the accumulator 402 can sort according to the column coordinates of the third element values. In this example, the accumulator 402 is sorted according to the row coordinates of the third element values ​​as an example for explanation. The accumulator 402 receives C1 and sorts the third element values ​​in C1 according to the row coordinates of the third element values. The accumulator 402 compares the position coordinates of the third element values ​​in C1 with the position coordinates of the third element values ​​in C0 in the order of the position coordinates of the third element values ​​in C1. For example, the accumulator 402 first compares the row coordinates, such as comparing k0 and i0. If k0 and i0 are consistent, the accumulator 402 continues to compare k0 and i1. After the accumulator 402 has compared the row coordinates, it compares the column coordinates l0 and j0, l0 and j1, etc. in the order of the position coordinates. When the position coordinates (k0, l0) and the position coordinates (i0, j1) are consistent, the accumulator 402 adds the third element value c0d0 corresponding to the position coordinates (k0, l0) and the third element value a0b0 corresponding to the position coordinates (i0, j1), and then writes the result of the addition into the first buffer 403.

[0131] Similarly, when the accumulator 402 receives C2, the accumulator 402 sorts the third element value in C2 according to the position coordinates, the accumulator 402 reads the position coordinates in the first buffer 403, and then compares the position coordinates of the third element value in C2 with the position coordinates in the first buffer 403, adds the third element value and the buffer value with the same position coordinates to obtain the accumulated value, and then stores the accumulated value in the first buffer 403. The calculation of the third element value in C3 by the accumulator 402 is similar to the calculation of the third element value in C2, and examples are not repeated here. The final result is that the accumulator 402 accumulates the third element values ​​with the same position coordinates in the intermediate result matrices C3, C2, C1 and C0 to obtain the fifth element value, and the accumulator 402 deletes the zero element value and the position coordinates of the zero element value in multiple fifth element values. The first buffer 403 outputs a result matrix, which contains the fifth element value and the position coordinates of the fifth element value. And the fifth element value included in the result matrix is ​​a non-zero element value, so the result matrix output by the first buffer 403 is a matrix in a compressed format.

[0132] In the second implementation, a matrix in a compressed format can be directly obtained, and the matrix in a compressed format output by the matrix calculation device can be applied to certain application scenarios where a matrix in a compressed format is required for subsequent calculations. And because the matrix calculation device outputs a matrix in a compressed format, the transmission resources for subsequent transmission of the matrix can be reduced. Moreover, in the second implementation, since the third element value cached in the first buffer 403 is cached in the order of the position coordinates, less cache space can be used to implement the implementation in this example, thereby saving the cache space of the first buffer 403.

[0133] Optionally, in this example, the format conversion unit 405 is also used to obtain the fifth matrix and the sixth matrix, and perform format conversion on the fifth matrix and the sixth matrix in the uncompressed format to obtain the first matrix and the second matrix in the compressed format. And the first matrix and the second matrix are output to the second buffer 406, wherein at least one of the fifth matrix and the sixth matrix is ​​a matrix in the uncompressed format. The vector outer product processing engine 401 obtains the first matrix and the second matrix in the compressed format from the second buffer 406. In this example, the matrix calculation device can receive a matrix in the uncompressed format, and then convert the matrix in the uncompressed format into a matrix in the compressed format, so that the matrix calculation device can support matrix calculations in multiple formats. In this example, the fifth matrix and the sixth matrix can include the following situations.

[0134] First case: Please refer to Fig. 9As shown, the fifth matrix and the sixth matrix are both matrices in uncompressed format. In this case, the format conversion unit 405 converts the fifth matrix into a first matrix in compressed format and the sixth matrix into a second matrix in compressed format. Optionally, the format conversion unit 405 can convert the fifth matrix and the sixth matrix into compressed matrices in COO format. Optionally, since the row coordinates of the element values ​​are retained in the CSC compressed format, and the column coordinates of the element values ​​are retained in the CSR compressed format, the format conversion unit can convert the fifth matrix in uncompressed format into a first matrix in CSC compressed format, and convert the sixth matrix in uncompressed format into CSR compressed format. Further, the format conversion unit 405 converts the first matrix in compressed format into N first column vectors, converts the second matrix in compressed format into N first row vectors, and writes the N first column vectors and the N first row vectors into the first buffer 406, and the vector outer product processing engine 401 obtains the first row vector and the second row vector from the second buffer 406.

[0135] The second case: one of the fifth matrix and the sixth matrix is ​​a matrix in an uncompressed format, and the other matrix is ​​a matrix in a compressed format. The fifth matrix is ​​a matrix in an uncompressed format, and the sixth matrix is ​​a matrix in a compressed format. If the sixth matrix is ​​a compressed matrix in CSC or CSR format. The format conversion unit converts both the fifth matrix and the sixth matrix into compressed matrices in COO format.

[0136] Optionally, the format conversion unit 405 is further configured to convert the matrix in the compressed format into a matrix in the target compressed format. For example, when both the first matrix and the second matrix are in CSC format or CSR format, the format conversion unit converts both the first matrix and the second matrix into COO format.

[0137] In this example, the matrix calculation device can convert a matrix in an uncompressed format into a matrix in a compressed format through a format conversion unit, so that the matrix calculation device can support both matrix calculations in compressed formats and matrix calculations in uncompressed formats. Optionally, the format conversion unit can convert a matrix in a non-target compressed format into a matrix in a target compressed format (such as a COO format). In this example, the matrix calculation device can convert matrices in other compressed formats into matrices in a target compressed format, and perform matrix calculations on matrices in the target compressed format. The matrix calculation device provided in this application can support matrix calculations in various formats.

[0138] Optionally, in order to improve the applicability of the matrix calculation unit, a high-precision matrix calculation can be implemented based on a low-precision matrix calculation device. The precision of the element values ​​included in the first column vector and the first row vector is the first precision. The format conversion unit 405 decomposes the first column vector into X second column vectors, and decomposes the first row vector into X second row vectors, wherein the second column vector and the second row vector contain the element values ​​of the second precision, and the first precision is higher than the second precision. Then, the vector outer product processing engine 401 calculates the vector outer product of the X second column vectors and the X second row vectors to obtain X 2 A fourth matrix is ​​formed, the fourth matrix including the fourth element value and the position coordinate of the fourth element value. Finally, the accumulator 402 converts X to 2 The fourth element values ​​with the same position coordinates in the fourth matrices are accumulated to obtain an intermediate result matrix, and the precision of the third element value in the intermediate result matrix is ​​the first precision.

[0139] The following example illustrates the decomposition of a vector of first precision into a vector of second precision in this example. After the format conversion unit 405 converts the first matrix in compressed format into N first column vectors and converts the second matrix in compressed format into N first row vectors, it can further decompose a first column vector into X second column vectors according to the precision of the element value, and decompose the first row vector into X second row vectors. Among them, the precision of the element values ​​included in the first column vector and the first row vector is the first precision. The second column vector and the second row vector both contain element values ​​of the second precision. For the convenience of explanation, the vector containing the element values ​​of the second precision is referred to as a "vector of the second precision", and the vector containing the element values ​​of the first precision is referred to as a "vector of the first precision". X is an integer greater than or equal to 2. The first precision and the second precision can be integer precision, or the first precision and the second precision can also be floating point precision, which is not specifically limited. Specifically, the format conversion unit 405 splits each element value in the vector of the first precision into multiple values ​​of the second precision, and disassembles the vector of the first precision into X vectors of the second precision. The following description is based on the case where the vector of the first precision and the vector of the second precision are integers, and the case where the vector of the first precision and the vector of the second precision are floating-point vectors.

[0140] In the first case, the vector of the first precision and the vector of the second precision are integers. The first precision is higher than the second precision, for example, the first precision is int32, and the second precision can be int2, int4, int8, or int16. Alternatively, the first precision is int16, and the second precision can be int4 or int8. Alternatively, the first precision is int8, and the second precision is int2 or int4. Alternatively, the first precision is int4, and the second precision is int2, etc.

[0141] The format conversion unit 405 splits a high-precision integer value into multiple low-precision integer values, and splits the high-precision integer value in order from high to low. Fig.10 As shown, the first precision is int32 and the second precision is int16. Int32 includes a 32-bit value range. The 32 bits in an int32 are split into 2 16 bits from high to low, that is, int32 is split into 2 int16. Or, the 32 bits in an int32 are split into 4 8 bits from high to low, that is, int32 is split into 4 int8. Similarly, int16 includes a 16-bit value range. The 16 bits in an int16 are split into 2 8 bits from high to low, that is, int16 is split into 2 int8.

[0142] In the second case, the vector of the first precision and the vector of the second precision are floating point types. For example, the first precision is FP32 or FP64, and the second precision is FP16. Alternatively, the first precision is FP64, and the second precision floating point can be FP32 or FP16. The following is an example of the case where the first precision floating point number is FP32 and the second precision floating point number is FP16.

[0143] 1. Decompose one FP32 to get three FP16.

[0144] At present, the composition of the standard format FP32 is shown in Table 1 above. FP32 includes a 1-bit sign, an 8-bit exponent and a 23-bit mantissa. In addition, there is an omitted 1-bit integer, and the omitted integer is 1. For a standard format FP32, the integer plus the mantissa is 24 bits in total. The composition of the standard format FP16 includes a 1-bit sign, a 5-bit exponent and a 10-bit mantissa. In addition, there is an omitted 1-bit integer, and the omitted integer is 1. For a standard format FP16, the integer plus the mantissa is 11 bits in total. If you want to decompose a standard format FP32 to get a standard format FP16, you need 3 standard format FP16s.

[0145] The integer and mantissa of the standard format FP32 can be divided into three parts. The first part is the integer and the first 10 bits of the mantissa, the second part is the mantissa from the 11th bit to the 21st bit, and the third part is the mantissa of the 22nd bit and the 23rd bit. These three parts are represented by a standard format FP16 respectively. It should be noted here that when the mantissa of the 22nd and 23rd bits of the third part is represented by the standard format FP16, 9 zeros can be added after the mantissa of the 23rd bit, that is, the mantissa of the 22nd and 23rd bits, as well as the added zeros, are represented by a standard format FP16.

[0146] In addition, the exponent range of FP16 is -15 to 15, which means that the decimal point can be shifted 15 bits to the left or 15 bits to the right. When the standard format FP16 is used to represent the first part of the above FP32, the fixed exponent displacement value is 0; when the standard format FP16 is used to represent the second part of the above FP32, the fixed exponent displacement value is -11; when the standard format FP16 is used to represent the third part of the above FP32, the fixed exponent displacement value is -22. It can be seen that when representing the third part, the corresponding fixed exponent displacement value alone has exceeded the exponent range of FP16. Therefore, the corresponding fixed exponent displacement value can be extracted for each standard format FP16 exponent.

[0147] Then, for a standard format FP32, it can be expressed as:

[0148] Among them, A1 is the standard format of FP32, EA1 is the exponent of A1, a0, a1 and a2 are three standard format FP16 obtained by decomposition, S1 is the minimum fixed exponent displacement value, and for the standard format FP16, S1=11.

[0149] In addition, for each standard format FP16 exponent, a common exponent displacement value can be extracted. Then, for a standard format FP32, it can be expressed as:

[0150] Among them, a0′, a1′ and a2′ are three standard formats of FP16 obtained by decomposition. In the above two representation methods, the decomposed FP16 has the following relationship: a1=a1′,

[0151] 2. Decompose one FP32 to get two FP16.

[0152] In order to reduce the number of FP16s obtained by decomposition, the current standard format FP16 can be adjusted to adjust its mantissa to 13 bits, and the number of bits of the sign and exponent remains unchanged. The adjusted FP16 can be called non-standard format FP16. The integer plus the mantissa of the non-standard format FP16 is 14 bits in total. Therefore, if the mantissa of a standard format FP32 is to be represented by a non-standard format FP16, only 2 non-standard format FP16s are needed.

[0153] The integer and mantissa of the standard FP32 format are divided into two parts, the first part is the integer and the first 13 bits of the mantissa, and the second part is the 14th to 23rd bits. These two parts are represented by non-standard FP16 format.

[0154] It should also be noted that when the second part is represented by non-standard FP16, four zeros can be added after the mantissa of the 23rd bit, that is, the mantissa from the 14th bit to the 23rd bit and the added zeros are represented by a non-standard format FP16. Similar to the above case 1, here, the corresponding fixed exponent displacement value can also be extracted for each standard format FP16 exponent.

[0155] Then, for a standard format FP32, it can be expressed as:

[0156] Among them, A2 is the standard format FP32, EA2 is the exponent of A2, a3 and a4 are two non-standard format FP16 obtained by decomposition, S2 is the fixed exponent displacement value, and for the non-standard format FP16, S2=14.

[0157] In addition, for each standard format FP16 exponent, a common exponent displacement value can be extracted. Then, for a standard format FP32, it can be expressed as:

[0158] Among them, a3′ and a4′ are two non-standard FP16 formats obtained by decomposition. In the above two representation methods, the decomposed FP16 has the following relationship: a4=a4′.

[0159] Of course, for the case where the first precision floating point number is FP64 and the second precision floating point number is FP32, the following situations may occur for decomposing FP64 to obtain multiple FP32s: decomposing one FP64 floating point number to obtain three FP32 floating point numbers; or, decomposing one FP64 floating point number to obtain two FP32 floating point numbers. Optionally, for the case where the first precision floating point number is FP64 and the second precision floating point number is FP16, the following situations may occur for decomposing FP64 to obtain multiple FP16s: decomposing one FP64 floating point number to obtain five FP16 floating point numbers; or, decomposing one FP64 floating point number to obtain four FP16 floating point numbers. The splitting principle is similar to the above-described case where the first precision floating point number is FP32 and the second precision floating point number is FP16, and will not be elaborated here.

[0160] For example, for the convenience of explanation, the first row vector is split into two second row vectors, and the first column vector is split into two second column vectors. For example, the first row vector A0 is a column vector [a0, a1, a2, a3] with a precision of FP32. T, as described above, the method of splitting the first precision value into two second precision values, the format conversion unit 405 splits the floating point number a0 with the precision of FP32 into two FP16 floating point numbers (such as a 0M and a 0L ). Similarly, split a1 into a 1M and a 1L . Split a2 into a 2M and a 2L . Split a3 into a 3M and a 3L That is, please refer to Figure 11, the format conversion unit 405 converts the column vector [a0 a1 a2 a3] with a precision of FP32 T Split into two column vectors with FP16 precision [a 0M a 1M a 2M a 3M ] T (remembered as "second column vector 1") and [a 0L a 1L a 2L a 3L ] T (Remembered as "second column vector 2"). Similarly, B0 is a row vector [b0b1 b2 b3] of FP32 precision, and [b0 b1 b2 b3] is also split into two FP16 row vectors [b 0M b 1M b 2M b 3M ] (remembered as "second row vector 1") and [b 0L b 1L b 2L b 3L ] (referred to as "second row vector 2"). It should be noted that the row coordinates of each element value in the first column vector and the second column vector are the same, that is, a 0M and a 0L The row coordinates are i0, a 1M and a 1L The row coordinates are i1, a 2M and a 2L The row coordinate of a is i2. 3M and a 3L The row coordinate of is i3. The column coordinates of each element value in the first row vector and the second row vector are the same, that is, b 0M and b 0L The column coordinates are j0, b 1M and b 1L The column coordinates are j1, b 2M and b 2L The column coordinates are j2, b 3M and b3L The column coordinate of is j3.

[0161] Further, the vector outer product processing engine 401 calculates the vector outer product of the X second column vectors and the X second row vectors to obtain X 2 A fourth matrix includes the fourth element value and the position coordinates of the fourth element value. The position coordinates of the fourth element value include the row coordinates of the element value in the first column vector and the column coordinates of the element value in the first row vector. The precision of the fourth element value is the first precision (such as FP32). In this example, refer to Fig.12 As shown, X is taken as 2 for example. The vector outer product processing engine 401 calculates [a 0M a 1M a 2M a 3M ] T and [a 0L a 1L a 2L a 3L ] T The two second column vectors are related to [b 0M b 1M b 2M b 3M ] and [b 0L b 1L b 2L b 3L ] The vector outer product of the two second row vectors. That is, the vector outer product processing engine 401 calculates [a 0M a 1M a 2M a 3M ] T and [b 0M b 1M b 2M b 3M ], we get the fourth matrix 1; calculate [a 0M a 1M a 2M a 3M ] T and [b 0L b 1L b 2L b 3L ], we get the fourth matrix 2; calculate [a 0L a 1L a 2L a 3L ] T and [b 0M b 1M b 2M b 3M ]'s vector outer product, we get the fourth matrix 3; [a0L a 1L a 2L a 3L ] T and [b 0L b 1L b 2L b 3L ] to obtain the fourth matrix 4. The vector outer product processing engine 401 obtains four fourth matrices through vector outer product calculation. The accumulator 402 obtains the four fourth matrices and, according to the index of the position coordinate of the fourth element value, accumulates the fourth element values ​​with the same position coordinates in the four fourth matrices to obtain an intermediate result matrix. The precision of the third element value in the intermediate result matrix is ​​the first precision. For example, the fourth element value a in the fourth matrix 1 0M b 0L , the fourth element value a in the fourth matrix 2 0M b 1M , the fourth element value a in the fourth matrix 3 1L b 0L , the fourth element value a in the fourth matrix 4 1L b 1M The position coordinates of a are the same, and the accumulator 402 adds a 0M b 0L 、a 0M b 1M 、a 1L b 0L and a 1L b 1M The four fourth element values ​​are accumulated, and the accumulated value is a third element value in the intermediate result matrix C0. 2 The specific method of accumulating the fourth element values ​​with the same position coordinates in the fourth matrix is ​​as follows: Figure 6 and Figure 8 The method of accumulating by the accumulator 402 in the corresponding example is similar. For details, please refer to the above description of Figure 5E , Figure 6 and Figure 8 The method performed by the accumulator 402 in the corresponding example is not described here. In this example, when calculating the vector outer product of two vectors, the vector of the first precision (high precision) can be further split into multiple vectors of the second precision (low precision), and then the vector outer product calculation is performed on the low-precision vectors, so that the outer product of the vector of the first precision can be accumulated by the outer product results of multiple vectors of the second precision, and the accuracy will not be lost.

[0162] In this example, if the precision of the element values ​​in the first row vector and the first column vector is large, the matrix calculation device can decompose the first row vector and the first column vector into multiple low-precision vectors. The matrix calculation device performs vector outer product calculation on the low-precision second column vector and the second row vector to obtain the result matrix, which can not only calculate the matrix in the compressed format, but also realize high-precision matrix calculation based on the low-precision matrix calculation device, thereby improving the applicability of the matrix calculation device. Moreover, during the matrix calculation process, the upper-level software applications (such as AI and HPC) based on the matrix calculation device do not perceive the specific process of the matrix calculation, thereby greatly reducing the cost of software adaptation.

[0163] Based on the matrix computing device provided in this application, great benefits can be obtained in multiple matrix computing scenarios. For example, the matrix computing device can be used in AI training and reasoning scenarios, and can fully support matrix calculations in compressed formats and matrix calculations in uncompressed formats. For the sparse characteristics of weights and feature data in AI calculations, the average is more than 50% (that is, more than 50% are compressed matrices). The matrix computing format in this application can directly calculate the matrix in compressed format, and there is no need to decompress the matrix in compressed format. In this way, the computing efficiency can be improved by more than 4 times. In addition, for HPC scenarios such as scientific computing, whether it is a matrix calculation in an uncompressed format with high computing power requirements or a matrix calculation scenario with limited memory bandwidth, the matrix computing device in this application can directly access the matrix in compressed format from the memory, thereby improving the computing benefits. The matrix computing device supports full-precision numerical calculations, and can also effectively cover calculations with various different precision requirements, such as AI training scenarios often require floating-point calculations such as FP32 and FP16, and some AI training scenes and HPC scientific computing scenarios that require FP64, which can be fully supported by the matrix computing device. In addition, the MAC in the matrix computing device can also support medium and low precision integer format calculations such as INT1, INT2, INT4, INT8, etc. For AI reasoning computing scenarios, it can improve computing power and reduce reasoning calculation time. It can also provide mixed reasoning calculation scenarios with various different precisions, greatly enhancing the applicability of the matrix computing device.

[0164] An embodiment of a matrix computing device is described above, and a method performed by the matrix computing device is described below. Fig.13 As shown, the embodiment of the present application provides a matrix calculation method, and the execution subject of the method can be Figure 2 The computing device shown, optionally, the execution subject of the method is Figure 3 The matrix calculation device shown in , optionally, the execution subject of the method can be Figure 4A The matrix computing device shown, optionally, the execution subject of the method can be Figure 7 The matrix calculation device shown.

[0165] Step 1301: Obtain a first calculation instruction, the first calculation instruction including N first column vectors and N first row vectors. The N first column vectors are obtained by transforming a first matrix in a compressed format, and the N first row vectors are obtained by transforming a second matrix in a compressed format, where N is an integer greater than or equal to 1.

[0166] Step 1302, calculate the vector outer products of the N first column vectors and the N first row vectors to obtain N intermediate result matrices, wherein the first column vector includes the first element value and the row coordinate of the first element value, the first row vector includes the second element value and the column coordinate of the second element value, the intermediate result matrix includes the third element value and the position coordinate of the third element value, and the position coordinate includes the row coordinate of the first element value and the column coordinate of the second element value.

[0167] For this step, please refer to the above matrix calculation device embodiment. Figure 5B , Figure 5C and Figure 5D In the corresponding example, the specific description of the functions performed by the vector outer product processing engine 401 is not repeated here.

[0168] Step 1303: According to the index of the position coordinate of the third element value, the third element values ​​with the same position coordinates in the N intermediate result matrices are accumulated to obtain a result matrix.

[0169] The N intermediate result matrices include at least a first intermediate result matrix and a second intermediate result matrix. The position coordinates of the third element value in the first intermediate result matrix are first position coordinates, and the position coordinates of the third element value in the second intermediate result matrix are second position coordinates.

[0170] In a first possible implementation, the matrix calculation device writes the third element value in the first intermediate result matrix into the corresponding position in the buffer according to the first position coordinate in the order in which the N intermediate result matrices are generated. Then, according to the second position coordinate of the third element value in the second intermediate result matrix, the cache value of the corresponding position in the buffer at the second position coordinate is read, and the third element value in the second intermediate result matrix and the cache value are accumulated to obtain a result matrix in an uncompressed format. Optionally, the matrix calculation device compresses the result matrix in an uncompressed format to obtain a result matrix in a compressed format.

[0171] In a first possible implementation, please refer to the above matrix calculation device embodiment. Figure 5E and Figure 6 In the corresponding example, the specific description of the function performed by the accumulator 402 is not repeated here.

[0172] In a second possible implementation, the matrix calculation device sorts the third element values ​​in the N intermediate result matrices according to the position coordinates of the third element values. The matrix calculation device compares the position coordinates in the sorted N intermediate result matrices, adds the third element values ​​with the same position coordinates, and deletes the position coordinates of the zero element value to obtain a result matrix in a compressed format.

[0173] In a second possible implementation, please refer to the above matrix calculation device embodiment. Figure 5E and Figure 8 In the corresponding example, the specific description of the function performed by the accumulator 402 is not repeated here.

[0174] In the embodiment of the present application, the matrix calculation device can directly calculate the matrix in the compressed format, without decompressing the matrix in the compressed format as in the traditional method and performing matrix calculation on the decompressed matrix. The matrix calculation device in the embodiment of the present application can improve the calculation efficiency of the matrix in the compressed format.

[0175] Optionally, see Fig.14 As shown, in order to support matrix calculations in multiple formats, the matrix calculation device can convert a matrix in an uncompressed format into a matrix in a compressed format, so that the matrix in the uncompressed format can be calculated. The format conversion of the matrix in the uncompressed format is shown in the following steps 1401 and 1402.

[0176] Step 1401: Obtain a third calculation instruction, where the third calculation instruction includes a fifth matrix and a sixth matrix, wherein at least one of the fifth matrix and the sixth matrix is ​​a matrix in an uncompressed format.

[0177] Step 1402: Perform format conversion on the fifth matrix to obtain a first matrix in a compressed format, and perform format conversion on the sixth matrix to obtain a second matrix.

[0178] For step 1401 and step 1402, please refer to the above-mentioned embodiment of the matrix calculation device. Fig. 9 In the corresponding example, the specific description of the functions performed by the format conversion unit 405 is not repeated here.

[0179] In an embodiment of the present application, the matrix calculation device can convert a matrix in an uncompressed format into a matrix in a compressed format through a format conversion unit, so that the matrix calculation device can support both matrix calculations in compressed formats and matrix calculations in uncompressed formats. Optionally, the format conversion unit can convert a matrix in a non-target compressed format into a matrix in a target compressed format (such as a COO format). In this example, the matrix calculation device can convert matrices in other compressed formats into matrices in a target compressed format, and perform matrix calculations on matrices in the target compressed format. The matrix calculation device provided in the present application can support matrix calculations in various formats.

[0180] Step 1403: Obtain a second calculation instruction, where the second calculation instruction includes a first matrix and a second matrix.

[0181] Step 1404: Convert the first matrix into N first column vectors, and convert the second matrix into N first row vectors.

[0182] For step 1403 and step 1404, please refer to the above-mentioned embodiment of the matrix calculation device. Figure 5A In the corresponding example, the description of the functions performed by the format conversion unit 405 is not repeated here.

[0183] Optionally, in order to improve the applicability of the matrix calculation unit, a high-precision matrix calculation can be implemented based on a low-precision matrix calculation device. The vector of the first precision is split into multiple vectors of the second precision, and then the vector outer product calculation is performed on the vectors of the second precision. Please refer to the following steps 1405 to 1407.

[0184] Step 1405: split the first column vector into X second column vectors, and split the first row vector into X second row vectors. The precision of the element values ​​included in the first column vector and the first row vector is the first precision, and the precision of the element values ​​included in the second column vector and the second row vector is the second precision, the first precision is higher than the second precision, and X is an integer greater than or equal to 2.

[0185] For this step, please refer to the above-mentioned embodiment of the matrix calculation device. Fig.10 and Fig.11 In the corresponding example, the specific description of the format conversion unit 405 splitting the integer value and the format conversion unit 405 splitting the floating-point value is not repeated here.

[0186] Step 1406: Calculate the vector outer product of the X second column vectors and the X second row vectors to obtain X 2 A fourth matrix is ​​formed, the fourth matrix includes a fourth element value and a position coordinate of the fourth element value, the position coordinate of the fourth element value includes a row coordinate of the first element value and a column coordinate of the second element value, and the precision of the fourth element value is the first precision.

[0187] For this step, please refer to the above-mentioned embodiment of the matrix calculation device. Fig.12 In the corresponding example, the description of the functions performed by the vector outer product processing engine 401 is not repeated here.

[0188] Step 1407: According to the index of the position coordinates of the fourth element value, 2 The fourth element values ​​with the same position coordinates in the fourth matrices are accumulated to obtain an intermediate result matrix, and the precision of the third element value in the intermediate result matrix is ​​the first precision.

[0189] Step 1408: According to the index of the position coordinate of the third element value, the third element values ​​with the same position coordinates in the N intermediate result matrices are accumulated to obtain a result matrix.

[0190] For steps 1407 and 1408, please refer to the above-mentioned embodiment of the matrix calculation device. Figure 5E , Figure 6 and Figure 8 The functions performed by the accumulator 402 in the corresponding example are not described in detail here.

[0191] In an embodiment of the present application, when calculating the vector outer product of two vectors, the vector of the first precision (high precision) can be further split into multiple vectors of the second precision (low precision), and then the vector outer product calculation is performed on the low-precision vectors, so that the outer product of the vector of the first precision can be obtained by accumulating the outer product results of multiple vectors of the second precision, and the accuracy will not be lost.

[0192] In one embodiment of the present application, a matrix calculation circuit is provided, and the matrix calculation circuit is used in one or more steps of steps 1301 to 1303 in the above method embodiment, or is used to execute one or more steps of steps 1401 to 1408. In practical applications, the matrix calculation circuit can be an ASIC, an FPGA, or a logic circuit, etc.

[0193] In another embodiment of the present application, a matrix computing system or chip is also provided. The structure of the system or chip can be as follows: Figure 3 As shown, it includes: a processor (taking a central processing unit as an example) 1 and a matrix calculation device 1. The processor 1 is used to send a calculation instruction to the matrix calculation device 1, and the matrix calculation device 2 is used to execute one or more steps of steps 1301 to 1303 in the above method embodiment, or one or more steps of steps 1401 to 1408.

[0194] In another embodiment of the present application, a matrix computing device is provided, the structure of which can be as follows: Figure 2As shown, the device may specifically be a PCIe card, a SoC, a processor, and a server including the above hardware. Figure 2 The device includes: a memory 201, a processor 202, a communication interface 203 and a bus 204. The communication interface 203 may include an input interface and an output interface.

[0195] The processor 202 may be configured to execute one or more steps of steps 1301-1303 in the above method embodiment, or one or more steps of steps 1401-1408. In some feasible embodiments, the processor 202 may include a matrix calculation unit, which may be used to support the processor to execute one or more steps in the above method embodiment. In practical applications, the matrix calculation unit may be an ASIC, an FPGA, or a logic circuit, etc. Of course, the matrix calculation unit may also be implemented by software, and the present application embodiment does not impose specific restrictions on this.

[0196] It should be noted that the various components of the above-mentioned matrix computing circuit, matrix computing system, matrix computing device, etc. provided in the embodiments of the present application are respectively used to implement the functions of the corresponding steps of the aforementioned method embodiments. Since each step has been described in detail in the aforementioned method embodiments, it will not be repeated here.

[0197] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid state drive (SSD).

[0198] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A matrix computing device, characterized in that: include: A vector outer product processing engine, used for obtaining N first column vectors and N first row vectors, calculating the vector outer products of the N first column vectors and the N first row vectors, and obtaining N intermediate result matrices, wherein the first column vector includes a first element value and a row coordinate of the first element value, the first row vector includes a second element value and a column coordinate of the second element value, the intermediate result matrix includes a third element value and a position coordinate of the third element value, and the position coordinate includes the row coordinate of the first element value and the column coordinate of the second element value; the N first column vectors are obtained by converting a first matrix in a compressed format, the N first row vectors are obtained by converting a second matrix in a compressed format, and N is an integer greater than or equal to 1; The accumulator is used to accumulate the third element values ​​with the same position coordinates in the N intermediate result matrices according to the index of the position coordinates of the third element value to obtain a result matrix.

2. The device according to claim 1, characterized in that The N intermediate result matrices include at least a first intermediate result matrix and a second intermediate result matrix, the position coordinates of the third element value in the first intermediate result matrix are first position coordinates, and the position coordinates of the third element value in the second intermediate result matrix are second position coordinates; The accumulator is also specifically used for: According to the generation order of the N intermediate result matrices, write the third element value in the first intermediate result matrix into the corresponding position in the buffer according to the first position coordinate; According to the second position coordinate of the third element value in the second intermediate result matrix, read the cache value of the position corresponding to the second position coordinate in the buffer, accumulate the third element value in the second intermediate result matrix and the cache value, and obtain a result matrix in an uncompressed format.

3. The device according to claim 2, characterized in that The device also includes a matrix compression unit; The matrix compression unit is used to compress the result matrix in the uncompressed format to obtain the result matrix in the compressed format.

4. The device according to claim 1, characterized in that The accumulator is also specifically used for: Sort the third element values ​​in the N intermediate result matrices according to the position coordinates of the third element values; The position coordinates in the sorted N intermediate result matrices are compared, the third element values ​​with the same position coordinates are added, and the position coordinates with zero element values ​​are deleted to obtain a result matrix in a compressed format.

5. The device according to claim 1, characterized in that The device also includes a format conversion unit; The format conversion unit is used to obtain the first matrix and the second matrix, convert the first matrix into N first column vectors, and convert the second matrix into N first row vectors.

6. The device according to claim 5, characterized in that The device also includes a format conversion unit; The format conversion unit is further used to obtain a fifth matrix and a sixth matrix, perform format conversion on the fifth matrix to obtain the first matrix, and perform format conversion on the sixth matrix to obtain the second matrix, wherein at least one of the fifth matrix and the sixth matrix is ​​a matrix in an uncompressed format.

7. The device according to any one of claims 1 to 6, characterized in that The precision of the element values ​​included in the first column vector and the first row vector is a first precision; The vector outer product processing engine is also used to calculate the vector outer product of the X second column vectors and the X second row vectors to obtain X 2 a fourth matrix, the fourth matrix including a fourth element value and a position coordinate of the fourth element value, the position coordinate of the fourth element value including a row coordinate of the first element value and a column coordinate of the second element value, and the precision of the fourth element value is a first precision; wherein the X second column vectors are obtained by splitting the first column vector, the X second row vectors are obtained by splitting the first row vector, the precision of the element values ​​included in the second column vector and the second row vector is a second precision, the first precision is higher than the second precision, and X is an integer greater than or equal to 2; The accumulator is further used to convert the X 2 The fourth element values ​​with the same position coordinates in the fourth matrices are accumulated to obtain the intermediate result matrix, and the precision of the third element values ​​in the intermediate result matrix is ​​the first precision.

8. A matrix calculation method, characterized in that: include: Get N first column vectors and N first row vectors; Calculate the vector outer products of N first column vectors and N first row vectors to obtain N intermediate result matrices, wherein the first column vector includes a first element value and a row coordinate of the first element value, the first row vector includes a second element value and a column coordinate of the second element value, the intermediate result matrix includes a third element value and a position coordinate of the third element value, and the position coordinate includes the row coordinate of the first element value and the column coordinate of the second element value; the N first column vectors are obtained by converting the first matrix in a compressed format, the N first row vectors are obtained by converting the second matrix in a compressed format, and N is an integer greater than or equal to 1; According to the index of the position coordinate of the third element value, the third element values ​​with the same position coordinate in the N intermediate result matrices are accumulated to obtain a result matrix.

9. The method according to claim 8, characterized in that The N intermediate result matrices include at least a first intermediate result matrix and a second intermediate result matrix, the position coordinates of the third element value in the first intermediate result matrix are first position coordinates, and the position coordinates of the third element value in the second intermediate result matrix are second position coordinates; According to the index of the position coordinate of the third element value, the third element values ​​with the same position coordinates in the N intermediate result matrices are accumulated to obtain a result matrix, including: According to the generation order of the N intermediate result matrices, write the third element value in the first intermediate result matrix into the corresponding position in the buffer according to the first position coordinate; According to the second position coordinate of the third element value in the second intermediate result matrix, read the cache value of the position corresponding to the second position coordinate in the buffer, accumulate the third element value in the second intermediate result matrix and the cache value, and obtain a result matrix in an uncompressed format.

10. The method according to claim 9, characterized in that The method further comprises: The result matrix in the uncompressed format is compressed to obtain a result matrix in a compressed format.

11. The method according to claim 8, characterized in that According to the index of the position coordinate of the third element value, the third element values ​​with the same position coordinates in the N intermediate result matrices are accumulated to obtain a result matrix, including: Sort the third element values ​​in the N intermediate result matrices according to the position coordinates of the third element values; The position coordinates in the sorted N intermediate result matrices are compared, the third element values ​​with the same position coordinates are added, and the position coordinates with zero element values ​​are deleted to obtain a result matrix in a compressed format.

12. The method according to claim 8, characterized in that Before obtaining the first computing instruction, the method further includes: Obtain a second calculation instruction, where the second calculation instruction includes the first matrix and the second matrix; The first matrix is ​​converted into N first column vectors, and the second matrix is ​​converted into N first row vectors.

13. The method according to claim 12, characterized in that Before obtaining the second computing instruction, the method further includes: Obtaining a third calculation instruction, the third calculation instruction comprising a fifth matrix and a sixth matrix, wherein at least one of the fifth matrix and the sixth matrix is ​​a matrix in an uncompressed format; The fifth matrix is ​​format-converted to obtain the first matrix in a compressed format, and the sixth matrix is ​​format-converted to obtain the second matrix.

14. The method according to any one of claims 8 to 13, characterized in that: The precision of the element values ​​included in the first column vector and the first row vector is a first precision; The vector outer products of the N first column vectors and the N first row vectors are calculated to obtain N intermediate result matrices, including: Calculate the vector outer product of the X second column vector and the X second row vector to get X 2 a fourth matrix, the fourth matrix including a fourth element value and a position coordinate of the fourth element value, the position coordinate of the fourth element value including a row coordinate of the first element value and a column coordinate of the second element value, and the precision of the fourth element value is a first precision; wherein the X second column vectors are obtained by splitting the first column vector, the X second row vectors are obtained by splitting the first row vector, the precision of the element values ​​included in the second column vector and the second row vector is a second precision, the first precision is higher than the second precision, and X is an integer greater than or equal to 2; The method further comprises: According to the index of the position coordinates of the fourth element value, the X 2 The fourth element values ​​with the same position coordinates in the fourth matrices are accumulated to obtain the intermediate result matrix, and the precision of the third element values ​​in the intermediate result matrix is ​​the first precision.

15. A circuit for matrix calculation, characterized in that: The matrix calculation circuit is used to execute the method according to any one of claims 8 to 14.

16. A system for matrix calculation, characterized in that: The system includes a processor and a matrix calculation device, wherein the processor is used to send calculation instructions to the matrix calculation device, and the matrix calculation device is used to execute the method as described in any one of claims 8-14.

17. A chip, characterized in that: The chip includes a processor, in which a matrix calculation device is integrated. The matrix calculation device is used to execute the matrix calculation method as described in any one of claims 8-14.

Images