Apparatus and method for matrix operations

By compressing sparse matrices and operating only on non-zero elements, the method addresses inefficiencies in matrix operations, reducing calculations and storage needs, thus improving efficiency and resource utilization.

JP2024166373AInactive Publication Date: 2024-11-28ARM LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
JP2024161294
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2019-05-21
Filing Date
2024-09-18
Publication Date
2024-11-28
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

Matrix operations, particularly in sparse matrices used in convolutional neural networks, involve numerous wasteful multiplications by zero, consuming circuit time and power, and require significant storage space, necessitating more efficient methods for performing and storing these operations.

Method used

A data processing device and method that compresses sparse matrices by determining element interactions based on correspondence data, performing operations only on non-zero elements, and using aggregation circuits to generate partial results, reducing the number of calculations and storage requirements.

Benefits of technology

This approach reduces the number of calculations and storage needs by identifying and operating on non-zero elements, thereby enhancing efficiency and resource utilization in matrix operations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024166373000001_ABST
    Figure 2024166373000001_ABST
Patent Text Reader

Abstract

To provide a data processing apparatus to perform an operation on a first matrix and a second matrix.SOLUTION: The data processing apparatus includes receiver circuitry to receive elements of the first matrix, elements of the second matrix, and correspondence data indicating where the elements of the first matrix are located in the first matrix. Determination circuitry performs, using the correspondence data, a determination of whether, for a given element of the first matrix in column i of the first matrix, a given element of the second matrix occurs in row i of the second matrix. Aggregation circuitry calculates an aggregation between a given row of the first matrix and a given column of the second matrix, and includes functional circuitry to perform, according to the determination, a function on the given element of the first matrix and the given element of the second matrix to produce a partial result.SELECTED DRAWING: Figure 9
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to data processing. [Background technology]

[0002] When performing a matrix operation such as matrix multiplication, many sub-steps may be performed. Often these sub-steps have little impact on the overall matrix operation. For example, when multiplying together two sparse matrices (ones that contain a relatively large percentage of entries with a value of 0), many multiplications by zero are performed. This may be particularly the case for convolutional neural networks (CNNs), which are often used, for example, in image classification. Although such multiplications may be wasteful because multiplications by zero result in a zero result, the process of actually performing the multiplication may still consume circuit time and power. Furthermore, storing such matrices may use a lot of storage space. It is therefore desirable to be able to perform such operations using fewer sub-steps. It is also desirable for such matrices to be stored using less storage space, thereby reducing resource consumption, as well as power and potentially circuit size. Summary of the Invention

[0003] Viewed from a first exemplary configuration, a data processing apparatus is provided for performing operations on a first matrix and a second matrix, the data processing apparatus comprising: a receiver circuit for receiving elements of the first matrix, elements of the second matrix, and corresponding data indicative of where the elements of the first matrix are located within the first matrix; a decision circuit for using the corresponding data to perform a decision whether, for a given element of the first matrix in row i, column ji of the first matrix, a given one of the elements of the second matrix occurs in row j, column i of the second matrix; and an aggregation circuit for computing an aggregation between a given row i of the first matrix and a given column i of the second matrix, the aggregation circuit comprising a function circuit for performing a function on the given element of the first matrix and the given element of the second matrix to generate a partial result in response to the decision.

[0004] Viewed from a second exemplary configuration, a data processing method is provided for performing an operation on a first matrix and a second matrix, the method comprising the steps of: receiving elements of the first matrix, elements of the second matrix, and corresponding data indicating where the elements of the first matrix are located within the first matrix; using the corresponding data to perform a determination whether, for a given element of the first matrix in column i of the first matrix, a given element of the second matrix occurs in row i of the second matrix; and calculating an aggregation between the given row of the first matrix and the given column of the second matrix, generating a partial result by performing a function on the given element of the first matrix and the given element of the second matrix in response to the determination.

[0005] Viewed from a third exemplary configuration, a data processing apparatus is provided, the data processing apparatus comprising a decode circuit for receiving one or more instructions including a matrix operation instruction, the matrix operation instruction referencing elements of a first matrix, elements of a second matrix, and corresponding data indicating where the elements of the first matrix are located within the first matrix, and in response to receiving the matrix operation instruction, the decode circuit generates one or more control signals to cause one or more execution units to perform a matrix operation on the first matrix and the second matrix. [Brief description of the drawings]

[0006] The invention will be further described, by way of example only, with reference to embodiments thereof illustrated in the accompanying drawings, in which: [Figure 1] 1 illustrates a schematic diagram of an exemplary data processing apparatus according to some embodiments. [Diagram 2] 1 illustrates a matrix multiply-add operation and shows how the location of a particular element in a matrix can be represented. [Diagram 3] 3 shows a portion of the matrix of FIG. 2 in compressed form, along with corresponding data showing how the elements of the compressed matrix are arranged in the (original) matrix. [Figure 4] 4 illustrates how the compression matrix shown in FIG. 3 can be stored in a register. [Diagram 5]1 illustrates a circuit capable of performing matrix operations within the compressed matrices illustrated previously, according to some embodiments. [Figure 6] 1 illustrates a matrix multiply-add operation where the second matrix is ​​uncompressed. [Figure 7] 7 illustrates how the uncompressed second matrix shown in FIG. 6 is stored in a register. [Figure 8] 1 illustrates a circuit capable of performing matrix operations within the compressed matrices illustrated previously, according to some embodiments. [Figure 9] 4 illustrates a circuit for decoding a matrix operation instruction according to some embodiments. [Figure 10] 1 shows a flowchart illustrating a method of data processing according to some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] Before discussing the embodiments with reference to the accompanying drawings, the following embodiments are described.

[0008] According to some embodiments, there is provided a data processing apparatus for performing operations on a first matrix and a second matrix, the data processing apparatus comprising: a receiver circuit for receiving elements of the first matrix, elements of the second matrix, and corresponding data indicative of where the elements of the first matrix are located within the first matrix; a decision circuit for using the corresponding data to perform a decision of whether, for a given element of the first matrix in column i of the first matrix, a given element of the second matrix occurs in row i of the second matrix; and an aggregation circuit for computing an aggregation between a given row of the first matrix and a given column of the second matrix, the aggregation circuit comprising a function circuit for performing a function on the given element of the first matrix and the given element of the second matrix to generate a partial result in response to the decision.

[0009] Some or all of the elements of the first and second matrices are received together with corresponding data. The corresponding data is used to indicate where not all elements are provided and where the providing elements are located in the first and / or second matrices. The decision circuit provides an indication of whether two of the providing elements (one of the first matrix and one of the second matrix) will "interact" with each other, e.g., for one of the given elements in column i of the first matrix, one of the given elements of the second matrix is ​​present in row i, such that the two elements will interact with each other if a particular operation (such as multiplication or addition) is performed between the first and second matrices. Based on this determination, the function circuit performs a function on the given elements of the first and second matrices. For example, if the two elements interact, the function is performed to generate a partial result, but if the two elements do not interact, the function is not performed (or simply returns a "0" as the partial result). The partial results are then aggregated together. The aggregation is performed for a given row of the first matrix and a given column of the second matrix. In this way, it is possible to perform operations between the first matrix and the second matrix without providing all elements of the first matrix and the second matrix. In particular, by determining where the interaction between the first matrix and the second matrix occurs, it is possible to limit the number of operations that must be calculated overall. Furthermore, storage requirements can be reduced because it is only necessary to store the elements of the first matrix, the elements of the second matrix, and the corresponding data rather than the entire first matrix and the entire second matrix.

[0010] In some embodiments, the elements of the first matrix are elements of a compressed form of the first matrix, and the compressed form of the first matrix has been generated by removing at least some entries that are zero from the first matrix, and removing at least some columns or rows whose entries are zero. By removing some of the entries that have zero, and removing at least some of the columns or rows whose entries are zero, it is possible to reduce the overall size of the matrix and the number of entries that need to be provided to the matrix. In some embodiments, removal of non-zero values ​​may be used. However, it should be understood that in the case of a sparse matrix (where a high percentage of the values ​​are zero), many operations may be performed in which one of the operands is zero. Such operations may be time consuming, even if the results are predictable or even irrelevant. By removing such values ​​from the matrix, the number of operations may be reduced.

[0011] In some embodiments, the correspondence data indicates a row number and a column number for each of the elements of the first matrix, where the row number and column number for a given element in the matrix corresponds to the row and column where the given element is located in the first matrix. Thus, the correspondence data indicates how to transform the elements of the first matrix into the first matrix. This can be used to compress the first matrix by omitting certain known values. Then, instead of providing all the elements of the matrix, compression can be achieved by only providing some of the elements of the matrix together with where these elements are actually located.

[0012] In some embodiments, the receiver circuit is adapted to receive further correspondence data indicative of where the elements of the second matrix are located within the second matrix, and the determination circuit is adapted to perform the determination using the further correspondence data in addition to the correspondence data. Similar to the correspondence data providing location information for the provided elements of the first matrix, the further correspondence data provides location information for the provided elements of the second matrix.

[0013] In some embodiments, the further corresponding data indicates a row number and a column number for each of the elements of the second matrix, the row number and column number for a given element within the elements corresponding to the row and column in which the given element is located in the second matrix. Note that there is no obligation for the corresponding data and the further corresponding data to have the same format, or even exactly the same information.

[0014] In some embodiments, the elements of the second matrix are elements of a compressed form of the second matrix, the compressed form of the second matrix being generated by removing at least some entries from the second matrix that are zero and removing at least some columns or rows whose entries are zero. By providing both the first matrix and the second matrix in compressed form, it is possible to reduce the amount of storage required to perform an operation on the two matrices and also to reduce the number of calculations performed in performing the operation.

[0015] In some embodiments, the operation includes a multiplication operation on a first matrix and a second matrix, the aggregation circuit includes a dot product circuit that calculates a dot product of a given row and a given column as the aggregation, the function circuit includes a multiplication circuit that performs multiplication as the function, and the dot product circuit includes an adder circuit that adds each partial result generated for a given row and a given column. Performing a multiplication between two matrices can be time consuming, with many calculations being performed. However, in many cases, only a portion of the calculations is of interest. For example, the calculations of interest may be those in which one of the values ​​is non-zero. Thus, by compressing one or both of the matrices and using the corresponding data, it is possible to determine where relevant interactions (e.g., between non-zero elements) will occur and to perform those specific calculations. This can reduce the number of calculations performed.

[0016] In some embodiments, the receiver circuit is adapted to receive elements of a third matrix, and the operation includes an addition operation of the third matrix with the results of the multiplication operation of the first and second matrices, and the adder circuit is adapted to add each partial result generated for a given row and a given column with one of the elements of the third matrix. In a multiply-accumulate operation, two operands are multiplied together and the result is added to an "accumulator" that accumulates the results of multiple such multiplications.

[0017] In some embodiments, the multiplication circuit is adapted to output a value "0" in response to determining that none of the elements of the second matrix occur in row i of the second matrix. If none of the elements of the second matrix occur in row i of the second matrix, there may be no associated interactions with those elements. For example, any such interactions are interactions with elements of the matrix that are not provided. This may occur, for example, when all elements are zero. As a result of no calculation being performed, a value of zero is directly output, thereby eliminating the need to perform a more complex calculation.

[0018] In some embodiments, the decision circuit is adapted to provide a value "0" as an input to the multiplication circuit in response to the decision circuit determining that a given element of the second matrix occurs elsewhere than at row j, column i of the second matrix. One way of causing the multiplication circuit to output the value "0" is to provide a "0" as an input to the multiplication circuit itself.

[0019] In some embodiments, the data processing apparatus comprises a storage circuit for storing the elements of the first matrix, the storage circuit including the corresponding data, in such an embodiment, the elements of the first matrix are stored together with the corresponding data, the corresponding data indicating where those elements appear within the first matrix.

[0020] In some embodiments, the storage circuit is a vector register. A vector register may be composed of multiple regular registers for the purpose of storing large amounts of data. A vector register is typically wired together or organized so that all data in the vector register can be accessed or provided simultaneously. Note that in some cases, a vector register may be scalable such that its amount of storage can be varied, and thus in such embodiments, it is possible to limit the size of the vector register to be sufficient to store the elements of the matrix and the corresponding data. In this way, a variety of different matrix sizes (or number of elements) can be supported without requiring a large vector register to handle worst case scenarios.

[0021] In some embodiments, a storage circuit is provided for storing the elements of the second matrix, the storage circuit including further corresponding data. In a similar manner to the storage circuit used for storing the elements of the first matrix, the second matrix can also be stored in the storage circuit, together with the further corresponding data. It should be noted that the storage circuits used for the first matrix and the second matrix can be the same or different.

[0022] In some embodiments, the storage circuit is a vector register. The vector register used to store the second matrix may be the same vector register used to store the first matrix or may be a different vector register.

[0023] In some embodiments, the width of the storage circuit is b bits, each of the elements of the first matrix has a width of c bits, and the number of elements of the first matrix is ​​at most b / c. Thus, the number of elements can vary depending on the width of the elements and also on the size of the storage. For example, if each element is only 1 bit and the storage circuit is 32 bits, the number of elements of the first matrix can be up to 32.

[0024] In some embodiments, the width of the correspondence data is x bits for each of the elements of the first matrix, and the number of elements of the first matrix is ​​at most b / (c+x). The correspondence data may be provided for each element to indicate that the element is in the first matrix. Such correspondence data may use a number of bits depending on the size of the first matrix. For example, if the correspondence data is provided for each row (including m columns), the size of the correspondence data may be log2(m) bits for each of the elements of the first matrix. Similarly, if the correspondence data is provided for each column (including n rows), the size of the correspondence data may be log2(n) for each of the elements of the first matrix. If the correspondence data is not provided on a row or column basis, such that each element of the first matrix must have a specified row and column, the correspondence data may be log2(m)+log2(n) bits.

[0025] In some embodiments, the width of the storage circuit is b bits, each of the elements of the second matrix has a width of c bits, and the number of elements of the second matrix is ​​at most b / c. As above, the number of elements of the second matrix can vary depending on the size of the storage circuit that stores the elements of the second matrix, as well as the number of bits used to represent each element.

[0026] In some embodiments, the width of the further corresponding data is x bits for each element of the second matrix, and the number of elements of the second matrix is ​​at most b / (c+x). As above, the further corresponding data has a width that depends on the size of the second matrix.

[0027] According to some embodiments, there is provided a data processing apparatus comprising a decode circuit for receiving one or more instructions including a matrix operation instruction, the matrix operation instruction referencing elements of a first matrix, elements of a second matrix, and corresponding data indicating where the elements of the first matrix are located within the first matrix, and in response to receiving the matrix operation instruction, the decode circuit generates one or more control signals to cause one or more execution units to perform a matrix operation on the first matrix and the second matrix.

[0028] According to such an aspect, it is possible to provide instructions for performing operations on a first matrix and a second matrix that reference elements of the first matrix and elements of the second matrix, and also corresponding data indicating how the elements of the first matrix relate to the first matrix (e.g., where the elements are located). Compression may be achieved by providing some (but not all) of the elements of the first matrix.

[0029] Specific embodiments will now be described with reference to the drawings.

[0030] FIG. 1 shows a schematic diagram of a data processing device 100 according to some embodiments. The device includes a receiver circuit 110 that receives some elements of a first matrix, at least some elements of a second matrix, and correspondence data indicating where the providing elements of the first matrix belong in the first matrix. This information is passed from the receiver circuit to a decision circuit 120. The decision circuit uses the correspondence data to determine which providing elements of the first matrix interact with elements of the second matrix when performing a matrix operation. Whether two elements will interact depends on their locations in the first and second matrices and on the operation being performed. For example, in case of matrix multiplication, an element in the first row, first column of the first matrix will not interact with an element in the second row, second column of the second matrix when the first and second matrices are multiplied together. As the correspondence data indicates where the providing elements of the first matrix are located in the first matrix, this information can be used to determine the pairs of elements that are interacting and their locations.

[0031] Information about the interactions that occur is passed to the aggregation circuit 130. The aggregation circuit 130 is used to calculate an aggregation over multiple interacting pairs. For example, the aggregation circuit can calculate an aggregation of pairs that interact with a given row / column when performing a matrix multiplication. The aggregation circuit 130 includes a functional circuit 140 that performs matrix operations. In particular, for a particular pair of elements (one from a first matrix and one from a second matrix) that the determination circuit 120 has determined to interact, the functional circuit performs the operation for which the interaction has been determined to occur. The functional circuit thereby generates a partial result of a single interacting pair, which is then aggregated by the aggregation circuit 130.

[0032] In some embodiments, the receiver circuit 110 also receives further correspondence data indicating where the provided elements of the second matrix occur within the second matrix. This may be the case, for example, where only a portion of the elements of the second matrix are provided. The further correspondence data is passed to the determination circuit 120, which uses the further correspondence data to determine the interacting pairs. In other embodiments, the further correspondence data may be ignored. This may be the case, for example, where the elements of the second matrix include all of the elements of the second matrix.

[0033] FIG. 2 illustrates how the location of elements of the first and second matrices may be indicated in the first and second matrices, respectively, using the correspondence data and optional further correspondence data. In FIG. 2, an example of a multiply-add operation is shown in which the result of the multiplication between matrix a and matrix b is added to a multiplication matrix c to provide a result matrix d. In this example, the multiplication matrix c is a 2×2 matrix, the first matrix a is an 8×2 matrix, the second matrix is ​​a 2×8 matrix, and the result matrix d is a 2×2 matrix. The nature of matrix multiplication means that the result of the multiplication of matrix a by matrix b is a 2×2 matrix. Of course, it will be understood that other size matrices can be used, and in fact the same process can be used to perform matrix multiplication without a multiplication operation.

[0034] In this example, the provided elements of the first matrix a are elements that are non-zero. Similarly, the provided elements of the second matrix b are values ​​that are non-zero. For the first matrix a, corresponding data is provided for each row and provides an index where each non-zero element is located within that row. For example, in the first row of the matrix a, there are non-zero elements at indexes 0, 5, and 7. In the second row, there are non-zero values ​​at indexes 1, 4, and 6. Thus, the corresponding data for the first row of the first matrix a represents values ​​0, 5, and 7, and the corresponding data would indicate values ​​1, 4, and 6 for the second row. For the second matrix b, further corresponding data is provided for each column. In this example, the first column of the matrix b includes non-zero values ​​at indexes 0, 3, and 7, so the further corresponding data for the first column would include indexes 0, 3, and 7. In the second column, non-zero values ​​are found at indexes 0, 3, and 6. Thus, the further correspondence data would indicate that at positions 0, 3, and 6 are located elements of the second matrix b.

[0035] Using the correspondence data and the further correspondence data, the first and second matrices can be reconstructed from the provided elements of the first matrix and the provided elements of the second matrix, respectively.

[0036] FIG. 3 shows how matrices a and b can be compressed using corresponding data to form compressed matrices a' and b', respectively. In this case, the compressed matrices are composed of non-zero elements of the original matrices a and b. In this way, the first matrix is ​​reduced from 8×2 to 3×2, and the second matrix b is reduced from 2×8 to 2×3. The storage space required to store or represent the matrices is therefore reduced compared to the situation of storing or representing the matrices in their full form, even though the corresponding data (and optional further corresponding data) are also stored. In this embodiment, the first matrix a and the second matrix b are compressed to corresponding sizes suitable for matrix multiplication operations. However, for other matrix operations, the compressed matrices a' and b' can be compressed such that their sizes do not correspond, depending on the operation to be performed.

[0037] 3 also shows how the corresponding data and the further corresponding data are represented: in particular, for the first matrix a, corresponding data is provided for each row of the compressed matrix a', whereas for the second matrix b, corresponding data is provided for each column of the compressed matrix b'.

[0038] 4 shows an example of how the compressed matrix may be stored using vector registers 410, 420, and 430. In this example, two vector registers VR a’ 410 and VR b’ 420 is used to store the compressed matrices a' and b', and a third vector register VR c 430 is used to represent the multiplication matrix c. Each of the vector registers 410, 420, 430 has a width of b bits.

[0039] For the first compressed matrix a', each of the elements of the second row is stored, followed by the corresponding data for the second row. This is followed by the elements of the first row, followed by the corresponding data for the first row. Each element is given a width of c bits. Thus, without corresponding data, the maximum number of elements that can be stored in one of the vector registers 410, 420, 430 is b / c. In this example, x bits are used to store each corresponding data for each of the elements. Since there are three elements in each row of the first compressed matrix a', 3x bits of corresponding data are provided for each row, for a total of 6x bits. Thus, the maximum number of elements that can be stored in each of the vector registers 410, 420, 430 is equal to (b-6x) / c. In other words, if x bits of corresponding data are provided for each element, the maximum number of elements that can be stored in the vector registers 410, 420, 430 is b / (c+x).

[0040] In this example, x is 5. It should be appreciated that this is more than sufficient to represent the index of an element. Indeed, in this example, the original matrix has 8 elements per row (for matrix a) or column (for matrix b), so only log2(8) = 3 bits per element are needed.

[0041] In this example, the first corresponding data 440 corresponding to elements 4, 5, and 6 is equal to 6<<10|4<<5|1. In other words, the value 1 is concatenated with the value 4 and shifted left 5 times, and concatenated with the value 6 and shifted left 10 times. This represents the fact that the value 6 belongs to the first index, the value 5 belongs to the fourth index, and the value 4 belongs to the sixth index. This is shown in the second row of matrix a in FIG. 2. The shifts are performed to prevent the indexes from conflicting with each other in storage. Similarly, the second corresponding data 450 corresponding to elements 1, 2, and 5 is equal to 7<<10|5<<5|0. In other words, the value 0 is concatenated with the value 5 and shifted left 5 times, and concatenated with the value 7 and shifted left 10 times. This represents the fact that the value 3 occurs at index 0, the value 2 occurs at index 5, and the value 1 occurs at index 7. This is shown in the first row of matrix a in Figure 2. Note that the corresponding data are grouped together by row, so that it is possible to know the location of a given value based on an index that forms part of the corresponding data.

[0042] Similarly, the second vector register VR b’The data stored in matrix b includes two items of further corresponding data 460, 470. The first corresponding data 460 corresponds to elements B, C, and D and is equal to 6<<10|3<<5|0. In other words, the value 0 is concatenated with the value 3 and shifted left 5 times, and concatenated with the value 6 and shifted left 10 times. This represents the fact that the value D is stored at index 0, the value C is stored at index 3, and the value B is stored at index 6. This is shown in the second column of matrix b in FIG. 2. The second further corresponding data 470 corresponds to values ​​8, 9, and A and is equal to 7<<10|3<<5|0. In other words, the value 0 is concatenated with the value 3 and shifted left 5 times, and concatenated with the value 7 and shifted left 10 times. This indicates that the value A occurs at index 0, the value 9 occurs at index 3, and the value 8 occurs at index 7. Again, this is shown as the first column of matrix b in Figure 2. Note that the further corresponding data is grouped together by column, so that it is possible to know the location of a given value based on an index that forms part of the corresponding data.

[0043] Note that the data elements of the multiplication matrix c do not need corresponding data because the multiplication matrix is ​​not compressed in the same way as matrices a' and b'. Therefore, the third vector register VR c 430 simply contains the four values ​​of the multiplication matrix c. Since there is no corresponding data and there are only four values, more bits can be allocated to the elements of the multiplication matrix c. This is also appropriate since the size of the elements of the multiplication matrix can be much larger, especially if they represent the summation results of many multiplications.

[0044] FIG. 5 shows a circuit used to perform a multiplication or multiply-and-accumulate operation between two compressed matrices a' and b' and a multiplication matrix c'. This process involves performing a series of dot-product operations. The circuit 500 is made of four units 530a, 530b, 530c, 530d, each of which performs a dot product. Each of the units 530a, 530b, 530c, 530d receives a different combination of elements of the first compressed matrix a' and the second compressed matrix b' according to a matrix multiplication operation. In addition, when a multiply-and-accumulate operation is performed, a multiplication element is passed to each of the four units 530a, 530b, 530c, 530d. For example, the dot product of the first column of the first compressed matrix a' and the first row of the second compressed matrix b' generates an element at (1,1) of the result matrix. If a multiply-and-accumulate operation is to be performed, the element at (1,1) of the multiplication matrix must also be added to the result of the dot-product operation to generate a result matrix. Similarly, the dot product of the first column of the first compressed matrix a' and the second row of the second compressed matrix b' generates an element at (2,1) (second column, first row) of the result matrix. If a multiply-and-accumulate operation is being performed, the element at (2,1) of the multiplication matrix is ​​also added to generate a result matrix. Regardless of whether a multiplication or multiply-and-accumulate operation is being performed, the output of each of the four units 530a, 530b, 530c, 530d is one of the elements of the result matrix d.

[0045] 5 also shows the structure of one of the units 530a. The structures of the other units 530b, 530c, 530d are similar except for the elements of the received compression matrices a', b' and the elements of the multiplication matrix c.

[0046] In the illustrated embodiment, the elements of the first row of the compressed matrix a' and the elements of the first column of the compressed matrix b' are received. The unit 530a includes three decision circuits 520aa, 520ab, 520ac, so that the circuit can support a compressed matrix b' with up to three elements per column (for the first compressed matrix b'). Each of the decision circuits 520aa, 520ab, 520ac receives a respective one of the three elements of the compressed matrix b'. Furthermore, each of the decision circuits receives the value z(0) as a further input. In this embodiment, each of the decision circuits 520aa, 520ab, 520ac takes the form of a multiplexer. The selection signal S of the multiplexer is composed of three components s2, s1, s0. Each of these components is a single bit, and the overall selection signal S is one-hot.

[0047] 5, in the case of the first determination circuit 520aa, s0=(a0id==b0id), s1=(a0id==b1id), and s2=(a0id==b2id). The first decision circuit 520aa has a first selection bit s0 equal to 1 if the index of the first provided element of the first compressed matrix a' is equal to the index of the first provided element of the second compressed matrix b'. Otherwise, the selection bit s0 is equal to 0.

[0048] The first decision circuit 520aa has a second selection bit s1 equal to 1 if the index of the first provided element of the first compressed matrix a' is equal to the index of the second provided element of the second compressed matrix b'. Otherwise, the selection bit s1 is equal to 0.

[0049] The first decision circuit 520aa has a third selection bit s2 equal to 1 if the index of the first provided element of the first compressed matrix a' is equal to the index of the third provided element of the second compressed matrix b'. Otherwise, the selection bit s2 is equal to 0.

[0050] This index is determined according to the corresponding data and the further corresponding data. Thus, the selection signal indicates to the first determination circuit 520aa whether the index of any of the first provided elements of the first compressed matrix a' is equal to the index of any of the three elements of the second compressed matrix b'. As can be seen, in a multiplication operation between a first matrix and a second matrix, the first element of the first matrix interacts with (is multiplied by) the second element of the second matrix if their indexes match, i.e., when the column number of the first element is the same as the row number of the second element. Thus, the selection signal indicates where any such match occurs, thereby causing the selection of the matching element from the second matrix.

[0051] Thus, the output of the decision circuit 520aa is either equal to the value 0 when there is no interaction, or equal to the value of one of the elements of the second compressed matrix b' when an interaction occurs between the first element of the first compressed matrix a' and an element of the second compressed matrix. This value is then passed to a first functional circuit 540aa in the form of a first multiplication circuit, which also receives as input the first element of the first compressed matrix a'. The first multiplication circuit 520aa then performs the multiplication. Its output is passed to an accumulation circuit 550a in the form of an adder, which performs the addition part of the dot product operation.

[0052] The second decision circuit 520ab is provided to determine whether any interaction occurs between the three provided elements of the second compressed matrix b' and the second provided element of the first compressed matrix a'. Again, the selection signal S is composed of three bits s0, s1, s2. However, for the second decision circuit 520ab, s0=(a1id==b0id), s1=(a1id==b1id), s2=(a1id==b2id). The output of the second decision circuit 520ab is either one of the elements of the second compressed matrix b' that is considered to interact with the second element of the first compressed matrix a', or z(0) if no interaction occurs. The result is passed to the second multiplication circuit 540ab. The second multiplication circuit multiplies the value passed together with the second element of the first compressed matrix a'. The result of the multiplication is then passed to a multiplication circuit 550a, in the form of an adder, which performs the addition portion of the dot product operation.

[0053] Similarly, the third decision circuit 520ac decides whether any of the three elements of the second compressed matrix b' interacts with the third element of the first compressed matrix a'. Here, the selection signal bits are s0=(a2id==b0id), s1=(a2id==b1id), s2=(a2id==b2id). Thus, the output of the second decision circuit 520ac is zero or one of the three provided elements of the second compressed matrix b' that is considered to interact with the second element of the first compressed matrix a'. The output is passed to the third multiplication circuit 540ac, where the result is multiplied by the third provided value of the first compressed matrix a', and the result is passed to the accumulator 550a.

[0054] The accumulator circuit 550a adds together the results of each of the multiplier circuits 540aa, 540ab, 540ac. The accumulator also adds provided elements from the accumulator matrix c (if provided). This result produces one of the values ​​of the result matrix d, as shown in FIG.

[0055] The other units 530b, 530c, 530d behave in a similar manner. Each of the decision circuits 520 compares the indexes of the three provided elements of the second compressed matrix b' with the indexes of one of the three provided elements of the first compressed matrix a'. The three decision circuits in one unit collectively compare each of the indexes of the provided elements of the second compressed matrix b' with each of the indexes of the provided elements of the first compressed matrix a'.

[0056] It can therefore be seen that by using the correspondence data and the further correspondence data in the form of an index, the decision circuit can determine where an interaction occurs between an element of the first matrix a' and an element of the second matrix b'. If an interaction is deemed to occur, an operation (for example a multiplication) between those elements is performed. Otherwise, the multiplication performed is a multiplication by zero.

[0057] In this embodiment, only three elements of each of the first compressed matrix a' and the second compressed matrix b' are considered. However, it should be understood that it can be scaled by providing further multiplication circuits 540 and by providing further inputs to the decision circuit 520. Furthermore, the output result provided is a 2x2 matrix, so four (2x2) units 530 are provided. However, if the output matrix d is larger, a larger array of units can be provided. For example, if the output matrix is ​​a 3x3 matrix, nine such units 530 will be provided. Of course, in such a case, the size of the compressed matrix is ​​likely to be larger as well, and therefore the size of the individual units 530 is likely to be expanded as well, as described above.

[0058] It will be appreciated that as a result of the apparatus 500, it is not necessary to perform the full set of operations that may be required, for example, for the uncompressed matrices a, b. For example, considering the number of elements in the uncompressed matrices a, b, a total of 32 multiplications would be required, but due to compression, in this embodiment, only 12 multiplications need to be performed. Furthermore, many multiplications occur in parallel across multiple units 530. As a result of both of these reasons, the matrix operations can be performed more quickly than for the uncompressed matrices. Furthermore, it will be appreciated that the compressed matrices a' and b' are significantly smaller than the uncompressed matrices a and b, and therefore can be stored using less storage space.

[0059] In this embodiment, both matrices are compressed. However, it should be understood that one of the two matrices may be in uncompressed form. If this occurs, no corresponding or further corresponding data is required, since the required indexes can be determined directly from the matrices themselves.

[0060] In other embodiments, operations other than multiplication (or multiply-add) can be performed. For example, when adding two matrices together, the addition of one or two zero operands is inherently a redundant operation. Thus, if two values ​​"interact" with each other in an addition, the actual addition operation can be ignored by simply returning a non-zero (or zero, if all operands are zero) operand.

[0061] FIG. 6 considers an example in which "tiling" may be used to perform an operation in which a first matrix and a second matrix have non-corresponding sizes. In particular, matrix a may be compressed by being a sparse matrix with many non-zero entries. It may therefore be compressed from a 2×8 matrix a to a 2×3 matrix a′ as previously shown. In contrast, matrix b is dense and has no non-zero entries, and therefore remains an 8×1 matrix. Normally, a matrix multiplication between these matrices produces a 2×1 matrix. However, with tiling, the 8×1 matrix may be treated as a 4×2 matrix with two columns 610, 620. The output is therefore a 2×2 matrix. As a result, element d 00 and d 10 is generated using the first half (bits 0 to 3) 610 of matrix b. Meanwhile, element d 01 and d 11 is generated using the second half (bits 4 through 7) 620 of the second matrix b.

[0062] Figure 7 shows how storage for the embodiment of Figure 6 is accomplished. In particular, storage for the first compressed matrix a' and the multiplication matrix c occurs in vector registers 710, 730 in the same manner as shown with respect to Figure 4. For the second matrix b, no further corresponding data is needed (as no compression occurs), so space is freed for each of the elements of the second matrix b to be stored.

[0063] Figure 8 shows a circuit 800 that can be used to perform a matrix operation where one of the two matrices has a non-corresponding size. In this example, matrix b remains in its uncompressed form and is therefore technically an 8x1 matrix, and the circuit of Figure 8 instead processes it as a 4x2 matrix using tiling.

[0064] The circuit 800 operates similarly to the circuit 500 shown in Figure 5. One important difference is that the uncompressed matrix b is processed to have four rows, and therefore each of the four units 830a, 830b, 830c, 830d receives four inputs relating to the second uncompressed matrix b.

[0065] Each of the decision circuits 820aa, 820ab, 820ac in the four units 830a, 830b, 830c, 830d receives four inputs together with a constant 0(z). Furthermore, the selection signal S is composed of four components s0, s1, s2, s3. For example, for the first unit 820aa, s0=(a0id==b0id), s1=(a0id==b1id), s2=(a0id==b2id), s3=(a0id==b3id). Again, the components of the selection signal S indicate whether an interaction occurs between a given element of the first compressed matrix a' and each of the presented elements of the second matrix b. In this case, no further correspondence data is available for the elements of the second matrix b, since no compression is performed. However, no further correspondence data is necessary, since the location of each of the elements of the second matrix b is known.

[0066] The circuit shown in Figure 8 can also be used with pairs of compressed matrices such as those for b'. This can be achieved by simply using an index provided as part of the (further) correspondence data. Similarly, the circuit of Figure 8 can be used for multiplication of matrices with non-corresponding sizes (in this case multiplying a compressed 2x3 matrix by an 8x1 matrix, which is treated as a 4x2 matrix), but The same circuit can also be used for smaller size matrices by ignoring any extra input signals to the decision circuit 820 and by setting the corresponding selection bits of the selection signal S for those inputs to 0 so that they are not selected. For example, the same circuit can be used to multiply a 2×3 matrix by another 2×3 matrix (as in FIG. 5) by ignoring the fourth input to the decision circuit 820 and setting the selection value by s3 to 0. If the first matrix is ​​smaller (e.g., if the compressed matrix a′ is 2×2), the extra decision units 820 (and their associated multiplication units 840) are ignored. For example, if the compressed matrix a′ is a 2×2 matrix, the third decision circuit 820ac and its associated multiplication unit 840ac are simply ignored.

[0067] FIG. 9 shows an apparatus 900 according to some embodiments. The apparatus includes a number of components 930 in the form of a pipeline. Within the pipeline 930, a fetch circuit 910 receives an instruction (MatrixOp) for performing a matrix operation. The instruction specifies an element of a first matrix (elemA), an element of a second matrix (elemB), correspondence data for the element of the first matrix (correspondenceA), optional further correspondence data for the element of the second matrix (correspondenceB), and an optional element of a third matrix multiplier (elemC). In each case, a reference to an element of the matrix may be provided in the form of a register or registers, or in the form of a location in memory where the element can be found. The fetch circuit 910 passes the fetch instruction to a decode circuit 920. The decode circuit decodes the instruction providing one or more control signals. These control signals are then passed to a matrix operation circuit 500 such as the one shown with respect to FIG. 5 or FIG. 8.

[0068] 10 illustrates a method 1000 of data processing, according to some embodiments. In step 1010, elements of a first matrix are received together with elements of a second matrix and corresponding data indicating where the elements of the first matrix occur within the first matrix. In step 1020, a determination is made as to which elements of the first matrix and the second matrix interact with each other according to the matrix operation. In step 1030, aggregation is performed by performing a matrix operation on the elements of the matrix and the second matrix that are deemed to interact with each other according to the determining step 1020.

[0069] In this application, the term "configured to..." is used to mean that an element of an apparatus has a configuration that allows it to perform a defined operation. In this context, "configuration" refers to a method of configuration or interconnection of hardware or software. For example, an apparatus may have dedicated hardware that provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured" does not imply that an apparatus element must be modified in any way to provide the defined operation.

[0070] Although exemplary embodiments of the invention have been described in detail herein with reference to the accompanying drawings, it will be understood that the invention is not limited to those precise embodiments, and that various changes, additions, and modifications may be made by those of ordinary skill in the art without departing from the scope and spirit of the invention as defined by the appended claims. For example, various combinations of the features of the dependent claims could be made with the features of the independent claims without departing from the scope of the invention.

Claims

[Claim 1] a decode circuit for receiving one or more instructions including a matrix operation instruction, the matrix operation instruction referencing elements of a first matrix, elements of a second matrix, and corresponding data indicating where the elements of the first matrix are located within the first matrix, the elements of the first matrix and the corresponding data being stored in a vector register; 13. A data processing apparatus comprising: a data processor configured to generate, in response to receiving the matrix operation instruction, one or more control signals to cause one or more execution units to perform a matrix operation on the first matrix and the second matrix.