Multiple product command

Abstract

The multiple cross product instruction specifies a plurality of first source vector operands, at least one second source vector operand, and correlation information associated with the second source vector operands, each vector operand including a plurality of data elements, the correlation information indicating, for each data element of a given second source vector operand, a corresponding first source vector operand. In response to the multiple cross product instruction, the instruction decoder circuit (50) controls the processing circuit (60) to execute calculations to perform a cross product operation, the cross product operation including, for a given first source vector operand, performing an associated cross product of the first source vector operand with a subset of the data elements of the second source vector operand. The processing circuit selects, for each data element of the second source vector operand, a corresponding first source vector operand to be used when performing the associated cross product operation in dependence on the correlation information.

Description

[Technical Field]

[0001] The present technology relates to the field of data processing.

[0002] The data processing apparatus may comprise processing circuitry capable of performing a cross product operation in which the cross product of two vectors is calculated. The processing circuitry may be further arranged to perform a multiple cross product operation on different pairs of vectors to multiply two matrices together.

[0003] Cross products and matrix multiplication have many applications, for example, the implementation of artificial neural networks (ANNs) typically involves matrix multiplication.

[0004] However, multiplication operations can be relatively slow, with each cross product operation (not to mention each matrix multiplication) typically involving multiple multiplication operations (e.g., multiplication for each pair of data elements in the input vector). Thus, there can be significant performance impacts associated with performing cross product operations, and it is therefore desirable to provide techniques for more efficiently performing cross product operations, particularly in situations where multiple cross product operations are to be performed.

[0005] In a first embodiment of the present technology, there is provided an apparatus, processing circuitry for performing vector operations; an instruction decoder circuit for decoding an instruction from the set of instructions to control a processing circuit to perform a vector operation specified by the instruction; The set of instructions includes a multiple cross product instruction specifying a plurality of first source vector operands, at least one second source vector operand, and correlation information associated with the at least one second source vector operand, each vector operand including a plurality of data elements, and for a given second source vector operand, the correlation information is adjusted to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; The instruction decoder circuit is responsive to the multiple cross product instruction to control the processing circuit to execute a plurality of calculations to perform a plurality of cross product operations, the plurality of cross product operations including, for a given first source vector operand, performing cross product operations associated with calculating a cross product of the given first source vector operand with a subset of the data elements of at least one second source vector operand; An apparatus is provided in which the processing circuitry is configured to, for each data element of the at least one second source vector operand, select, in dependence on the correlation information, a corresponding first source vector operand to be used when performing the associated cross product operation.

[0006] In a second embodiment of the present technology, there is provided a method, comprising: performing vector operations using processing circuitry; decoding an instruction from the set of instructions to control a processing circuit to perform the vector operation specified by the instruction; The set of instructions includes a multiple cross product instruction specifying a plurality of first source vector operands, at least one second source vector operand, and correlation information associated with the at least one second source vector operand, each vector operand including a plurality of data elements, and for a given second source vector operand, the correlation information is adjusted to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; The method includes: executing, in response to a multiple cross product instruction, a plurality of calculations to perform a plurality of cross product operations, the plurality of cross product operations including, for a given first source vector operand, performing cross product operations associated with computing a cross product of the given first source vector operand with a subset of data elements of at least one second source vector operand; For each data element of the at least one second source vector operand, selecting a corresponding first source vector operand to be used when performing the associated cross product operation in dependence on the correlation information.

[0007] In a third embodiment of the present technology, there is provided a computer program comprising instructions, the instructions, when executed on a computer, processing program logic for performing vector operations; instruction decoder program logic for decoding a target instruction from a set of target instructions to control processing program logic to perform the vector operation specified by the target instruction; the set of target instructions includes a multiple cross product instruction specifying a plurality of first source vector operands, at least one second source vector operand, and correlation information associated with the at least one second source vector operand, each vector operand including a plurality of data elements, and for a given second source vector operand, the correlation information is adjusted to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; the instruction decoder program logic controls the processing program logic to execute a plurality of calculations to perform a plurality of cross product operations in response to the multiple cross product instruction, the plurality of cross product operations including, for a given first source vector operand, performing cross product operations associated with calculating a cross product of the given first source vector operand with a subset of the data elements of at least one second source vector operand; A computer program is provided, wherein the processing program logic is configured to select, for each data element of the at least one second source vector operand, a corresponding first source vector operand to be used when performing an associated cross product operation in dependence on the correlation information.

[0008] The above-described computer program is, in some embodiments, stored on a computer-readable storage medium, which can be transitory or non-transitory. [Brief explanation of the drawings]

[0009] Further aspects, features, and advantages of the present technology will become apparent from the following description of examples, read in conjunction with the accompanying drawings. [Figure 1] FIG. 1 is a block diagram of a data processing device. [Figure 2] 1 illustrates an example of an architectural register that may be provided in a device including a vector register for storing vector operands and an array register for storing 2D arrays of data elements, including an example of a physical implementation of the array register. [Figure 3A] 1 illustrates schematically how access to a square 2D array in array storage can be performed. [Figure 3B] 1 illustrates schematically how access to a square 2D array in array storage can be performed. [Figure 4A] Indicates the cross product operation. [Figure 4B] Denotes a matrix multiplication operation. [Figure 5] Demonstrates N:M structured sparsity in matrices. [Figure 6A] Demonstrates N:M structured sparsity in matrices. [Figure 6B] Demonstrates N:M structured sparsity in matrices. [Figure 7] An example is given in which a multiply-accumulate (MAC) array is used to multiply a matrix of activations by a matrix of weights. [Figure 8A] We show how matrices with 2:4 and 4:8 structured sparsity can be compressed. [Figure 8B] We show how matrices with 2:4 and 4:8 structured sparsity can be compressed. [Figure 9] 1 illustrates matrix multiplication in which one of the input matrices to be multiplied is a compressed matrix. [Figure 10] 1 is a block diagram of a data processing apparatus showing how processing circuitry is used to perform a cross product operation; [Figure 11A] It shows how the generated cross product result can be used to update the associated storage element in a 2D array of array storage. [Figure 11B] It shows how the generated cross product result can be used to update the associated storage element in a 2D array of array storage. [Figure 12] 10 schematically illustrates fields that may be provided in a double cross product instruction; [Figure 13] 10 shows an example of a multiple cross product instruction and an example of how correlation information can be represented. [Figure 14] 10 shows an example of a multiple cross product instruction and an example of how correlation information can be represented. [Figure 15] 10 shows an example of a multiple cross product instruction and an example of how correlation information can be represented. [Figure 16] 1 illustrates an embodiment of a circuit for performing an outer product-accumulation operation. [Figure 17] 1 illustrates an embodiment of a circuit for performing an outer product-accumulation operation. [Figure 18] FIG. 10 is a flow diagram illustrating the steps performed when decoding a double cross product instruction. [Figure 19] 1 illustrates a simulator implementation that may be used.

[0010] Before discussing example implementations with reference to the accompanying drawings, the following description of example implementations and associated advantages is provided.

[0011] According to one exemplary configuration, an apparatus is provided that includes a processing circuit and an instruction decoder circuit. The instruction decoder circuit (also referred to herein as a decoder circuit or an instruction decoder) is adapted to decode an instruction from a set of instructions to control a processing circuit to perform a vector operation specified by the instruction. For example, the instruction decoder circuit may be responsive to an instruction in the set of instructions to generate a control signal, and the control signal may control the processing circuit to perform the vector operation.

[0012] A vector operation is an operation performed on vector operands, e.g., operands that include multiple data elements. A vector operation can include any operation involving at least one vector operand, such as a load or storage operation to load / store a vector from / to a storage location (e.g., memory or cache), or an arithmetic operation (e.g., addition, multiplication) performed on a vector operand. In the present technology, a processing circuit is capable of performing vector operations including at least a cross product operation. For example, vector operands can be (but are not necessarily) stored in vector registers, and a single vector register can store an entire vector operand, or a single vector operand can be distributed among multiple vector registers (thus a single vector register can store elements from multiple vector operands).

[0013] The set of instructions that the instruction decoder circuitry is configured to decode for execution by the processing circuitry includes at least a "multiple-outer-product" (MOP) instruction. The MOP instruction is defined in an instruction set architecture (ISA) and includes (e.g., directly or indirectly, e.g., in a respective field of the instruction, by identifying a register that stores a corresponding data value) at least: multiple (e.g., two or more) first source vector operands; At least one (e.g., one or more) second source vector operands, and • Identifying correlation information associated with at least one second source vector operand.

[0014] Note that the terms "first" and "second" are merely labels, and the first and second source vector operands do not necessarily have to be the first and second operand(s), respectively, specified by the instruction. Rather, the first and second source vector operand(s) may be identified by the MOP instruction in either order.

[0015] Each source vector operand (e.g., the second source vector operand and each of the plurality of first source vector operands) includes multiple data elements, and for a given second source vector operand, the correlation information is tailored to indicate (e.g., directly or indirectly identify) the corresponding first source vector operand for each data element in the given second source vector operand. The correlation information may also (optionally) identify corresponding first source vector operands for data elements in any other second source vector operands identified by the instruction (e.g., when two or more second source vector operands are identified by the MOP instruction). Thus, the correlation information identifies any data elements in the second source vector operands associated with a respective first source vector operand.

[0016] The instruction decoder circuit is configured to control the processing circuit to execute a plurality of calculations to implement a plurality of cross product operations in response to the MOP instruction; e.g., in response to a single MOP instruction being decoded by the instruction decoder circuit, the processing circuit is configured to execute a calculation that produces a result equivalent to executing two or more cross product operations. The plurality of cross product operations executed by the processing circuit in response to the decoded MOP instruction include a cross product operation on each of at least a subset of the first source vector operands identified by the MOP instruction. More specifically, the processing circuit executes an associated cross product operation for a given first operand to calculate the cross product of that first source vector operand with a suitable subset (e.g., this may be some, but not all) of the data elements of at least one second source vector operand. A subset of data elements of the at least one second source vector operand to be used in each cross product operation is identified based on the correlation information; specifically, for each data element of the at least one second source vector operand, a corresponding first source vector operand to be used in performing the associated cross product operation is selected based on the correlation information. (Note that the cross product operation does not necessarily have to be performed on all of the first source vector operands or on all subsets of the at least one second source vector operand.)

[0017] In this manner, by providing correlation information for associating a first source vector operand with data elements in at least one second source vector operand, a multiple cross product operation can be performed in response to a single instruction, even when only one second source vector operand is specified by the instruction. This is in contrast to a typical (e.g., "single") cross product operation, in which a single cross product operation is performed by multiplying each data element of one source vector by each data element of another source vector.

[0018] By defining an instruction (MOP instruction) that allows multiple cross product operations to be performed in response to a single instance of the instruction, the throughput of instructions executed by a processing circuit can be significantly improved. Thus, the MOP instruction of the present technology allows the performance of the processing circuit to be improved. Indeed, in one exemplary implementation, these multiple cross product operations can be performed in parallel, further improving throughput.

[0019] In addition to increasing throughput, providing correlation information to associate a subset of data elements in the at least one second source vector operand with each first source vector operand allows the at least one second source vector operand to be defined such that multiple vectors are effectively compressed into a single source vector operand. Furthermore, using the correlation information in this manner provides flexibility as to how multiple vectors are compressed into a single second source vector operand; for example, the correlation information allows multiple vectors to be compressed such that a data element from any one of the multiple vectors can occupy any data element position in the at least one second source vector operand, including contiguous and / or non-contiguous data elements.

[0020] Providing instructions that support performing a multiple cross product operation based on source vector operands that may include data elements from multiple vectors compressed into fewer source vector operands can be advantageous in some scenarios. In one particular, non-limiting example, this may allow vectors that include one or more "zero" elements to be represented in a more compact form.

[0021] For example, because a cross-product operation relies on multiplication, e.g., multiplying each data element in one vector by each data element in another vector, and because any number of multiplications by zero is zero, it is possible to effectively compress the input vector for the cross-product operation by removing some or all zero elements. It may be assumed that vector elements equal to zero need to be reinserted into the input vector before performing the cross-product operation because otherwise, corresponding elements of the result vector (e.g., having a zero value) may be removed from the resulting cross product (leading to an inaccurate result). However, techniques of the present technology that use correlation information to identify which first source vector operands correspond to which elements of one or more second source vector operands can allow input vector elements equal to zero to be removed from one or more source vector operands without changing the result of the cross-product operation.

[0022] The present technology can thus enable such vectors to be represented so that they occupy less space in any data storage structure that stores compressed operands (e.g., by removing data elements equal to zero from the vector, such that the total number of data elements to be recorded, and therefore the space in the storage structure required to store those structures, is reduced. Reducing the amount of space occupied in storage provides numerous benefits, including reduced energy consumption when the data is stored in volatile storage, as well as reduced latency and power consumption in loading, manipulating, and storing vectors. Furthermore, by reducing the number of data elements in one or more of the source vectors, the total number of multiplications to be performed is reduced, leading to a reduction in the overall latency associated with the operations and an increase in data throughput.

[0023] Therefore, providing support in an instruction set architecture (ISA) for MOP instructions can provide significant performance improvements in data processing devices.

[0024] It should be understood that it is not required that the at least one second source vector operand represent a compressed form of multiple vectors, or that the multiple vectors contain zeros in one or more data elements. This is just one example of how MOP instructions can be used to improve performance and reduce power consumption.

[0025] It should be understood that, while the result produced by executing the MOP instruction is equivalent to computing a multiple cross product operation, the order in which the individual data elements of each vector operand are used / consumed by the processing circuit when executing the MOP instruction is an implementation matter. For example, the cross product operation does not necessarily have to be performed one at a time (e.g., by considering each first source vector operand in turn). Instead, in at least some example implementations of the present technology, each data element in the at least one second source vector operand can be considered in turn (e.g., selecting an appropriate first source vector operand for a given data element in the at least one second source vector operand). In other implementations, all of the data elements in the at least one second source vector can be considered in parallel, with a corresponding first source vector operand being selected in parallel for each element of the second source operand. These techniques, in which a first source vector operand is selected for each data element in at least one second source vector, may be more efficient because only a single pass of each second source vector operand may need to be made, rather than having to scan each second source vector operand multiple times (as would be the case if the cross product operations were performed one at a time). However, it will be understood that other implementations may be used to produce equivalent results.

[0026] In some embodiments, the apparatus comprises an array storage circuit comprising storage elements for storing data elements, the array storage circuit being arranged to store at least one two-dimensional (2D) array of data elements accessible to the processing circuit when performing the vector operation. In such embodiments, the multiple cross product instruction may specify a given two-dimensional array of data elements in the array storage forming a destination operand, and the processing circuit may be configured to perform an associated cross product operation for the given first source vector operand by multiplying each data element of the given first source vector operand by each data element in a subset of the data elements of at least one second source vector operand to generate a plurality of cross product result elements, and using each cross product result element to update a value held in an associated storage element in the given two-dimensional array of storage elements.

[0027] For example, an array storage circuit, which may comprise a set of array registers, may provide a useful mechanism for performing certain types of operations, such as a cross product operation. Specifically, a matrix of data elements produced as a result of performing the cross product operation may be stored in associated data elements of a 2D array represented in the array storage circuit.

[0028] However, the inventors of the present technology have realized that in some exemplary use cases, when performing a cross product operation using two source vectors, as described above, some of the elements in one or both of the source vectors may be zero. This can result in inefficient use of the storage elements of the 2D array, as a significant number of those storage elements may be unused or may only be used to store zero values. It can also result in inefficient use of the resources of the hardware components forming the processing circuit (which may be capable of performing calculations to generate results for each of the storage elements). However, according to the technology described herein, through the use of correlation information that associates a subset of data elements in at least one second source vector operand with a corresponding first source vector operand, a single instruction (i.e., the above-mentioned MOP instruction) can be defined that enables a multiple cross product operation to be performed, with the result of each cross product operation being stored in the associated storage element of the two-dimensional (2D) array. This can significantly improve throughput (as described above) while also more efficiently utilizing the available storage elements in the array storage.

[0029] Note that there need not necessarily be a 1:1 correlation between each cross product result and the associated storage element in the 2D array. For example, in some cases, a double cross product result may be used to determine the value held in a given storage element, and extra operations, such as accumulation (addition) operations, are performed to combine the double cross product results.

[0030] In some embodiments, the processing circuitry comprises multiplier circuitry for generating each cross product result when performing the multiple cross product operations, and multiplexer circuitry associated with the multiplier circuitry for selecting, under control of the correlation information, selected data elements of the multiple first source vector operands and selected data elements of the at least one second source vector operand to be multiplied to generate associated cross product result elements.

[0031] For example, the multiplication circuit may include a multiplier circuit, which may be, for example, a simple multiplier for multiplying two values ​​together, or a multiply-accumulate (MAC) circuit associated with each storage element in the 2D array of storage elements that multiplies two values ​​together and adds the result to an accumulated value. In the case of a MAC circuit, the same MAC circuit may both multiply selected data elements together to generate an associated cross product result and add the associated cross product result to the value (which may be zero) currently stored in the associated storage element of the array storage circuit. However, there may not necessarily be a 1:1 correlation between each multiplier circuit or MAC circuit and each storage element in the 2D array; for example, there may be one multiplier / MAC per storage element (in which case a cross product result may be stored in each storage element to be calculated in parallel), or there may be less than one multiplier / MAC per storage element (in which case each multiplier / MAC is used to calculate the cross product results of multiple storage elements in series).

[0032] However, many multiplier circuits may be provided, and the provision of multiplexer circuits associated with the multiplier circuits allows the data elements for each multiplication operation to be selected, thus enabling implementation of the MOP instructions of the present technology.

[0033] In some embodiments, the correlation information includes at least one set of indexes, the at least one set of indexes including an index associating each data element of a given second source vector operand with a corresponding first source vector operand.

[0034] For example, an index may be provided for each data element of a given second source vector operand. Alternatively, each index may be associated with multiple data elements within a given second source vector operand.

[0035] In some embodiments, the correlation information is provided by at least one correlation source operand specified by the multiple cross product instruction, the given correlation source operand including, for each data element of the given second source vector operand, a corresponding element including an index relating the data element of the given second source vector operand to the corresponding first source vector operand.

[0036] Providing an index for each data element of at least one second source vector in the set of indices forming the correlation information provides a simple mechanism for determining which index corresponds to which data element of the second source vector operand, which in turn can simplify processing and therefore provide improved performance.

[0037] As a particular example, the set of indexes may be stored as a multi-bit scalar value (e.g., in a scalar register or predicate register), where each bit is associated with a different vector element in one of one or more vector registers. In another example, the set of indexes may be stored as a vector (e.g., in a vector register), with each data element in the vector holding one or more indexes for corresponding data elements in at least one second source vector operand. It will be understood that these are just a few examples of how the set of indexes may be stored, and other examples are possible.

[0038] In some embodiments, the correlation information includes a set of indices for each second source vector operand.

[0039] For example, if the MOP instruction specifies multiple second source vector operands, the MOP instruction may also identify a separate set of indexes for each second source vector operand. Thus, the number of sets of indexes may be at least as large as the number of second source vector operands.

[0040] In some embodiments, the at least one set of indexes includes a set of indexes that provides correlation information for a plurality of second source vector operands.

[0041] Therefore, instead of indicating a set of indexes for each second source vector operand, each set of indexes identified by the MOP instruction may provide correlation information for two or more second source operands. Thus, in this embodiment, the number of sets of indexes may be less than the number of second source vectors. Advantageously, the size / width of the registers used is fixed, and in relatively large implementations, this allows the amount of storage space occupied by the correlation information to be reduced.

[0042] In some embodiments, the correlation information is provided by at least one correlation source operand specified by the multiple cross product instruction, and each element of the given correlation source operand includes a plurality of indices, the plurality of indices including indices for corresponding data elements of each of a plurality of second source vector operands.

[0043] Thus, in these embodiments, each element of a given correlation source operand provides multiple indices, including at least one index for each second source vector. In particular embodiments, the most significant bits within a given element can provide an index for a data element of a first source vector, and the least significant bits provide an index for a data element of a second source vector. This approach takes advantage of the fact that the indices can each be formed with fewer bits than there is space within each element of the correlation source operand. Thus, because multiple indices can be stored in each element of the correlation source operand, less storage space is required to store the correlation information.

[0044] In some embodiments, each cross product operation performed by the processing circuitry is based on a different subset of data elements of the at least one second source vector operand.

[0045] For example, each subset of data elements of the at least one second source vector operand may contain different data elements from each other subset (e.g., each cross product operation may be based on different data elements of the second source vector operand). In these examples, the subsets of data elements of the at least one second source vector operand used for any two of the cross product operations differ by at least one data element. Note that "data element" generally refers to a particular data element position within a vector operand, rather than a particular numerical value held at that position. Thus, since multiple data elements within a given source vector operand may hold data having the same numerical value, the numerical values ​​of data elements in different subsets are not necessarily different. Furthermore, it is noted that the subsets may differ by more than one data element, and as a particular example, each data element in the at least one second source vector operand may be part of at most one subset of data elements.

[0046] In this way, multiple vectors can be represented in each second source vector operand, with each subset representing a different vector, and a multiply cross product operation can be performed based on the multiple vectors.

[0047] In some embodiments, the double cross product instruction includes a cross-product-sum instruction such that multiple cross product results have the same associated storage elements within a given two-dimensional array of storage elements, and the processing circuitry is configured to combine those multiple cross product results to update the values ​​held in the associated storage elements.

[0048] Although there are many possible uses for the MOP instruction, in this example, the MOP instruction is a sum-of-multiple-outer-products instruction (which may also be referred to as a "sum-of-multiple-outer-products instruction (SMOP)"), the execution of which involves accumulating multiple outer product results into a single storage element of a 2D array. This variation of the MOP instruction can be advantageous because it allows two matrices to be multiplied together in response to a single instruction.

[0049] In some embodiments, the apparatus comprises a set of vector registers accessible to the processing circuitry, each vector register arranged to store a vector comprising a plurality of data elements, and the plurality of first source vector operands and the at least one second source vector operand comprise a vector contained within a vector register of the set of vector registers.

[0050] A vector is a one-dimensional (1D) array containing multiple (two or more) data elements. Mathematically, a vector of data elements can be represented as a single column or a single row of data elements, and in a data processing system such as the device in this example, the data elements of a vector are stored in a single vector register. A vector operand is contrasted with a scalar operand, as each scalar operand contains a single data item (e.g., each data element in a vector can be a scalar operand). Storing and manipulating (performing operations on) data elements in the form of vectors is advantageous because it allows multiple data elements to be operated on in parallel (e.g., using single-instruction-multiple-data (SIMD) processing). This can significantly improve performance by increasing throughput, particularly when performing operations on large data arrays (e.g., matrix multiplication).

[0051] In some embodiments, the apparatus comprises a set of predicate registers accessible to the processing circuitry, each predicate register adapted to store predicate information comprising a plurality of elements, each element providing a predicate value, and the correlation information being stored in at least one predicate register of the set of predicate registers.

[0052] A data processing system such as the apparatus of this embodiment may be provided with a set of predicate registers for storing predicates (e.g., predicate information). Each predicate may be, for example, a mask of true / false (e.g., 1 / 0) values ​​to be used in vector processing. For example, the predicate may indicate which data elements in a vector should and should not be operated on. This embodiment utilizes the predicate registers for another purpose: storing correlation information. This approach is advantageous because it utilizes circuitry (predicate registers) that may already be used to perform cross-product operations if each subset of data elements were provided in a separate vector register (e.g., to predicate against zero values). Thus, this approach provides correlation information without occupying additional storage space (e.g., without occupying additional architectural registers or additional space in memory, cache, or some other storage structure).

[0053] In some embodiments, the data elements in each second source vector operand represent data values ​​from multiple rows and multiple columns of the source matrix.

[0054] As mentioned above, there are several scenarios in which the MOP instruction of the present technique may be useful. However, in a particular embodiment, the MOP instruction may be used in matrix multiplication. In this embodiment, each subset of data elements in at least one second source vector operand may represent a different row / column of the source matrix, and thus the present technique provides a more compact representation of the matrix.

[0055] In some embodiments, each element of a given correlation source operand is adjusted to allow reconstruction of a source matrix from at least one second source vector operand.

[0056] For example, each element of a given correlated source operand may indicate which row / column of the original matrix one or more corresponding elements of the second source vector operand came from. As a particular example, each element may be a bitmap of 1s and 0s indicating which row / column of the original matrix held a non-zero value (e.g., if the original matrix has two rows and the first bitmap is (1, 0), this may indicate that the first data element in a given second source vector operand represents the first data element in the first row of the matrix). An advantage of this approach is that it captures information about the original matrix format, making it straightforward to restore the second source vector operand to the original matrix.

[0057] In some embodiments, each data element in each second source vector operand is associated with a corresponding first dimension in the source matrix, the corresponding first dimension comprising a corresponding row or a corresponding column in the source matrix, and each data element in each second source vector operand provides a data value selected from among the data values ​​in the corresponding first dimension in the source matrix.

[0058] For example, the first data element in a given second source vector operand may be the first data element in a given row / column of the source matrix, and the correlation information may indicate which row / column the data element came from. Thus, in this manner, a matrix composed of multiple rows / columns (e.g., multiple vectors) can be compressed into fewer vector operands.

[0059] In some embodiments, the source matrix comprises a matrix having N:M structured sparsity, where each defined group of M data values ​​in the source matrix contains at most N non-zero data values.

[0060] In a matrix having structured sparsity, a particular (e.g., defined) group of M data elements is constrained to have at most N non-zero values. This may be the result of "pruning" an ANN, such that some of the data elements within the matrix may be replaced with zeros. The values of N and M depend on the implementation form. For example, ratios of 2:4 and 4:8 are common, and more generally, a ratio where M = 2N with the condition that N is less than M (N < M), or any other ratio, may be used.

[0061] When a matrix has structured sparsity, it is possible to compress the matrix to occupy less storage space by removing zero values, and in this way, the amount of storage space occupied by the matrix is reduced (this specifically leads to performance improvement and power consumption reduction when loading / storing data elements). The drawback of this approach may be considered to be that additional processing is required to decompress a second source vector into the original source matrix before any data processing operation (e.g., outer product operation) is performed on the data elements. However, the inventors have noticed that these compressed matrices can be used as operands of MOP instructions in combination with correlation information indicating from which row or column of the source matrix the data elements came. In this way, not only is the amount of storage space occupied by the matrix reduced, but the latency involved in performing matrix multiplication is reduced (leading to further performance improvement) because multiple outer product operations can be executed in response to a single MOP instruction.

[0062] Note that each group of M data elements within the source matrix is a defined / specific group, and not just any group of M elements within the source matrix can be used. For example, a given group of M elements can be taken and aligned in one dimension (e.g., a row or a column). For example, when M = 4, the first four elements within a given row can form one group.

[0063] In some embodiments, the source matrix comprises a matrix of weights or a matrix of activations for use in implementing an artificial neural network (ANN).

[0064] As mentioned above, the nodes of an ANN are typically represented as matrices of weights, and these matrices can be large. Similarly, data inputs to the nodes of an ANN typically take the form of activation matrices. Because these matrices can be large and the number of nodes in an ANN is typically very large, a significant amount of data is typically required to represent the ANN. Therefore, it can be useful to prune the neural network by clearing some of the data elements in the weight matrices (e.g., setting them to zero). Specifically, this can be done in a structured manner according to a defined N:M sparsity. This allows the matrices representing the ANN to be compressed into a smaller number of vector operands, providing all of the benefits discussed above (e.g., more efficient use of data storage, better performance, and lower power consumption). It will be appreciated that these benefits are particularly important when implementing ANNs when large amounts of data are involved.

[0065] The techniques described above may be implemented in a hardware device having circuit hardware implementing the processing circuitry, instruction decoder circuitry and other device features described above, that supports a multiple outer product (MOP) instruction as part of a native instruction set architecture supported by the decoding circuitry and processing circuitry.

[0066] However, in another embodiment, the same technique may be implemented in a computer program (e.g., an architecture simulator or model) that may be provided to control a host data processing apparatus to provide an instruction execution environment for executing instructions from the target code. The computer program may include instruction decode program logic for decoding instructions of the target code to control the host data processing apparatus to perform data processing, including performing vector operations. The instruction decode program logic thus emulates the functionality of the instruction decoder (instruction decoder circuit) of the hardware apparatus as described above. The program may also include processing program logic for performing data processing (thus emulating the functionality of the processing circuit described above) when executed on the host data processing apparatus. The program may also include register maintenance program logic for maintaining (in the memory or architectural registers of the host apparatus) data structures that represent (emulate) the architectural registers of the instruction set architecture being simulated by the program; for example, these may include any or all of vector registers, scalar registers, array registers, and predicate registers. The instruction decode program logic includes support for MOP instructions, which have the same functionality as described above for the hardware embodiment. Thus, such a simulator computer program can present, for target code executed on the simulator computer program, an instruction execution environment similar to that which would be provided by an actual hardware device capable of directly executing a target instruction set, even though there is no actual hardware providing these capabilities in the host computer running the simulator program. Thus, by providing a simulation of the above-described device, the simulator computer program provides all of the advantages described above in connection with the device.Additionally, simulation can be useful for running code written for one instruction set architecture on a host platform that does not actually support that architecture. Simulators can also be useful during the development of software for new versions of an instruction set architecture, while software development is performed in parallel with the development of hardware devices that support the new architecture. This can allow software to be developed and tested on a simulator so that software development can begin before hardware devices that support the new architecture are available.

[0067] The simulator program may be stored on a storage medium, which may be transitory or non-transitory.

[0068] Specific embodiments will now be described with reference to the drawings.

[0069] FIG. 1 schematically illustrates a data processing system 10 comprising a processor 20 coupled to a memory 30 storing data values ​​32 and program instructions 34. The processor 20 includes an instruction fetch unit 40 for fetching the program instructions 34 from the memory 30 and providing the fetched program instructions to an instruction decoder circuit 50. The decoder circuit 50 decodes the fetched program instructions and generates control signals for controlling a processing circuit 60 to perform processing operations on data values ​​held in storage elements of a register storage 65 as specified by a decoded vector instruction. As shown in FIG. 1, the register storage 65 may be formed from a number of different blocks. For example, a scalar register file 70 may be provided that includes a number of scalar registers that may be specified by an instruction, and a vector register file 80 may be provided that includes a number of vector registers that may be specified by an instruction.

[0070] Also, as shown in Figure 1, processor 20 may access array storage 90. In the embodiment shown in Figure 1, array storage 90 is provided as part of processor 20, although this is not a requirement. In various embodiments, array storage may be implemented as any one or more of architecturally addressable registers, architecturally non-addressable registers, scratchpad memory, and cache.

[0071] The processing circuitry 60, in one exemplary implementation, may include both vector processing circuitry and scalar processing circuitry. A general distinction between scalar and vector processing is as follows: Vector processing may involve applying a single vector processing instruction to data elements of a data vector having multiple data elements at each position within the data vector. The processing circuitry may also perform vector processing to perform operations on multiple vectors within a two-dimensional array (sometimes referred to as a subarray) of data elements stored within the array storage 90. Scalar processing effectively operates on single data elements rather than data vectors. Vector processing may be useful when processing operations are performed on many different instances of the data being processed. In a vector processing arrangement, a single instruction may be applied to multiple data elements (of a data vector) simultaneously. This may improve efficiency and throughput of data processing compared to scalar processing.

[0072] Processor 20 may be arranged to process two-dimensional arrays of data elements stored in array storage 90. The two-dimensional arrays may, in at least some embodiments, be accessed in multiple directions as one-dimensional vectors of data elements. In one exemplary implementation, array storage 90 may be arranged to store one or more two-dimensional arrays of data elements, where each two-dimensional array of data elements may form a square array portion of a larger or higher dimensional array of data elements in memory.

[0073] Register storage 65 also includes a predicate register file 75, which stores predicate information (e.g., masks) for use in data processing operations (e.g., masking out certain data elements of a vector so that they are excluded from a particular processing operation).

[0074] 2 illustrates one example of architectural registers 65 of processor 20 that may be provided in an exemplary implementation. The architectural registers (as defined in the instruction set architecture (ISA)) may include a set of scalar registers (not shown) and a set of predicate registers 100 for storing predicate information. The predicate registers may also store correlation information for executing a multiple-op instruction (MOP), as discussed below. For example, a specific number of predicate registers 100 may be provided, e.g., 16 registers P0-P15 in this example. The predicate registers may have a fixed size, although some bits within each element may not necessarily be used depending on the data type of the element stored in the predicate register.

[0075] Additionally, the architectural registers available for selection by program instructions in the ISA supported by decoder 50 may include a fixed number of vector registers 105 (labeled Z0 through Z31 in this embodiment). Of course, providing the number of predicate / vector registers shown in FIG. 2 is not required, and other embodiments may provide a different number of registers specifiable by program instructions. Each vector register may store vector operands including a variable number of data elements, where each data element may represent an independent data value. In response to vector processing (SIMD) instructions, the processing circuitry may perform vector operations on the vector operands stored in the registers to generate a result. For example, vector processing may include lane-by-lane operations, in which a corresponding operation is performed on each lane of elements in one or more operand vectors to generate a corresponding result of the elements of the result vector. When performing vector or SIMD operations, each vector register may have a vector length VL, where vector length refers to the number of bits in a given vector register. The vector length VL used in vector processing mode may be fixed or variable for a given hardware implementation. The ISA supported by processor 20 may support variable vector lengths, such that different processor implementations may choose to implement vector registers of different sizes, but the ISA may be vector length agnostic, such that instructions are designed so that code can function correctly regardless of the particular vector length implemented on a given CPU that executes the program.

[0076] Vector registers Z0-Z31 may also function as operand registers for storing vector operands that provide input to processing and accumulation operations performed by processing circuitry 60 on two-dimensional arrays of data elements stored in array storage 90.

[0077] As shown in FIG. 2, the architectural registers also include a certain number N A of array registers 110, ZA0 through ZA(NA-1), that form the aforementioned array storage 90. Each array register can be considered a single 2D array of data elements, e.g., a set of register storage for storing the results of processing and accumulation operations. However, processing and accumulation operations may not be the only operations that can use array registers. Array registers can also be used to store square arrays, such as when performing row / column transpositions of array structures in memory. When a program instruction references one of the array registers 110, it is referenced as a single entity using an array identifier ZAi; however, some types of instructions (e.g., data transfer instructions) can also select a subportion of the array by defining an index value that selects a portion of the array (e.g., one horizontal / vertical group of elements).

[0078] In practice, the physical implementation of the register storage corresponding to the array registers may include a specific number NR of array vector registers ZAR0 to ZAR(NR-1), as also shown in FIG. 2. The array vector registers ZAR forming the array register storage 110 may be a separate register set from the vector registers Z0 to Z31 used for vector input to SIMD and array operations. Each of the array vector registers ZAR may have a vector length VL, and thus each array vector register ZAR may store a 1D vector of length VL that can be logically divided into a variable number of data elements. For example, if VL is 512 bits, this could be, for example, a set of 64 8-bit elements, 32 16-bit elements, 16 32-bit elements, 8 64-bit elements, or 4 128-bit elements. It will be understood that not all of these options need to be supported in a given implementation. Supporting variable element sizes provides flexibility for computations involving data structures of different precisions. To represent a 2D array of data, a group of vector registers ZAR0 to ZAR(NR-1) can be logically considered as a single entity assigned a given one of the array register identifiers ZA0 to ZA(NA-1), and a 2D array is thus formed with elements spread within a single vector register corresponding to one dimension of the array and elements of the other dimension of the array striped across multiple vector registers.

[0079] As noted above, processing circuitry 60 is arranged to access scalar registers 70, vector registers 80, and / or array storage 90 under the control of instructions decoded by decoder circuitry 50. Further details of this latter arrangement will now be described with reference to Figure 3A, which merely provides one illustrative example of how array storage may be accessed, particularly considering access to a square 2D array within the array storage.

[0080] In the illustrated embodiment, the square 2D array in array storage 90 is arranged as an array 205 of n x n storage elements / locations 200, where n is an integer greater than 1. In this embodiment, n is 16, which means that the granularity of access to storage locations 200 is 1 / 16 of the total storage of the 2D array in either the horizontal or vertical array direction.

[0081] From the perspective of the processing circuitry, the array of n x n locations is accessible as n linear (one-dimensional) vectors in a first direction (e.g., horizontal as depicted) and n linear vectors in a second array direction (e.g., vertical as depicted). Thus, the n x n storage locations are accessible from the perspective of processing circuitry 60 as 2n linear vectors of n data elements each.

[0082] The array of storage locations 200 is accessible by access circuits 210, 220, column select circuit 230 and row select circuit 240 under the control of control circuit 250 which communicates with at least processing circuit 60 and optionally with decoder circuit 50.

[0083] Referring to FIG. 3B, the n linear vectors in a first direction (the horizontal or "H" direction as depicted) for an exemplary square 2D array designated "ZA1" (note that, as discussed below, there may be more than one such 2D array provided within array storage 90, e.g., ZA0, ZA1, ZA2, etc.), may be referenced (in hexadecimal notation) to each of 16 data elements 0 through F, in this example, as ZA1H0 through ZA1H15. The same underlying data stored in the 256 entries (16×16 entries) of array storage 90ZA1 of FIG. 3B may instead be referenced in a second direction (the vertical or "V" direction as depicted) as ZA1V0 through ZA1V15. Note, for example, that data element 260 is item F of ZA1H0 but is referenced as item 0 of ZA1V15. It should be noted that the use of "H" and "V" does not imply any spatial or physical layout requirements regarding the storage of the data elements that make up array storage 90, and is unrelated to whether, in any exemplary application, the 2D arrays within the array storage store row data or column data.

[0084] 4A illustrates a cross product operation. The cross product operation may take as input two vectors A and B, which may be stored in a vector register file as described above. The result of the cross product operation is a matrix (e.g., a 2D array)

[0085]

number

[0086] 4B illustrates a matrix multiplication operation. Specifically, FIG. 4B illustrates an operation that involves multiplying two matrices C and D (which may be stored, for example, in vector registers (e.g., one row or one column held in each register) or in an array storage circuit) together to produce a matrix CD. As can be seen, the result of multiplying two n×n matrices together is an n×n matrix (more generally, an n×k matrix multiplied by a k×m matrix results in an n×m matrix).

[0087] While there are several ways to calculate the elements of the output matrix CD, a technique typically used by processors is to perform a multiple cross product operation and accumulate (add) the results to each other. For example, to perform the matrix multiplication shown in FIG. 4B, the processing circuitry may first calculate the cross product of the leftmost column i of matrix A with the topmost row w of matrix B to generate 16 cross product results and populate a 4×4 array in the array storage. The processing circuitry may then calculate the cross product of the next column j of matrix A with the next row x of D to generate another 16 cross product results that are added to the cross product results already stored in the array. This process then continues with the last two pairs of vectors (

[0088]

number

[0089]

number

[0090] It can thus be seen that matrix multiplication can be performed by performing multiple cross product operations and accumulating the results, e.g., by executing one or more multiple cross product instructions. Note that the order in which pairs of vectors are multiplied together is not limited to the order described above, and the cross products can be calculated in any order. In addition, the cross product operations need not be performed sequentially.

[0091] As can be seen from the example of Figure 4B, matrix multiplication involves performing a significant number of operations; for example, multiple cross product operations may be performed, each of which involves multiple multiplication operations. As a result, matrix multiplication can be a particularly time- and energy-consuming process. This is especially true in situations such as implementing artificial neural networks (ANNs), which may involve performing a significant number of matrix multiplications. Therefore, there is an interest in improving the performance of matrix multiplication operations.

[0092] Figure 5 illustrates a matrix with N:M structured sparsity. In the matrix illustrated in Figure 5, shaded elements represent non-zero elements, and blank (unshaded) elements represent zero elements. In the particular example illustrated in Figure 5, a 10x11 matrix with 3:5 sparsity is shown. More specifically, each column of 10 data elements is considered to include two groups of M = 5 consecutive data elements, of which at most N = 3 are non-zero (e.g., shaded data elements).

[0093] As explained above, structured sparsity can be introduced into a matrix (e.g., by clearing / setting some of its elements to zero) to reduce the number of data elements in the matrix that will be involved in at least some matrix operations performed on that matrix. In addition, the inventors have realized that introducing structured sparsity into a matrix (as discussed in more detail below) allows the matrix to be compressed into a smaller number of vector operands, which reduces the amount of space occupied by the matrix.

[0094] 6A and 6B, similar to FIG. 5, show further examples of sparse matrices for information, where shaded elements represent non-zero elements and blank (unshaded) elements represent zero elements. Specifically, FIG. 6A shows a matrix with 2:4 structured sparsity (N=2, M=4), and FIG. 6B shows a matrix with 4:8 structured sparsity (N=4, M=8). Both FIG. 6A and FIG. 6B show 8×8 matrices with a maximum of four non-zero data elements per column, but note that the matrix shown in FIG. 6A is further constrained in that each group of four data elements in a column (e.g., the first four elements in a column or the last four elements in a column) has a maximum of two non-zero elements. This can be seen, for example, by comparing the leftmost column of each matrix; in the matrix of FIG. 6B, the first four data elements in the leftmost column are all non-zero, which is not allowed in a matrix with 2:4 structured sparsity, where at most two of these four elements can be non-zero (as shown in the leftmost column of the matrix of FIG. 6A).

[0095] Data processing operations can be performed on sparse matrices just as they can be performed on other matrices. For example, Figure 7 shows how an array of multiply-accumulate (MAC) units can be used to calculate the result of multiplying an 8x4 matrix of activations by a 4x8 matrix (with 2:4 sparsity) of weights (e.g., weights at a given node of an ANN). Predicate information can be used, for example, to mask out zero values.

[0096] However, as explained above, the inventors have realized that there are several advantages to compressing a sparse matrix into a smaller number of vector operands. An example of this is shown in FIG. 8A, where a 4×8 matrix with a 2:4 sparsity is compressed into two vector operands. Specifically, FIG. 8 shows how a matrix that initially occupies four vector registers (Z8, Z9, Z10, Z11) can be compressed into two vector registers (Z4, Z5), each of which stores data elements from one or more rows of the source matrix. This frees up two vector registers in the vector register file (and / or an equivalent amount of space in memory / another data store).

[0097] A set of indices Id is maintained for each compressed source vector operand that can be used to reconstruct the source matrix. For example, the indices may indicate which row of the source matrix each element in Z4 or Z5 comes from.

[0098] Another example of how a sparse matrix can be compressed into fewer vector operands is shown in FIG. 8B, where an 8×8 sparse matrix (with 4:8 structured sparsity) is compressed into four vector operands.

[0099] Once the sparse matrix is ​​compressed, the inventors have realized that a multiple outer product (MOP) operation can be performed on the resulting vector operands without first decompressing them to form the original matrix. This is shown in Figure 9, where an 8x4 matrix of activations is multiplied by a 2x8 compressed matrix of weights using multiplication circuit 255 (which may be a MAC array, for example). For example, this may be performed as two MOP operations, one based on Z4 and the other on Z5, or as a sum-of-outer-products (SMOP) operation.

[0100] To multiply matrices together, four cross-product operations are performed. These involve computing the cross product of each column of the activation matrix (e.g., vectors Z0, Z1, Z2, and Z3) with data elements in Z4 and Z5 that have the same shading. For example, this means that the first, second, fourth, and sixth data elements from the left in Z5 are multiplied by the corresponding elements in Z3; the third, fifth, and seventh elements from the left in Z5 and the first element from the left in Z4 are multiplied by the corresponding elements in Z2, and so on. An index may be used to identify which vector in the first group of vector operands (Z0:Z3) should be multiplied with each data element in the second group of vector operands (Z4:Z5).

[0101] The multiplication circuit 255 shown in FIG. 9 includes several multiplexers 270, each of which provides an input to a corresponding multiplier circuit (not shown). In this particular embodiment, an 8×8 array of multiplexers 270 is provided, including a multiplexer (and corresponding multiplier) for each multiplication to be performed. Each multiplexer receives as input a data element from a corresponding position in each of the activation vectors Z0 through Z3 and, based on correlation information, selects one of the data elements for multiplication with the data element in the corresponding position in Z4 or Z5. However, it will be understood that in other embodiments, there may be fewer multiplexers and multipliers, with each multiplexer / multiplier pair being used to perform several of the multiplication operations to be performed.

[0102] Thus, as described above, the apparatus of the present technology is configured to support execution of a multiple-op instruction that identifies as input a plurality of first vector operands (e.g., vector registers Z0:Z3), at least one second source vector operand (e.g., Z4 and / or Z5), and correlation information (e.g., one or both of a set of indexes).

[0103] While the figure shows an example in which the weight matrix is ​​a compressed sparse matrix, it should be noted that it is equally possible for the activation matrix to be a compressed sparse matrix instead of, or in addition to, the weight matrix. More generally (since multiplying an activation matrix with a weight matrix is ​​just one exemplary use case for the present technology), it does not matter whether the first vector operand(s) or the second vector operand(s) specified by the MOP instruction represent a compressed sparse matrix; it can be either or both.

[0104] FIG. 10 is a block diagram of an apparatus according to an exemplary implementation, illustrating how processing circuitry is used to perform a cross-product operation. A vector register file 80 provides multiple vector registers that can be used to store vectors of data elements. As previously described, the MOP instruction can be tailored to identify multiple first source vector operands 300 and at least one second source vector operand 320. The at least one second source vector operand 320 (and optionally the multiple first source vector operands 300) includes multiple subsets of data elements, each subset for a different cross-product operation. Note that the terms “first” and “second” used herein to refer to two source vector operand(s) are used purely as labels to distinguish them and do not imply any particular ordering with respect to how those operands are specified by the instruction. Thus, either of the source operand fields of the instruction may be used to specify the first source vector operand referred to above, and then the other of the source operand fields may be used to specify the second source vector operand referred to above.

[0105] Furthermore, according to the techniques described herein, at least one of the two source vector operands may include multiple sub-vectors of data elements for use in different cross-product operations, while the other source vector operand may not; however, in some example implementations, both source vector operands may also include multiple subsets of data elements. Similarly, two or more second source vector operands 320 may be specified in addition to specifying multiple first source vector operands. In addition, the number of first source vector operands and the number of second source vector operands specified by the MOP are not limited to either one vector operand or two vector operands; in fact, more than two first / second vector operands may be specified (e.g., four vector operands, eight vector operands, etc.).

[0106] The MOP instruction also specifies correlation information that identifies which data elements in the at least one second source vector operand 320 are to be multiplied by which vector operands in the plurality of first vector operands. In this example, the correlation information is stored in predicate registers 325 in predicate register file 75, and thus the MOP instruction also identifies one or more predicate registers 325.

[0107] The processing circuit 60 is controlled in dependence on control signals received from the decoder circuit (decoder) 50, and when the decoder circuit 50 decodes the aforementioned MOP instruction, it sends control signals to the processing circuit to execute a plurality of calculations to perform a plurality of cross product operations, the plurality of cross product operations including, for a given first source vector operand, executing associated cross product operations to calculate the cross product of the given first source vector operand with a subset of data elements of at least one second source vector operand. As part of this process, the control signals control the selection circuit 340 provided by the processing circuit 60 to select appropriate data elements to be processed by each cross product operation. Each cross product operation includes multiplying each data element of the associated subset of data elements in the at least one second source vector operand by each data element in the associated first source vector operand, and then using each cross product result element to update a value held in an associated storage element in a given two-dimensional array 380 of storage elements in the array storage 90.

[0108] The selection circuit 340 can be organized in a variety of ways, but in one exemplary implementation comprises a multiplexer circuit provided for each of a plurality of multipliers used to generate a cross product result from two input data elements, the multiplexer circuit being used to select the appropriate two input data elements for each multiplier.

[0109] The selected input data elements are then forwarded to multiplication circuit 350, which, as described above, in one exemplary implementation may include a multiplication circuit for each cross product result generated. Each cross product result element is generated by multiplying two input data elements provided to a corresponding multiplier in multiplication circuit 350. The cross product result elements may be provided directly to array update circuit 370, which uses them to update storage elements in 2D array 380, with each cross product result element having an associated storage element in 2D array 380 and being used to update the value held in that associated storage element. However, it is often the case that the cross product operation being performed is an accumulation operation, and each generated cross product result element is combined with an existing value stored in an associated storage element of 2D array 380 (e.g., by adding or subtracting the cross product result from an existing value) using optional accumulation circuit 360. Although the multiplier circuit 350 and the optional accumulate circuit 360 are shown as separate blocks in FIG. 10, in an exemplary implementation they may be provided as a combined block formed from multiplier accumulate circuits.

[0110] Array update circuit 370 is used to control access to associated storage elements in 2D array 380 to ensure that each value received by the array update circuit is used to update an associated storage element in 2D array 380.

[0111] Cross product operations are usefully used within data processing systems for a variety of different reasons; therefore, the ability to perform multiple cross product operations in response to a single instruction can provide significant processing power / throughput improvements and may more efficiently use available storage resources provided by the two-dimensional array within array storage 90. As just an example of how cross product operations may be used, a cross product operation may be used to implement a matrix multiplication operation. Matrix multiplication may include, for example, multiplying a first m×k matrix of data elements by a k×n matrix of data elements to produce an m×n matrix of data elements result. This operation may be decomposed into multiple cross product operations (more specifically, k cross product operations, where k may be referred to as the “depth”), each of which involves performing a sequence of multiply-accumulate operations to multiply each data element of an m vector of data elements from a first matrix by each data element of an n vector of data elements from a second matrix to produce an m×n matrix of resulting data elements stored within the 2D array. The results of multiple cross product operations can be accumulated within the same 2D array to produce the m×n matrix produced by performing the matrix multiplication described above.

[0112] Matrix multiplication has several potential uses: In addition to applications in implementing ANNs such as those described above, matrix multiplication can be used, for example, in image processing.

[0113] Figure 11A shows how a cross product result element may be associated with a particular storage element within a 2D array. In the example of Figure 11A, a data element 570 from a first source vector operand is multiplied by a data element 572 from a second source vector operand using a multiplication function 574 to generate a cross product result element, which is then subjected to an accumulation operation by an accumulation function 576 to add or subtract the cross product result to or from the current value stored in the associated storage element 578 to generate an updated value that is stored in the associated storage element.

[0114] 11B illustrates a cross-product-sum operation in which two cross-product result elements are associated with the same storage element in a 2D array. In this example, a data element 580 from a first source vector operand is multiplied by a data element 582 from a second source vector operand using a multiplication function 584 to generate a first cross-product result element. Similarly, a data element 586 from the first source vector operand and a data element 588 from the second source vector operand are multiplied by a multiplication function 590 to generate a second cross-product result element. The two cross-product results are then added using an addition function 592, and an accumulation function 594 is performed to generate an updated data value for storage in an associated storage element 596. Thus, it will be appreciated that in some implementations, there may be more than one cross-product result element associated with the same storage element in a 2D array.

[0115] 12 is a diagram that schematically illustrates fields that may be provided in a MOP instruction, according to one exemplary implementation. The opcode field 605 is used to identify the type of instruction, in this case, identifying that the instruction is a MOP instruction.

[0116] One or more control information fields 610 may be provided to identify one or more predicates, for example, as mentioned above. Additionally, field 615 identifies correlation information to be used in the operation; for example, this may be an identifier of a register (e.g., a predicate register) that stores the correlation information. Note that this is separate from any predicate information held in control information field 610 that is used as a predicate / mask in the operation. Field 620 is then used to identify multiple first source vector operands (e.g., by specifying vector registers in vector register file 80, one or more vector operands are implicitly associated with the identified registers). Similarly, field 625 may be used to identify one or more second source vector operands, again, for example, by specifying one or more vector registers in vector register file 80. This is just one example; in practice, either field 620, 625 may be used to specify the aforementioned first source vector operand, with the other field specifying the second source vector operand. Finally, field 630 may be used to identify a destination 2D array within array storage 90 that will be used to store the matrix produced as a result of performing the multiple cross product operation specified by the multiple cross product instruction.

[0117] As mentioned above, there are multiple ways in which correlation information can be represented. Figures 13-15 show some examples of how correlation information can be represented in one or more predicate registers.

[0118] 13 shows a first example in which each predicate register P0, P1 holds a set of indexes for a corresponding vector register Z4, Z5, and each set of indexes forms correlation information for that corresponding vector register. Each index in this example identifies the register of the source matrix from which the corresponding data element comes (and thus indirectly identifies a row of the source matrix), with the first (bottom) row being row "0" and the fourth (top) row being row "3." For example, the leftmost element in predicate register P0 (which holds correlation information for vector register Z4 in this example) is "2," indicating that the corresponding element (leftmost element) in vector register Z4 comes from the third row of the source matrix (e.g., from vector register Z10 in this example).

[0119] The right side of Figure 13 shows how a MOP instruction (in this case identified as an "FMOPA" instruction) may be represented in this example. In this example, a separate MOP instruction is being executed for each of vector registers Z4 and Z5, with each MOP instruction being: a destination array (ZAi) into which the cross product results calculated by the processing circuit are written; A predicate register (P0 or P1) that holds the index used when executing the MOP instruction. a plurality of first source vector registers (Z0 to Z3) that hold a plurality of first source vector operands; and Identify a second source vector register (Z4 or Z5) that holds the second source vector operand for the operation.

[0120] 14 shows another example of how correlations can be represented, in which the same index is used, but the indexes for both Z4 and Z5 are stored in the same predicate register P0. This example takes advantage of the fact that the number of bits required to represent each index is typically significantly less than the number of bits available in each element of the predicate register; for example, the indexes in this example are at most two bits long (indices 0, 1, 2, and 3 are represented in binary as 00, 01, 10, and 11, respectively), while the number of bits in each element of the predicate register can be four or more. Therefore, the inventors realized that the indices of the two vector registers Z4 and Z5 can be packed into a single predicate register, reducing the number of predicate registers occupied by correlation information. Specifically, as shown in FIG. 14, the upper half (e.g., the upper two bits) of each data element can be used to store an index for the corresponding data element in one of the vector registers (e.g., the upper half of the leftmost element stores the value "3", which is the index for the leftmost element in Z5), while the lower half (e.g., the lower two bits) of each data element can be used to store an index for the corresponding data element in the other of the vector registers (e.g., the lower half of the leftmost element stores the value "2", which is the index for the leftmost element in Z4).

[0121] Thus, as shown by the instruction assembly syntax on the right side of Figure 14, this embodiment requires two slightly different MOP instructions (i.e., an "FMOPA1" instruction and an "FMOPA2" instruction), both of which identify the same predicate register P0, but one that causes the processing circuit to read the most significant two bits of each element and the other that causes the processing circuit to read the least significant two bits of each element. To distinguish between the two types of MOP instructions, each of the two MOP instructions may have, for example, a different opcode. Alternatively, the encoding of the two instructions may differ in some other way.

[0122] Therefore, each MOP instruction in Figure 14 is ● Destination array (ZAi), ●Same predicate register (P0), a plurality of first source vector registers (Z0 to Z3) that hold a plurality of first source vector operands; and Identify a second source vector register (Z4 or Z5) that holds the second source vector operand for the operation.

[0123] 15 shows another example of how correlation information can be represented, where the correlation information in two vector registers Z4 and Z5 is compressed into a single predicate register P0. In this example, each data element in predicate register P0 holds a bitmap indicating which data elements in the corresponding row of the source matrix held non-zero values. For example, the left-most data element in the predicate register reads "1100," indicating that the left-most column of the source matrix has non-zero elements in the top two rows and zeros in the bottom two rows of the same column. From this, it can be determined that the left-most data elements in vector registers Z4 and Z5 are from the top two rows (second and third) of the source matrix.

[0124] Thus, when executing the MOP instruction in this embodiment, the processing circuitry determines the location of either the first or second "1" in the corresponding bitmap to identify which of the first source vectors should be used in the associated cross product operation.

[0125] Each command is ● Destination array (ZAi), ●Same predicate register (P0), a plurality of first source vector registers (Z0 to Z3) that hold a plurality of first source vector operands; and Identify a second source vector register (Z4 or Z5) that holds the second source vector operand for the operation.

[0126] 14 and 15 show an example in which a separate MOP instruction is executed for each second vector operand (Z4, Z5), it should be noted that it is also possible to define a single MOP instruction that specifies both second vector operands.

[0127] As explained above, structured sparsity is an optimization technique for inference of neural network models (but not for training of neural network models). Therefore, it can be beneficial to support models using data as signed 8-bit integers (int8) in Brain Floating Point (bfloat16 / BF16) format. In some data processors, two-way data interleaving is typically used for the cross product of bf16 data, so that, for example, the two vectors of weights (Z4, Z5) in Figure 9 can be stored in a single vector register of bf16 elements (Z m ). Similarly, the four activation vectors (Z0:Z3) are packed into two vector registers (Z n , Z n+1 ) Figure 16 shows the principle of a multi-register outer multiply-accumulate scheme with bf16 elements. The predicate register can contain a list of indexes as shown in Figure 14 or a bitmap as shown in Figure 15.

[0128] 16, the MOP instruction in this embodiment is a sum-of-outer-products (SMOPA) instruction for performing a basic two-way dot product operation. The SMOPA instruction in this embodiment is: ● Destination array (ZAi), ● Predicate register (P i ), Multiple first source vector registers (Z) that hold multiple first vector operands n , Z n+1 ), and A second source vector register (Z) that holds multiple second source vector operands. m ), specify

[0129] For 8-bit data types (int8 or uint8), the base instruction is a cross-product-accumulate instruction with 32-bit accumulation. The basic operation is a 4-way dot product operating on 4-way interleaved data. As shown in FIG. 17, in a variation of the invention operating on 8-bit data, two source registers are used for activation (left operand) and one source register is used for weight (right operand). In this variation of the instruction, two predicate registers are used because four elements out of eight are selected (e.g., two times two elements out of four). Four 8-bit 4-to-1 multiplexers are provided before each multiplier to achieve a throughput of one operation per cycle.

[0130] The right side of Figure 17 shows the format of this variation of the MOP instruction. As shown, the MOP instruction in this example is a sum-of-outer-products instruction (SMOPA), which: ● Destination array (ZAi), ● Predicate register (P0, P1), Multiple first source vector registers (Z) that hold multiple first vector operands n , Z n+1 ), and A second source vector register (Z) that holds multiple second source vector operands. m ), specify

[0131] While Figure 17 assumes that a 2:4 sparsity pattern is used, in an alternative variation of the instruction, a 4:8 sparsity pattern is used in conjunction with a bitmap to select the source element of the left operand. This variation requires an 8:1 multiplexer instead of a 4:1. However, the format of the alternative variation of the instruction is the same as that shown in Figure 17, since the operands are identical.

[0132] Regarding the use of the command, below is a code snippet showing how the command can be used. LD1B{Z0-Z3},p0 / Z,[&activations] LD1B{Z4-Z5},p1 / Z,[&weights] LDR p2,[&indexes] LDR p3,[&indexes] SMOPA ZA0.S,P2,{Z0.H-Z1.H},Z4.H SMOPA ZA1.S,P3,{Z0.H-Z1.H},Z5.H SMOPA ZA2.S,P2,{Z2.H-Z3.H},Z4.H SMOPA ZA3.S,P3,{Z2.H-Z3.H},Z5.H

[0133] 18 is a flow diagram illustrating the steps performed when decoding a multiple cross product instruction according to one exemplary implementation. At step 650, it is determined whether a MOP instruction has been encountered. If not, at step 655, a standard decode of the associated instruction is performed and processing circuitry is controlled to perform the required operations defined by the instruction.

[0134] However, if a MOP instruction is encountered, then in step 660 the instruction is decoded to identify the source vector operands (e.g., a plurality of first source vector operands and at least one second source vector operand), the destination 2D array, correlation information, and the type of cross product to be performed (e.g., whether an accumulating cross product or a non-accumulating transform is being performed, and e.g., whether a regular cross product operation or a cross-product-accumulation operation is being performed).

[0135] Next, in step 665, the processing circuitry is controlled to perform the necessary cross product operations and perform the necessary updates to the 2D array storage elements. As part of this process, the selection circuitry is controlled to select data elements for each multiplication operation in response to the correlation information.

[0136] FIG. 19 illustrates a simulator implementation that may be used. While the above embodiments implement the present invention in terms of apparatus and methods for operating specific processing hardware supporting the technology, it is also possible to provide an instruction execution environment according to the embodiments described herein implemented through the use of a computer program. Such computer programs are often referred to as simulators, insofar as they provide a software-based implementation of a hardware architecture. Various simulator computer programs include emulators, virtual machines, models, and binary translators, including dynamic binary translators. Typically, a simulator implementation may execute on a host processor 715, optionally running a host operating system 710 and supporting the simulator program 705. In some arrangements, there may be multiple layers of simulation between the hardware and the provided instruction execution environment, and / or there may be multiple different instruction execution environments provided on the same host processor. Historically, powerful processors have been required to provide simulator implementations that run at reasonable speeds, but such an approach may be justified in certain situations, such as when it is desirable to execute code native to another processor for compatibility or reuse reasons. For example, a simulator implementation may provide an instruction execution environment with additional functionality not supported by the host processor hardware, or may provide an instruction execution environment typically associated with a different hardware architecture. An overview of simulation is given in "Some Efficient Architecture Simulation Techniques," Robert Bedichek, Winter 1990 USENIX Conference, pp. 53-63.

[0137] While embodiments have been described above with reference to particular hardware components or features, equivalent functionality can be provided in simulated embodiments by suitable software components or features. For example, a particular circuit may be implemented in a simulated embodiment as computer program logic; e.g., simulator program 705 may include processing program logic 720 for emulating the behavior of the processing circuit described above and instruction decode program logic for emulating the behavior of the instruction decoder circuit described above. Similarly, memory hardware such as registers or cache may be implemented in a simulated embodiment as software data structures; in this particular example, array storage emulation program logic 722 is provided to emulate the array storage described above. In arrangements where one or more of the hardware elements referenced in the foregoing embodiments reside on host hardware (e.g., host processor 715), some simulated embodiments may use the host hardware where suitable.

[0138] The simulator program 705, which may be stored on a computer-readable storage medium (which may be a non-transitory medium), provides a program interface (an instruction execution environment) to the target code 700 (which may include applications, an operating system, and a hypervisor) that is the same as the interface of the hardware architecture being modeled by the simulator program 705. Thus, the program instructions of the target code 700, including the MOPs described above, may be executed from within an instruction execution environment using the simulator program 705, thereby allowing a host computer 715, which does not actually have the hardware capabilities of the devices described above, to emulate those capabilities.

[0139] In this application, the term "configured to" is used to mean that elements of a device have a configuration that is capable of performing a defined operation. In this context, "configuration" refers to a manner of arrangement or interconnection of hardware or software. For example, a device may have dedicated hardware that provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not imply that device elements need to be modified in any way to provide the defined operation.

[0140] Although illustrative embodiments of the present invention have been described in detail herein with reference to the accompanying drawings, it will be understood that the invention is not limited to those precise embodiments, and that various changes and modifications can be made to the embodiments by those skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. 1. An apparatus comprising: processing circuitry for performing vector operations; an instruction decoder circuit for decoding an instruction from a set of instructions so as to control the processing circuit to perform the vector operation specified by the instruction; the set of instructions includes a multiple cross product instruction specifying a plurality of first source vector operands, at least one second source vector operand, and correlation information associated with the at least one second source vector operand, each vector operand including a plurality of data elements, and for a given second source vector operand, the correlation information is adjusted to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; the instruction decoder circuit is responsive to the multiple cross product instruction to control the processing circuit to execute a plurality of calculations to perform a plurality of cross product operations, the plurality of cross product operations including, for a given first source vector operand, performing cross product operations associated with calculating a cross product of the given first source vector operand with a subset of data elements of the at least one second source vector operand; the processing circuitry is configured to, for each data element of the at least one second source vector operand, select, in dependence on the correlation information, a corresponding first source vector operand to be used when performing the associated cross product operation.

2. an array storage circuit comprising storage elements for storing data elements, the array storage circuit being arranged to store at least one two-dimensional array of data elements accessible to said processing circuit when performing said vector operation; said double cross product instruction specifying a given two-dimensional array of data elements in said array storage forming a destination operand; the processing circuitry is configured to perform the associated cross product operation on the given first source vector operand by multiplying each data element of the given first source vector operand by each data element in the subset of data elements of the at least one second source vector operand to generate a plurality of cross product result elements, and using each cross product result element to update a value held in an associated storage element in the given two-dimensional array of storage elements; 10. The apparatus of claim 1.

3. The processing circuitry a multiplication circuit for generating each cross product result when performing the plurality of cross product operations; and multiplexer circuitry associated with the multiplier circuitry for selecting, under control of the correlation information, selected data elements of the plurality of first source vector operands and selected data elements of the at least one second source vector operand to be multiplied to generate associated cross product result elements.

4. the correlation information includes at least one set of indexes; the at least one set of indices includes an index associating each data element of the given second source vector operand with the corresponding first source vector operand; An apparatus according to any one of claims 1 to 3.

5. the correlation information is provided by at least one correlation source operand specified by the multiple cross product instruction; a given correlation source operand includes, for each data element of said given second source vector operand, a corresponding element including said index relating said data element of said given second source vector operand to said corresponding first source vector operand; 5. The apparatus of claim 4.

6. the correlation information includes a set of indices for each second source vector operand; 6. An apparatus according to claim 4 or 5.

7. the at least one set of indices includes a set of indices providing the correlation information for a plurality of second source vector operands.

6. An apparatus according to claim 4 or 5.

8. the correlation information is provided by at least one correlation source operand specified by the multiple cross product instruction; each element of a given correlation source operand includes a plurality of indices, the plurality of indices including indices for corresponding data elements of each of the plurality of second source vector operands; 8. The apparatus of claim 7.

9. each cross product operation performed by the processing circuitry is based on a different subset of data elements of the at least one second source vector operand; An apparatus according to any one of claims 1 to 8.

10. the multiple cross product instruction includes a cross product sum instruction, multiple cross product results have the same associated storage element within the given two-dimensional array of storage elements, and the processing circuitry is configured to combine those multiple cross product results to update the value held in the associated storage element. Apparatus according to any one of claims 1 to 9 when dependent on claim 2.

11. a set of vector registers accessible to said processing circuitry; Each vector register is adapted to store a vector containing a plurality of data elements; the plurality of first source vector operands and the at least one second source vector operand comprise vectors contained within vector registers of the set of vector registers; An apparatus according to any one of claims 1 to 10.

12. a set of predicate registers accessible to said processing circuitry; Each predicate register is adapted to store predicate information including a plurality of elements, each element providing a predicate value; the correlation information is stored in at least one predicate register of the set of predicate registers; 12. The apparatus of claim 11.

13. the data elements in each second source vector operand represent data values ​​from multiple rows or multiple columns of a source matrix; An apparatus according to any one of claims 1 to 12.

14. each element of the given correlation source operand is adjusted to enable reconstruction of the source matrix from the at least one second source vector operand; Apparatus according to claim 13 when dependent on claim 8.

15. each data element in each second source vector operand is associated with a corresponding first dimension in said source matrix, said corresponding first dimension comprising a corresponding row or a corresponding column in said source matrix; each data element in each second source vector operand provides a data value selected from among the data values ​​in the corresponding first dimension in the source matrix; 15. Apparatus according to claim 13 or 14.

16. the source matrix comprises a matrix having N:M structured sparsity, and each defined group of M data values ​​in the source matrix contains at most N non-zero data values; 16. Apparatus according to any one of claims 13 to 15.

17. The source matrix comprises a matrix of weights or a matrix of activations for use in implementing an artificial neural network. An apparatus according to any one of claims 13 to 16.

18. 1. A method comprising: performing vector operations using processing circuitry; decoding an instruction from a set of instructions to control the processing circuit to perform the vector operation specified by the instruction, the set of instructions including a multiple cross product instruction specifying a plurality of first source vector operands, at least one second source vector operand, and correlation information associated with the at least one second source vector operand, each vector operand including a plurality of data elements, and for a given second source vector operand, the correlation information is adjusted to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; The method includes: executing, in response to the multiple cross product instruction, a plurality of calculations to perform a plurality of cross product operations, the plurality of cross product operations including, for a given first source vector operand, performing cross product operations associated with computing a cross product of the given first source vector operand with a subset of data elements of the at least one second source vector operand; and for each data element of the at least one second source vector operand, selecting a corresponding first source vector operand to be used when performing the associated cross product operation in dependence on the correlation information.

19. A computer program comprising instructions that, when executed on a computer, processing program logic for performing vector operations; instruction decoder program logic for decoding a target instruction from a set of target instructions so as to control the processing program logic to perform the vector operation specified by the target instruction; the set of target instructions includes a multiple cross product instruction specifying a plurality of first source vector operands, at least one second source vector operand, and correlation information associated with the at least one second source vector operand, each vector operand including a plurality of data elements, and for a given second source vector operand, the correlation information is adjusted to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; the instruction decoder program logic controls the processing program logic to execute a plurality of calculations to perform a plurality of cross product operations in response to the double cross product instruction, the plurality of cross product operations including, for a given first source vector operand, performing cross product operations associated with calculating a cross product of the given first source vector operand with a subset of data elements of the at least one second source vector operand; the processing program logic is configured to select, for each data element of the at least one second source vector operand, a corresponding first source vector operand to be used when performing the associated cross product operation in dependence on the correlation information.

20. 20. A computer readable storage medium storing a computer program according to claim 19.

Images