Multi-outer product instruction

Abstract

A multi-outer product instruction specifies a plurality of first source vector operands, each vector operand including a plurality of data elements, at least one second source vector operand, and correlation information associated with the second source vector operand, and the correlation information indicates a corresponding first source vector operand for each data element of a given second source vector operand. In response to the multi-outer product instruction, the instruction decoder circuitry (50) controls the processing circuitry (60) to perform calculations to perform outer product operations that include performing, for a given first source vector operand, an associated outer product of a subset of data elements of the first source vector operand and the second source vector operand. The processing circuitry selects, for each data element of the second source vector operand, a corresponding first source vector operand to be used in performing the associated outer product operation, depending on the correlation information.

Description

[0001] The present technology relates to the field of data processing.

[0002] The data processing apparatus may comprise processing circuitry capable of performing an outer product operation in which the outer product of two vectors is calculated. The processing circuitry may further be arranged to perform a plurality of outer product operations on different pairs of vectors to multiply two matrices together.

[0003] Outer products and matrix multiplications have several applications. For example, the implementation of artificial neural networks (ANNs) generally involves matrix multiplications.

[0004] However, multiplication operations can be relatively slow, and each outer product operation (not to mention each matrix multiplication) generally involves a large number of multiplication operations (e.g., one multiplication for each pair of data elements in the input vectors). Thus, there can be a significant performance impact associated with performing outer product operations, and thus it would be desirable to provide techniques for performing outer product operations more efficiently, particularly in situations where multiple outer product operations are to be performed.

[0005] In a first example of the present technology, there is provided an apparatus comprising:

[0006] Processing circuitry for performing vector operations; and

[0007] instruction decoder circuitry that decodes instructions from a set of instructions to control the processing circuitry to perform the vector operations specified by the instructions,

[0008] in:

[0009] The set of instructions includes specifying a plurality of first source vector operands, at least one second source

[0010] vector operand and dependency information associated with the at least one second source vector operand

[0011] Multiple outer product instructions, where each vector operand includes multiple data elements and

[0012] For a given second source vector operand, the dependency information is arranged to be

[0013] Each data element of the vector operand indicates a corresponding first source vector operand;

[0014] The instruction decoder circuit controls the processing circuit to execute in response to the multiple outer product instruction

[0015] Perform multiple calculations to implement multiple outer product operations, wherein the multiple outer product operations include

[0016] The first source vector operand performs an associated outer product operation to calculate the given first source vector

[0017] an outer product of the operand and a subset of the data elements of the at least one second source vector operand;

[0018] and

[0019] The processing circuit is configured to, depending on the correlation information,

[0020] Each data element of the two source vector operands is selected to be subjected to the associated outer product operation

[0021] The corresponding first source vector operand to use when .

[0022] In a second example of the present technology, a method is provided, the method comprising:

[0023] Perform vector operations using processing circuits;

[0024] decoding instructions from a set of instructions to control the processing circuitry to perform the vector operations specified by the instructions,

[0025] in:

[0026] The set of instructions includes specifying a plurality of first source vector operands, at least one second source

[0027] vector operand and dependency information associated with the at least one second source vector operand

[0028] Multiple outer product instructions, where each vector operand includes multiple data elements and

[0029] For a given second source vector operand, the dependency information is arranged to be

[0030] Each data element of the vector operand indicates a corresponding first source vector operand;

[0031] The method includes executing a plurality of calculations in response to the multiple outer product instruction to implement a plurality of

[0032] Outer product operations, wherein the plurality of outer product operations include performing on a given first source vector operand

[0033] The outer product operation is associated to calculate the given first source vector operand and the at least one

[0034] the outer product of a subset of the data elements of the two source vector operands; and

[0035] for each of the at least one second source vector operands depending on the dependency information

[0036] data element selects the corresponding first source to be used in performing the associated outer product operation

[0037] Vector operand.

[0038] In a third example of the present technology, there is provided a computer program comprising instructions that, when executed on a computer, control the computer to provide:

[0039] Processor logic to perform vector operations; and

[0040] an instruction decoder program logic that decodes target instructions from a set of target instructions to control the processing program logic to perform the vector operations specified by the target instructions,

[0041] in:

[0042] The set of instruction targets includes specifying a plurality of first source vector operands, at least one

[0043] two source vector operands and a correlation associated with the at least one second source vector operand

[0044] Multiple outer product instructions with vector information, where each vector operand includes multiple data elements and

[0045] and for a given second source vector operand, the dependency information is arranged to be

[0046] Each data element of the second source vector operand indicates a corresponding first source vector operand;

[0047] The instruction decoder program logic controls the processing in response to the multiple outer product instruction.

[0048] The sequential logic performs multiple calculations to implement multiple outer product operations, wherein the multiple outer product operations include

[0049] The associated outer product operation is performed on a given first source vector operand to compute the given first source vector operand.

[0050] A sub-array of a data element of a source vector operand and the at least one second source vector operand

[0051] the outer product of the sets; and

[0052] The processing program logic is configured to, depending on the correlation information,

[0053] Each data element of the second source vector operand is selected to be the one on which the associated outer product is to be performed

[0054] The corresponding first source vector operand to use in the operation.

[0055] In some examples, the computer program described above is stored on a computer-readable storage medium. The computer-readable storage medium may be transient or non-transient.

[0056] Further aspects, features and advantages of the present technology will be apparent from the following description of examples read in conjunction with the accompanying drawings, in which:

[0057] Figure 1 A block diagram showing a data processing apparatus;

[0058] Figure 2 shows examples of architectural registers that may be provided within a device including vector registers for storing vector operands and array registers for storing 2D arrays of data elements, including examples of physical implementations of the array registers;

[0059] Figure 3A and Figure 3B schematically illustrates how access may be performed to a square 2D array within an array storage device;

[0060] Figure 4A Show the outer product operation;

[0061] Figure 4B shows the matrix multiplication operation;

[0062] Figure 5 , Fig. 6A and Figure 6B Shows N:M structured sparsity in the matrix;

[0063] Figure 7 Shows an example of multiplying an activation matrix by a weight matrix using a multiply-accumulate (MAC) array;

[0064] Fig. 8A and Figure 8B shows how matrices with 2:4 and 4:8 structured sparsity can be compressed;

[0065] Fig. 9 A matrix multiplication showing that one of the input matrices to be multiplied is a compressed matrix;

[0066] Fig.10 is a block diagram of a data processing apparatus showing how processing circuitry is used to perform an outer product operation;

[0067] Fig.11A and Fig. 11B shows how the generated outer product results can be used to update associated storage elements within a 2D array of array storage devices;

[0068] Fig.12 schematically illustrates fields that may be provided within a multiple outer product instruction;

[0069] Figures 13 to 15 shows examples of multiple outer product instructions and examples of how dependency information may be represented;

[0070] Fig.16 and Fig.17 An example of a circuit for performing an outer product sum operation is shown;

[0071] Fig.18 is a flow chart showing the steps performed when decoding a multiple outer product instruction; and

[0072] Fig.19 A simulator implementation that can be used is shown.

[0073] Before discussing example implementations with reference to the accompanying drawings, the following description of example implementations and associated advantages is provided.

[0074] According to an example configuration, a device is provided, the device comprising processing circuitry and instruction decoder circuitry. The instruction decoder circuitry (also referred to herein as decoder circuitry or instruction decoder) is arranged to decode instructions from a set of instructions and control the processing circuitry to perform the vector operations specified by the instructions. For example, the instruction decoder circuitry may generate control signals in response to instructions in the set of instructions, and the control signals may control the processing circuitry to perform the vector operations.

[0075] A vector operation is an operation performed on a vector operand - for example, an operand comprising multiple data elements. A vector operation may include any operation involving at least one vector operand, such as a load or store operation to load / store a vector from / to a storage location (e.g., a memory or cache), or an arithmetic operation (e.g., addition, multiplication) performed on a vector operand. In the present technology, the processing circuitry is capable of performing vector operations including at least an outer product operation. For example, a vector operand may (but need not necessarily) be stored in a vector register, where a single vector register may store the entire vector operand, or where a single vector operand may be dispersed among multiple vector registers (and thus a single vector register may store elements from multiple vector operands).

[0076] The instruction decoder circuit is configured to decode for execution by the processing circuit the set of instructions comprising at least a "multiple outer product" (MOP) instruction. The MOP instruction is defined in an instruction set architecture (ISA) and identifies (directly or indirectly, e.g., by identifying a register storing a corresponding data value, e.g., identifying a corresponding field of the instruction) at least the following:

[0077] a plurality of (e.g., two or more) first source vector operands;

[0078] at least one (e.g., one or more) second source vector operands; and

[0079] • Dependency information associated with the at least one second source vector operand.

[0080] It should be noted that the terms "first" and "second" are only labels, and the first and second source vector operands do not necessarily need to be the first and second operands (groups) specified by the instruction, respectively. On the contrary, the first and second source vector operands (groups) can be identified by the MOP instruction in any order.

[0081] Each source vector operand (e.g., the second source vector operand and each first source vector operand of the plurality of first source vector operands) comprises a plurality of data elements, and for a given second source vector operand, the dependency information is arranged to indicate (e.g., identify, directly or indirectly) a corresponding first source vector operand for each data element in the given second source vector operand. The dependency information may also (optionally) identify a corresponding first source vector operand for a data element in any other second source vector operand identified by the instruction (e.g., if more than one second source vector operand is identified by a MOP instruction). Thus, the dependency information identifies any data element in a second source vector operand that is associated with an individual first source vector operand.

[0082] The instruction decoder circuit controls the processing circuit to perform multiple calculations to implement multiple outer product operations in response to the MOP instruction - for example, in response to a single MOP instruction decoded by the instruction decoder circuit, the processing circuit is configured to perform calculations that generate results equivalent to performing two or more outer product operations. The multiple outer product operations to be performed by the processing circuit in response to the decoded MOP instruction include outer product operations for each of at least a subset of the first source vector operands identified by the MOP instruction. More specifically, the processing circuit performs an associated outer product operation on a given first operand to calculate the outer product of a subset of the data elements of the first source vector operand and at least one second source vector operand (for example, this may be a true subset - for example, some but not all data elements). The subset of data elements of at least one second source vector operand to be used for each outer product operation is identified based on the correlation information - specifically, the corresponding first source vector operand to be used when performing the associated outer product operation is selected for each data element of the at least one second source vector operand based on the correlation information. (It should be noted that the outer product operation need not necessarily be performed on every first source vector operand, or on every subset of the at least one second source vector operand.)

[0083] In this manner, multiple outer product operations are performed in response to a single instruction by providing dependency information to associate a first source vector operand with data elements in at least one second source vector operand, even if only one second source vector operand is specified by the instruction. This is in contrast to a general (e.g., "single") outer product operation, where a single outer product operation is performed by multiplying each data element of one source vector by each data element of another source vector.

[0084] By defining an instruction (MOP instruction) that enables multiple outer product operations to be performed in response to a single instance of the instruction, the throughput of instructions executed by the processing circuit can be significantly improved. Therefore, the MOP instruction of the present technology allows for improved performance of the processing circuit. In fact, in one example implementation, these multiple outer product operations can be performed in parallel, even further improving throughput.

[0085] In addition to increasing throughput, providing dependency information to associate a subset of data elements in at least one second source vector operand with a respective first source vector operand allows the at least one second source vector operand to be defined such that multiple vectors are efficiently compressed into one source vector operand. Furthermore, using dependency information in this manner provides freedom in how multiple vectors are compressed into a single second source vector operand - for example, dependency information allows multiple vectors to be compressed such that a data element from any of the multiple vectors can occupy any data element position in the at least one second source vector operand - including consecutive and / or non-consecutive data elements.

[0086] Providing instructions that support performing multiple outer product operations based on source vector operands may be advantageous in several scenarios, where the source vector operands may include data elements from multiple vectors that have been compressed into a smaller number of source vector operands. In one specific non-limiting example, this may allow vectors that include one or more "zero" elements to be represented in a more compact form.

[0087] For example, because the outer product operation relies on multiplication - e.g., multiplying each data element in one vector by each data element in another vector - and because any number multiplied by zero is zero, it is feasible to effectively compress the input vector for the outer product operation by removing some or all of the zero elements. Because other corresponding elements of the result vector (e.g., which will have zero values) can be removed from the resulting outer product (leading to incorrect results), it can be assumed that vector elements equal to zero must be reinserted into the input vector before performing the outer product operation. However, the method of the present technology - wherein dependency information is used to identify which first source vector operands are associated with which elements in one or more second source vector operands - can allow input vector elements equal to zero to be removed from one or more source vector operands without changing the result of the outer product operation.

[0088] Thus, the present technique may allow such vectors to be represented in a manner such that they may take up less space in any data storage structure storing the compressed operands (e.g., by removing data elements from the vectors that are equal to zero, such that the total number of data elements to be recorded—and thus the space required in the storage structure to store these structures—is reduced). Reducing the amount of space taken up in the storage device provides a number of advantages, including reducing energy consumption if the data is stored in volatile storage, and reducing latency and power consumption when loading, manipulating, and storing vectors. Further, by reducing the number of data elements in one or more of the source vectors, the total number of multiplications to be performed is reduced, resulting in a reduction in the overall latency associated with the operation and increased data throughput.

[0089] Therefore, providing support for MOP instructions in an instruction set architecture (ISA) may provide significant improvements in performance in a data processing device.

[0090] It will be appreciated that it is not necessary that the at least one second source vector operand represents a compressed form of the plurality of vectors or that the plurality of vectors include zeros in one or more data elements.This is just one example of how a MOP instruction may be used to improve performance and reduce power consumption.

[0091] It should be understood that, although the result generated by executing the MOP instruction is equivalent to calculating multiple outer product operations, when executing the MOP instruction, the order in which the individual data elements of each vector operand are used / consumed by the processing circuit depends on the specific implementation. For example, the outer product operation does not necessarily need to be performed one at a time (e.g., by considering each first source vector operand in turn). Instead, in at least some example implementations of the present technology, each data element in at least one second source vector operand may be considered in turn (e.g., selecting an appropriate first source vector operand for a given data element in at least one second source vector operand). In other specific implementations, all data elements in at least one second source vector may be considered in parallel, wherein the corresponding first source vector operand is selected in parallel for each element of the second source operand. These methods-wherein the first source vector operand is selected for each data element in at least one second source vector-are more efficient because they may only need to generate each second source vector operand of a single round, rather than having to scan each second source vector operand multiple times (because if the outer product operation is performed one at a time, there may be such a situation). However, it will be understood that other specific implementations may also be used to generate equivalent results.

[0092] In some examples, the apparatus includes array storage circuitry including storage elements storing data elements, the array storage circuitry being arranged to store at least one two-dimensional (2D) array of data elements accessible by the processing circuitry when performing the vector operations. In such examples, the multiple outer product instruction may specify a given two-dimensional array of data elements within the array storage device forming a destination operand, and the processing circuitry may be configured to perform the associated outer product operation of the given first source vector operand by multiplying each data element of the given first source vector operand by each data element in a subset of the data elements of the at least one second source vector operand to generate a plurality of outer product result elements, and using each outer product result element to update a value held in an associated storage element within the given two-dimensional array of storage elements.

[0093] An array storage circuit (which may, for example, include a set of array registers) may provide a useful mechanism for performing certain types of operations (e.g., outer product operations). Specifically, a matrix of data elements resulting from performing an outer product operation may be stored within associated data elements of a 2D array represented in the array storage circuit.

[0094] However, the inventors of the present technology have recognized in some example use cases that, as mentioned above, when performing an outer product operation using two source vectors, there may be situations where some elements in one or both of the source vectors are zero. This can result in inefficient use of the storage elements of the 2D array, because a significant number of these storage elements are then unusable or will only be used to store zero values, and can also result in inefficient use of resources of hardware components that form processing circuits (which may be able to perform calculations to produce results for each of the storage elements). However, according to the techniques described herein, a single instruction (i.e., the MOP instruction discussed above) can be defined that enables multiple outer product operations to be performed through the use of dependency information associated with a subset of data elements in at least one second source vector operand and a corresponding first source vector operand, wherein the result of each outer product operation is stored in an associated storage element of a two-dimensional (2D) array. This can significantly improve throughput (as mentioned above) while also making more efficient use of available storage elements within the array storage device.

[0095] It should be noted that there does not necessarily need to be a 1:1 correlation between each outer product result and the associated storage element in the 2D array. For example, in some cases, multiple outer product results may be used to determine the value held in a given storage element, where additional operations (e.g., accumulation (addition) operations) are performed to combine the multiple outer product results.

[0096] In some examples, the processing circuit includes a multiplication circuit that generates each outer product result when performing multiple outer product operations, and a multiplexer circuit associated with the multiplication circuit, which selects selected data elements of the multiple first source vector operands and selected data elements of the at least one second source vector operand to be multiplied under control of the correlation information to generate associated outer product result elements.

[0097] For example, the multiplication circuit may include a multiplier circuit - for example, this may be a simple multiplier for multiplying two values ​​together, or a multiply-accumulate (MAC) circuit that multiplies two values ​​together and adds the result to an accumulated value - that is associated with each storage element in the 2D array of storage elements. In the case of a MAC circuit, the same MAC circuit may multiply selected data elements together to generate an associated outer product result, and add the associated outer product result to the value (which may be zero) currently stored in the associated storage element of the array storage circuit. However, there may not necessarily be a 1:1 correlation between each multiplier circuit or MAC circuit and each storage element in the 2D array - for example, there may be one multiplier / MAC per storage element (in which case it is feasible for the outer product results stored in each storage element to be calculated in parallel), or there may be less than one multiplier / MAC per storage element (in which case each multiplier / MAC is used to calculate the outer product results of multiple storage elements in series).

[0098] However, a number of multiplier circuits are provided, and multiplexer circuits associated with the multiplication circuits are provided to allow selection of a data element for each multiplication operation, thereby facilitating implementation of the MOP instruction of the present technique.

[0099] In some examples, the dependency information includes at least one set of indices, and the at least one set of indices includes an index associating each data element of the given second source vector operand with the corresponding first source vector operand.

[0100] For example, an index may be provided for each data element of a given second source vector operand. Alternatively, each index may be associated with a plurality of data elements in a given second source vector operand.

[0101] In some examples, the dependency information is provided by at least one dependency source operand specified by the multiple outer product instruction, and the given dependency source operand includes a corresponding element for each data element of the given second source vector operand, the corresponding element including the index associating the data element of the given second source vector operand with the corresponding first source vector operand.

[0102] Providing an index for each data element of at least one second source vector in a set of indices forming the dependency information provides a simple mechanism for determining which index corresponds to which data element of the second source vector operand, which may in turn simplify processing and therefore provide an improvement in performance.

[0103] As a specific example, the set of indices may be stored as a multi-bit scalable value (e.g., in a scalar register or a predicate register), where each bit is associated with a different vector element in one of the one or more vector registers. In another example, the set of indices may be stored as a vector (e.g., in a vector register), where each data element in the vector holds one or more indices for a corresponding data element in at least one second source vector operand. It will be appreciated that these are only some examples of how the set of indices may be stored, and other examples are possible.

[0104] In some examples, the dependency information includes a set of indices for each second source vector operand.

[0105] For example, if the MOP instruction specifies multiple second source vector operands, it may also identify a separate set of indices for each second source vector operand.Thus, the number of index sets may be at least as large as the number of second source vector operands.

[0106] In some examples, the at least one set of indices includes a set of indices that provide the dependency information for a plurality of second source vector operands.

[0107] Therefore, instead of indicating a set of indices for each second source vector operand, it is also feasible that each set of indices identified by the MOP instruction provides dependency information for more than one second source operand. Therefore, in this example, the number of index groups may be less than the number of second source vectors. Advantageously, in a specific implementation where the size / width of the registers used is fixed and relatively large, this allows reducing the amount of storage space occupied by the dependency information.

[0108] In some examples, the dependency information is provided by at least one dependency source operand specified by the multiple outer product instruction, and each element of a given dependency source operand includes multiple indices including an index of a corresponding data element for each second source vector operand of the multiple second source vector operands.

[0109] Thus, in these examples, each element of a given dependency source operand provides multiple indices, including at least one index for each second source vector. In a specific example, the top bits in a given element may provide indices for data elements of a first source vector, while the bottom bits provide indices for data elements of a second source vector. This method exploits the fact that the indexes may each be formed with fewer bits than the space present in each element of the dependency source operand. Thus, multiple indices may be stored in each element of the dependency source operand, so that less storage space is required to store dependency information.

[0110] In some examples, each outer product operation performed by the processing circuit is based on a different subset of the data elements of the at least one second source vector operand.

[0111] For example, each subset of the data elements of at least one second source vector operand may include data elements different from each other subset (e.g., each outer product operation may be based on different data elements of the second source vector operand). In these examples, the subsets of the data elements of any two of the at least one second source vector operand for the outer product operation differ by at least one data element. It should be noted that "data element" generally refers to a specific data element position in a vector operand, rather than a specific numerical value maintained in that position. Therefore, the numerical values ​​of the data elements in different subsets may not necessarily differ, because it is possible for multiple data elements in a given source vector operand to maintain data with the same numerical value. Further, it should be noted that a subset may differ by more than one data element--as a specific example, each data element in at least one second source vector operand may be part of a subset of at most one data element.

[0112] In this way, multiple vectors may be represented in each second source vector operand, with each subset representing a different vector, and multiple outer product operations may be performed based on the multiple vectors.

[0113] In some examples, the multiple outer product instructions include outer product and instructions such that multiple outer product results have the same associated storage element within a given two-dimensional array of storage elements, and the processing circuit is configured to combine the multiple outer product results to update the value held in the associated storage element.

[0114] There are many possible applications of the MOP instruction, but in this example, the MOP instruction is an outer product sum instruction (which may also be referred to as a "multiple outer product sum instruction (SMOP)"), the execution of which involves accumulating multiple outer product results into a single storage element of a 2D array. This variation of the MOP instruction can be advantageous because it allows two matrices to be multiplied together in response to a single instruction.

[0115] In some examples, the apparatus includes a set of vector registers accessible by the processing circuitry, wherein each vector register is arranged to store a vector comprising a plurality of data elements, and the plurality of first source vector operands and the at least one second source vector operand comprise vectors contained within a vector register of the set of vector registers.

[0116] A vector is a one-dimensional (1D) array that includes multiple (more than one) data elements. Mathematically, a one-dimensional array can represent a data element vector as a single column or a single row of data elements; in a data processing system (such as the device in this example), the data elements of a vector are stored in a single vector register. Vector operands are contrasted with scalar operands because each scalar operand includes a single data item (e.g., each data element in a vector can be a scalar operand). Storing and manipulating data elements in the form of vectors (performing operations on them) is advantageous because it allows operations on multiple data elements to be performed in parallel (e.g., using single instruction multiple data (SIMD) processing). This can significantly improve performance, especially when performing operations (e.g., matrix multiplication) on large data arrays, where the ability to operate on multiple data elements in response to a single instruction can significantly improve performance by increasing throughput.

[0117] In some examples, the apparatus comprises a set of predicate registers accessible by the processing circuitry, wherein each predicate register is arranged to store predicate information comprising a plurality of elements, each element providing a predicate value, and the dependency information is stored within at least one predicate register of the set of predicate registers.

[0118] A data processing system, such as the apparatus of this example, may be provided with a set of predicate registers to store predicates (e.g., predicate information). For example, each predicate may be a mask of true / false (e.g., 1 / 0) values ​​used in vector processing—for example, a predicate may indicate which data elements in a vector should or should not be operated on. This example uses the predicate registers for another purpose: storing dependency information. If each subset of data elements is already provided in a separate vector register (e.g., to assert zero values), this approach is advantageous because it uses circuitry (predicate registers) that may already be used to perform outer product operations. Thus, this approach provides dependency information without taking up additional storage space (e.g., taking up additional architectural registers, or additional space in memory, cache, or some other storage structure).

[0119] In some examples, these data elements in each second source vector operand represent data values ​​from multiple rows and multiple columns of a source matrix.

[0120] As mentioned above, there are several scenarios in which the MOP instructions of the present technology can be useful. However, in a particular example, the MOP instructions can be used in matrix multiplication. In this example, each subset of data elements in at least one second source vector operand can represent a different row / column of the source matrix, and thus the present technology provides a more compact representation of the matrix.

[0121] In some examples, each element of the given correlation source operand is arranged such that the source matrix can be reconstructed from the at least one second source vector operand.

[0122] For example, each element of the given correlation source operand can indicate which row / column of the original matrix one or more corresponding elements of the second source vector operand are from. As a particular example, each element can be a bit map of 1s and 0s indicating which rows / columns of the original matrix hold non-zero values (e.g., if the original matrix has two rows and the first bit map is (1,0), this can indicate that the first data element in the given second source vector operand represents the first data element of the first row of the matrix). The advantage of this method is that it captures information related to the original matrix format, so simply decompressing the second source vector operand back to the original matrix.

[0123] In some examples, each data element in each second source vector operand is associated with a corresponding first dimension in the source matrix, where the corresponding first dimension includes a corresponding row or column in the source matrix, and each data element in each second source vector operand provides a data value selected from these data values in the corresponding first dimension in the source matrix.

[0124] For example, the first data element in a given second source vector operand can be the first data element in a given row / column of the source matrix, and the correlation information can indicate which row / column the data element is from. Thus, in this way, a matrix composed of multiple rows / columns (e.g., multiple vectors) can be compressed into a smaller number of vector operands.

[0125] In some examples, the source matrix includes a matrix having N:M structured sparsity, where each group of M defined data values in the source matrix includes at most N non-zero data values.

[0126] In a matrix with structured sparsity, a particular (e.g., defined) group of M data elements is restricted to having at most N non-zero values. This can be the result of "pruning" an ANN, which can cause some data elements in the matrix to be replaced with zeros. The values of N and M are implementation-dependent - for example, ratios of 2:4 and 4:8 are common; more generally, a ratio of M = 2N can be used, or any other ratio can be used as long as N is less than M (N < M).

[0127] When the matrix has structured sparsity, it is feasible to compress the matrix to occupy less storage space by removing zero values; in this way, the amount of storage space occupied by the matrix is ​​reduced (which leads to improved performance and reduced power consumption, especially when loading / storing data elements). It may be considered that the disadvantage of this method is the additional processing of decompressing the second source vector back to the source matrix before performing any data processing operation (e.g., outer product operation) on the data elements. However, the inventors recognize that these compressed matrices can be combined with the correlation information indicating which rows or columns of the source matrix the data elements come from and used as the operands of the MOP instruction. In this way, not only the amount of storage space occupied by the matrix is ​​reduced, because multiple outer product operations can be performed in response to a single MOP instruction, but also the delay involved in performing matrix multiplication is improved (leading to further improvement in performance).

[0128] It should be noted that each group of M data elements in the source matrix is ​​a defined / specific group - not any group of M elements in the source matrix can be used. For example, a given group of M elements may be taken and aligned in one dimension (e.g., row or column) (e.g., if M=4, the first four elements in a given row may form a group).

[0129] In some examples, the source matrix includes a weight matrix or an activation matrix used in an artificial neural network (ANN) execution.

[0130] As mentioned above, the nodes of ANN are generally represented as weight matrices, and these matrices can be large. Similarly, the data input to the nodes of ANN generally takes the form of activation matrices. Because these matrices can be large, and the number of nodes in ANN is generally very large, a considerable amount of data is generally required to represent ANN. Therefore, it is useful to prune the neural network by clearing some data elements in the weight matrix (e.g., setting them to zero). Specifically, this can be done in a structured manner according to the defined N:M sparsity. This allows the matrix representing the ANN to be compressed into a smaller number of vector operands, providing all the advantages discussed above (e.g., more efficient use of data storage devices, better performance, and lower power consumption). It will be understood that when executing ANN, assuming that a large amount of data is involved, these advantages are particularly significant.

[0131] The techniques discussed above may be implemented in a hardware device having circuit hardware that implements the processing circuitry, instruction decoder circuitry, and other device features described above to support multiple outer product (MOP) instructions as part of a native instruction set architecture supported by the decode circuitry and processing circuitry.

[0132] However, in another example, the same technology may be implemented in a computer program (e.g., an architecture simulator or model) that can provide an instruction execution environment for controlling a host data processing device to provide for the execution of instructions from a target code. The computer program may include instruction decoding program logic for decoding instructions of the target code to control the host data processing device to perform data processing including performing vector operations. Therefore, the instruction decoding program logic emulates the functionality of the instruction decoder (instruction decoder circuit) of the hardware device discussed above. The program may also include processing program logic to perform data processing when executed on the host data processing device (and thus emulate the functionality of the above-mentioned processing circuit). Furthermore, the program may include register maintenance program logic that maintains a data structure (in the memory or architecture registers of the host device) representing (simulating) the architecture registers of the instruction set architecture simulated by the program - for example, these architecture registers may include any or all vector registers, scalar registers, array registers, and predicate registers. The instruction decoding program logic includes support for MOP instructions, which have the same functionality as described above for the hardware example. Therefore, this type of simulator computer program can present an instruction execution environment similar to the environment provided by the actual hardware device that can directly execute the target instruction set to the target code executed on the simulator computer program, even if there may not be any actual hardware providing these features on the host computer that is executing the simulator program. Therefore, by providing the simulation of the above-mentioned device, the simulator computer program is provided in all the advantages related to the device discussed above. In addition, simulation can be useful for executing the code written for the architecture on a host platform that does not actually support an instruction set architecture. Furthermore, when software development is performed in parallel with the development of the hardware device that supports the new architecture, the simulator can be useful during the software of the instruction set architecture of the new version. This can allow software to be developed and tested on the simulator so that software development can start before the hardware device that supports the new architecture is available.

[0133] The simulator program may be stored on a storage medium, and the storage medium may be transient or non-transitory.

[0134] Certain embodiments will now be described with reference to the accompanying drawings.

[0135] Figure 1A data processing system 10 is schematically shown which includes a processor 20 coupled to a memory 30 which stores data values ​​32 and program instructions 34. The processor 20 includes an instruction fetch unit 40 for fetching program instructions 34 from the memory 30 and supplying these fetched program instructions to an instruction decoder circuit 50. The decoder circuit 50 decodes these fetched program instructions and generates control signals to control the processing circuit 60 to perform processing calculations on the data values ​​held in the storage elements of the register storage device 65 as specified by the decoded vector instructions. Figure 1 As shown, register storage device 65 may be formed of a plurality of different blocks. For example, a scalar register file 70 may be provided, which includes a plurality of scalar registers that can be specified by instructions, and similarly, a vector register file 80 may be provided, which includes a plurality of vector registers that can be specified by instructions.

[0136] like Figure 1 As shown, the processor 20 can access the array storage device 90. Figure 1 In the example shown, array memory device 90 is provided as part of processor 20, but this is not required. In various examples, the array memory device may be implemented as any one or more of the following: architecturally addressable registers; non-architecturally addressable registers; high-speed scratchpad memory; and cache.

[0137] In an example implementation, the processing circuit 60 may include both vector processing circuitry and scalar processing circuitry. The general difference between scalar processing and vector processing is as follows. Vector processing may involve applying a single vector processing instruction to a data element of a data vector having a plurality of data elements at corresponding positions in the data vector. The processing circuit may also perform vector processing to perform operations on a plurality of vectors in a two-dimensional array of data elements (which may also be referred to as a subarray) stored in the array storage device 90. Scalar processing effectively operates on a single data element, but not on a data vector. Vector processing may be used in instances where processing operations are performed on many different instances of data to be processed. In a vector processing arrangement, a single instruction may be applied to a plurality of data elements (of a data vector) simultaneously. Compared to scalar processing, this may improve the efficiency and throughput of data processing.

[0138] The processor 20 may be arranged to process two-dimensional array data elements stored in the array storage device 90. In at least some examples, the two-dimensional array may be accessed in multiple directions as one-dimensional vector data elements. In an example implementation, the array storage device 90 may be arranged to store one or more two-dimensional array data elements, and each two-dimensional array data element may form a square array portion of a larger or even higher dimensional array data element in memory.

[0139] The register store 65 also contains a predicate register file 75. This stores predicate information (eg masks) for use in data processing operations (eg to mask off certain data elements of a vector so that they are excluded from a particular processing operation).

[0140] Figure 2 An example of an architectural register 65 of a processor 20 that may be provided in an example implementation is shown. The architectural registers (as defined in an instruction set architecture (ISA)) may include a set of scalar registers (not shown) and a set of predicate registers 100 for storing predicate information. The predicate registers may also store dependency information for executing a multiple outer product (MOP) instruction, as will be discussed below. For example, there may be a certain number of predicate registers 100, in this example, 16 registers P0 to P15 are provided, for example. The predicate registers may have a fixed size, although some bits in each element may not necessarily be used depending on the data type of the elements stored in the predicate register.

[0141] Furthermore, the architectural registers selectable by program instructions in the ISA supported by decoder 50 may include a number of vector registers 105 (labeled Z0 to Z31 in this example). Figure 2 The number of predicates / vector registers shown is not necessary, and other examples may provide different numbers of registers that may be specified by program instructions. Each vector register may store a vector operand including a variable number of data elements, wherein each data element may represent an independent data value. In response to a vector processing (SIMD) instruction, the processing circuit may perform vector processing on the vector operands stored in the register to generate a result. For example, vector processing may include channel-by-channel calculations, wherein the corresponding calculations are performed on each element channel in one or more operand vectors to generate the corresponding results of the elements of the result vector. When performing vector or SIMD processing, each vector register may have a certain vector length VL, wherein the vector length refers to the number of bits in a given vector register. The vector length VL used in the vector processing mode may be fixed or variable for a given hardware implementation. The ISA supported by the processor 20 may support variable vector lengths, so that different processors may be selected to implement vector registers of different sizes, but the ISA is vector length-independent, so that instructions are designed so that the code may function correctly regardless of the specific vector length implemented on the given CPU that executes the program.

[0142] Vector registers Z0 to Z31 may also function as operand registers for storing vector operands, which provide input to processing and accumulation operations performed by processing circuitry 60 on two-dimensional array data elements stored in array storage device 90 .

[0143] like Figure 2As shown, the architectural registers also include a certain number NA of array registers 110 forming the previously mentioned array storage devices 90 (ZA0 to ZA(NA-1)). Each array register can be considered as a group of register storage devices for storing a single 2D array data element (e.g., the result of a process and accumulate operation). However, process and accumulate operations may not be the only operations that can use array registers. Array registers can also be used to store square arrays while performing a row / column direction transposition of the array structure in memory. When a program instruction refers to one of the array registers 110, it is referenced as a single entity using the array identifier ZAi, but some types of instructions (e.g., data transfer instructions) can also select a sub-portion of the array (e.g., a horizontal / vertical group of elements) by defining an index value that selects a portion of the array.

[0144] In practice, the physical implementation of the register storage device corresponding to the array register may include a certain number NR of array vector registers ZAR0 to ZAR(NR-1), such as Figure 2 As shown. The array vector registers ZAR forming the array register storage device 110 may be a group of registers different from the vector registers Z0 to Z31 used for SIMD processing and vector input to the array processing. Each of the array vector registers ZAR may have a vector length VL, so each array vector register ZAR may store a 1D vector of length VL, which may be logically divided into a variable number of data elements. For example, if VL is 512 bits, this may be a group of 64 8-bit elements, 32 16-bit elements, 16 32-bit elements, 8 64-bit elements, or 4 128-bit elements. It should be understood that not all of these options need to be supported in a given specific implementation. By supporting variable element sizes, this provides the flexibility of handling calculations involving data structures of different precisions. To represent a 2D data array, the group of array vector registers ZAR0 to ZAR(NR-1) may be logically considered to be a single entity assigned a given one of the array register identifiers ZA0 to ZA(NA-1) such that the 2D array is formed with elements extending within a single vector register corresponding to one dimension of the array and elements in another dimension of the array serialized across multiple vector registers.

[0145] As discussed above, the processing circuitry 60 is arranged to access the scalar registers 70, the vector registers 80, and / or the array storage device 90 under the control of instructions decoded by the decoder circuitry 50. Reference will now be made to Figure 3A Further details of this latter arrangement are described, which merely provides one illustrative example of how an array storage device may be accessed, particularly considering accessing a square 2D array within the array storage device.

[0146] In the illustrated example, the square 2D array within the array storage device 90 is arranged as an array 205 of n×n storage elements / locations 200, where n is an integer greater than 1. In this example, n is 16, which implies that the granularity of access to the storage locations 200 is 1 / 16 of the total storage of the 2D array in either the horizontal or vertical array direction.

[0147] From the perspective of the processing circuitry, the array of n×n locations is accessible as n linear (one-dimensional) vectors in a first direction (e.g., the horizontal direction as shown) and n linear vectors in a second array direction (e.g., the vertical direction as shown). Thus, from the perspective of the processing circuitry 60, the n×n storage locations can be accessed as 2n linear vectors each having n data elements.

[0148] The array of storage locations 200 is accessible by access circuits 210 , 220 , column select circuits 230 , and row select circuits 240 under the control of control circuitry 250 in communication with at least processing circuitry 60 and optionally decoder circuitry 50 .

[0149] Reference Figure 3B In the case of an example square 2D array designated as "ZA1" (note that, as discussed below, more than one such 2D array may be provided within array storage device 90, e.g., ZA0, ZA1, ZA2, and so on), the n linear vectors in a first direction (the horizontal or "H" direction as shown) each have 16 data elements 0...F (expressed in hexadecimal notation) and may be referenced in this example as ZA1H0...ZA1H15. The ... in a first direction (expressed in hexadecimal notation as shown) in an array storage device 90. The n linear vectors in a first direction (the horizontal or "H" direction as shown) each have 16 data elements 0...F (expressed in hexadecimal notation) in a first direction (expressed in hexadecimal notation as shown) in a first direction (expressed in hexadecimal notation as shown) in a first direction (expressed in hexadecimal notation as shown) in a first direction (expressed in hexadecimal notation as shown) Figure 3B 15. It should be noted that, for example, data element 260 is referenced as item F of ZA1H0, rather than item 0 of ZA1V15. It should be noted that the use of "H" and "V" does not indicate any spatial or physical layout requirements regarding the storage of the data elements making up array storage device 90, nor does it have any relevance as to whether the 2D arrays within the array storage device store row data or column data in any example application.

[0150] Figure 4A The outer product operation is shown. The outer product operation takes as input two vectors A and B which may be stored in a vector register file as discussed above. The result of the outer product operation is a matrix (e.g., a 2D array) As shown, each data element in the output matrix is ​​determined by multiplying together the corresponding data elements in each of the input vectors—for example, the upper-left element in the result matrix is ​​calculated by multiplying together element a0 of vector A and element b0 of vector B. In populating the result matrix, each data element in vector A is multiplied by each data element in vector B; thus, the result of computing the outer product of a vector of n elements with a vector of m elements is an n×m matrix.

[0151] Figure 4B shows the matrix multiplication operation. Specifically, Figure 4B An operation is shown involving multiplying two matrices C and D (which may be stored, for example, in vector registers (e.g., with one row or column held in each register) or in array storage circuits) together to generate a matrix CD. As can be seen from the figure, the result of multiplying two n×n matrices together is an n×n matrix (more generally, an n×k matrix multiplied by a k×m matrix will result in an n×m matrix).

[0152] There are several ways to calculate the elements of the output matrix CD, but a technique generally adopted by processors is to perform multiple outer product operations and accumulate (add) the results together. For example, to perform the operation shown in Figure 4B For matrix multiplication in , the processing circuitry may first compute the outer product of the leftmost column i of matrix A and the top row w of matrix B to generate 16 outer product results, and use these outer product results to fill a 4×4 array in the array storage device. The processing circuitry may then compute the outer product of the next column j of matrix A and the next row x of D to generate another 16 outer product results that are added to the outer product results already stored in the array. This process may then be repeated for the last two pairs of vectors ( and ) to generate the final result CD.

[0153] Thus, it can be seen that matrix multiplication can be performed by performing multiple outer product operations and accumulating the results - for example, this can be performed by executing one or more multiple outer product instructions. It should be noted that the order in which these vector pairs are multiplied together is not limited to the above order - the outer products can be calculated in any order. In addition, the outer product operations do not need to be performed sequentially.

[0154] If available from Figure 4B As seen in the examples in , matrix multiplication involves performing a considerable number of operations - for example, multiple outer product operations may be performed where each outer product operation involves multiple multiplication operations. Therefore, matrix multiplication can be a particularly time consuming and energy consuming process. This is particularly true in the case of performing, for example, an artificial neural network (ANN), which may include performing a considerable number of matrix multiplications. Therefore, there is interest in improving the performance of matrix multiplication operations.

[0155] Figure 5Displays a matrix with N:M structured sparsity. Figure 5 In the matrix shown, shaded elements represent non-zero elements, while blank (unshaded) elements represent zero elements. Figure 5 In the particular example shown, a 10x11 matrix with 3:5 sparsity is shown. More specifically, each column of ten data elements is considered to include two groups of M=5 consecutive data elements of which at most N=3 are non-zero (eg, shaded data elements).

[0156] As explained above, structured sparsity can be introduced into a matrix (e.g., by clearing / setting some of its elements to zero) to reduce the number of data elements in the matrix that are involved in at least some matrix operations to be performed on the matrix. In addition, the inventors have recognized (as discussed in more detail below) that introducing structured sparsity into a matrix can allow the matrix to be compressed into a smaller number of vector operands, which reduces the amount of space occupied by the matrix.

[0157] Fig. 6A and Figure 6B Show further examples of sparse matrices used for information—e.g. Figure 5 , shaded elements represent non-zero elements, while blank (unshaded) elements represent zero elements. Specifically, Fig. 6A shows a matrix (N=2, M=4) with 2:4 structured sparsity, and Figure 6B A matrix (N=4, M=8) with 4:8 structured sparsity is shown. It should be noted that although Fig. 6A and Figure 6B Both show that each column of an 8×8 matrix with at most four data elements non-zero will be shown in Fig. 6A The matrices in are further constrained so that each group of 4 data elements in a column (e.g., the first four elements in a column or the last four elements in a column) has at most two non-zero elements. This can be seen by comparing, for example, the leftmost column of each matrix - for example, in Figure 6B In a matrix with , the first four data elements in the leftmost column are all non-zero; this is not allowed in a matrix with 2:4 structured sparsity, where at most 2 of these four elements can be non-zero (e.g. Fig. 6A ) as shown in the leftmost column of the matrix in .

[0158] Data processing operations can be performed on sparse matrices just as they can be performed on other matrices. For example, Figure 7 Shows how an array of multiply-accumulate (MAC) units can be used to compute the result of an 8×4 activation matrix multiplied by a 4×8 weight matrix (with 2:4 sparsity) (e.g., the weights at a given node of an ANN). For example, predicate information can be used to mask out zero values.

[0159] However, as explained above, the inventors have recognized that there are several advantages to compressing sparse matrices into a smaller number of vector operations. An example of this is shown in Fig. 8A , where a 4×8 matrix with 2:4 sparsity is compressed into two vector operands. Specifically, FIG8 shows how a matrix that originally occupies four vector registers (Z8, Z9, Z10, Z11) can be compressed into two vector registers (Z4, Z5) each storing data elements from one or more rows of the source matrix. This frees up two vector registers in the vector register file (and / or an equivalent amount of space in memory and / or another data storage device).

[0160] A set of indices Id is maintained for each of the compressed source vector operands, which can be used to reconstruct the source matrix. For example, the index may indicate which row of the source matrix each element in Z4 or Z5 comes from.

[0161] Another example of how a sparse matrix can be compressed into a smaller number of vector operations is shown in Figure 8B , where an 8×8 sparse matrix (with 4:8 structured sparsity) is compressed into four vector operands.

[0162] Once a sparse matrix has been compressed, the inventors have recognized that multiple outer product (MOP) operations can be performed on the resulting vector operands without first decompressing them to form the original matrix. This is shown in Fig. 9 , where an 8×4 activation matrix is ​​multiplied by a 2×8 compressed weight matrix (e.g., this may be a MAC array) using multiplication circuit 255. For example, this may be performed as two MOP operations, one based on Z4 and one based on Z5, or as a sum of outer products (SMOP) operation.

[0163] Four outer product operations are performed to multiply the matrices together. These operations involve computing the outer products of the columns of the activation matrix (e.g., vectors Z0, Z1, Z2, Z3) with the data elements in Z4 and Z5 that have the same shading. For example, this means that the first, second, fourth, and sixth data elements from the left in Z5 are multiplied by the corresponding elements in Z3; the third, fifth, and seventh elements from the left in Z5 and the first element from the left in Z4 are multiplied by the corresponding elements in Z2, and so on. An index may be used to identify which vector in the first vector operand (Z0:Z3) group each data element in the second vector operand (Z4:Z5) group should be multiplied by.

[0164] Shown in Fig. 9The multiplication circuit 255 in includes a number of multiplexers 270, each of which provides input to a corresponding multiplier circuit (not shown). In this particular example, an 8×8 array of multiplexers 270 is provided, including a multiplexer (and corresponding multiplier) for each multiplication to be performed. Each multiplexer takes as input the data elements from the corresponding position in each of the activation vectors Z0 to Z3, and selects one of these data elements for multiplication with the data element at the corresponding position in Z4 or Z5 based on the dependency information. However, it will be understood that in other examples, there may be fewer multiplexers and multipliers, with each multiplexer / multiplier pair being used to perform a number of multiplication operations to be performed.

[0165] Thus, as discussed above, an apparatus of the present technology is configured to support execution of a multiple outer product (MOP) instruction that identifies as input a plurality of first vector operands (e.g., vector registers Z0:Z3 in this case), at least one second source vector operand (e.g., Z4 and / or Z5), and dependency information (e.g., one or both of the index groups).

[0166] It should be noted that while the figures show examples where the weight matrix is ​​a compressed sparse matrix, it is equally feasible for the activation matrix to be a compressed sparse matrix instead of or in addition to the weight matrix. More generally (because multiplying the activation and weight matrices is just one example use case for the present technique), it does not matter which of the first vector operand (group) or the second vector operand (group) specified by the MOP instruction represents a compressed sparse matrix - it can be either or both.

[0167] Fig.10 8 is a block diagram of a device specifically implemented according to an example, which shows how to use processing circuits to perform outer product operations. A vector register file 80 provides a plurality of vector registers that can be used to store vectors of data elements. As discussed earlier, the MOP instruction can be arranged to identify a plurality of first source vector operands 300 and at least one second source vector operand 320. At least one second source vector operand 320 (and also optionally a plurality of first source vector operands 300) includes a plurality of subsets of data elements, wherein each subset is used for different outer product operations. It should be noted that the terms "first" and "second" used herein in reference to (groups of) two source vector operands are used purely as marks to distinguish between them, and do not imply any particular ordering of how these operands are specified by instructions. Therefore, any one of the source operand fields of the instruction can be used to specify the first source vector operand referred to above, and another of the source operand fields will then be used to specify the second source vector operand referred to above.

[0168] Further, although according to the techniques described herein, at least one of the two source vector operands will include multiple subsets of data elements for use in different outer product operations, and the other source vector operand may not, in some example implementations there may also be situations where both source vector operands include multiple subsets of data elements. Similarly, in addition to specifying multiple first source vector operands, more than one second source vector operand 320 may be specified. In addition, the number of first source vector operands and the number of second source vector operands specified by the MOP are not limited to one vector operand or two vector operands, but in fact more than two first / second vector operands may be specified (e.g., four vector operands or eight vector operands, etc.).

[0169] The MOP instruction also specifies dependency information that identifies which data elements in the at least one second source vector operand 320 are to be multiplied by which vector operands in the plurality of first vector operands. In this example, the dependency information is stored in predicate registers 325 in the predicate register file 75, and thus the MOP instruction also identifies one or more predicate registers 325.

[0170] The processing circuitry 60 is controlled in accordance with control signals received from the decoder circuitry (decoder) 50, and when the decoder circuitry 50 decodes the MOP instruction mentioned earlier, it sends control signals to the processing circuitry to control the processing circuitry to perform a plurality of calculations to implement a plurality of outer product operations, wherein the plurality of outer product operations include performing associated outer product operations on a given first source vector operand to calculate the outer product of the given first source vector operand and a subset of the data elements of at least one second source vector operand. As part of this process, those control signals will control the selection circuitry 340 provided by the processing circuitry 60 to select the appropriate data element to be processed by each outer product operation. Each outer product operation includes multiplying each data element of the subset of associated data elements in at least one second source vector operand by each data element in the associated first source vector operand, and then using each outer product result element to update the value in the associated storage element within the given two-dimensional array 380 of storage elements held within the array storage device 90.

[0171] The selection circuit 340 may be organized in a variety of ways, but in one example implementation includes a multiplexer circuit provided for each multiplier of a plurality of multipliers used to generate outer product results from two input data elements, the multiplexer circuit being used to select the appropriate two input data elements for each multiplier.

[0172] Next, the selected input data elements are forwarded to the multiplication circuit 350, which, as mentioned above, may include a multiplier circuit for each outer product result to be generated in the example implementation. Each outer product result element is generated by multiplying two input data elements provided to the corresponding multiplier within the multiplication circuit 350. The outer product result elements may be provided directly to the array update circuit 370, which is used to update the storage elements within the 2D array 380, each outer product result element having an associated storage element within the 2D array 380 and used to update the value held in the associated storage element. However, it is typically the case that the outer product operation performed is an accumulation operation, and each outer product result element generated is combined with an existing value stored in an associated storage element of the 2D array 380 using the optional accumulation circuit 360 (e.g., by adding the outer product result to the existing value or subtracting the outer product result from the existing value). Although the multiplication circuit 350 and the optional accumulation circuit 360 are used in Fig.10 , they are shown as separate blocks, in one example implementation, they may be provided as a combined block formed by multiply-accumulate circuits.

[0173] The array update circuit 370 is used to control access to related storage elements within the 2D array 380 to ensure that each value received by the array update circuit is used to update the associated storage element within the 2D array 380 .

[0174] Outer product operations are effectively used within data processing systems for a variety of different reasons, and thus the ability to perform multiple outer product operations in response to a single instruction can provide significant performance / throughput improvements, as well as making more efficient use of available storage resources provided by the two-dimensional array within the array storage device 90. As to how the outer product operation can be used, by way of example only, it can be used to implement a matrix multiplication operation. Matrix multiplication can, for example, involve multiplying a first m×k matrix of data elements by a k×n matrix of data elements to produce an m×n matrix of data elements result. This operation can be decomposed into multiple outer product operations (more specifically, k outer product operations, where k can be referred to as the depth), where each outer product operation involves performing a series of multiply-accumulate operations to multiply each data element of the m data element vector from the first matrix by each data element of the n data element vector from the second matrix to produce an m×n matrix of result data elements stored within the 2D array. The results of multiple outer product operations can be accumulated within the same 2D array to produce an m×n matrix that has been generated by performing the matrix multiplication mentioned earlier.

[0175] Matrix multiplication has several potential applications. In addition to the applications of executing ANNs as mentioned above, matrix multiplication can be used in, for example, image processing.

[0176] Fig.11A shows how the outer product result elements can be associated with specific storage elements in a 2D array. Fig.11A In the example, a multiplication function 574 is used to multiply a data element 570 from a first source vector operand by a data element 572 from a second source vector operand to produce an outer product result element, which is then accumulated by an accumulation function 576 to add the outer product result to a current value stored in an associated storage element 578 (or subtract the outer product result from the current value stored in the associated storage element 578) to produce an updated value that is then stored in the associated storage element.

[0177] Fig. 11B The sum of the outer product operation is shown, wherein two outer product result elements are associated with the same storage element in the 2D array. In this example, a data element 580 from a first source vector operand is multiplied by a data element 582 from a second source vector operand using a multiplication function 584 to produce a first outer product result element. Similarly, a data element 586 from the first source vector operand is multiplied by a data element 588 from a second source vector operand by a multiplication function 590 to produce a second outer product result element. Then, an addition function 592 is used to add the two outer product results together, and an accumulation function 594 is performed to produce an updated data value for storage in an associated storage element 596. Therefore, it will be appreciated that in some implementations, more than one outer product result element associated with the same storage element in the 2D array may exist.

[0178] Fig.12 is a diagram schematically illustrating fields that may be provided within a MOP instruction according to one example implementation. The opcode field 605 is used to identify the type of instruction, in this case, that the instruction is a MOP instruction.

[0179] One or more control information fields 610 may be provided, for example, to identify one or more predicates as mentioned earlier. In addition, field 615 identifies dependency information to be used in the operation - for example, this may be an identifier of a register (e.g., a predicate register) storing dependency information. It should be noted that this is separate from any predicate information held in control information field 610, which will be used as a predicate / mask in the operation. Field 620 is then used to identify a plurality of first source vector operands (e.g., by specifying a vector register within vector register file 80, wherein one or more vector operands are implicitly associated with the identified registers). Similarly, field 625 may be used to identify one or more second source vector operands, for example, again by specifying one or more vector registers within vector register file 80. It should be noted that this is only an example, and in practice either of fields 620, 625 may be used to specify the first source vector operand mentioned earlier, with the other field then specifying the second source vector operand. Finally, field 630 may be used to identify a destination 2D array within array storage device 90 that is used to store a matrix generated as a result of performing the multiple outer product operations specified by the multiple outer product instruction.

[0180] As mentioned above, there are a variety of ways in which dependency information can be represented. Figures 13 to 15 Some examples are shown of how dependency information may be represented in one or more predicate registers.

[0181] Fig.13 A first example is shown in which each predicate register P0, P1 holds a set of indices for a corresponding vector register Z4, Z5, each set of indices forming dependency information for its corresponding vector register. Each index in this example identifies the register of the source matrix from which the corresponding data element comes (and thus indirectly identifies the row of the source matrix), where the first (bottom) row is row "0" and the fourth (top) row is row "3". For example, the leftmost element in predicate register P0 (which holds the dependency information for vector register Z4 in this example) is "2", indicating that the corresponding element (the leftmost element) in vector register Z4 is from row 3 of the source matrix (e.g., in this example, from vector register Z10).

[0182] Fig.13 The right side of shows how a MOP instruction can be represented in this example (in this case, identified as an "FMOPA" instruction). In this example, a separate MOP instruction is executed for each of the vector registers Z4, Z5, where each MOP instruction identifies:

[0183] The destination array (ZAi) to which the outer product results computed by the processing circuits are written;

[0184] A predicate register (P0 or P1), which holds the index to be used when executing the MOP instruction;

[0185] a plurality of first source vector registers (Z0 to Z3) for holding a plurality of first source vector operands;

[0186] and

[0187] - A second source vector register (Z4 or Z5) holding the second source vector operand for this operation.

[0188] Fig.14 Another example of how dependencies may be represented is shown; in this example, the same index is used, but the index for both Z4 and Z5 is stored in the same predicate register P0. This example exploits the fact that the number of bits required to represent each index is generally significantly smaller than the number of available bits in each element of the predicate register - for example, the indexes in this example are at most 2 bits long (indices 0, 1, 2, and 3 are represented in binary as 00, 01, 10, and 11, respectively), whereas the number of bits in each element of the predicate register may be 4 or more. Therefore, the inventors have recognized that the indexes for the two vector registers Z4, Z5 may be packed into a single predicate register, reducing the number of predicate registers occupied by dependency information. Specifically, if Fig.14 As shown, the upper half (e.g., the top two bits) of each data element may be used to store the index of the corresponding data element of one of the vector registers (e.g., the upper half of the leftmost element stores the value "3", which is the index of the leftmost element in Z5), and the lower half (e.g., the bottom two bits) of each data element may be used to store the index of the corresponding data element of another of the vector registers (e.g., the lower half of the leftmost element stores the value "2", which is the index of the leftmost element in Z4).

[0189] Therefore, if Fig.14 As shown in the instruction assembly language syntax on the right side of , this example requires two slightly different MOP instructions (i.e., a "FMOPA1" instruction and a "FMOPA2" instruction), both of which identify the same predicate register P0, but one causes the processing circuit to read the top two bits of each element, and the other causes the processing circuit to read the bottom two bits of each element. To distinguish between the two types of MOP instructions, each of the two MOP instructions may, for example, have a different opcode. Alternatively, the encoding of the two instructions may be somewhat different.

[0190] therefore, Fig.14 Each MOP instruction in the command indicates:

[0191] Destination array (ZAi);

[0192] Same predicate register (P0);

[0193] a plurality of first source vector registers (Z0 to Z3) for holding a plurality of first source vector operands;

[0194] and

[0195] - A second source vector register (Z4 or Z5) holding the second source vector operand for this operation.

[0196] Fig.15 Another example of how dependency information can be represented is shown, where the dependency information of two vector registers Z4, Z5 is compressed into a single predicate register P0. In this example, each data element in the predicate register P0 holds a bit image indicating which data elements in the corresponding row of the source matrix hold non-zero values. For example, the leftmost data element in the predicate register reads "1100", indicating that there are non-zero elements in the top two rows of the leftmost column of the source matrix, and zeros are in the bottom two rows of the same column. From this, it can be determined that the leftmost data elements in the vector registers Z4, Z5 are from the top two (2nd and 3rd) rows of the source matrix.

[0197] When executing the MOP instruction in this example, therefore, the processing circuitry determines the position of one of the first or second "1"s in the corresponding bitmap to identify which of the first source vectors should be used in the associated outer product operation.

[0198] Each directive specifies:

[0199] Destination array (ZAi);

[0200] Same predicate register (P0);

[0201] a plurality of first source vector registers (Z0 to Z3) for holding a plurality of first source vector operands;

[0202] and

[0203] - A second source vector register (Z4 or Z5) holding the second source vector operand for this operation.

[0204] It should be noted that although Fig.14 and Fig.15 While an example of executing a separate MOP instruction for each second vector operand (Z4, Z5) is shown, it is also feasible to define a single MOP instruction that specifies both second vector operands.

[0205] As explained above, structured sparsity is an optimization technique for inference of neural network models (and not for their training). Therefore, it is advantageous to support models that use data in the Brain Floating Point (bfloat16 / BF16) format and as signed 8-bit integers (int8). Since 2-way data interleaving is commonly used in some data processors for outer products of bf16 data, for example, Fig. 9 The two weight vectors (Z4, Z5) in are packed into a single vector register (Z) of bf16 elements. m ), such as Fig.16 Similarly, the four activation vectors (Z0:Z3) are packed into two vector registers (Z n ,Z n+1 )middle. Fig.16 The principle of a multi-register outer product-sum scheme using bf16 elements is shown. The predicate register may contain Fig.14 The index list shown or Fig.15 The bitmap is shown.

[0206] like Fig.16 As shown, the MOP instruction in this example is a Sum of Outer Products (SMOPA) instruction that is used to perform a basic 2-way dot product operation. In this example, the SMOPA instruction specifies:

[0207] Destination array (ZAi);

[0208] Predicate register (P i );

[0209] Multiple first source vector registers (Z n ,Z n+1 ), holding a plurality of first vector operands; and

[0210] · Second source vector register (Z m ), holding multiple second source vector operands.

[0211] In the case of 8-bit data types (int8 or uint8), the basic instruction is the outer product and instruction with accumulation on 32 bits. The basic operation is a 4-way dot product operating on 4-way interleaved data. Fig.17 As shown, in a variant of the invention operating on 8-bit data, 2 source registers are used for activation (left operand) and one source register is used for weight (right operand). Since 4 elements are selected from 8 elements in this variant of the instruction (e.g., 2 times 2 elements from 4), two predicate registers are used. An 8-bit 4-to-1 multiplexer is provided in front of each multiplier to achieve a throughput of one operation per cycle.

[0212] Fig.17 The form of this variant of the MOP instruction is shown on the right side of . As shown, the MOP instruction in this example is the outer product sum instruction (SMOPA), which specifies:

[0213] Destination array (ZAi);

[0214] Predicate registers (P0, P1);

[0215] Multiple first source vector registers (Z n ,Z n+1 ), holding a plurality of first vector operands; and

[0216] · Second source vector register (Z m ), holding multiple second source vector operands.

[0217] Although Fig.17 Assuming a 2:4 sparsity pattern is used, in an alternative variant of the instruction, a 4:8 sparsity pattern is used with a bitmap to select the source element of the left operand. This variant will require an 8-to-1 multiplexer instead of a 4-to-1 multiplexer. However, the form of the alternative variant of the instruction will be the same as Fig.17 The forms shown are identical because the operands will be the same.

[0218] As far as the use of the directive is concerned, the following is a code snippet showing how the directive may be used:

[0219] LD1B{Z0-Z3},p0 / Z,[&activations]

[0220] LD1B{Z4-Z5},p1 / Z,[&weights]

[0221] LDR p2,[&indexes]

[0222] LDR p3,[&indexes]

[0223] SMOPA ZA0.S,P2,{Z0.H-Z1.H},Z4.H

[0224] SMOPA ZA1.S,P3,{Z0.H-Z1.H},Z5.H

[0225] SMOPA ZA2.S,P2,{Z2.H-Z3.H},Z4.H

[0226] SMOPA ZA3.S,P3,{Z2.H-Z3.H},Z5.H

[0227] Fig.186 is a flow chart showing the steps performed when decoding a multiple outer product instruction according to an example implementation. At step 650, it is determined whether a MOP instruction has been encountered. If not, a standard decode of the associated instruction is performed at step 655, where the processing circuit is then controlled to perform the desired operation defined by the instruction.

[0228] However, if a MOP instruction is encountered, then at step 660, the instruction is decoded to identify source vector operands (e.g., multiple first source vector operands and at least one second source vector operand), a destination 2D array, dependency information, and the form of the outer product to be performed (e.g., whether an accumulate outer product is being performed, or a non-accumulate variant is being performed, and, for example, whether a normal outer product operation is to be performed or a sum of outer products is to be performed).

[0229] Next, at step 665, the processing circuitry is controlled to perform the required outer product operations and to perform the required updates to the 2D array storage elements. As part of this process, the selection circuitry is controlled to select the data elements for each multiplication operation depending on the dependency information.

[0230] Fig.19 The simulator that can be used is shown to be implemented. Although the embodiment described earlier is implemented with the device and method for operating the specific processing hardware of the technology concerned, it is also possible to provide an instruction execution environment according to the embodiment described herein, which is implemented by using a computer program. Such computer programs are often called simulators because they provide a software-based implementation of the hardware architecture. The types of simulator computer programs include emulators, virtual machines, models, and binary converters (including dynamic binary converters). In general, the simulator is implemented and can be run on a host processor 715 that optionally runs a host operating system 710 and supports the simulator program 705. In some arrangements, there can be multiple layers of simulation between the hardware and the provided instruction execution environment and / or multiple different instruction execution environments provided on the same host processor. Historically, powerful processors have been needed to provide simulators that are implemented at reasonable speeds, but this method may be justified in some cases, such as when it is necessary to execute another processor local code because of compatibility or reuse reasons. For example, the simulator can be implemented to provide an instruction execution environment with additional functionality that is not supported by the host processor hardware, or to provide an instruction execution environment that is generally associated with different hardware architectures. An overview of simulation is given in "Some Efficient Architecture Simulation Techniques", Robert Bedichek, Winter 1990 USENIX Conference, pp. 53-63.

[0231] Where an embodiment has been previously described with reference to a particular hardware architecture or feature, in a simulated embodiment, equivalent functionality may be provided by a suitable software architecture or feature. For example, a particular circuit may be implemented as computer program logic in a simulated embodiment - for example, the simulator program 705 may include: a processing program logic 720, which is used to simulate the behavior of the processing circuit described above; and an instruction decoding program logic, which is used to simulate the behavior of the instruction decoder circuit described above. Similarly, memory hardware (e.g., registers or caches) may be implemented as software data structures in a simulated embodiment - in this particular example, array storage simulation program logic 722 is provided to simulate the array storage device described above. One or more of the hardware elements mentioned in the previously described embodiments are present in an arrangement on host hardware (e.g., host processor 715), and some simulated embodiments may utilize host hardware when appropriate.

[0232] The simulator program 705 may be stored on a computer-readable storage medium (which may be a non-transitory medium) and provide a program interface (instruction execution environment) to the target code 700 (which may include an application program, an operating system, and a hypervisor), which is the same as the interface of the hardware architecture modeled by the simulator program 705. Therefore, the program instructions of the target code 700 including the above-mentioned MOP can be executed within the instruction execution environment using the simulator program 705, so that the host computer 715 that does not actually have the hardware features of the above-mentioned device can simulate these features.

[0233] In this application, the phrase "configured to..." is used to mean that the elements of a device have a configuration that enables the defined operation to be performed. In this context, "configuration" means the arrangement or manner of interconnection of hardware or software. For example, the device may have dedicated hardware that provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not mean that the elements of the device need to be changed in any way to provide the defined operation.

[0234] Although illustrative embodiments of the present invention have been described herein in detail with reference to the accompanying drawings, it should be understood that the invention is not limited to these precise embodiments and that various changes and modifications may be effected therein by one skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. A device, comprising: A processing circuit, the processing circuit is used to perform vector operations; and an instruction decoder circuit, the instruction decoder circuit being configured to decode an instruction from a set of instructions to control the processing circuit to perform the vector operation specified by the instruction, wherein: The set of instructions includes multiple outer product instructions specifying a plurality of first source vector operands, at least one second source vector operand, and dependency information associated with the at least one second source vector operand, wherein each vector operand includes a plurality of data elements, and for a given second source vector operand, the dependency information is arranged to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; the instruction decoder circuitry, responsive to the multiple outer product instruction, controlling the processing circuitry to perform a plurality of calculations to implement a plurality of outer product operations, wherein the plurality of outer product operations include performing an associated outer product operation on a given first source vector operand to calculate an outer product of a subset of data elements of the given first source vector operand and the at least one second source vector operand; and The processing circuitry is configured to select, for each data element of the at least one second source vector operand, a corresponding first source vector operand to be used in performing the associated outer product operation in dependence on the dependency information.

2. The device according to claim 1, comprising array storage circuitry comprising storage elements storing data elements, the array storage circuitry being arranged to store at least one two-dimensional array of data elements accessible by the processing circuitry when performing the vector operation, in: The multiple outer product instruction specifies a given two-dimensional array of data elements within an array storage device forming a destination operand; and The processing circuitry is configured to perform the associated outer product operation of the given first source vector operand by multiplying each data element of the given first source vector operand by each data element of the subset of data elements of the at least one second source vector operand to generate a plurality of outer product result elements, and using each outer product result element to update a value held in an associated storage element within the given two-dimensional array of storage elements.

3. The apparatus of claim 2, wherein the processing circuit comprises: a multiplication circuit, the multiplication circuit being configured to generate each outer product result when performing the plurality of outer product operations; and A multiplexer circuit is associated with the multiplication circuit, and the multiplexer circuit selects selected data elements of the multiple first source vector operands and selected data elements of the at least one second source vector operand to be multiplied to generate associated outer product result elements under control of the correlation information.

4. An apparatus according to any preceding claim, wherein: The relevance information includes at least one set of indexes; and The at least one set of indices includes an index associating each data element of the given second source vector operand with the corresponding first source vector operand.

5. The device according to claim 4, wherein: The dependency information is provided by at least one dependency source operand specified by the multiple outer product instruction; and A given dependency source operand comprises, for each data element of the given second source vector operand, a corresponding element comprising the index associating the data element of the given second source vector operand with the corresponding first source vector operand.

6. The device according to claim 4 or claim 5, wherein The dependency information includes a set of indices for each second source vector operand.

7. The device according to claim 4 or claim 5, wherein The at least one set of indices comprises a set of indices providing the dependency information for a plurality of second source vector operands.

8. The device according to claim 7, wherein: The dependency information is provided by at least one dependency source operand specified by the multiple outer product instruction; and Each element of a given dependency source operand comprises a plurality of indices including an index for a corresponding data element of each second source vector operand of the plurality of second source vector operands.

9. A device according to any preceding claim, wherein Each outer product operation performed by the processing circuitry is based on a different subset of the data elements of the at least one second source vector operand.

10. Apparatus according to any preceding claim, when dependent on claim 2, wherein: The multiple outer product instructions include an outer product sum instruction; and A plurality of outer product results have the same associated storage element within the given two-dimensional array of storage elements, and the processing circuit is configured to combine the plurality of outer product results to update the value held in the associated storage element.

11. An apparatus according to any preceding claim, comprising: a set of vector registers accessible by the processing circuitry, wherein: Each vector register is arranged to store a vector comprising a plurality of data elements; and The plurality of first source vector operands and the at least one second source vector operand include vectors contained within vector registers of the set of vector registers.

12. The device according to claim 11, comprising a set of predicate registers, the set of predicate registers being accessible by the processing circuitry, wherein: Each predicate register is arranged to store predicate information comprising a plurality of elements, each element providing a predicate value; and The dependency information is stored in at least one predicate register of the set of predicate registers.

13. An apparatus according to any preceding claim, wherein The data elements in each second source vector operand represent data values ​​from multiple rows or multiple columns of a source matrix.

14. Apparatus according to claim 13, when dependent on claim 8, wherein Each element of the given correlation source operand is arranged to enable the source matrix to be reconstructed from the at least one second source vector operand.

15. An apparatus according to claim 13 or claim 14, wherein: Each data element in each second source vector operand is associated with a corresponding first dimension in the source matrix, wherein the corresponding first dimension comprises a corresponding row or a corresponding column in the source matrix; and Each data element in each second source vector operand provides a data value selected from the data values ​​in the corresponding first dimension in the source matrix.

16. The device according to any one of claims 13 to 15, wherein The source matrix comprises a matrix having N:M structured sparsity, wherein each group of defined M data values ​​in the source matrix comprises at most N non-zero data values.

17. The device according to any one of claims 13 to 16, wherein The source matrix comprises a weight matrix or an activation matrix for use in executing an artificial neural network.

18. A method comprising: Perform vector operations using processing circuits; decoding an instruction from a set of instructions to control the processing circuit to perform the vector operation specified by the instruction, in: The set of instructions includes multiple outer product instructions specifying a plurality of first source vector operands, at least one second source vector operand, and dependency information associated with the at least one second source vector operand, wherein each vector operand includes a plurality of data elements, and for a given second source vector operand, the dependency information is arranged to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; The method comprises, in response to the multiple outer product instruction, performing a plurality of calculations to implement a plurality of outer product operations, wherein the plurality of outer product operations comprises performing an associated outer product operation on a given first source vector operand to compute an outer product of a subset of data elements of the given first source vector operand and the at least one second source vector operand; as well as A corresponding first source vector operand to be used in performing the associated outer product operation is selected for each data element of the at least one second source vector operand in dependence on the dependency information.

19. A computer program comprising instructions which, when executed on a computer, control the computer to provide: Processor logic to perform vector operations; and an instruction decoder program logic, the instruction decoder program logic being configured to decode a target instruction from a set of target instructions to control the processing program logic to perform the vector operation specified by the target instruction, in: the set of target instructions comprising multiple outer product instructions specifying a plurality of first source vector operands, at least one second source vector operand, and dependency information associated with the at least one second source vector operand, wherein each vector operand comprises a plurality of data elements, and for a given second source vector operand, the dependency information is arranged to indicate, for each data element of the given second source vector operand, a corresponding first source vector operand; the instruction decoder program logic, in response to the multiple outer product instruction, controlling the processing program logic to perform a plurality of calculations to implement a plurality of outer product operations, wherein the plurality of outer product operations include performing associated outer product operations on a given first source vector operand to calculate an outer product of a subset of data elements of the given first source vector operand and the at least one second source vector operand; and The processing program logic is configured to select, for each data element of the at least one second source vector operand, a corresponding first source vector operand to be used in performing the associated outer product operation in dependence on the dependency information.

20. A computer-readable storage medium storing the computer program according to claim 19.

Images