Masking row or column positions for matrix processing

Abstract

and an operand storage circuit configured to store information for forming the first and second input operands for the matrix processing circuit, and a masking circuit configured to perform a masking operation to mask at least a portion of the matrix processing operation or information stored in the operand storage circuit based on masking state data indicating the locations of one or more masked rows or columns that are processed to represent masked values. This is useful for improving performance of 2D convolution operations because masking can be used to mask selected rows or columns when performing 2D convolution as a series of 1×1 convolution operations applied to different kernel positions.

Description

[Technical Field]

[0001] The present technique relates to the field of data processing, and more particularly to matrix processing.

[0002] Matrix processing operations that produce a two-dimensional matrix as a result matrix can be important operations in several fields of data processing, for example machine learning or image processing.

[0003] At least some examples provide an apparatus comprising: a matrix processing circuit that performs a matrix processing operation on first and second input operands to generate a result matrix, the result matrix being a two-dimensional matrix; an operand storage circuit that stores information for forming the first and second input operands for the matrix processing circuit; and a masking circuit that performs a masking operation to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuit based on masking state data indicating the positions of one or more masked rows or columns that are processed to represent a masking value.

[0004] At least some examples provide an apparatus comprising: means for performing a matrix processing operation on first and second input operands to generate a result matrix, the result matrix being a two-dimensional matrix; means for storing information for forming the first and second input operands for the means for performing; and means for performing a masking operation to mask at least a portion of the matrix processing operation or information stored in the operand storage circuitry based on masking state data indicating the locations of one or more masked rows or columns to be processed to represent a masking value.

[0005] At least some examples provide a data processing method including: storing information for forming a first input operand and a second input operand for a matrix processing operation in an operand storage circuit; performing the matrix processing operation on the first input operand and the second input operand to generate a result matrix, the result matrix being a two-dimensional matrix; and performing a masking operation to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuit based on masking state data indicating the positions of one or more masked rows or columns to be processed to represent a masking value. [Brief explanation of the drawings]

[0006] Further aspects, features, and advantages of the present technology will become apparent from the following description of examples, read in conjunction with the accompanying drawings. [Figure 1] FIG. 1 illustrates an example of an unpadded two-dimensional (2D) convolution. [Figure 2] FIG. 1 illustrates an example of a padded 2D convolution. [Figure 3] FIG. 1 illustrates an example in which 2D convolution is applied to input data containing multiple channels to generate output data containing multiple channels. [Figure 4] FIG. 10 is a diagram illustrating an example of a memory layout for storing input data in a memory. [Figure 5] For comparison, we show a technique in which input channel data stored in memory is rearranged to generate several rows of data stored in memory in order to simplify the subsequent 2D convolution process applied to the remapped rows. [Figure 6] We show a different approach where a 2D convolution operation is split into several 1x1 convolutions. [Figure 7] We show how masking selected rows or columns of an operand matrix allows a 2D convolution to be performed by a series of 1x1 convolutions without requiring a step of rearranging the data in memory. [Figure 8]FIG. 10 illustrates how applying a variable position shift between the input and output of a given matrix operation allows the same set of input channel data loaded from memory to be reused across multiple different 1×1 convolution operations for different kernel positions. [Figure 9] FIG. 1 is a diagram illustrating a data processing device having a matrix processing circuit. [Figure 10] FIG. 2 is a diagram illustrating a part of a matrix processing circuit and a register used by the matrix processing circuit. [Figure 11] 1 illustrates various ways of representing addressing and masking state information for matrix processing operations. [Figure 12] 1 illustrates various ways of representing addressing and masking state information for matrix processing operations. [Figure 13] 1 illustrates various ways of representing addressing and masking state information for matrix processing operations. [Figure 14] FIG. 10 illustrates an example in which the matrix processing operation is a cross product and the device has a position shift circuit that applies a variable position shift. [Figure 15] FIG. 10 illustrates an example of processing a load instruction that loads a target row or column of a matrix processing operation. [Figure 16] FIG. 1 illustrates a method for processing matrix processing instructions. [Figure 17] A second example of processing a matrix processing command is shown below. DETAILED DESCRIPTION OF THE INVENTION

[0007] Row or column masking for matrix processing operations Two-dimensional (2D) convolution is a common operation in the field of machine learning, particularly in the field of neural networks. 2D convolution can also be used for other purposes, such as applying a filter to an image. In a 2D convolution, a kernel is provided to define the filter or other operation to be applied. The kernel is applied to one or more input channels, each of which typically comprises a matrix of a size larger than the kernel. In a 2D convolution, for a given output element position in the output matrix, the value of the given output element position depends on the sum of the products of each pair of kernel values ​​and input channel values. For each output matrix position, the selection of the input channel value to be multiplied with the corresponding kernel value is different. For a given output element position, the kernel value multiplied with the corresponding input matrix element is the kernel value that is aligned in position when the kernel is logically positioned so that the center kernel element is above the input matrix element that corresponds to the given output element position. Examples of 2D convolution are further described below.

[0008] One reason 2D convolution operations are relatively complex to implement in data processing is that they may require the calculation of several pairwise sums of products of kernels and input elements for many different combinations of kernel values ​​and input elements, including adding products involving input matrix elements that may not be stored at adjacent addresses in memory address space. Therefore, a typical approach to performing 2D convolution is to perform several remapping (rearrangement) operations (before the sum-of-products calculation itself) to remap the data stored for the input matrix in memory to generate several custom data structures corresponding to the values ​​to be calculated for each respective kernel position of the kernel. However, this remapping involves many instances of copying data from one memory location to another, incurring extra latency and wasting memory space. Therefore, it may be desirable to find a way to implement 2D convolution without requiring such remapping, so that the required operations can be applied directly based on the layout of the input channel data in memory space.

[0009] In the following example, an apparatus includes a matrix processing circuit that performs a matrix processing operation on a first input operand and a second input operand to generate a result matrix, the result matrix being a two-dimensional matrix. The first and second input operands themselves need not be two-dimensional and may be one-dimensional vectors in some instances, while in other instances the matrix processing operation can be applied to two-dimensional input operands. An operand storage circuit is provided that stores information for forming the first and second input operands for the matrix processing circuit. The masking circuit performs a masking operation to mask at least a portion of the matrix processing operation or information stored in the operand storage circuit based on masking state data that indicates the location of one or more masked rows or columns to be processed to represent a masked value. The masking state data can be defined as an operand of a matrix processing instruction that instructs the matrix processing circuit to perform the matrix processing operation, or it can be some stored state data that is separately configured and not explicitly referenced by the matrix processing instruction.

[0010] By providing masking based on masking state data indicating the masked row / column positions, this allows the matrix processing to skip particular rows or columns of input data, which may be particularly useful for 2D convolution operations. The masking circuitry can perform the masking operation either when loading operands into the operand storage circuitry, or when performing the matrix processing operation itself, or both when loading the operand storage circuitry and when performing the matrix processing operation.

[0011] This technique helps support more efficient 2D convolution operations. For example, a 2D convolution operation can be split (by software) into several separate 1x1 convolution operations that apply kernel values ​​from a single kernel position in a larger kernel matrix to several input matrix elements for a given input channel and update the respective elements in the output matrix based on the results. (In some cases, multiple channels of such 1x1 convolution operations can be performed in parallel.) Such 1x1 convolutions allow the operation for a given kernel position to be applied without requiring remapping of structures in memory; successive results of 1x1 convolutions for different kernel positions are accumulated together (to account for which kernel position is being applied, appropriate shifts of the output matrix elements are updated relative to the input matrix elements used to calculate those outputs). As a result, after performing a 1x1 convolution for each kernel position, the result is equivalent to the result of a 2D convolution.

[0012] To support this, it may be useful to provide a masking circuit that can be controlled to mask a given row or column position based on masking state data, such that data from some rows / columns of the corresponding input channel can be treated as if they represent masked values ​​rather than actual data stored in memory. This is because, although a 2D convolution is divided into successive 1×1 convolutions, for most output element positions, the correct result for a given 1×1 convolution can be achieved by reading the corresponding input matrix element, multiplying that element by the corresponding kernel value, and writing the result to the corresponding output matrix element (with a shift in position between the relative position of the input matrix element in the input matrix and the relative position of the corresponding output matrix element in the output matrix, the shift being by the same number of element positions for each multiplication performed on a given kernel position). However, there are some elements at the edges of the matrix where this approach gives incorrect results, for example, because elements at one edge of the output matrix are updated based on elements at the opposite edge of the input matrix, causing an error referred to below as a "wraparound" error. By providing a masking operation, it is possible to mask rows or columns of input data that should not affect the output. Thus, by providing support for row / column masking, this can enable improved performance of 2D convolution operations, which can be important for neural network performance.

[0013] It will be appreciated that the control of which particular rows / columns of a matrix are masked is controlled by software and is therefore not a feature of a particular processor implementation. The apparatus provides a feature that allows software to select the rows / columns to be masked.

[0014] When a given row or column of a given operand matrix is ​​indicated as masked by the masking state data, there may be different options for selecting the masking value to be used for that row / column position. In many practical applications, a masking value of 0 may be useful. This may be useful for supporting row skipping to address the "wraparound" problem mentioned above, which should prevent rows / columns on one edge of the input matrix from affecting the calculation of output matrix elements on the opposite edge. A masking value of 0 may also be useful for enabling a padding value to be supplied to be multiplied with kernel elements located outside the boundary of the input matrix when a padded 2D convolution operation is applied and the kernel is located near an edge of the input matrix. Therefore, in some hardware implementations, it may be sufficient for the masking circuitry to support only a fixed masking value, e.g., a masking value of 0, to be used for any masked row / column position.

[0015] However, in some applications using 2D convolution, it may be desirable to use a padding value other than 0 (e.g., when a matrix is ​​represented using a quantization scheme in which each value is offset from its true value by a specific number, such that a "zero point" is represented by a numeric value other than 0). To support such operations, it may be useful to provide the ability to select a non-zero value as the masking value. Thus, in some implementations, in a masking operation, the masking value can be selected from among multiple masking values ​​(e.g., 0 or another preset value) based on at least one of a masking value selection parameter specified by an instruction causing the masking operation to be performed (e.g., a load instruction for loading information into an operand storage circuit or a matrix processing instruction for controlling a matrix processing circuit to perform a matrix processing operation), a control value stored in a control register, or a masking vector specifying distinct masking values ​​for multiple elements of a masked row / column. In the last option, the masking vector can be read from a vector register.

[0016] The masking state data may have an encoding that identifies elements within a two-dimensional array of elements that should be treated as representing a masking value. Thus, the masking state data may identify (fully or partially) the location of masked elements across two dimensions. Providing state data that allows masking to be applied in two dimensions may be useful for dealing with several issues related to 2D convolution processing, including the "wraparound" error problem mentioned above, the fact that at the end of a loop there may be some unused "out-of-bounds" elements that extend beyond the end of the data structure being processed, and for providing support for a "position shift" function, which is described in more detail below.

[0017] For example, the masking state data may specify first masking state data indicating one or more masked row or column positions where all elements at the masked row or column positions should be treated as representing a masked value, and second masking state data indicating whether individual element positions within a given row or column should be masked. Masking an entire row or column using the first masking state data may be useful for handling “wraparound” errors and / or “out-of-bounds” rows / columns in the first dimension, while individual masking of specific elements within a row or column that are not fully masked may be useful for “out-of-bounds” columns / rows in the second dimension and / or to support the position shifting function described below (or for more general element-by-element predication). The first masking state data may include a set of elements identifying masked / unmasked row / column positions in one dimension (row or column), and the second masking state data may include a set of elements identifying masked / unmasked positions in the orthogonal dimension (column or row). In some cases, the same set of second masking state data can be shared across rows / columns, so that the second masking state data can specify individual display of masked / unmasked elements only for a single row / column (or, if different patterns of masked / unmasked elements are required for different rows / columns, the second masking state data can be coordinated between processing one row / column and processing the next row / column).

[0018] The masking state data may have an encoding that can indicate at least two non-adjacent row or column locations as masked row or column locations, separated by at least one unmasked row or column location. This recognizes that when a 2D convolution is divided into several 1x1 convolutions, there may be several non-adjacent row or column locations that need to be masked to prevent input values ​​on one edge of the input matrix from affecting output values ​​on the opposite edge of the output matrix. Also, the padded locations for a padded 2D convolution may not correspond to consecutive addresses in memory.

[0019] The masking state data can be represented in several different ways. In general, the masking state data may be any set of information capable of indicating which row / column positions within a matrix structure should be masked. One approach may be for the masking state data (e.g., the first masking state information described above) to include several masking state indicators, each corresponding to a respective row or column position of a given operand matrix and indicating whether the corresponding row or column position is a masked row or column position. For example, the masking state data may include a bitmap, with each bit corresponding to a given row or column position and set to one value if that row or column position is masked and set to another value if that row or column position remains unmasked. Similarly, the second masking information may include a second bitmap indicating masked row / element positions within a particular row / column.

[0020] The masking state data need not distinguish whether it refers to respective rows of a given operand matrix or respective columns of a given operand matrix. Different software applications may choose different layouts for matrices in memory (e.g., row-major or column-major), but the format of the masking state data may be the same regardless.

[0021] The operand storage circuitry can be implemented in different ways. In some examples, the operand storage circuitry can comprise a set of input registers from which the first and second operands can be read when performing a given matrix processing operation.

[0022] However, it may be useful to provide a matrix transposition circuit as part of the operand storage circuit, the matrix transposition circuit including several storage units for storing each matrix element of a given operand matrix. The storage units of the matrix transposition circuit may be readable in row groups corresponding to the rows of the given operand matrix and in column groups corresponding to the columns of the given operand matrix. Providing such a matrix transposition circuit can be very useful in addressing the fact that different machine learning algorithms may store input channel data in memory using different layouts. For example, some algorithms may use a row-major layout in memory, in which the offset between the memory addresses of adjacent elements in the same row of a matrix is ​​smaller than the offset between the memory addresses of adjacent elements in the same column of a given operand matrix. Other algorithms may use a column-major layout, in which the offset between the addresses of adjacent elements in the same column is smaller than the offset between adjacent elements in the same row. Because the matrix transpose circuitry allows for on-the-fly remapping of whether row-major or column-major format is used if a given operand matrix is ​​written to a row-group matrix transpose circuitry, and vice versa, the matrix transpose circuitry allows for on-the-fly remapping of whether row-major or column-major format is used, so that subsequent matrix processing operations can be in a consistent format regardless of whether the input matrix data stored in memory is row-major or column-major, which can simplify code development and avoid the need for remapping or rearrangement of data within the memory storage itself.

[0023] It should be noted that the storage units of the matrix transpose circuit need not be physically arranged in rows and columns. The storage units of the matrix transpose circuit need only be logically readable in groups of storage elements corresponding to rows or in groups corresponding to columns. For example, the matrix transpose circuit can be implemented as a set of registers with multiple read / write ports so that portions of the registers can be addressed in different combinations. For example, if each register stores a row group, the column group can be considered to be formed by a set of portions of data (the set includes one portion per register, in corresponding locations within each register). Alternatively, the opposite mapping may be used, where each column group is mapped to one register and the row group is a stripe of portions of data in corresponding locations within each register. It should also be noted that, although possible, it is not necessary for the "rows" of a matrix stored in memory to be written to "row groups" of the matrix transpose circuit, but such rows of the matrix could equally well be written to "column groups" of the matrix transpose circuit. Thus, "row groups" and "column groups" of storage units in a matrix transpose circuit refer to orthogonal groups from which storage units of the matrix transpose circuit can be read, but do not necessarily follow the same row / column orientation as the matrix in memory. Indeed, to improve pipelining of reads / writes of the matrix transpose circuit, it may be useful to alternate the selection of whether successive groups of rows (either rows or columns) of the input matrix are written to the matrix transpose circuit in row groups or column groups.

[0024] Thus, when loading data into the matrix transpose circuit, the load circuit can select whether to load at least one row group or at least one column group of the storage unit of the matrix transpose circuit based on a portion of the matrix data structure in memory. The selection of whether to load at least one row group or at least one column group can be based on one or both of row / column direction selection information specified by the load instruction and row / column direction selection information stored in a control register that can be updated in response to a row / column direction switching instruction. Some implementations can use only one of these options to determine whether to load a row group or a column group (either the information specified by the load instruction or the information specified in the control register). Alternatively, an implementation can combine both of these pieces of information. For example, a control register bit can indicate either row mode or column mode, while a bit in the load instruction can indicate whether the meaning of the stored bit should be inverted (so that, for a load instruction with the “invert” bit set, the instruction loads a row when the stored bit indicates a column and loads a column when the stored bit indicates a row). Similarly, when reading data from the matrix transpose circuit to supply operands for a matrix processing operation (or to transfer information to operand registers from which operands can then be obtained for the matrix processing operation), row / column direction selection information can specify whether to read a row group or a column group of the matrix transpose circuit (again, the selection information can be specified by an instruction and / or control register, with the option to combine both the row / column direction bits in the register and the "invert" bit in the instruction to be used for a store instruction similar to the load instruction described above).

[0025] The masking operation based on the masking state data may be performed at a different time relative to the loading of operands for the matrix processing and the processing of the matrix processing operation itself.

[0026] In some implementations, the matrix processing circuit may include a masking circuit. In response to the masking information, the masking circuit of the matrix processing circuit may perform the matrix processing operation using a portion of one of the first operand and the second operand corresponding to one or more masked row or column positions that is treated as representing a masked value instead of the actual value of the portion of one of the first operand and the second operand stored in the operand storage circuit. Thus, while actual data from the input channels may be loaded from memory into the operand storage circuit as usual, replacing such input data with a masked value to provide padding or avoid the wraparound error described above may be controlled by masking data read from the operand storage circuit at input to the matrix processing circuit. This technique may be particularly useful for implementations that also support the option of applying a variable position shift, as described further below.

[0027] In some embodiments, the masking circuit may be configured by a load circuit that, in response to a load instruction, loads information corresponding to a target row or column of a given operand matrix into the operand storage circuit based on a portion of a matrix data structure stored in memory. In this case, the load circuit includes a masking circuit, and when the target row or column corresponds to a masked row or column position indicated by the masking state data, the load circuit may load the portion of the operand storage circuit corresponding to the target row or column with data having a masking value instead of data based on the portion of the matrix data structure stored in memory. In this approach, masking can be applied at the time of loading the operand from memory, avoiding unnecessary loading of matrix elements that would be masked anyway. Out-of-bounds data (corresponding to addresses beyond the end of the data structure being processed, referenced by the load instruction in the last iteration of the loop because the amount of data to be processed does not correspond to an exact multiple of the amount of data that can be processed in one iteration) can also be masked using the masking circuit to prevent them from being loaded, thus preventing address faults caused by accessing addresses that may be invalid.

[0028] Some hardware implementations may support both types of masking, which may be useful because, for example, padding and masking of out-of-bounds data may be handled more efficiently by masking at the time of loading, but if variable position shifting is supported, handling of "wraparound" errors of the type described above may require masking at different input rows / columns for different instances reading the same set of input data, in which case it may be more effective to apply masking at the time of reading the operand storage circuitry to perform a particular matrix processing operation. Thus, to provide maximum flexibility, some implementations may support both types of masking.

[0029] In those implementations that provide a load circuit with a masking circuit for applying masking at the time of loading operand data from memory, if, in response to a load instruction, the masking state data corresponding to a target row or column indicates that the target row or column corresponds to a masked row or column position, the load circuit may determine whether each of the matrix elements of the target row or column should be masked based on a shared item of masking state data shared among two or more matrix elements of the target row or column. Thus, it is not necessary to provide an individual masking state for each individual element in the target row or column (although this is possible if necessary, as described above with the example of second masking state data providing 2D masking). To support a "split into 1x1 convolution" approach for processing 2D convolutions, a common memory layout for input channel data may be to group input elements at the same x and y locations for multiple input channels together in a contiguous memory block, in which case masking can be applied to entire rows or columns of the input matrix structure that defines the input data for each of those input channels. This means that it may be sufficient to share an item of masking state data across the rows or columns of the operand matrix being processed.

[0030] In the load masking example, the masking status data may be represented using a set of masking status indicators (eg, bitmaps), as described above.

[0031] However, another approach is that the masking state data may include several offset values, each corresponding to a respective row or column location of a given operand matrix and indicating the offset of the address of the corresponding portion of the matrix data structure in memory relative to the base address. In this case, the masked row or column location may be indicated by the offset value of the masked row or column location having a predetermined reserved offset value. This approach may be useful because it means that the masking state data may be represented using part of the addressing information used to identify the memory address where the portion of the matrix data structure in memory should be loaded. Thus, for each row or column location, the base address and the corresponding offset value of that row or column location may be used to identify the address in memory where the portion of the matrix data structure should be loaded if the offset value does not have the predetermined reserved offset value. However, if the offset value of a given row or column location has the predetermined reserved offset value, instead of loading the corresponding portion of the matrix data structure in memory, the masking value may be written to the portion of the operand storage circuit that would otherwise store the portion of the matrix for that row or column. Thus, this approach avoids the need to provide separate masking state data beyond the state data used to address the matrix data structure in memory. The predetermined reserved offset value may be any reserved value that is designated as not being usable for an actual offset value, such as -1 (e.g., in signed binary representation, a value where all offset bits are 1).

[0032] In one example, the masking state data may be stored in at least one masking state register provided within the processing unit, for example, there may be specific instructions to write the masking state data to the masking state register before executing a load instruction to load a portion of the operand matrix under control of the masking state data.

[0033] The masking status register may be a dedicated register specifically provided for controlling masking when performing matrix operations and / or loading operands for matrix operations.

[0034] In another example, the at least one masking state register may include at least one predicate register. In response to a vector instruction (or single-instruction multiple-data instruction) for controlling the processing circuit to perform vector processing using one or more vector operands including one-dimensional arrays of elements, the vector predicate register may be read to provide a predicate value that controls whether each lane of the vector processing is masked. Thus, the same register may be shared between indicating a vector predicate for a vector operation and indicating masking state data for a matrix operation.

[0035] At least one masking state addressing register may be provided to store masking state addressing information that identifies a location in memory from which the masking state data can be obtained. For example, when the masking state data is represented using a set of offset values ​​as described above, the set of offset values ​​may be stored in memory, and the masking state addressing information in the masking state addressing register may identify where in memory the array is stored. This approach may reduce the number of registers required architecturally to support matrix operations, which may be preferable for some low-power microarchitecture implementations.

[0036] Nevertheless, even if it is not architecturally required to provide registers to store the masking state information itself (because microarchitectures that do not want to provide dedicated hardware to store this information can instead load it from memory as needed), some microarchitecture designers may choose to provide a masking state cache to cache masking state data retrieved from memory so that it can be accessed more quickly for future accesses to help improve performance. This can be useful because the masked / unmasked row / column pattern may be the same for several matrix operations, so caching can save a significant number of memory accesses.

[0037] Regardless of the format of the masking state data, the load circuitry may determine a target address of the portion of the matrix data structure in memory based on the addressing information, which may be defined in a variety of ways: the addressing information may be obtained from a register explicitly referenced by the instruction causing the load to be performed, or may be obtained from a default register implicitly referenced to the load instruction.

[0038] In one example, the addressing information may include a set of address pointers, each address pointer indicating an address of a portion of a matrix data structure corresponding to a respective row or column location of a given operand matrix.

[0039] In another example, the addressing information may include a base address of a matrix data structure stored in memory and offset information for determining the address of a portion of the matrix data structure corresponding to a given row or column of a given operand matrix relative to the base address. In some examples, this offset information may be represented using the same set of offset values ​​used for the masking state data, but this is not required, and in other examples, the offset information may be separate from the masking state data. The offset information may be represented in various ways, for example, using a stride value indicating the difference between the address of a portion of the matrix data structure corresponding to one row or column of a given operand matrix and the address of a portion of the matrix data structure corresponding to the next row or column of the given operand matrix, or by explicitly recording multiple row / column offsets in the offset data structure as described above. The use of a stride value avoids the need to explicitly encode each individual offset value for each row, while the use of a more explicit offset data structure allows the masking state to be represented in the same structure as the offsets, enabling the processing of matrices with irregular patterns of memory accesses for each row / column. In either approach, representing addresses using offset information relative to a base address allows addressing information to be represented using fewer bits than if the addressing information indicated absolute addresses corresponding to each row / column location in a given operand matrix.

[0040] In some examples, the addressing information may include further information providing sub-portion selection information for selecting which sub-portion of the portion of the matrix data structure in memory identified based on the addressing information to load into the operand storage circuit when loading a given target row or column. This recognizes that, given a limit on the maximum size of a matrix that can be processed by hardware, when processing an input matrix of a larger size, it may be necessary to divide the operation into several sub-operations, each operating on a smaller portion of the input matrix. Because the layout of the matrix data in memory may include rows or columns of a larger size than the block of matrix data operated on by a given set of matrix processing instructions, the sub-portion selection information may be used to narrow down which sub-portion of the row or column to process for a given operation.

[0041] Thus, there are numerous options for representing addressing information that identifies the location in memory where a given target row or column should be loaded. At least one addressing register may be provided for storing the addressing information. Prior to executing a load instruction or a matrix processing instruction, the program being executed may load the at least one addressing register with appropriate addressing information for selecting the portion of the matrix data structure to be processed.

[0042] In some implementations, a prefetch circuit may be provided to generate a prefetch request to prefetch a portion of a given operand matrix from memory according to addressing information stored in at least one addressing register. For example, if the addressing information includes an array of offset values, while loading a row or column of a given operand matrix relative to a previous row or column, the prefetch circuit can look ahead and start prefetching data based on the offset of a later row / column, resulting in improved performance. Alternatively, other microarchitectures may prefer not to provide a prefetch circuit to save power and circuit area.

[0043] In some implementations, the first and second input operands for the matrix processing operation may be two-dimensional matrix operands. For example, the matrix processing circuit may support a complete matrix multiplication operation executed in a single instruction, which may be beneficial for performance. However, this approach may be more expensive in terms of power consumption and circuit area.

[0044] Therefore, other implementations may prefer to provide a matrix processing circuit that supports performing matrix processing operations on one-dimensional vector operands to generate a two-dimensional result matrix. For example, the matrix processing operation may include a cross product operation applied to one-dimensional vector operands to generate a two-dimensional result matrix. This recognizes that, in practice, a matrix multiplication operation applied to two 2D matrix operands to generate a 2D result matrix can be decomposed into several separate cross product operations applied to each individual row / column combination of the input matrix operands, and the results of the cross product operations are accumulated together to generate a final result equivalent to the 2D matrix multiplication result. Therefore, it may be particularly useful for a cross product operation to include a cross product and accumulation operation in which the result matrix includes an updated value for each element of an accumulator matrix, and the updated value for a given element of the accumulator matrix corresponds to the result of adding the previous value of a given element of the accumulator matrix to the corresponding element of the cross product result matrix that corresponds to the result of performing the cross product operation on a first input operand and a second input operand, represented as one-dimensional vectors. This operation may be useful for supporting the 2D convolution operation described above.

[0045] The matrix processing circuitry can respond to a single instruction to generate a result matrix as a two-dimensional matrix based on first and second input operands. Thus, even if a matrix multiplication operation is broken down into multiple instructions that perform separate cross-product operations, with each cross-product operation operating on a one-dimensional vector operand, each individual cross-product operation can still generate a two-dimensional result matrix. This can provide improved performance compared to approaches that use the vector processing circuitry to perform a series of vector operations equivalent to the matrix operation, where each vector operation processes a 1D vector operand and generates a 1D vector result.

[0046] Position shift for matrix processing An example apparatus includes a matrix processing circuit that performs a matrix processing operation on first and second operands to generate a result matrix, the result matrix being a 2D matrix. An operand storage circuit stores information for forming first and second input operands for the matrix processing circuit. A position shift circuit is provided for applying a variable position shift to vary which row or column of the result matrix is ​​updated based on a given element of one of the first and second input operands stored in the operand storage circuit during a given matrix processing operation. The variable position shift is based on one of several alternative shift amounts selectable for a given matrix processing operation. Each alternative shift amount corresponds to a position shift of one of the first and second input operands relative to the result matrix by a different number of rows or columns.

[0047] The position shifting circuit is useful for supporting techniques in which a 2D convolution operation is decomposed into several separate 1×1 convolutions that accumulate into a result matrix. The inventors recognized that in such a series of 1×1 convolutions, 1×1 convolution operations corresponding to several adjacent kernel positions require very similar input data, but with a relative shift of one or more row / column positions between the inputs for each kernel position. Thus, by providing a circuit for applying variable row / column position shifts of the inputs to a given matrix processing operation on the output, this means that the same operand data loaded from memory can serve as input for matrix processing operations on several different kernel positions during the series of 1×1 convolutions that perform the 2D convolution operation, thereby reducing the number of load operations required to load data from memory to perform the given 2D convolution operation.

[0048] As described above, while some implementations may implement a full matrix multiplication operation, to limit hardware costs, other implementations may implement the matrix processing operation as a cross product operation applied to one-dimensional vector operands as first and second input operands to generate a two-dimensional result matrix. Thus, in this case, a variable position shift can vary which row or column of the result matrix is ​​updated based on a given element in one of the first and second input vector operands. Again, for reasons similar to those described above, it may be particularly useful for the matrix processing operation to be a cross product and accumulation operation, where the result matrix includes updated values ​​for each element of the accumulator matrix, formed based on the previous value of the accumulator matrix and the corresponding element generated for the outer product result. This operation may be useful for supporting a 1×1 convolution technique for processing 2D convolutions.

[0049] The position shift circuit can select between each alternative shift amount based on a parameter specified by a matrix processing instruction for controlling the matrix processing circuit to perform the matrix processing operation. In some implementations, the parameter identifying the shift amount can be part of the opcode of the matrix processing instruction, such that several different opcodes can be assigned respective shift amounts, each corresponding to the same type of matrix processing operation (other than having different shift amounts). Alternatively, a separate parameter in the instruction encoding can be defined, e.g., a shift amount selection field separate from the opcode that identifies the particular matrix processing operation to be performed. The parameter for selecting the shift amount can be represented as an immediate value in the instruction encoding or can be identified in a register specified by the matrix processing instruction.

[0050] Alternatively, in some implementations, a specific dedicated register may be provided for storing the shift amount selection parameter, so that the register that is read in response to a matrix processing instruction to obtain the shift amount selection parameter is implicit and therefore does not require explicit encoding in the instruction encoding.

[0051] The matrix processing circuitry may also support predication, which may identify specific rows or columns in the result matrix as active or inactive row or column positions, as identified by predicate information accessible to the matrix processing circuitry. Thus, when a given row or column of the result matrix corresponds to an active row or column position indicated by the predicate information, the matrix processing circuitry may generate elements of the given row or column of the result matrix having values ​​that depend on the corresponding row or column of one of the first and second input operands (which row or column is the corresponding row or column depends on one of the alternative shift amounts selected for that particular matrix processing operation). When a given row or column of the result matrix corresponds to an inactive row or column position indicated by the predicate information, the elements of the given row or column of the result matrix are generated to have values ​​that are independent of the corresponding row or column of one of the first and second input operands. For example, if a given row or column of the result matrix is ​​inactive, the corresponding elements may retain their previous values ​​without being updated based on the corresponding row or column of the input operands. This helps address the "wraparound" problem mentioned above by providing the ability to prevent particular rows or columns of input operands from affecting the output. This predication may be an example of a masking operation as previously mentioned.

[0052] Again, with respect to the masking example described above, the operand storage circuitry may comprise a matrix transpose circuitry that allows reading and writing in either row groups or column groups of the storage units of the matrix transpose circuitry. This helps to support more efficient processing of matrix data structures stored in memory represented in either row-major or column-major format. When the position shifting example is used, all of the features described above for the matrix transpose circuitry may be provided.

[0053] Where a matrix transposition circuit is provided, the operand storage circuit may also comprise an operand register, separate from the matrix transposition circuit itself, for storing first and second input operands for the matrix processing operation. The operand register may be a storage circuit from which the operands of a given processing operation are read in response to a matrix processing instruction for controlling the processing circuit to perform the matrix processing separation.

[0054] Dedicated shift instructions can be provided to control the operand shift circuit to read at least one row or column of a given operand matrix from the matrix transpose circuit and write at least one row or column to an operand register. This can simplify the encoding of the matrix processing instruction, as any additional parameters for selecting which column or row is read from the matrix transpose circuit (or for selecting which particular row or column should be read) can be encoded into the shift instruction, such that such parameters require less encoding space within the matrix processing instruction.

[0055] However, another approach is that the operands may be read out of the matrix transpose circuit in response to a matrix processing instruction and provided directly to circuit logic for performing the matrix processing operation, without having to run through a series of operand registers.

[0056] Although such operand movement circuitry in response to a movement instruction, or the ability to read operands directly from the matrix transpose circuitry, is not explicitly mentioned above for the embodiment using masking, these features may also be provided in that example.

[0057] Also, the masking function described in the previous section can be combined with the position shifting function described above. Thus, even in the position shifting example, it is possible to provide a masking circuit that performs a masking operation based on the masking state data described above.

[0058] In fact, combining both a masking function on loads and a position shift (including predication applied at the input to a matrix processing operation) can be particularly useful. While one might expect that predication is simply redundant when masking on loads is supported, in practice, providing both functions can be useful. This is because masking on loads can be used to insert padding values ​​to support padded 2D convolutions, even if the predication applied at the input to the matrix processing operation is further masking to prevent certain rows from affecting the output (to address the wraparound issue discussed above). This is because, when a position shift function is used to allow multiple kernel positions to be calculated based on a data set loaded for a single kernel position, predication based on a predicate value can be used to select individual rows to be suppressed for each individual kernel position, because the location of rows affected by the wraparound issue can vary for each kernel position, which would be difficult to handle if such wraparound were handled only at the time of loading data from memory. Nevertheless, a masking technique can be useful for providing padding values.

[0059] Nevertheless, in the above example, if position shifting is not supported, and separate loads are performed for each kernel location, masking at the time of performing the load operation may be sufficient to address the wraparound issue, or alternatively, masking on loads may not be supported at all, and instead masking / predication may be applied when performing matrix processing operations.

[0060] Again, with respect to the masking example, the result matrix generated for the matrix processing operation may be a two-dimensional result matrix generated from first and second input operands in response to a single instruction, thus not requiring separate processing of individual vector instructions each producing a one-dimensional vector result.

[0061] 2D convolution Figure 1 shows an example of a 2D convolution operation performed on an input matrix and a kernel matrix to generate an output matrix. In this example, the input matrix is ​​a 4x4 matrix, the kernel is a 3x3 matrix, and the output is a 2x2 matrix. It will be understood that it is not necessary for the matrices involved to be square matrices with the same dimensions in terms of number of rows and columns, and the particular set of matrix sizes shown in Figure 1 is merely an example.

[0062] In a 2D convolution operation, for each output element in the output matrix, a kernel is centered on the element of the input matrix at the position corresponding to the output element being generated, and the output element is generated with a value corresponding to the sum of the products of each kernel element and the input matrix element at the corresponding position relative to the centered kernel. For example, for an output matrix element F' corresponding to the position of input element F, the value of F' is generated by multiplying each pair of input and kernel elements at the corresponding position, assuming that center kernel element K5 is positioned above the input element F corresponding to output position F'. Thus, F'=A * K1+B * K2+C * K3+E * K4+F * K5+G * K6+I * K7+J * K8+K * K9.

[0063] Similarly, for each other matrix element in the output matrix, an element is generated based on a sum of products, but with kernels generated across different elements of the input matrix. For example, for output element G', the kernel matrix has its center element K5 above the input matrix element G, which means that the sum of products is G'=B * K1+C * K2+D * K3+F * K4+G * K5+H * K6+J * K7+K * K8+L *Similar operations are performed to generate the output elements J' and K'.

[0064] 1 illustrates an unpadded 2D convolution operation, meaning that output elements F', G', J', and K' are generated only for input positions F, G, J, and K, where it is possible to center the kernel at that input position without any kernel elements of the kernel matrix extending outside the bounds of the input matrix. For example, input elements A, B, C, D, E, H, I, L, N, M, O, and P do not have corresponding elements in the output matrix because they require portions of the kernel that extend outside the bounds of the input matrix. Thus, for unpadded 2D convolution, the output may generally be smaller than the input.

[0065] It is also possible to perform padded 2D convolution, in which an output matrix is ​​produced with the same dimensions as the input matrix, by supplying padding values ​​(PVs) to element positions outside the boundary of the input matrix that are needed to apply a kernel centered at a position near the edge of the input matrix, as shown in Figure 2. In the example of Figure 2, the input matrix and kernel may be the same as in Figure 1, but the output matrix is ​​also a 4x4 matrix, and in addition to elements F', G', J', and K', which are calculated in the same way as in Figure 1, it also contains surrounding elements A' through P' to place the output matrix on the same side as the input matrix.

[0066] For calculations where the kernel is centered on one of these outer element positions, the kernel elements located on the outside of the input matrix are multiplied by a padding value (PV). For example, for the calculation to generate output element A', this requires that center kernel position K5 be located above element A of the input matrix, so while there are valid input values ​​for positions A, B, E, and F in the input matrix corresponding to kernel elements K5, K6, K8, and K9, the other kernel elements K1, K2, K3, K4, and K7 are multiplied by padding values ​​when generating the sum of products to generate the new value of output matrix A'.

[0067] Similarly, for other elements around the boundary of the output matrix, the padding values ​​are at different positions relative to the kernel depending on which edge of the input matrix that kernel overlaps. For example, for output position L', padding values ​​are required for the right columns of kernels K3, K6, and K9 because these are positions that extend outside the input matrix when the kernel is centered over position L. Similarly, for output element N', kernel position K5 is centered over position N, which means that the bottom row of kernel positions K7, K8, and K9 extend outside the input matrix and therefore require padding.

[0068] In one example, the padding value may simply be 0. However, some 2D convolution operations may require other types of padding values. For example, in some cases, a quantization scheme may be used in which an offset is applied to the true value of the matrix when generating the stored numerical value for each matrix element, so that "0" may actually be represented using a non-zero numerical value. In this case, the padding value may be a non-zero value representing a 0 point. The padding value may also be set based on an averaging of other elements in the input matrix. The exact rules for setting the padding value may depend on the particular application being performed. Therefore, it may be useful to support the ability to select between several alternative types of padding values ​​(e.g., based on parameters specified by a control register and / or matrix processing instruction).

[0069] Although not shown in the examples of FIGS. 1 and 2, strided convolution is also possible, where, when centered on a given input element, kernel values ​​are applied to neighboring input elements separated from the central input element by intervals of a constant stride (in contrast to FIGS. 1 and 2, where the stride is 1, other examples may have a stride of 2 or more).

[0070] Unpadded and padded 2D convolution operations can be useful in a variety of processing applications. For example, 2D convolution can be useful for applying filters to images, such as for blurring, sharpening, edge detection, etc. The kernel applied may be selected based on the type of filter desired and may have specific values ​​of kernel elements that bring out some feature, such as an edge. Effectively, the kernel slides over each successive image pixel and applies an operation to generate a new value for the output pixel based on that pixel and a number of surrounding pixels, using the relationship defined by the kernel.

[0071] Another type of processing that may involve 2D convolution is found in the field of machine learning, for example, when implementing neural networks. For example, a neural network trained to detect features in image data may be implemented using a set of kernels applied to the image data in a 2D convolution operation. More generally, feature maps representing some data being processed may be processed with kernels to make inferences about the data.

[0072] As shown in Figure 3, it may be useful for machine learning algorithms to support multiple channels of input and output data and multiple sets of kernel weights to allow several different inferences to be derived from a dataset. Each input / output channel may comprise a two-dimensional matrix of elements. For example, the number of input channels may be IC, and the height and width of each input channel may be IH (Input Height) and IW (Input Width). The number of output channels may be OC, and the height and width of each output channel may be OH (Output Height) and OW (Output Width). OC sets of kernel weights are provided, where OC corresponds to the number of output channels. Each set of kernel weights is a function of KH * KW *(where KH and KW are the kernel height and width, and IC is the number of input channels.) A given output channel is generated by performing IC instances of a basic 2D convolution operation of the type shown in Figure 1 or Figure 2, where each instance convolves a single input channel, IC, into KH. * The results of the elementary 2D convolutions of each input channel are accumulated together (or, as will be explained later, by performing other sequences of operations that yield the same result) to combine them with the corresponding subset of KW kernel weights and produce the corresponding output channel. The other output channels use similar operations, but with a different KH weight for each output channel. * KW * It is calculated using a set of IC kernel weights. Whether OH and OW are equal to or smaller than the input height IH and input width IW may depend on whether a padded or unpadded 2D convolution is being performed.

[0073] In this example, the number of output channels OC is equal to the number of input channels IC, but this is not required. Other examples can have different numbers of IC and OC. Also, the 2D convolution shown in Figure 3 may be only one step in a tree of 2D convolutions, so that the input channels themselves can be formed as outputs from previous convolutions, and the output channels of Figure 3 can themselves be processed by further convolutions.

[0074] When 2D convolution is applied to several input channels, there are several possible layouts for storing the input channel data in memory. Figure 4 shows one possible memory layout, called the NHWC memory layout, where C refers to the input channel, W refers to the width, H refers to the height, and N refers to several distinct objects represented by separate sets of IC input channels. The NHWC notation indicates that when reading data from consecutive addresses in a data structure in memory, the input channel identification variable C is the fastest-changing variable and the object identification variable N is the slowest-changing variable. Thus, in the NHWC layout, when traversing consecutively increasing addresses in a data structure in memory, the input matrix elements for a given x-y matrix location of each IC input channel are first stored in consecutive address blocks in memory. Then, the elements in each input channel at the next location in the same row as the first matrix element are laid out for each other x-y location, and so on. That is, the elements first cycle through all input channels for one element position, then move to the next element in the same row (because the width W is the next fastest changing variable after the channel ID), and then once all positions in the same row (elements with the same y matrix coordinate) have been stored for all channels, the next element stored will be the next row at the next highest y position.

[0075] Thus, referring to Figure 3, the first row of the memory layout shown in Figure 4 may correspond to elements in the cross-hatched boxes corresponding to location A in each input channel, the next row may correspond to elements shown in dotted shading corresponding to location B in each input channel, and so on for the remaining elements C, D in that first row. When the end of the row is reached, the same is done for the next row starting with the element at location E in each input channel. If multiple objects to be processed (e.g., several separate images) are each represented using a separate set of IC input channels, all of the data for one object (N=0) is stored in memory before the data for the next object (N=1).

[0076] For ease of understanding, FIG. 4 shows elements for a given input matrix location in all channels within one "row" of the address space, then moves to the next "row" of the 2D representation of FIG. 4 to store an element at the next input location B; however, it will be understood that in reality, the address space is simply a monotonically increasing series of addresses, and there is no 2D arrangement of addresses as shown in FIG. 4. The 2D representation shown in FIG. 4 is a graphical representation used for simplicity in fitting information to a page. Nevertheless, the information stored in memory represents multiple channels of matrices, and those matrices are two-dimensional structures logically arranged in rows and columns.

[0077] 4 is one possible layout, other implementations may store the matrix structure in different layouts. For example, if an NCHW memory layout is used, the layout may provide all X / Y values ​​for channel 0, then all X / Y values ​​for channel 1, etc.

[0078] Regardless of the particular memory layout selected for a given application, one problem with 2D convolution techniques is that the elements needed to be combined with a kernel element to produce a given output element in the output matrix may not be in contiguous memory addresses in the memory address space. For example, to compute the top-left output location A' in the padded 2D convolution of Figure 2, this may require that the input elements at locations A, B, E, and F be retrieved from memory, but when stored in the NHWC memory layout, as shown in Figure 4, these are not in a contiguous portion of the address space because they are separated by the elements at input locations C and D. Each kernel location may require a different, custom-tailored subset of elements to be extracted from the data structure defining the input matrix in memory.

[0079] Figure 5 illustrates one approach to addressing this problem, called im2row. In im2row, before performing the 2D convolution operation itself, the input matrix structure representing the input channels is first rearranged to generate several rows 2 of data stored in a different portion of the address space than the original input data structure. Each row 2 corresponds to the data to be operated on by the kernel matrix for a particular output element location in the output matrix. For example, for output location A', the necessary elements A, B, E, and F of each input channel can be gathered together and combined with appropriate padding so that they are in the correct locations corresponding to the ordering of kernel elements K1 through K9. This means that subsequent matrix processing operations can simply multiply each kernel element of multiple kernel channels by the corresponding data in the corresponding location in row 2 and add the resulting products to generate the data for that output location. Note that a given row 2 has respective input values ​​for each of the input channels IC located adjacent to each other, which are operated on by respective kernel values ​​for the same kernel location in different kernel channels.

[0080] Similarly, for each other output position in the output matrix, a different row 2 is generated by collecting each of the input elements required to generate that output position. * KW * OH for additional data containing IC elements * This requires generating 2D rows. While this can create a lot of overhead in extracting respective subsets of elements from data stored in memory and copying them elsewhere in memory to generate the rows, this can greatly simplify subsequent 2D convolution operations, after which kernel values ​​can be applied directly to contiguous memory blocks in a matrix processing operation to generate the corresponding output matrix.

[0081] However, this approach has several problems. One problem is that the performance of matrix processing operations implemented in a given data processing system is increasingly improving. As matrix processing performance improves, Amdahl's Law implies that other operations executed alongside the matrix processing operation have an increasingly significant impact on overall performance. Even if the matrix processing operation itself can continue to improve in performance, the full benefit of performance improvements in matrix processing cannot be realized if other operations, such as the im2row operation shown in Figure 5, cannot show the same performance improvement as the matrix processing operation (because the im2row operation is memory bandwidth limited). Therefore, the overhead of executing im2row as shown in Figure 5 is becoming increasingly unacceptable in some processing applications. Another problem is that these remapping operations consume a lot of memory. For example, note that in the example in Figure 5, the input matrix element at position F is shown multiple times within row 2. Therefore, this also wastes memory address space due to duplication of input values ​​simply to provide the appropriate relative position of the input matrix to the kernel matrix. For example, im2row can require as much as 8 to 9 times the memory of the original input data structure in some machine learning algorithms.

[0082] Another type of convolution is the 1×1 convolution, which is similar to the 2D convolution described above, but instead of having a two-dimensional extent, has a kernel that is a 1×1 matrix. With a 1×1 kernel, the result of a 1×1 convolution is simply an output matrix where each element corresponds to the result of multiplying the corresponding element of the input matrix by the same kernel element. As shown in Figure 6, by using a series of 1×1 convolutions, it is possible to produce the same result as a 2D convolution by accumulating the results of several 1×1 convolutions with a relative shift in the position where the result of a given 1×1 convolution is added to the result from the previous 1×1 convolution.

[0083] In the 2D convolution example above, the product-sum computations are shown separately for each location in the output matrix, with each product group being for a different pair of input / kernel locations but the same output location.

[0084] However, it is also possible to divide the multiplications into different groups, with the set of multiplications associated with a single kernel position considered a group, which generates one of the products to be summed for each output position. Considering the example of Figure 2, if we consider a single kernel position, such as position K1, that kernel value K1 needs to be multiplied by a padding value when generating output value A', by input value G when generating output value L', and by input value I when generating output element N'. Thus, the top part of Figure 6 shows the relationship between the input elements multiplied by K1 to form the single partial product used in the summation for each of the corresponding output elements A'-P' in the output matrix.

[0085] Similarly, for each other kernel position K2 through K9, it can be determined which input element (or padding value) that kernel element should be multiplied with to generate another of the summed products for each of the output positions. Note that a given input matrix element contributes to a different element of the output matrix for each kernel position. For example, consider input element F, which when multiplied with kernel element K1 contributes to output element K', when multiplied with kernel element K2 contributes to output element J', when multiplied with kernel element K3 contributes to output element I', and so on until F when multiplied with kernel element K9 contributes to output element A'.

[0086] Thus, between each kernel element position, there is a relative shift between the position of a given output element in the output matrix and the positions of the corresponding input elements that contribute to that given output element for that particular kernel element position. For example, the shift of the effective input matrix between a K1 multiplication and a K2 multiplication is a shift to the left by one column position.

[0087] This means that by performing a series of 1x1 convolutions and accumulating the result of each 1x1 convolution into an accumulator matrix that represents a running sum of the output matrix, the result can be equivalent to the result of a 2D convolution operation performed over a kernel size greater than 1x1. For example, the result of each of the K2 multiplications shown is the result of K1 multiplications (e.g., K1 in a K1 1x1 convolution). * K2 at position F' set based on A * The result of each of the K3 multiplications may then be added to the corresponding element of the accumulator matrix that is the result of K1 and K2 multiplications (K3 * The result of C is that F' is K1 * A+K2 * B+K3 * (The sum of the input and output elements F' is added to the accumulated value of the output element F' so that it equals C.) This continues for each successive kernel position, so that by the end of the ninth 1×1 convolution operation, the output matrix has the same result as if the 2D convolution operation had been performed with a 3×3 kernel matrix. It will be appreciated that computing the 1×1 convolutions in the order K1, K2, K3, ..., K9 shown in FIG. 6 is not required, and any order of kernel points can be used. However, when position shifting examples are used as described below, computing adjacent kernel positions consecutively can help improve performance because there is a smaller shift between the input positions used to compute a given output position of successive 1×1 convolutions, and therefore can facilitate more frequent reuse of data loaded from memory across multiple 1×1 convolutions when the variable position shifting technique described below with respect to FIG. 8 is used.

[0088] As shown in Figure 7, the advantage of using the split 1x1 convolution technique shown in Figure 6 is that this means that the multiplications required for a given kernel position Kn can be applied to data loaded from a memory block that is either a single contiguous memory block, or several such contiguous blocks separated by regular stride intervals. This means that the 1x1 convolution operation can be applied directly to data in a format similar to the in-memory data structure, without the need for the performance-intensive and memory-hungry im2row technique shown in Figure 5.

[0089] FIG. 7 illustrates how a 1×1 convolution, similar to the previous example, can be extended to process multiple input and output channels. FIG. 7 illustrates the matrix multiplication operations for computing the set of products corresponding to a single kernel position in the x and y dimensions, e.g., kernel position K1 in the example of FIG. 7. That is, FIG. 7 illustrates the computation of the products for only the top portion of FIG. 6, but extended to process multiple input / output channels. It will be appreciated that similar operations may then be performed for each of the other kernel positions.

[0090] 7 shows an example for implementing part of a 2D convolution operation where there is intersection between input channels to generate each output channel (i.e., the results of the 2D convolutions applied to each kernel / input channel pair are summed to obtain the matrix for a given output channel). This means that for a 1×1 convolution corresponding to a given kernel point K1, the value at a given location F′ in a given output channel is the sum of products ΣK1 i * A i where i is incremented across all input channels and K1 i are the kernel values ​​at the corresponding positions in each kernel channel, and A i means that x is the input element at the corresponding position in each input channel. The corresponding operation can be performed in parallel on several different sets of kernel channels (to allow multiple features to be detected in parallel) to produce multiple output channels.

[0091] Thus, as shown in FIG. 7, a 1×1 convolution for a given kernel position K1 when evaluated across multiple input / output channels can be expanded to become a matrix multiplication operation, which multiplies a ZxIC input matrix 10, providing a set of Z input element values ​​A through K for each of the IC input channels, by an ICxOC kernel matrix 11, providing a set of kernel values ​​for kernel position K1 for each IC input channel within each of the OC sets of distinct kernel channels corresponding to the respective output channels. The result of the matrix multiplication is a ZxOC output matrix 12, providing a set of Z output elements F' through P' for each output channel OC. The Z dimension of the input / output matrices 10, 12 changes depending on which kernel position Kn is being processed; for K1, the range of required unpadded element positions spans A through K, while for a different element position (e.g., K2), the range of unpadded element positions may be larger (e.g., extending from A to L). Also, if non-zero padding values ​​are used, additional matrix rows may be required in the input / output matrices to accommodate the non-zero padding.

[0092] Because each row of input matrix 10 contains a set of elements for a single x-y location in the input matrix across each of the IC input channels, input matrix 10 can be loaded directly from memory from a data structure laid out as shown in FIG. 4. For example, the top row of input matrix 10 provides the "A" elements for each of the different input channels (e.g., x=0, y=0), the next row of input matrix 10 provides all the "B" elements (x=0, y=1), and so on. Thus, when data is arranged in memory in an NHWC layout as shown in FIG. 4, this input matrix 10 simply corresponds exactly to the format of the data stored in memory and can therefore be loaded as a single contiguous memory block. Alternatively, the number of input channels IC that can be processed in a single operation by the processing hardware can be determined by the number of channels C used in the matrix structure stored in memory. maxIf the actual number of inputs is less than the number of inputs, the input matrix 10 can accommodate any number of discontinuous chunks separated by a fixed stride, which is still much easier to load from memory than if the 2D convolution were performed in the manner shown in Figure 2, which requires a large number of irregularly patterned memory accesses as shown in the im2row example. Thus, the 1 × 1 convolution technique means that no remapping of the matrix structure stored in memory is required before performing the multiplications to compute the 1 × 1 convolution.

[0093] Similarly, the output matrix 12 has a layout that corresponds to the input matrix 10, so that once all the 1×1 convolutions of the 2D convolution have been accumulated together, the result can be written directly back into an in-memory matrix data structure laid out as in Figure 4.

[0094] Considering the upper-left kernel weight K1, as shown at the top of Figure 6, the relative shift between the input and output positions is such that row A of the input matrix is ​​multiplied by kernel weight K1 to produce the output in row F of the output matrix, row B of the input matrix contributes to row G of the output matrix, and so on. This generally works for most rows because there is a constant downward shift of five row positions between the input and output matrices for the K1 weight example. However, there are some rows D, H of the input matrix that multiply these rows by kernel weights and accumulate the results at corresponding shift positions I', M' in the output matrix, as shown in Figure 6. This means that the leftmost elements of the output matrix are updated based on multiplication with elements at the opposite right end of the input matrix, which is inaccurate for 2D convolutions. This problem is sometimes referred to as the "wraparound" problem. The wraparound problem can be avoided by splitting the matrix multiplication between matrices 10 and 11 shown in Figure 7 into several separate operations, each corresponding to a chunk of input matrix 10 containing only a block of rows A-C (or E-G or I-K), all of which need to contribute to the output matrix, but this requires additional instructions to be executed, reducing performance.

[0095] Therefore, it may be useful to support a masking operation that allows certain rows of the input to be skipped when generating the output, so that the 1 × 1 convolution can be applied to more rows, even if some selected rows encounter wraparound problems. This is indicated by the "X" marked on the path between input rows D and H and output rows I' and M'. The masking operation can be controlled by masking state data that defines the location of the masked row (or masked column, if the matrix is ​​instead placed with the input elements for a given input channel position extending within the same column). An example of encoding the masking state data is described below. The masking operation can be performed when loading data from memory into a register (so that, instead of loading the actual data elements from memory, a masking value is instead loaded into the corresponding portion of the operand storage device for storing information for forming the input channel matrix 10). Alternatively, the masking operation can be performed when performing the matrix processing operation itself, so that when the matrix processing circuitry reads operands for processing, predication is applied to mask out rows of elements that are read, ensuring that the matrix processing circuitry treats those elements as if they represent the masked value rather than the actual values ​​stored in the operand storage. The masked value may be zero, or may be non-zero if zeros are represented using non-zero values. In either case, this means that wraparound issues are prevented from causing errors, which allows 1 × 1 convolutions to be performed with fewer instructions, since they can be applied to matrix sizes larger than a block of contiguous rows without encountering wraparound issues.

[0096] For the other kernel weight positions K2-K9, matrix multiplication operations similar to those shown in FIG. 7 for K1 can be performed and the results accumulated together.

[0097] FIG. 8 illustrates a further observation that can be used to improve performance by reducing the number of times data must be loaded from an in-memory matrix data structure to perform 1×1 convolutions for a set of kernel weight positions. From FIG. 6, it is observed that when evaluating each 1×1 convolution for different kernel positions within the same row, the input matrices required for each of those kernel positions are very similar. For example, FIG. 8 shows input matrix 10 for center-left, center, and center-right kernel positions K4, K5, and K6, respectively. For center kernel weight K5, the input matrix is ​​precisely aligned with the output matrices for each of the other positions in input / output matrices 10 and 12, such that kernel weight K5 is multiplied with position A when producing output A, multiplied with position B when producing output B, and so on.

[0098] For center-left kernel position K4, K4 needs to be multiplied with element A of the input matrix when producing output element B (because K4 is multiplied with A when the center position of kernel K5 is above element B). Similarly, for each of the other positions in the input / output matrices 10, 12, there is a one position shift between the input and output elements.

[0099] Similarly, center-right kernel position K6 must be multiplied with input element B to produce output element A, and with input element C to produce output element B, and so on.

[0100] As shown in Figure 8, for the center-left and center-right positions, there are some rows that are skipped due to the wraparound problem described with respect to Figure 7, and the specific locations of the skipped rows vary depending on the kernel weight position (e.g., the skipped input rows for K4 are rows D, H, L, but for K6 they are rows E, I, M, and there are no skipped input rows for K5).

[0101] However, it will be appreciated that in general, the input data for rows A-P of input matrix 10 is essentially the same for each of the three kernel weight positions K4, K5, K6, except that for center-left position K4, input matrix 10 is shifted down one row position relative to the output, relative to center position K5, so that input row A is used to generate output row B instead of generating row A as in center position K5. Similarly, for the center-right position, input matrix 10 is shifted up one row relative to output matrix 12, such that input row B is fed to output row A.

[0102] Thus, by providing circuitry that performs variable position shifting of the input relative to the output, it is possible to tailor which rows of the output matrix are updated based on specific rows of the input matrix, and by supporting multiple different alternative shift amounts that can be selected, it is observed that this allows a block of matrix data loaded from memory to be reused for 1×1 convolutions of multiple different kernel locations. This means that the memory bandwidth associated with loading input rows A-P can be amortized across multiple different matrix processing operations, thereby significantly improving performance. When this position shifting is used, the position of masked rows to handle wraparound issues will be different for each kernel location, and therefore masking is required at the time of reading operands previously loaded from registers or matrix transpose boxes.

[0103] Data processing device supporting matrix processing FIG. 9 illustrates a schematic diagram of an example of a data processing device 20. The data processing device has a processing pipeline 24 including several pipeline stages. In this example, the pipeline stages include a fetch stage 26 for fetching instructions from an instruction cache 28; a decode stage 30 for decoding the fetched program instructions to generate micro-ops that are processed by the remaining stages of the pipeline; an issue stage 32 for checking whether operands required for the micro-op are available in a register file 34 and for issuing the micro-op for execution once the operands required for a given micro-op are available; an execution stage 36 for performing a data processing operation corresponding to the micro-op by processing operands read from the register file 34 to generate a result value; and a writeback stage 38 for writing the results of the operation back to the register file 34. It will be appreciated that this is only one example of a possible pipeline architecture, and other systems may have additional stages or different configurations of stages. For example, an out-of-order processor may include a register renaming stage for mapping architectural registers specified by a program instruction or micro-op to physical register specifiers that identify physical registers in the register file 34.

[0104] The execution stage 36 includes several processing units for performing different classes of processing operations. For example, the execution units may include a scalar arithmetic / logic unit (ALU) 40 for performing arithmetic or logical operations on scalar operands read from the registers 34, a floating-point unit 42 for performing operations on floating-point values, a branch unit 44 for evaluating the results of branch operations and adjusting a program counter representing the current execution point accordingly, a matrix processing unit 46 for matrix processing (described in more detail below), and a load / store unit 48 for performing load / store operations to access data in the memory systems 28, 50, 52, 54.

[0105] In this example, the memory system includes a level 1 data cache 50, a level 1 instruction cache 28, a shared level 2 cache 52, and a main system memory 54. It will be understood that this is only one example of a possible memory hierarchy and that other arrangements of caches can be provided. The particular types of processing units 40-48 shown in the execution stage 36 are merely an example; other implementations may have different sets of processing units or may include multiple instances of the same type of processing unit such that multiple micro-operations of the same type can be processed in parallel. It will be understood that FIG. 1 is only a simplified representation of some components of a possible processor pipeline architecture and that a processor may include many other elements that are not shown for the sake of brevity.

[0106] In some implementations, data processing device 20 may be a multiprocessor device that includes multiple CPUs (central processing units, or processor cores) 60, each having a processing pipeline 24 similar to that shown for one of CPUs 60 in Figure 9. Device 20 may also include at least one graphics processing unit (GPU) 62 and / or other master devices 64 that can communicate with each other and with the CPUs via an interconnect 66 used to access memory 54.

[0107] One approach to supporting matrix processing operations may be to decompose the individual multiplications of a given matrix processing operation into separate integer or vector instructions that can be processed on the processing pipeline 24 of a given CPU 60. However, this may be relatively slow.

[0108] Another approach to accelerating matrix processing may be to provide a hardware accelerator with dedicated hardware designed to process matrix operations as one of the devices 64 connected to the interconnect 66. To interact with such a hardware accelerator, the CPU 24 executes load / store instructions using the load / store unit 48 to define the matrix operands to be read from memory by the hardware accelerator and to write configuration data to the hardware accelerator that defines the processing operations to be applied to the operands. The CPU can then read the results of the matrix processing from the hardware accelerator using a load instruction that specifies an address mapped to a register in the hardware accelerator. While this approach may be faster than using integer operations in a pipeline, there may nevertheless be overhead associated with using a load / store mechanism to transfer information between the general-purpose processor 60 and the hardware accelerator 64, and the hardware accelerator approach may pose challenges when different virtual machines running on the same processing system need to share access to the hardware accelerator. Therefore, this approach may not scale well in virtualization implementations with several virtual machines.

[0109] 9, a matrix processing circuit 46 may be provided within the normal processing pipeline 24 of a given CPU 60, which can be controlled to perform matrix operations in response to matrix operation program instructions decoded by the pipeline's decode stage 30 (similar to using the ALU 40 or floating point unit 42 to control normal integer or floating point operations). This eliminates the need to transfer data back and forth between the CPU 60 and the hardware accelerator, making it much easier to enable many different virtual machines to perform matrix operations.

[0110] Although FIG. 9 shows a multiprocessor device 20 having several CPUs 60, this is not required and the matrix processing circuitry 46 can also be implemented in a single core system.

[0111] 10 shows in more detail a portion of the matrix processing circuit 46 and associated registers for supporting matrix processing. The matrix processing circuit 46 may include an operand storage circuit including a set of input operand registers 70, a set of output matrix registers 72, and a matrix transpose circuit 74 (hereinafter referred to as the matrix transpose box). The matrix processing circuit also includes a matrix load circuit 80 for handling the loading of data from a matrix structure in memory into the operand storage circuits 70, 74, an operand movement circuit 82 for moving operand data between the matrix transpose box 74 and the input operand registers 70, and matrix processing logic 84 for performing the matrix processing operation itself on the input operands stored in the input operand registers 70 to generate a two-dimensional result matrix stored in the output matrix registers 72.

[0112] The matrix transpose box 74 includes several storage elements 88, each for storing a different matrix element of a given operand (input) matrix. The storage elements 88 are logically arranged in rows and columns so that they can be accessed either as a row group 90, in which all of the storage elements 88 corresponding to the same row of the input matrix are readable / writable, or as a column group 92, in which all of the storage elements 88 corresponding to the same column of the input matrix are readable / writable. The physical arrangement of the storage elements 88 on the integrated circuit need not follow the logical arrangement of rows and columns and can have any physical arrangement. The ability to read or write elements 88 in row groups 90 and column groups 92 is instead provided by providing read / write ports and multiplexing circuitry, which allows the associated elements corresponding to a given row or a given column to be read regardless of their physical location within the chip.

[0113] This means that when loading data from the matrix data structure into memory, the matrix load circuit 80 can select (in response to row / column direction select parameter 89) whether to load individual row groups 90 of the matrix transpose box 74 or individual column groups 92 with data from a portion of the matrix structure in memory selected based on addressing information 94. A load instruction 98 decoded by the instruction decoder 30 to control the matrix load circuit 80 can specify a row / column ID 99 that identifies which particular row or column is to be loaded. The instruction can specify the row / column ID 99 directly as an immediate parameter or indirectly by specifying a register containing the row / column ID 99.

[0114] The row / column selection parameter 89 can be explicitly coded in the load instruction 98 using a field in the instruction encoding that selects whether a row group 90 or a column group 92 of the matrix transpose box 74 is loaded with data from memory. Alternatively, the row / column direction selection parameter can be coded implicitly. For example, there may be a control parameter stored in a control register that specifies whether the matrix load instruction 98 should currently select whether a row or a column of the matrix transpose box 74 should be loaded. The control parameter in the control register can switch state when a row / column direction switch instruction is executed. This eliminates the need for every matrix load instruction to specify an explicit row / column direction selection parameter. Alternatively, both the parameter specified in the instruction encoding and the parameter stored in the control register can be used, and in the instruction encoding, a combination of a control register bit and a row / column selection bit selects which row / column direction to use. For example, a control register bit can indicate whether a row / column is selected, but a bit in the instruction encoding can select whether the bit in the control register is inverted, e.g.,

[0115] [Table 1] Of course, other encodings may be used instead, this is just an example.

[0116] The load circuit 80 also selects whether to replace the value loaded into the matrix transpose box 74 with a masked value instead of the value loaded from memory in response to the masking state information 96, 97. In this example, the masking state information includes first masking state information 96 and second masking state information 97.

[0117] The first masking state information 96 is used to control the masking of specific row / column positions so that the corresponding row / column group in the matrix transpose box 74 is not updated based on the corresponding value in the memory. For each row / column position in the matrix transpose box 74, the first masking state information 96 identifies whether the row / column position is a masked row / column position or an unmasked row / column position. That is, if the row / column selection parameters 89 indicate that elements should be written row by row, the masking indication of the first masking state information corresponds to a different row position. If the row / column selection parameters 89 indicate that elements should be written column by column in the matrix transpose box 74, the masking indication of the first masking state information corresponds to a different column position.

[0118] If the first masking state information 96 specifies that the target row / column to be loaded is an unmasked row / column, the second masking state information 98 can be used to identify which individual element positions within the target row / column are masked, and the matrix load circuit 80 retrieves corresponding data from a matrix structure stored in memory and writes the unmasked elements of the target row / column to corresponding elements 88 of the selected row / column group of the matrix transpose box 74 (instead, the masked-out elements within the selected row / column group are set to a masked value). Thus, the second masking state information 98 may provide a set of masking indications, each corresponding to a different position extending in the opposite dimension from the position associated with the masking indication of the first masking state information. That is, if the row / column selection parameter 89 indicates that an element should be written to a row, the masking indication of the second masking state information corresponds to a different column position. If the row / column selection parameter 89 indicates that elements should be written to the matrix transpose box 74 column-by-column, the masking indications of the first masking state information correspond to different column positions.

[0119] The first and second masking state information 96, 97 together represent two-dimensional masking state information, as they indicate the positions of masked elements across two dimensions of the matrix to be loaded into the matrix transpose box 74. However, each individual instruction only uses the portion of the first masking state information that corresponds to a single target row / column (the portions of the first masking state information relating to other rows / columns are ignored). Nevertheless, the first and second masking state information 96, 97 may together define masking positions across the entire 2D matrix transpose box, such that the masking state data does not need to be changed between loading one row / column and loading the next row / column.

[0120] On the other hand, if the selected row / column location is indicated by the first masking state information 96 to be a masked row / column location, then instead of supplying the data loaded from memory, a masking value is written to each of the matrix elements 88 in the selected row / column. Here, each element in the selected row / column may share the same item of first masking state data 96, either identifying all elements in the selected row / column as masked or identifying all matrix elements 88 in the selected row / column as unmasked. If the load instruction specifies a masked row / column, then in response to the masking state information 96, the matrix load circuit 80 instead writes a masking value to each of the elements in the masked row / column.

[0121] Whether a masking value is supplied to a particular element 88 of the matrix transpose box 74 due to masking of the entire row based on the first masking state data 96 or masking of an individual element based on the second masking state data 97, the masking value may be a predetermined value such as zero, or may be one of several alternative masking values ​​selectable based on masking selection information that may be stored in a register or in a parameter explicitly specified by the load instruction.

[0122] The addressing information 94 may be stored in the CPU's general purpose registers 34, which are also used for general integer operands, or in some instances may be stored in some dedicated matrix addressing information registers that store information specific to identifying the portion of the matrix structure being loaded from memory.

[0123] Figures 11-13 show some examples of how the masking state information and addressing information 94 may be encoded. In the example of Figure 11, the addressing information 94 is specified within general purpose registers 34 that are also used for integer operands. In this case, before executing a matrix load instruction 98, a previous instruction may need to ensure that the referenced general purpose register contains the appropriate address operand to represent the address of the required row or column of the matrix, and between executions of successive load instructions 98 that target different rows of the input matrix, these address operands may need to be updated to point to the next row or column.

[0124] 11, the first masking state information (mask1) 96 is represented as a bitmap including several bit flag indicators 100, each corresponding to a given row / column position in the matrix transpose box 74. A row / column number 99 specified by the load instruction 98 is used to select which bit flag indicator 100 in the masking bitmap 96 to read, and depending on the value of the read bit flag 100, controls whether the corresponding row is masked or not (e.g., a bit flag of 1 can indicate an unmasked row / column, and a bit flag of 0 can indicate a masked row / column, or vice versa).

[0125] Similarly, the second masking state information (mask2) 97 is represented as a bitmap including several bit flag indicators 101 each corresponding to a column / row position (the opposite dimension to the position indicated by each bit flag indicator 100 in mask1 bitmap 96), so that mask2 indicates the position of each masked element in the target row / column having the row / column number 99 specified by the load instruction 98 as described above.

[0126] The registers that store the first / second masking state information 96, 97 can be dedicated registers for storing masking state information for masking matrix operands / processing (and serve no other purpose), or they can serve a dual function such that the same registers are also used for other information when processing instructions other than matrix processing related instructions. For example, the masking state information 96, 97 can be read from a predicate register, which can also be used to store vector predicates that control the masking of lanes of vector processing when a vector instruction is executed.

[0127] 12 shows another example in which the first / second masking state information 96, 97 is again represented as a bitmap, as in FIG. 11. In this case, however, the matrix processing circuitry has access to a set of matrix addressing registers 102 that specify at least a base address 104 and a stride value 106, and optionally a row / column offset (lower portion selection information) 108. In this approach, the addressing information registers 102 can be set before executing a group of loads to load all of the rows or columns of a given input matrix, and the matrix load circuitry 80 can calculate the addresses of the individual rows / columns based on the addressing information 102 and row / column selection number 99 specified by the load instruction 98, so there is no need to change the addressing information 102 between individual loads of different rows or columns in the same input matrix. 4 , the base address 104 can be set to point to the start of a memory region corresponding to the portion of the matrix being processed, and the stride value 106 can be set to reference the offset between the address marking the start of one row of the matrix data structure and the address marking the start of the next row (or column, if a column-major layout is used instead). The intra-row / column offset 108 can be used to select individual portions within a row of the entire matrix structure stored in memory, which can be useful when the entire matrix structure in memory is larger than the maximum row / column length supported by the hardware in the transpose box 74 and matrix processing logic 84. This allows the processing of large data structures in memory to be divided into smaller chunks that can be processed by the hardware in multiple passes. Thus, the intra-row / column offset can select individual portions within a “row” stored in memory. Supporting the row / column offset value 108 is not required; alternatively, between processing one chunk of a given row and processing the next chunk, the base address 104 may be updated to point to the location of the next chunk instead of updating the row / column offset value 108. Also, the offset value 108 may instead be provided in a general-purpose register referenced as a source register by the load instruction.

[0128] In this approach, when processing an individual load instruction 98, the matrix load circuitry 80 can calculate the address of the portion of data to be loaded into the selected row or column of the matrix transpose box 74 by adding the base address to the product of the stride value 106 and the row / column number 99 specified by the instruction, optionally offset by an intra-row / column offset value 108 as needed.

[0129] 13 shows another example representing addressing information 94 and masking state information 96, 97. In this example, addressing information 94 again includes base address 104, but this time the addressing information also includes offset data structure 110 stored in memory at a location identified by offset structure base address 112. Here, offset data structure 110 stored in memory functions both as part of addressing information 94 and as first masking state information 96. Second masking state information 97 may still be provided as a separate mask register "mask2," similar to the examples of FIGS. 11 and 12.

[0130] The offset data structure 110 defines an array of offset values, with each offset 114 corresponding to a particular row / column number that can be selected by an individual matrix load instruction 98. When a load instruction specifies a given row / column number (e.g., column 2 in the example shown in FIG. 10), a corresponding offset value 114-2 for that column is selected, and the address of the corresponding row / column of data in the matrix structure stored in memory can be derived by adding the selected offset value to the base address stored in the base address register 104. In most cases where the selected row / column is designated as an unmasked row or column, the load will proceed normally.

[0131] However, certain offset values ​​cannot be used for valid offsets and are instead reserved to indicate masked row / column locations. For example, a reserved offset value may be -1 (i.e., a binary value with the most significant bit being 1 and all other bits set to 0 for complementary representation). Thus, when calculating the address of an individual load instruction, if the selected offset value 114-2 for a selected row / column number is determined to have a reserved value, this is interpreted as a masked row or column location, and therefore, instead of performing an actual load from a portion of the matrix data structure stored in memory, the associated row or column group 90, 92 of the matrix transpose box 74 is filled with each element 88 of that row having a masking value, e.g., 0.

[0132] Thus, in this approach, the offsets that define the locations in memory where each row or column of the input matrix is ​​loaded into the matrix transpose box also serve as masking state information, avoiding the need for a separate register for the masking state values.

[0133] An advantage of using the array 110 of offset values ​​114 as part of the addressing information is that it requires much less storage capacity, as the offsets can be indicated relative to a common base address and therefore can be represented using fewer bits, compared to an alternative approach of storing in memory a table of absolute addresses indicating the address of each row / column of matrix data. Nevertheless, other implementations may omit the base register 104 of the example of Figure 13, such that each offset is effectively an offset relative to zero, but this would require more bits for each offset value 114.

[0134] Also, the use of special reserved values ​​in offset field 110 to represent masked row / column locations may be more efficient than if padding were instead supported by storing padding values ​​in memory itself and representing the masked row / column by specifying, in the field of offset array 110 corresponding to the masked row / column, an offset value that points to the actual location in memory where the padding value is stored. With the special reserved value approach, there is no need to perform an actual memory load to obtain the padding value, since the padding value may instead be generated on the fly by load circuitry 80 based on detection of the reserved offset value.

[0135] 13 shows an example in which the offset structure 110 is stored in a memory system at an address derived from the offset structure base address 112, some microarchitecture designs may choose to provide hardware with an offset cache 116 that can cache the values ​​of the offset structure for faster access by the matrix load circuitry 80 to avoid the need to fetch them again from memory in the future. This recognizes that the pattern of offsets applied may be the same for multiple different positions in the matrix, and thus it is efficient to maintain the same offset structure that can be reused. However, other implementations may provide the architecturally necessary offset registers to store the offset structure 110, so that they do not need to allocate space in memory at all.

[0136] Regardless of how the particular masking state information 96, 97 and addressing information 94 are represented, this function allows the required portion of a matrix stored in memory to be loaded into the matrix transpose box 74 so that the 1×1 convolution of the aforementioned operation can be applied to that portion of the matrix. Masking allows certain lines of the input to be skipped, as shown in FIG. 7, to address wraparound issues. Also, by allowing certain rows or columns in a matrix to be masked, this can be useful for providing padding values ​​to address padded convolutions of the type shown in FIG. 2. Also, in some cases, a 2D convolution operation may be applied to a matrix having a width or height that is smaller than the maximum width or height supported by the hardware, so the masking state can be used to mask out the last unused rows or columns of the matrix.

[0137] After writing rows or columns of a given operand matrix to the matrix transpose box 74, the data can be read out by the operand shift circuit 82 in row or column groups and transferred to the input operand registers 70 ready for matrix processing. The operand shift circuit 82 is not limited to reading data from the matrix transpose box 74 in the same row / column direction as the data was loaded by the matrix load circuit 80. In fact, if the data structure stored in memory for the input operands is stored in a different row / column-major format compared to the output data structure, it may be useful for the operand shift circuit 82 to read the data in the opposite row / column direction from that used when loading. This on-the-fly transposition of the matrix as it is loaded into the matrix transpose box 74 and read out for processing can be performed in hardware much more efficiently than would be possible from remapping the data layout in memory. This can therefore significantly improve performance when processing input matrices of potentially different memory layouts.

[0138] Note that for any given memory layout of a matrix structure stored in memory, it is possible to load that same layout into the matrix transpose box 74 either column-by-column or row-by-row, so whether the row / column select parameter 89 specifies a row-wise or column-wise orientation can be selected completely independently of the actual layout used in the underlying matrix structure in memory. This is because, to transpose the orientation of a matrix using the matrix transpose box, it is irrelevant whether data is loaded column-wise and read row-wise, or whether data is loaded row-wise and read column-wise; both achieve the same result. In fact, when performing such on-the-fly transpositions, it may be useful to alternate between loading matrix data row-wise and loading them column-wise to achieve better pipelining of reading from earlier rows or columns of the matrix and loading later rows or columns of the matrix for processing.

[0139] For example, assume a sequence of operations in which a series of rows of a matrix structure in memory are loaded into rows 0 through 7 of matrix transpose box 74, but the output data structure to which they are combined has the opposite memory layout and is therefore read out column by column. In this case, after loading the final row 7 into the matrix transpose box, operand shift circuitry 82 can begin reading out columns one by one, starting with column 0 and ending with column 7. However, as soon as the data for column 0 is read out, matrix load circuitry 80 can begin loading further rows of the matrix structure from memory for the next chunk of the matrix to be processed, while operand shift circuitry 82 continues to read out successive columns 1 through 7 for processing by matrix processing object 84. Because columns 1 through 7 may still be needed by matrix processing logic 84, it is more efficient to begin loading those further rows of the matrix columns into respective columns 0, 1, 2, etc., as those columns become successively free for the operand shift circuitry to read them out for processing. Thus, loads of subsequent portions of the matrix can be loaded into respective columns of the matrix transpose box 74 at previous column positions 0 and 1 while readouts of subsequent columns associated with the previous chunk of the matrix are still in progress. For example, as the matrix is ​​moved by the operand shift circuit 82, once data for a particular column, e.g., column 2, has been read, loads to that column for the next pass can begin, thus enabling some performance improvement through pipelining. Then, once all columns have been loaded for the next chunk of the matrix in memory to be processed, the next set of operand shift operations performed by the operand shift circuit 82 can be performed row-by-row while loads proceed immediately behind to fill the row group 90 of the matrix transpose box just read out by the operand shift circuit 82. Thus, by alternating which direction is used for sets of loads (if on-the-fly transposition is used), it can be seen that this can provide better performance than if the same row / column direction were used throughout the entire matrix.

[0140] Alternatively, if a particular set of operations is being performed that does not require on-the-fly transposition of the matrix layout (e.g., because the output data structure has the same layout in memory as the input data structure), a fixed one of the row / column orientations can be chosen for both the matrix load and operand move operations. Nevertheless, pipelining can still be present so that operands can be read from a particular row / column for processing while loads are being performed to other rows / columns.

[0141] In the example of FIG. 10 , to limit the hardware complexity of matrix processing logic 84 and the latency associated with individual instructions, matrix processing logic 84 does not support performing a full matrix multiplication operation on two two-dimensional matrix operands within a single instruction; instead, such a 2D matrix multiplication operation can be decomposed into several separate cross-product and accumulate operations, each performed on a pair of one-dimensional vector operands. The example of FIG. 7 is used to explain the operation of the cross-product operation. In the example of FIG. 7 , to generate output matrix 12 from input matrix 10 and kernel matrix 11, the example of FIG. 7 requires matrix multiplication of 11×4 input matrix 12 by 4×4 kernel matrix 11 to give 11×4 output matrix 10. A full matrix multiplication operation requires that a given output element of output matrix 12 (e.g., the element labeled 200 in position F′ in FIG. 7 ) be generated based on the sum of pairwise products of each element in the corresponding row 202 of input matrix 10 and the corresponding element in the corresponding column 204 of kernel matrix 11. Since the matrix multiplication is being performed as part of a series of 1x1 convolutions that are accumulated to produce the equivalent of a larger 2D convolution, the result of adding the pairwise products of row 202 and column 204 is added to the previous value of output matrix 12 for element F' to produce an updated value for element F'.

[0142] However, such a matrix multiplication operation requires that four separate products be calculated for each output element position in the output matrix 12, followed by an addition of five terms (the four products and the previous value of the output element), which can be slow to implement and difficult to fit into pipeline timing of other operations.

[0143] In contrast, the cross product operation operates on a first vector operand u = (u1, u2, ..., u m ) and the second vector operand v=(v1,v2,...,v n ) and combine them to form a two-dimensional result matrix W, where

[0144]

number

[0145] For cross product and accumulate operations, each element of the updated result matrix W' is also the result matrix W:

[0146]

number

[0147] A full matrix multiplication operation can be decomposed into individual cross-product operations. For example, taking a vector operand 206 shown in FIG. 7 corresponding to one column of an 11×4 input matrix and a second vector operand 208 corresponding to one row of kernel matrix 11, multiplying each element of first vector operand 206 by the corresponding element of second vector operand 208 for each pair of column and row positions results in a 2D array of intermediate results; for example, element 200 identified in FIG. 7 results from the product of the element marked A in column 206 with the top-left K1 kernel position in row 208 extracted from kernel matrix 11. After each combination of input column and kernel row is processed, by performing iterations of the cross-product and accumulate operation over each respective combination of column position of input matrix 10 and row position of kernel matrix 11, the result is the same as if the full matrix multiplication operation had been performed, but at a lower hardware cost.

[0148] Thus, to support the cross product and accumulate operations performed by matrix processing logic 84, input operand registers 70 store one-dimensional vector operands, and operand shift circuit 82 reads out portions of the input matrix in matrix transpose box 74 row-by-row or column-by-column. Thus, even though a given underlying operand matrix on which operations are being performed is a two-dimensional matrix structure, at the time of applying the matrix processing operations it is treated as a series of one-dimensional vector operands, but matrix processing logic 84 can nevertheless generate a result matrix as a two-dimensional matrix structure within a single instruction that corresponds to the result of applying the cross product / accumulate operations to pairs of vector operands. This means that operation is still faster than if individual vector processing instructions were processed that could only generate a single row / column of the result matrix at a time.

[0149] In the example of Figure 10, the input registers 70 of the matrix processing logic 84 include two input registers A0, A1, each for storing a first vector operand, and two input registers B0, B1, each for storing a second vector operand. Also provided are four result matrix registers C0-C3 72, each capable of storing a result matrix of two-dimensional extent (although Figure 10 shows a square matrix of dimensions NxN, other examples may support different heights / widths of the result matrix). In some implementations, the matrix processing logic may be hardware-implemented with respect to which combination of input registers is used while generating the result matrix placed in a given result matrix register 72. For example, the result matrix registers C0-C3 may store the input operands A0, A1, B0, B1, and B1, respectively. * B0 and A0 * B1 and A1 * B0 and A1 *B1 and B2, respectively. This recognizes that when performing matrix processing, it is often necessary to process the same set of rows or columns of one input matrix with a corresponding set of rows or columns of a second input matrix in different combinations. For example, in the 1×1 combination example of FIG. 7 , column 206 of input matrix 10 not only needs to be multiplied with the elements of row 208 of kernel matrix 11 for the first cross product operation, but also with each element of the next row of kernel matrix 11 for the subsequent cross product operation, and so on for the remaining rows. Similarly, kernel row 208 may need to be multiplied with several different columns 206 in the input matrix. By providing input register storage 70 sufficient to store multiple rows or columns at once, different combinations of rows or columns of operand A and rows or columns of operand B can be implemented with a single set of operand load / move operations to fill registers 70, and then several different matrix processing operations on multiple different combinations of operands can be applied to those operands without having to repeat the load / move for each individual matrix processing operation. 10, which uses four output matrix registers, allows for an increased number of matrix processing instructions to be processed per matrix load instruction. Other examples may provide additional input / output registers 70, 72, although the exact number of registers selected may be a trade-off between hardware cost and performance.

[0150] Alternatively, other approaches may provide only enough input operand register storage 70 for a single vector operand pair, in which case a single pair of vector registers needs to be loaded with new values ​​for each different combination of input matrix rows / columns being multiplied.

[0151] Also, it is not necessary to provide separate register banks for the two operands A and B. In another example, both operands A and B may be selected from respective registers within a single combined register file.

[0152] 10, each matrix processing instruction 240 may specify a given result destination register 72, a pair of input vector registers 70 for providing the source operands for the operation, and control information including predicate (masking state) information 242 and shift select information 244. As explained above, in some implementations, the selection of the result matrix register 72 used for a given operation may be implicit from the combination of source registers 70 selected, so in this case the instruction may not need to specify a separate destination register identifier, although it may be useful to provide an additional destination register specifier if more arbitrary selection of the destination is permitted.

[0153] Figure 14 illustrates in more detail the matrix processing logic 84, including the use of predicate information 242 and shift select information 244. Figure 14 illustrates a vector cross product operation applied to a first vector operand 250 stored in a given one of the "A" input vector registers 70 of the operand storage device and a second vector operand 252 stored in a given one of the "B" input vector registers. For example, in the convolution example above, the "A" register may be used for the input matrix 10 and the B register may be used for the kernel weights 11.

[0154] The matrix processing logic 84 includes a position shift circuit 260 for applying a variable position shift between an element of one of the input operands 250 and a corresponding element position in the output matrix 270 generated in response to the matrix processing instruction 240. The shift information 244 may be represented as an explicit parameter in the matrix processing instruction 240 or by a control parameter stored in a control register. The shift parameter 244 specifies one of several variable shift amounts. Based on the selected shift amount, the position shift circuit activates several multiplexers to select which input element from the first vector operand 250 is provided to each element position in the shifted input operand 272. For example, if a variable shift amount of 0 is selected, each element of the input vector 250 passes through the element at the corresponding position in the shifted input vector 272, and if a variable shift amount of 1 is selected, the element at a given element position in the shifted input vector 272 is set to the value of the element at the next higher element position in the original input vector 250. For the element at the highest element position in the shifted input vector 272, a padding value 274 can be supplied because there is no higher element position in the original input vector to inject when a variable shift amount greater than 0 is selected. Similarly, for higher values ​​of the shift amount, a larger shift in position can be applied to adjust which position of the input vector 250 is supplied to the shifted position in the shifted input vector 272. The shift is not applied to the second vector operand 252, which is simply used in its original position.

[0155] Next, the matrix processing logic 84 computes the equation C'[i,j]=C[i,j]+P[i].A s hiftA cross product operation is performed to generate each element C'[i,j] according to B[i]×B[j], where i is repeated over all rows of the result matrix C'[i,j] and j is repeated over all columns of the result matrix C'[i,j]. Here, a predicate bit P[i] corresponding to a given row position i in the result matrix specifies whether the row is masked (inactive) or unmasked (active). In this example, inactive rows of output matrix 270 are indicated by a predicate bit equal to 0, and active rows are indicated by a predicate bit equal to 1, although it will be understood that other examples may have the opposite mapping of predicate values, such that inactive rows may be identified using a predicate bit equal to 1 and active rows may be identified by a predicate bit equal to 0. For inactive rows, in this example, the corresponding element of shifted input vector 272 is assumed to be replaced with a masking value of 0, although other examples may use non-zero masking values.

[0156] Thus, in this approach, the variable position shift provided by position shift circuit 260 helps support the approach shown in FIG. 8 , whereby once input operand register 70 is loaded with a particular vector 250 representing a given row or column of the input matrix, several matrix processing instructions specifying different values ​​of variable shift amount 244 can be executed, operating on the exact same contents of input vector 250 in register 70 to account for the relative position shift between input vector 250 and output matrix 270 required to apply kernel weights for different kernel positions, as shown in FIG. 8 . This eliminates the need to reload vector operand register 250 for each kernel position. Additionally, providing a predication function using predicate value 242 helps address the need to skip certain rows, as shown in FIG. 8 , to account for the wraparound issue discussed with respect to FIG. 7 . Predication can also help address cases where the number of column rows is insufficient to fill the entire vector supported by the hardware.

[0157] While Figure 14 shows the position shift circuitry 260 being provided during the reading of input vector operands 250 from a given input register 70 and providing the shifted operands to the matrix processing logic 84 to perform the cross product / accumulate operation, it is also possible to apply the position shift during the matrix processing logic 84 generating the result of the cross product / accumulate operation and writing the result back to the result matrix register 72, although this approach is slightly more complicated as it also requires a shift of some of the previous values ​​of the output matrix that are read as input to the cross product / accumulate operation when an accumulate operation is being performed (i.e., C[i,j] in the above equation).

[0158] 10-14 thus help matrix processing functions within a processing pipeline to more efficiently handle 2D convolution operations, which are very common in the field of machine learning. It will be appreciated that programmers may find other uses for the same functions, and they need not be used exclusively for such 2D convolution operations.

[0159] 10 shows a matrix transpose box 74 that is useful for enabling different layouts of the matrix structure in memory to be processed using the same instruction set regardless of their stored layout, the matrix transpose box 74 is not required, and some implementations may omit it. In this case, if there are differences in the memory layout of the input and output matrices, any transpositions must be handled separately by remapping the data stored in memory using load / store instructions before applying any matrix processing operations, or by converting its format before generating the output and then writing it back to the data structure in memory that corresponds to the output. If the matrix transpose box 74 is not provided, the matrix load circuit 80 may instead directly load the rows or columns of the matrix structure in memory into the input registers 70, which can be read by the matrix processing logic when performing the matrix processing operations.

[0160] Also, in some implementations, it may not be necessary at all to provide input operand registers 70, as if a matrix transpose box 74 were provided; another approach could be for the matrix processing logic 84 to read its operands directly from storage elements 88 of the matrix transpose box 74. Thus, while in general some operand storage circuitry is provided into which rows or columns of a matrix are loaded by the matrix load circuitry 80, from which operands can be obtained by the matrix processing logic 84, it is not necessary to provide both the matrix transpose box 74 and the input operand registers 70; they could be provided alone or in combination as in the example of FIG. 10.

[0161] While FIG. 10 shows an example applied to a square matrix with equal numbers of rows and columns in the matrix, this is not required and other examples can support asymmetric numbers of rows and columns.

[0162] Performance is best improved when both the row / column masking and position shifting functions described above are provided, although this is not required and some implementations may provide only one or the other of these functions.

[0163] 15 is a flow chart showing a method for processing a matrix load instruction in an example where masking is applied when performing the load operation. When such an instruction is encountered in step 300, in step 302, the instruction decoder 30 decodes the load instruction to generate a control signal that controls the matrix load circuit 80 to obtain first masking state data 96 from either an internal register in the CPU 60 (e.g., in the register bank 34 or in an internal register associated with the matrix load circuit 80), a data structure 110 in memory, or an offset cache 116. The first masking state data 96 is "entire row / column" masking state data that indicates whether an entire row / column is masked. It is not essential that the entire first masking state data 96 be obtained by the matrix load circuit 80; it may be sufficient to simply refer to the masking indication 100 or 114 corresponding to the row / column number 99 of the target row / column to be loaded. Thus, in step 304, the matrix load circuitry determines, based on the obtained first masking state data 96, whether the row / column number 99 specified by the matrix load instruction corresponds to a masked row or column position in the input matrix being processed. If the specified row / column is a masked row / column, in step 306, the corresponding portion of the operand storage circuitry 74, 70 corresponding to the target row / column is loaded with data having a masking value instead of actually performing a memory load of the corresponding portion of the matrix data structure stored in memory. The masking value can be selected from among several options based on a selection parameter encoded in the load instruction or specified elsewhere in a control register. Alternatively, some implementations may always default to a fixed masking value, such as 0.

[0164] On the other hand, if the target row or column location is not a masked row or column location, then in step 308, the matrix load circuit 80 obtains second masking state data 97, which is element-by-element masking state data indicating the location of any individual masked column / row location within the target row / column. In step 310, the matrix load circuit determines whether there are any active elements within the target row / column (even if the first masking state data 96 indicates that the target row / column is unmasked, the second masking state data 97 may have set all elements within the target row / column to inactive). If there is at least one active element in the target row / column, then in step 312, the matrix load circuit 80 triggers a load operation to read from memory a portion of the matrix data structure corresponding to the target row or column. The address where the data is loaded may be derived from the addressing information 94, for example, by adding the base address 104 to the multiplication of the row / column number and the specified stride 106 in the example of FIG. 12 . Upon retrieving the relevant data chunk from memory, for any active elements in that row or column, the loaded data is written to the corresponding storage element 88 of the matrix transpose box 74 or loaded directly into the corresponding portion of the selected input operand register 70. In contrast, for any inactive elements in the target row / column indicated by the second masking state data 97, the corresponding storage element 88 or portion of the selected input operand register 70 is filled with a masking value, which may also be zero or non-zero and may be fixed or programmably controlled.

[0165] If, in step 310, the matrix load circuit 80 determines that all elements in the target row / column are indicated as inactive by the second masking state data 97, then, in step 314, the load operation is prevented from being performed and each element of the target row / column in the operand storage circuit (i.e., the matrix transpose box 74 or the storage element 88 of the input operand register 70) is filled with the masking value without the need to perform any load from memory.

[0166] Although FIG. 15 shows two separate steps 302, 308 for obtaining the first and second masking state data 96, 97, other examples may obtain both masking state data 96, 97 in step 302 before checking whether the target row / column is masked by the first masking state data 96.

[0167] FIG. 16 illustrates a first example of processing a matrix processing instruction 240 in an embodiment that supports masking applied at the time of matrix processing. In step 320, the pipeline's instruction decoder 30 identifies the instruction being processed as a matrix processing instruction and generates control signals to control the matrix processing circuit 46 to process the instruction. In response to these control signals, in step 322, the matrix processing logic 84 obtains first and second operands that depend on information stored in the operand storage circuits 70, 74. As previously mentioned, these operands can be obtained directly from the matrix transpose box 74 or from the input operand registers 70. The matrix processing circuit also obtains masking state data 96 (e.g., predicate vector 242 as shown in FIG. 14) that indicates the masked row / column positions where the input values ​​are treated as if they represent masked values. In step 324, the matrix processing circuit 46 performs the matrix processing operation on the first and second operands to generate a two-dimensional result matrix that can be written back to one of the result matrix registers 72. For example, the operation may be a cross product and accumulate operation as described above where the first and second operands are vector operands. For any inactive rows / columns indicated as masked by masking state data 96, the corresponding elements of the result matrix may retain their previous values ​​or may be set to the values ​​that would result if the corresponding input values ​​were set to the masking values.

[0168] FIG. 17 illustrates a second example of processing a matrix processing instruction in an embodiment supporting the variable position shift functionality described with respect to FIGS. 8 and 14. Steps 320, 322, and 324 are similar to the corresponding steps in FIG. 16 (though the masking feature is not explicitly shown in FIG. 17, it may still be provided in some embodiments). However, FIG. 17 also supports the position shift functionality shown in FIG. 14. In step 326, one of several alternative shift amounts is selected by the matrix processing circuit 46 depending on the variable shift amount 244 specified by the matrix processing instruction. While FIG. 14 illustrates an example with three different possible shift amounts to correspond to the three options shown in FIG. 8, it will be understood that other implementations supporting larger kernel sizes may require more than three different selectable shift amounts. Alternatively, to limit the complexity of the position shift circuit 260, even if larger kernel sizes are supported, the position shift may be limited to a certain maximum size; if additional load is required to support larger kernel sizes, this is still possible.

[0169] Thus, in step 328, a variable position shift is applied by position shift circuit 260 based on the shift amount selected in step 326, thereby changing which row or column of 2D result matrix 270 is updated based on a given element of one of input operands 250. Next, in step 324 of Figure 17, a matrix processing operation is applied based on the variable position shift to generate result matrix 270.

[0170] So, in summary, these ideas help support more efficient hardware to support the processing of 2D convolution operations, which are common operations in the fields of machine learning and image processing. Further examples are described in the following clauses. (1) A device comprising: a matrix processing circuit that performs a matrix processing operation on the first input operand and the second input operand to generate a result matrix, the result matrix being a two-dimensional matrix; an operand storage circuit for storing information for forming a first input operand and a second input operand for the matrix processing circuit; a position shift circuit for applying a variable position shift to change which row or column of a result matrix is ​​updated based on a given element of one of the first and second input operands stored in the operand storage circuit during a given matrix processing operation, the variable position shift being based on one of a plurality of alternative shift amounts selectable for the given matrix processing operation, each alternative shift amount corresponding to a position shift of one of the first and second input operands relative to the result matrix by a different number of rows or columns; 1. An apparatus comprising: (2) The apparatus of clause (1), wherein the first input operand and the second input operand include one-dimensional vector operands. (3) The apparatus of clause (2), wherein the matrix processing operation includes a cross product operation applied to the first input operand and the second input operand to generate a result matrix. (4) The apparatus of clause (3), wherein the cross product operation includes a cross product and accumulation operation in which a result matrix includes an updated value for each element of the accumulator matrix, and the updated value for a given element of the accumulator matrix corresponds to the result of adding the previous value of the given element of the accumulator matrix to the corresponding element of the cross product result matrix that corresponds to the result of performing the cross product operation on the first input operand and the second input operand. (5) The apparatus of any one of clauses (1) to (4), wherein the position shift circuit is configured to select the one of a plurality of alternative shift amounts based on parameters specified by a matrix processing instruction for controlling the matrix processing circuit to perform a matrix processing operation. (6) The apparatus of any one of clauses (1) to (5), wherein when a given row or column of the result matrix corresponds to an active row or column position indicated by predicate information accessible to the matrix processing circuit, the matrix processing circuit is configured to generate an element of the given row or column of the result matrix having a value according to the corresponding row or column of the one of the first and second input operands, the corresponding row or column being selected according to one of a plurality of alternative shift amounts selected for the given matrix processing operation; and when the given row or column corresponds to an inactive row or column position indicated by the predicate information, the matrix processing circuit is configured to generate an element of the given row or column of the result matrix having a value independent of the corresponding row or column of the one of the first and second input operands. (7) The apparatus of any one of clauses (1) to (6), wherein the operand storage circuit comprises a matrix transpose circuit including a plurality of storage units for storing respective matrix elements of a given operand matrix, and the storage units of the matrix transpose circuit are readable in row groups corresponding to rows of the given operand matrix and also in column groups corresponding to columns of the given operand matrix. (8) The apparatus of clause (7), wherein when a given operand matrix is ​​written to the matrix transpose circuit of a row group, the matrix transpose circuit is configured to support readout of the given operand matrix from the matrix transpose circuit of a column group, and when a given operand matrix is ​​written to the matrix transpose circuit of a column group, the matrix transpose circuit is configured to support readout of the given operand matrix from the matrix transpose circuit of a row group. (9) the operand storage circuit includes operand registers that store first and second input operands for the matrix processing operation; 10. The apparatus of claim (7) or (8), further comprising: operand shift circuitry responsive to a shift instruction to read at least one row or column of a given operand matrix from the matrix transpose circuitry and write the at least one row or column to an operand register. (10) The apparatus of any one of clauses (7) to (9), further comprising an operand movement circuit responsive to a matrix processing instruction to read at least one row or column of a given operand matrix from the matrix transposition circuit and provide the at least one row or column to the matrix processing circuit as one of the first and second input operands. (11) The apparatus of any one of clauses (1) to (10), wherein a load circuit loads, in response to a load instruction, information corresponding to a target row or column of a given operand matrix into the operand storage circuit based on a portion of a matrix data structure stored in the memory; in response to the load instruction, the load circuit is configured to obtain masking state data indicating the location of one or more masked rows or columns in the given operand matrix; and when the target row or column corresponds to the masked row or column location indicated by the masking state data, the load circuit is configured to load a portion of the operand storage circuit corresponding to the target row or column with data having a masking value instead of data based on the portion of the matrix data structure stored in the memory. (12) The apparatus of any one of clauses (1) to (11), wherein the matrix processing circuit is configured to generate a result matrix from the first input operand and the second input operand in response to a single instruction. (13) An apparatus comprising: means for performing a matrix processing operation on first and second input operands to generate a result matrix, the result matrix being a two-dimensional matrix; means for storing information to form the first and second input operands for the performing means; and means for applying a variable position shift to change which row or column of the result matrix is ​​updated based on a given element of one of the first and second input operands stored in the storing means during a given matrix processing operation, the variable position shift being based on one of a plurality of alternative shift amounts selectable for the given matrix processing operation, each alternative shift amount corresponding to a position shift of one of the first and second input operands relative to the result matrix by a different number of rows or columns. (14) A data processing method comprising: performing a matrix processing operation on a first input operand and a second input operand to generate a result matrix, the result matrix being a two-dimensional matrix, the first input operand and the second input operand depending on information stored in an operand storage circuit; and applying a variable position shift to change which row or column of the result matrix is ​​updated during a given matrix processing operation based on a given element of one of the first and second input operands stored in the operand storage circuit, the variable position shift being based on one of a plurality of alternative shift amounts selectable for the given matrix processing operation, each alternative shift amount corresponding to a position shift of one of the first and second input operands relative to the result matrix by a different number of rows or columns.

[0171] In this application, the term "configured to..." is used to mean that elements of an apparatus have a configuration that is capable of performing a defined operation. In this context, "configuration" refers to a way of arranging or interconnecting hardware or software. For example, an apparatus may have dedicated hardware that provides the defined operation, or a processor or other processing device may be programmed to perform the function. "Configured to" does not imply that the apparatus elements need to be modified in any way to provide the defined operation. Although exemplary embodiments of the present invention are described in detail herein with reference to the accompanying drawings, it will be understood that the invention is not limited to these precise embodiments, and that various changes and modifications can be made to the embodiments by those skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. 1. An apparatus comprising: a matrix processing circuit configured to perform a matrix processing operation on a first input operand and a second input operand to generate a result matrix, the result matrix being a two-dimensional matrix; an operand storage circuit for storing information for forming the first input operand and the second input operand for a matrix processing circuit; a masking circuit for performing a masking operation to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuit based on masking state data indicating a masked row or column position of one of the first input operand and the second input operand, whose input value is processed to represent a masked value; An apparatus comprising:

2. The apparatus of claim 1 , wherein the masking value is zero.

3. The masking value is a masking value selection parameter specified by an instruction to perform the masking operation; a control value stored in a control register; 10. The apparatus of claim 1, wherein the masking value is selected from a plurality of masking values ​​in response to at least one of: a masking vector specifying distinct masking values ​​for multiple elements of the masked row / column.

4. The apparatus of claim 1 , wherein the masking state data comprises coded information that identifies elements within a two-dimensional array of elements that should be treated as representing the masking value.

5. The masking state data is first masking state data indicating one or more masked row or column positions where all elements at said masked row or column positions should be treated as representing said masked value; second masking status data indicating whether an individual element position within a given row or column should be masked; The apparatus of claim 4 , wherein the apparatus specifies:

6. 6. The apparatus of claim 1, wherein the masking state data comprises coding information capable of indicating as masked row or column positions at least two non-adjacent row or column positions separated by at least one unmasked row or column position.

7. 7. The apparatus of claim 1, wherein the operand storage circuitry comprises a matrix transpose circuitry including a plurality of storage units for storing respective matrix elements of a given operand matrix, the storage units of the matrix transpose circuitry being readable in row groups corresponding to rows of the given operand matrix and also in column groups corresponding to columns of the given operand matrix.

8. the matrix transpose circuit is configured to support reading out of the given operand matrix from the matrix transpose circuit by column groups when the given operand matrix is ​​written into the matrix transpose circuit by row groups; the matrix transpose circuit is configured to support reading out of the given operand matrix from the matrix transpose circuit by row groups when the given operand matrix is ​​written into the matrix transpose circuit by column groups.

8. The apparatus of claim 7.

9. the matrix processing circuit comprises the masking circuit, and in response to the masking state data, performs the matrix processing operation using the portion of either the first input operand or the second input operand corresponding to the one or more masked row or column positions that are treated as representing the masked value instead of an actual value of the portion of either the first input operand or the second input operand stored in the operand storage circuit.

9. An apparatus according to any one of claims 1 to 8.

10. a load circuit responsive to a load instruction to load information corresponding to a target row or column of a given operand matrix into the operand storage circuit based on a portion of a matrix data structure stored in memory; Equipped with 10. The apparatus of claim 1, wherein the loading circuitry includes the masking circuitry, and when the target row or column corresponds to a masked row or column position indicated by the masking state data, the loading circuitry is configured to load a portion of the operand storage circuitry corresponding to the target row or column with data having the masking value instead of data based on the portion of the matrix data structure stored in memory.

11. 11. The apparatus of claim 10, wherein, in response to the load instruction, if the masking state data corresponding to the target row or column indicates that the target row or column corresponds to a masked row or column position, the load circuitry is configured to determine whether each of the plurality of matrix elements of the target row or column should be masked based on a shared item of masking state data shared among a plurality of matrix elements of the target row or column.

12. the masking state data includes a plurality of offset values, each corresponding to a respective row or column position of the given operand matrix and indicating an offset of an address of a corresponding portion of the matrix data structure in memory relative to a base address; 12. The apparatus according to claim 10 or 11, wherein the positions of the masked rows or columns are indicated by the offset values ​​of the masked row or column positions having predetermined reserved offset values.

13. 13. The apparatus of claim 10, wherein the load circuitry is configured to retrieve the masking state data from memory based on masking state addressing information stored in at least one masking state addressing register.

14. 14. Apparatus according to any one of claims 11 to 13, wherein the load circuitry is configured to determine a target address of the portion of the matrix data structure in memory based on addressing information.

15. 15. The apparatus of claim 14, wherein the addressing information comprises a plurality of address pointers, each address pointer indicating an address of a portion of the matrix data structure corresponding to a respective row or column location of the given operand matrix.

16. The addressing information is a base address of the matrix data structure; and a stride value indicating a difference between an address of the portion of the matrix data structure corresponding to one row or column of the given operand matrix and an address of the portion of the matrix data structure corresponding to a next row or column of the given operand matrix.

17. The addressing information is a base address of the matrix data structure; offset information, a plurality of offset values, each corresponding to a respective row or column position of the given operand matrix and indicating an offset of an address of a corresponding portion of the matrix data structure in memory relative to the base address; and an offset data structure address indicating an address of a data structure in memory that provides the plurality of offset values.

18. 18. The apparatus of claim 14, wherein the addressing information further includes subportion selection information for selecting which subportion of the portion of the matrix data structure in memory identified based on the addressing information is to be loaded into the operand storage circuit.

19. at least one addressing register for storing said addressing information; 19. The apparatus of claim 14, further comprising: a prefetch circuit for generating a prefetch request to prefetch a portion of the given operand matrix from memory in response to the addressing information stored in the at least one addressing register.

20. The apparatus of claim 1 , wherein the first input operand and the second input operand are one-dimensional vector operands.

21. 21. The apparatus of claim 1, wherein the matrix processing operation comprises a cross product operation applied to the first input operand and the second input operand to generate the result matrix.

22. The apparatus of claim 21, wherein the result of the cross product operation is accumulated by an accumulation operation, and the updated value of a given element of the accumulator matrix corresponds to the result of adding the previous value of the given element of the accumulator matrix to the corresponding element of a cross product result matrix that corresponds to the result of performing the cross product operation on the first input operand and the second input operand.

23. 23. Apparatus according to any one of claims 1 to 22, wherein the matrix processing circuitry is configured to generate the result matrix from the first input operand and the second input operand in response to a single instruction.

24. 1. An apparatus comprising: means for performing a matrix processing operation on a first input operand and a second input operand to generate a result matrix, the result matrix being a two-dimensional matrix; means for storing information for forming the first input operand and the second input operand for the means for executing; means for performing a masking operation to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuitry based on masking state data indicating the location of one or more masked rows or columns that are processed to represent a masking value; An apparatus comprising:

25. 1. A data processing method comprising: storing, in an operand storage circuit, information for forming a first input operand and a second input operand for a matrix processing operation; performing a matrix processing operation on the first input operand and the second input operand to generate a result matrix, the result matrix being a two-dimensional matrix; performing a masking operation to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuitry based on masking state data indicating the location of one or more masked rows or columns that are processed to represent a masking value; A method comprising:

Images