Apparatus and method for masking row or column positions for matrix processing

Abstract

An apparatus is provided that includes a matrix processing circuit to perform a matrix processing operation on a first input operand and a second input operand to generate a result matrix, where the result matrix is a two-dimensional matrix, an operand storage circuit to store information used to form the first input operand and the second input operand for the matrix processing circuit, and a masking circuit to perform a masking operation to mask at least a portion of the matrix processing operation or information stored to the operand storage circuit based on masking state data indicating one or more masked row or column positions to be treated as representing a masked value. This is useful to improve performance of a two-dimensional convolution operation as the masking can be used to mask out selected rows or columns when performing the 2D convolution as a series of 1x1 convolution operations applied to different kernel positions.

Description

Background Technology

[0001] This technology relates to the field of data processing. More specifically, this invention relates to matrix processing.

[0002] Matrix processing operations that generate two-dimensional matrices as result matrices can be important operations in certain data processing fields (e.g., in machine learning or image processing).

[0003] At least some examples provide an apparatus comprising: a matrix processing circuit for performing a matrix processing operation on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix; an operand storage circuit for storing information for forming the first input operand and the second input operand; and a masking circuit for performing a masking operation to mask the matrix processing operation or at least a portion of the information stored in the operand storage circuit based on masking state data indicating row or column positions of one or more masks that will be considered as masking values.

[0004] At least some examples provide an apparatus comprising: means for performing a matrix processing operation on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix; means for storing information forming the first input operand and the second input operand for the means for performing the operation; and means for performing a masking operation to mask the matrix processing operation or at least a portion of the information stored in the operand storage circuit based on masking state data indicating row or column positions of one or more masks that will be considered as masking values.

[0005] At least some examples provide a data processing method comprising: storing information in an operand storage circuit for forming a first input operand and a second input operand for a matrix processing operation; performing a matrix processing operation on the first input operand and the second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix; and performing a masking operation to mask the matrix processing operation or at least a portion of the information stored in the operand storage circuit based on masking state data indicating row or column positions of one or more masks that will be considered as masking values. Attached Figure Description

[0006] Further aspects, features, and advantages of this technology will become apparent from the following description, which is taken in conjunction with the accompanying drawings, in which: Figure 1 An example of padding-free 2D convolution is shown; Figure 2 An example of a padded 2D convolution is shown; Figure 3 An example is shown of applying 2D convolution to input data that includes multiple channels to generate output data that includes multiple channels; Figure 4 An example of a memory layout for storing input data in memory is shown; Figure 5 One method for comparison is shown, in which input channel data stored in memory is rearranged to produce multiple rows of data stored in memory, in order to simplify subsequent 2D convolution processing applied to the remapped rows; Figure 6 A different approach is shown where the 2D convolution operation is broken down into multiple 1×1 convolutions; Figure 7 The steps shown demonstrate how masking selected rows or columns of the operand matrix enables 2D convolutions to be achieved through a series of 1×1 convolutions without rearranging the data in memory. Figure 8 This demonstrates how applying variable position shift between the input and output of a given matrix operation enables the same set of input channel data loaded from memory to be reused on multiple different 1×1 convolution operations for different kernel locations; Figure 9 A data processing device with matrix processing circuitry is schematically shown. Figure 10 A portion of the matrix processing circuit and the registers used by the matrix processing circuit are schematically shown; Figures 11 to 13 Different ways of representing addressing information and masking state information for matrix processing operations are shown; Figure 14 An example is shown in which the matrix processing operation is an outer product information and the device has a position shifting circuit to apply a variable position shift; Figure 15 An example of processing a load instruction to load a target row or column for matrix processing operations is shown; Figure 16 The method for processing matrix processing instructions is shown; and Figure 17 A second example of processing matrix instructions is shown. Detailed Implementation

[0007] Row or column masking for matrix processing operations Two-dimensional (2D) convolution is a popular operation in machine learning, particularly for neural networks. 2D convolution can also be used for other purposes, such as applying filters to images. In a 2D convolution operation, a kernel is provided to define the filter or other operation to be applied. The kernel is applied to one or more input channels, each containing a matrix that is typically larger than the kernel. In a 2D convolution operation, for a given output element position within the output matrix, the value at that position depends on the sum of the products of the corresponding kernel value and the input channel value. The choice of the input channel value to be multiplied by the corresponding kernel value is different for each output matrix position. For a given output element position, the kernel value multiplied by the corresponding input matrix element is the kernel value that is positionally aligned when the kernel is logically positioned, such that the center kernel element is above an element of the input matrix that corresponds positionally to the given output element position. An example of 2D convolution is further described below.

[0008] One reason 2D convolution operations are relatively complex to implement in data processing is that, for many different combinations of kernel values ​​and input elements, they may require computing the sum of several products of kernel values ​​and input elements, including adding products involving input matrix elements at adjacent addresses that may not be stored in the memory address space. Therefore, a typical approach to performing 2D convolution is to perform some remapping (rearrangement) operations (before the product and computation itself) to remap the data stored in memory for the input matrix, resulting in multiple custom data structures corresponding to the values ​​to be operated on at each corresponding kernel location. However, this remapping involves many instances of copying data from one memory location to another, which introduces additional latency and wastes memory space. Therefore, it may be desirable to find a way to implement 2D convolution that allows the desired operation to be applied directly based on the layout of the input channel data within the memory space, without such remapping.

[0009] In the examples below, an apparatus includes: a matrix processing circuit for performing a matrix processing operation on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix. The first and second input operands themselves do not need to be two-dimensional, and in some examples may be one-dimensional vectors, although other examples may apply the matrix processing operation to two-dimensional input operands. An operand storage circuit is provided to store information for forming the first and second input operands of the matrix processing circuit. A masking circuit performs a masking operation to mask the matrix processing operation or at least a portion of the information stored in the operand storage circuit based on masking state data indicating row or column positions of one or more masks that will be considered as masking values. The masking state data may be defined as an operand of a matrix processing instruction instructing the matrix processing circuit to perform a matrix processing operation, or it may be some stored state data configured separately and not explicitly referenced by the matrix processing instruction.

[0010] Masking is provided by masking state data based on the row / column positions indicating the masking location. This allows matrix processing to skip certain rows or columns of the input data, which is particularly useful for 2D convolution operations. The masking circuitry can perform the masking operation while operands are being loaded into the operand storage circuitry, while the matrix processing operation itself is being performed, or simultaneously while loading the operand storage system and performing the matrix processing operation.

[0011] This approach helps support more efficient 2D convolution operations. A 2D convolution operation can be (via software) broken down into multiple individual 1×1 convolution operations. These operations apply kernel values ​​from a single kernel location within a larger kernel matrix to multiple elements of the input matrix for a given input channel, and update individual elements within the output matrix based on the result (in some cases, multiple channels of this 1×1 convolution processing can be performed in parallel). This 1×1 convolution allows for the application of operations at a given kernel location without requiring a remapping of the structure in memory, where successive results of 1×1 convolutions at different kernel locations are accumulated together (where appropriate shifts of the output matrix elements are updated relative to the input matrix elements used to compute these outputs to indicate which kernel location is being applied), such that after performing a 1×1 convolution on each kernel location, the result is equivalent to the result of a 2D convolution.

[0012] To support this, it is useful to provide a masking circuit that can be controlled based on masking state data to mask a given row or column position, such that some row / column data from the corresponding input channel can be treated as representing masked values ​​rather than the actual data stored in memory. This is because when a 2D convolution is broken down into consecutive 1×1 convolutions, for most output element positions, the correct result for a given 1×1 convolution can be achieved by reading the corresponding input matrix element, multiplying that element by the corresponding kernel value, and writing the result to the corresponding output matrix element (where there is a positional shift between the relative position of the input matrix element within the input matrix and the position of the corresponding output matrix element within the output matrix, and this shift is the same number of element positions for each multiplication performed for a given kernel position). However, there are some elements at the edges of the matrix for which this approach may give incorrect results, leading to the preferred error, the "wrapback" error, as elements on one edge of the output matrix are updated based on elements on the opposite edge of the input matrix. By providing a masking operation, this allows rows or columns of input data that should not affect the output to be masked. Therefore, by providing support for row / column masking, the performance of 2D convolution operations can be improved, which is very important for the performance of neural networks.

[0013] It should be understood that the control over which specific rows / columns of the matrix are masked is software-controlled and therefore not a feature implemented by a specific processor. The device provides a feature that allows software to select which rows / columns to mask.

[0014] When a given row or column of a given operand matrix is ​​indicated to be masked by masking state data, there may be different options for selecting the mask value to use for that row / column position. For many practical applications, a mask value of zero can be useful. This helps support skipping rows to handle the aforementioned "wrap-around" problem, where a row / column on one edge of the input matrix should be prevented from affecting the computation of output matrix elements on the opposite edge. Furthermore, when applying a padded 2D convolution operation and the kernel is centered near the edge of the input matrix, a mask value of zero can be used to multiply the padded value to be provided with kernel elements located outside the boundaries of the input matrix. Therefore, in some hardware implementations, it may be sufficient for the masking circuitry to support only a fixed mask value (e.g., a mask value of zero) for any masked row / column position.

[0015] However, for some applications that use 2D convolution, it may be desirable to use padding values ​​other than zero (e.g., if a quantization scheme is used to represent a matrix, where each value is offset from its true value by a certain amount, such that "zeros" are represented by values ​​other than zero). To support such operations, it may be useful to provide the ability to select non-zero values ​​as masking values. Therefore, in some implementations, in a masking operation, a masking value can be selected from a plurality of masking values ​​(e.g., zero or another pre-configured value) based on at least one of the following: a masking value selection parameter specified by an instruction that causes the masking operation to be performed (e.g., a load instruction for loading information into operand storage circuitry, or a matrix processing instruction for controlling matrix processing circuitry to perform matrix processing operations); a control value stored in a control register; and a masking vector that specifies individual masking values ​​for multiple elements of the masked rows / columns. Using the last option, the masking vector can be read from a vector register.

[0016] Masking state data can have encodings that identify the elements to be considered as representing mask values ​​within a two-dimensional array of elements. Therefore, masking state data can (fully or partially) identify the position of masking elements in two dimensions. Providing state data that allows masking to be applied in two dimensions can be useful for handling many problems involved in 2D convolution processing, including the "wrap-back" error discussed above, where there may be many unused "out-of-bounds" elements at the end of the loop that extend to the end of the data structure being processed, and supporting the "position shifting" functionality described in more detail below.

[0017] For example, masking state data may specify: first masking state data indicating one or more masked row or column positions, for which all elements in the masked row or column position are treated as representing masking values; and second masking state data indicating whether individual element positions within a given row or column should be masked. Using the first masking state data to mask an entire row or column can be useful for handling “wrap-around” errors and / or “out-of-bounds” rows / columns in a first dimension, and individually masking specific elements in a row or column that is not fully masked can be useful for supporting “out-of-bounds” columns / rows in a second dimension and / or the positional shift features (or more generally, per-element predicates) described below. The first masking state data may include a set of elements identifying masked / unmasked row / column positions in a dimension (row or column), while the second masking state data may include a set of elements identifying masked / unmasked positions in an orthogonal dimension (column or row). In some cases, the second masking state data can specify individual indications of masked / unmasked elements for only a single row / column, because the same set of second masking state data can be shared across rows / columns (or the second masking state data can be adjusted between processing one row / column and the next if different rows / columns require masked / unmasked elements of different modes).

[0018] The masking state data may have an encoding capable of indicating at least two non-adjacent row or column positions separated by at least one unmasked row or column position as masked row or column positions. This recognizes that when a 2D convolution is split into multiple 1×1 convolutions, there may be multiple non-adjacent row or column positions that need to be masked to prevent input values ​​on one edge of the input matrix from affecting output values ​​at opposite edges of the output matrix. Furthermore, the positions to be filled in the 2D convolution may not correspond to contiguous addresses in memory.

[0019] Masking state data can be represented in a variety of different ways. Generally, masking state data can be any set of information that indicates which row / column positions within a matrix structure will be masked. One approach is for the masking state data (e.g., the first masking state information described above) to include multiple masking state indicators, each corresponding to a specific row or column position in a given operand matrix and indicating whether that row or column position is a masked row or column position. For example, the masking state data could include a bitmap where each bit corresponds to a given row or column position and is set to one value if that row or column position is to be masked and another value if it is to remain unmasked. Similarly, the second masking information could include a second bitmap indicating the masked row / element positions within a specific row / column.

[0020] The masking state data does not need to distinguish whether it refers to a corresponding row or a corresponding column of a given operand matrix. Different software applications may choose different layouts for matrices in memory (e.g., row-major or column-major), but the format of the masking state data can be the same.

[0021] Operand storage circuitry can be implemented in different ways. In some examples, operand storage circuitry may include a set of input registers from which the first and second operands can be read when a given matrix processing operation is performed.

[0022] However, it is useful to provide a matrix transpose circuit as part of the operand storage circuit, which includes multiple storage cells to store the corresponding matrix elements of a given operand matrix. The storage cells of the matrix transpose circuit are readable in row groups corresponding to the rows of the given operand matrix and also in column groups corresponding to the columns of the given operand matrix. Providing such a matrix transpose circuit is very helpful in handling the fact that different machine learning algorithms may use different layouts to store input channel data in memory. For example, some algorithms may use a row-major layout in memory, where the offset between the memory addresses of adjacent elements in the same row of the matrix is ​​less than the offset between the memory locations of adjacent elements in the same column of the given operand matrix. Other algorithms may use a column-major layout, where the offset between the addresses of adjacent elements in the same column is less than the offset between adjacent elements in the same row. The matrix transpose circuit can instantly remap whether row-major or column-major format is used. This is because if a given operand matrix is ​​written to the transpose circuit as a row group, it can be read from the transpose circuit as a column group, and vice versa. This allows subsequent matrix processing operations to use a consistent format, regardless of whether the input matrix data stored in memory is row-major or column-major. This simplifies code development and avoids the need to remap or rearrange the data within the memory itself.

[0023] It is important to note that the memory cells of a matrix transpose circuit do not need to be physically arranged in rows and columns. It is sufficient that the memory cells of the matrix transpose circuit are logically readable in the groups of memory elements corresponding to rows or the groups corresponding to columns. For example, a matrix transpose circuit can be implemented as a register group with multiple read / write ports, allowing for addressing of portions of the registers in different combinations. For instance, if each register stores a row group, then the column group can be considered to consist of a set of data portions (this set includes a portion of each register, located at the corresponding location within each register). Alternatively, the reverse mapping can be used, where each column group maps to a register and the row group is a stripe of the data portion within the corresponding location in each register. Furthermore, it should be noted that writing the “rows” of the matrix stored in memory into the “row groups” of the matrix transpose circuit is not necessary, although it is possible, but these rows of the matrix can be written just as well into the “column groups” of the matrix transpose circuit. Therefore, the "row groups" and "column groups" of memory cells in a matrix transpose circuit refer to orthogonal groups. Memory cells of the transpose matrix can be read through orthogonal groups, but the row / column orientation does not need to be consistent with the matrix in memory. In practice, to improve the read / write pipeline of the matrix transpose circuit, it is sometimes possible to alternately select whether consecutive row groups (rows or columns) of the input matrix are written into the matrix transpose circuit as row groups or column groups.

[0024] Therefore, when loading data into the matrix transpose circuit, the loading circuit can select whether to load at least one row group or at least one column group of the matrix transpose circuit's memory cells based on a portion of the matrix data structure in memory. The selection of whether to load at least one row group or at least one column group can be based on one or both of the following: row / column direction selection information specified by the load instruction; and row / column direction selection information stored in the control register, which can be updated in response to a row / column direction switching instruction. Some implementations may use only one of these options to determine whether to load a row group or a column group (the information specified by the load instruction or the information specified in the control register). Alternatively, implementations may combine these two pieces of information. For example, a control register bit may indicate row mode or column mode, but a bit in the load instruction may indicate whether the meaning of the storage bit should be reversed (therefore, for a load instruction with the "reverse" bit set, the instruction will load rows when the storage bit indicates columns, and columns when the storage bit indicates rows). Similarly, when reading data from the matrix transpose circuit to provide operands for matrix processing operations (or transferring information to the operand register, from which operands can then be obtained for matrix processing operations), the row / column direction selection information can specify whether to read the row group or column group of the matrix transpose circuit (again, the selection information can be specified by the instruction and / or in the control register, and the row / column direction bit in the register can be combined with the "invert" bit in the instruction to store the instruction, similar to the load instruction described above).

[0025] Masking operations based on masking state data can be performed at different times relative to the loading of operands used for matrix processing and the processing of the matrix processing operation itself.

[0026] In some implementations, the matrix processing circuitry may include masking circuitry. The masking circuitry of the matrix processing circuitry can perform matrix processing operations in response to masking information, wherein the row or column position of one of the first and second operands corresponding to one or more masks is treated as representing a masked value, rather than the actual value of a portion of the first and second operands stored in the operand storage circuitry. Therefore, although the actual data from the input channel can be loaded normally from memory into the operand storage circuitry, the replacement of such input data with masked values ​​can be controlled to provide filling or avoid the aforementioned wraparound errors by masking the data read from the operand storage circuitry when it is input to the matrix processing circuitry. This approach is particularly useful for implementations that also support the application of variable position shifting options, as discussed further below.

[0027] In some implementations, the masking circuit may include a loading circuit that, in response to a load instruction, loads information corresponding to a target row or column of a given operand matrix into an operand storage circuit based on a portion of a matrix data structure stored in memory. In this case, when the target row or column corresponds to a masked row or column position indicated by masking state data, the loading circuit may load data with masked values ​​for a portion of the target row or column in the operand storage circuit, rather than data based on a portion of the matrix data structure stored in memory. Using this method, masking can be applied when loading operands from memory, avoiding the unnecessary loading of matrix elements that would be masked anyway. Masking circuits can also be used to mask out-of-bounds data (corresponding to addresses outside the end of the data structure to be processed, which are referenced by the load instruction in the last iteration of the loop because the amount of data to be processed does not correspond to an exact multiple of the amount of data that can be processed in a single iteration), preventing them from being loaded and thus preventing address errors due to accessing potentially invalid addresses.

[0028] Some hardware implementations support both types of masking, which can be useful because, for example, masking at load time can more efficiently handle padding and masking of out-of-bounds data. However, if variable position shifting is supported, handling the aforementioned types of "wrap-back" errors might require masking at different input rows / columns for different instances of reading the same set of input data. In this case, applying masking when reading operand storage circuitry to perform a specific matrix processing operation might be more efficient. Therefore, to provide maximum flexibility, some implementations support both types of masking.

[0029] For specific implementations that provide loading circuitry including masking circuitry to apply masking when loading operand data from memory, the loading circuitry can determine whether each of the multiple matrix elements in the target row or column should be masked based on a shared item of masking state data shared among two or more matrix elements in the target row or column, when the masking state data corresponding to the target row or column indicates that the target row or column corresponds to the masked row or column position. Therefore, it is not necessary to provide a separate masking state for each individual element within the target row or column (although this would be possible if needed, as described above in the example of providing second masking state data for 2D masking). To support the "split into 1×1 convolution" approach for processing 2D convolutions, a common memory layout for the input channel data is to group the input elements at the same xy position of multiple input channels together in a contiguous block of memory. In this case, masking can be applied to an entire row or column of the input matrix structure that defines the input data for each of these input channels. This means that sharing a single masking state data across an entire row or column of the operand matrix being processed is sufficient.

[0030] For the example of loading a mask, a set of mask state indicators (e.g., bitmaps) can be used to represent the mask state data, as described above.

[0031] However, another approach could be to include multiple offset values ​​in the masking state data, each offset corresponding to a specific row or column position of a given operand matrix, and indicating the offset of the corresponding portion of the matrix data structure in memory relative to the base address. In this case, the masked row or column position could be indicated by the offset value of the masked row or column position with a predetermined reserved offset value. This approach can be useful because it means that a portion of the addressing information can be used to represent the masking state data, which identifies the memory address from which the portion of the matrix data structure should be loaded into memory. Thus, for each corresponding row or column position, the address in memory can be identified using the base address of that row or column position and the corresponding offset value, from which a portion of the matrix data structure should be loaded when the offset value does not have a predetermined reserved offset value. However, if the offset value of a given row or column position has a predetermined reserved offset value, the masking value can be written to a portion of the operand storage circuit (which originally stored the portion of the matrix for that row or column) instead of being loaded into the corresponding portion of the matrix data structure in memory. Therefore, this approach avoids the need to provide separate masking state data in addition to the state data used to address the matrix data structure in memory. The predetermined reserved offset value can be any reserved value that is specified as not allowed to be used for the actual offset value, such as -1 (e.g., a value where all offset bits are 1 in a signed binary representation).

[0032] In one example, the masking state data may be stored in at least one masking state register provided within the processing device. For example, prior to executing a load instruction for loading a portion of the operand matrix under the control of the masking state data, there may be specific instructions for writing the masking state information into the masking state register.

[0033] The masking status register can be a dedicated register specifically provided for controlling masking when performing matrix processing and / or loading operands for matrix processing.

[0034] In other examples, at least one masking state register may include at least one predicate register. In response to a vector instruction (or single instruction multiple data instruction) for controlling the processing circuitry to perform vector processing using one or more vector operands comprising an array of one-dimensional elements, the vector predicate register may be read to provide a predicate value controlling whether a corresponding channel of the vector processing is masked. Therefore, the same register can be shared between the vector predicate indicating vector operations and the masking state data indicating matrix operations.

[0035] At least one masking state addressing register can be provided to store masking state addressing information that identifies the location in memory from which masking state data can be obtained. For example, when a set of offset values ​​is used to represent masking state data as described above, this set of offset values ​​can be stored in memory, and the masking state address information in the masking state addressing register can identify where the array is stored in memory. This approach reduces the number of registers required in the architecture to support matrix processing, which may be preferred for some low-power microarchitecture implementations.

[0036] However, even if the architecture doesn't require registers to store the masking state information itself (since microarchitectures that don't want to provide dedicated hardware for storing this information can load it from memory when needed), some microarchitecture designers may still choose to provide a masking state cache to cache the masking state data retrieved from memory. This allows for faster access to the masking state data for future accesses, thus helping to improve performance. This can be useful because for many matrix operations, the masked / unmasked row / column patterns may be the same, so caching can save a significant amount of memory accesses.

[0037] Regardless of the form of the masking state data, the loading circuit can determine the target address in memory for a portion of the matrix data structure based on addressing information, which can be defined in various ways. The addressing information can be obtained from a register explicitly referenced by the instruction that caused the load to be executed, or from a default register implicitly referenced by the load instruction.

[0038] In one example, addressing information may include a set of address pointers, where each address pointer indicates the address of a portion of a matrix data structure corresponding to a specific row or column position of a given operand matrix.

[0039] In another example, the addressing information may include the base address of a matrix data structure stored in memory and offset information for determining the address of a portion of the matrix data structure corresponding to a given row or column of a given operand matrix relative to the base address. While in some examples this offset information may be represented using the same set of offset values ​​used for the masking state data, this is not necessary, and in other examples the offset information may be separate from the masking state data. The offset information may be represented differently, for example, using a stride value indicating the difference between the address of the portion of the matrix data structure corresponding to one row or column of a given operand matrix and the address of the portion of the matrix data structure corresponding to the next row or column of the given operand matrix, or by explicitly recording the offsets of multiple rows / columns in an offset data structure, as described above. The use of a stride value avoids the need for explicit encoding of each individual offset value for the corresponding row, but the use of a more explicit offset data structure allows the masking state to be represented in the same structure as the offsets and will allow processing of matrices with irregular patterns of memory access for corresponding rows / columns. Regardless of the approach, using offset information relative to the base address to represent the address allows for the use of fewer bits to represent the addressing information than the addressing information indicates the absolute address corresponding to each row / column position of a given operand matrix.

[0040] In some examples, the addressing information may also include additional information that provides sub-part selection information to select which sub-part of the matrix data structure portion in memory, identified based on the addressing information, should be loaded into the operand storage circuit when loading a given target row or column. This recognizes that, given the limitation on the maximum size of matrices that can be processed in hardware, when processing large input matrices, it may be necessary to break down operations into multiple sub-operations, each acting on a smaller portion of the input matrix. Since the layout of the matrix data in memory may include rows or columns larger than the block of matrix data to be operated on by a given set of matrix processing instructions, the sub-part selection information can be used to narrow down which sub-part of the row or column should be processed for a given operation.

[0041] Therefore, there are many options for representing addressing information that identifies the location in memory to load a given target row or column. At least one addressing register can be provided to store the addressing information. Before executing a load instruction or matrix processing instruction, the executing program can load the appropriate addressing information into at least one addressing register to select the portion of the matrix data structure to be processed.

[0042] In some implementations, prefetch circuitry may be provided to generate a prefetch request for prefetching a portion of a given operand matrix from memory, based on addressing information stored in at least one addressing register. For example, if the addressing information includes an array of offset values, the prefetch circuitry can look forward and begin prefetching data based on the offset of the later row / column when loading rows or columns of the given operand matrix for earlier rows or columns, thereby improving performance. Alternatively, other microarchitectures may prefer not to provide prefetch circuitry to save power and circuit area.

[0043] In some specific implementations, the first and second input operands for matrix processing operations can be two-dimensional matrix operands. For example, matrix processing circuitry can support full matrix multiplication operations executed in a single instruction, which can be beneficial for performance. However, this approach may be more expensive in terms of power consumption and circuit area.

[0044] Therefore, other implementations may be more inclined to provide matrix processing circuitry that supports performing matrix processing operations on one-dimensional vector operands to generate a two-dimensional result matrix. For example, the matrix processing operation may include an outer product operation applied to 1D vector operands to generate a 2D result matrix. It is recognized that, in practice, a matrix multiplication operation applied to two 2D matrix operands to generate a 2D result matrix can be decomposed into multiple separate outer product operations applied to corresponding combinations of corresponding rows / columns of the input matrix operands, where the results of the outer product operations are accumulated to generate a final result equivalent to the 2D matrix multiplication result. Therefore, it may be particularly useful for the outer product operation to include an outer-product-and-accumulate operation, for which the result matrix includes updated values ​​of corresponding elements of the accumulator matrix, where the updated value of a given element of the accumulator matrix corresponds to the result of adding the previous value of the given element of the accumulator matrix to the corresponding element of the outer product result matrix, the corresponding element of which corresponds to the result of performing the outer product operation on the first and second input operands. This operation can be used to support the aforementioned 2D convolution operation.

[0045] Matrix processing circuitry can respond to a single instruction to generate a two-dimensional matrix from the result matrix based on a first input operand and a second input operand. Therefore, even if a matrix multiplication operation is broken down into multiple instructions performing individual outer product operations, each acting on a one-dimensional vector operand, each individual outer product operation can still generate a two-dimensional result matrix. This provides improved performance compared to methods using vector processing circuitry to perform a series of vector operations equivalent to matrix operations, where each vector operation processes 1D vector operands to produce a 1D vector result.

[0046] Position shifting for matrix processing An exemplary apparatus includes: a matrix processing circuit for performing a matrix processing operation on a first operand and a second operand to generate a result matrix, wherein the result matrix is ​​a 2D matrix. An operand storage circuit stores information about the first input operand and the second input operand used to form the matrix processing circuit. A position shifting circuit is provided to apply a variable position shift to change which row or column of the result matrix is ​​updated during a given matrix processing operation based on a given element of one of the first and second input operands stored in the operand storage circuit. The variable position shift is based on one of a plurality of alternative shift amounts selectable for a given matrix processing operation. Each alternative shift amount corresponds to a position shift of one of the first and second input operands relative to a different number of rows or columns of the result matrix.

[0047] Position shifting circuitry is useful for methods that support decomposing 2D convolution operations into multiple individual 1×1 convolutions accumulated in the resulting matrix. The inventors recognized that in such a series of 1×1 convolutions, 1×1 convolution operations corresponding to multiple adjacent kernel positions require very similar input data, but there is a relative shift of one or more row / column positions between the inputs at the corresponding kernel positions. Therefore, by providing circuitry to apply variable row / column position shifts of the inputs to a given matrix processing operation relative to the output, it means that during a series of 1×1 convolutions implementing a 2D convolution operation, the same operand data loaded from memory can serve as input for matrix processing operations targeting multiple different kernel positions, which reduces the number of loading operations required to load data from memory to perform a given 2D convolution operation.

[0048] As mentioned above, while some implementations can perform full matrix multiplication, other implementations, to limit hardware costs, can implement the matrix processing operation as an outer product operation applied to one-dimensional vector operands, which serve as the first and second input operands, to generate a two-dimensional result matrix. Therefore, in this case, variable position shifts can change which row or column of the result matrix is ​​updated based on a given element within one of the first and second vector operands. Similarly, for reasons similar to those discussed above, it may be particularly useful to implement the matrix processing operation as an outer accumulation operation, where the result matrix includes updated values ​​of corresponding elements of the accumulator matrix, formed based on previous values ​​of the accumulator matrix and the corresponding elements generated for the outer product result. This operation can be used to support 1×1 convolution methods for processing 2D convolutions.

[0049] The position shifting circuit can select between corresponding alternative shift amounts based on parameters specified by a matrix processing instruction used to control the matrix processing circuit to perform a matrix processing operation. In some implementations, the parameter identifying the shift amount can be part of the opcode of the matrix processing instruction, allowing multiple different opcodes to be assigned to the corresponding shift amount, each opcode corresponding to the same type of matrix processing operation (except for different shift amounts). Alternatively, a separate parameter can be defined in the instruction encoding, such as a shift amount selection field separate from the opcode identifying the specific matrix processing operation to be performed. The parameter used to select the shift amount can be represented as an immediate value within the instruction encoding or identified in a register specified by the matrix processing instruction.

[0050] Alternatively, in some implementations, a specific dedicated register may be provided for storing the shift selection parameter, such that the register for obtaining the shift selection parameter in response to a matrix processing instruction read is implicit, and therefore does not need to be explicitly encoded in the instruction encoding.

[0051] The matrix processing circuitry can also support predicates, where certain rows or columns within the resulting matrix can be identified as active or inactive row or column positions, as indicated by predicate information accessible to the matrix processing circuitry. Therefore, when a given row or column of the resulting matrix corresponds to an active row or column position indicated by the predicate information, the matrix processing circuitry can generate elements of that given row or column of the resulting matrix, these elements having values ​​for the corresponding row or column that depend on one of the first and second input operands (which row or column is the corresponding row or column depends on one of the alternative shift amounts selected for a particular matrix processing operation). When a given row or column of the resulting matrix corresponds to an inactive row or column position indicated by the predicate information, elements of that given row or column of the resulting matrix are generated, these elements having values ​​for the corresponding row or column that are independent of one of the first and second input operands. For example, when a given row or column of the resulting matrix is ​​inactive, the corresponding elements can retain their previous values ​​without needing to be updated based on the corresponding row or column of the input operand. This helps address the “wraparound” problem discussed above by providing the ability to prevent certain rows or columns of the input operands from affecting the output. The predicate could be an example of the masking operation described earlier.

[0052] Similarly, for the masking example discussed above, the operand storage circuitry may include a matrix transpose circuit capable of reading and writing to its memory cells in either row or column group form. This facilitates more efficient processing of matrix data structures stored in memory, represented in either row-major or column-major order. All the features of the matrix transpose circuitry discussed above can also be provided when using the position-shifting example.

[0053] When a matrix transpose circuit is provided, the operand storage circuit may further include an operand register for storing a first operand and a second input operand for matrix processing operations, the operand register being separate from the matrix transpose circuit itself. The operand register may be a storage circuit that reads operands for a given processing operation in response to a matrix processing instruction that controls the processing circuit to execute matrix processing separately.

[0054] Dedicated move instructions can be provided to control the operand move circuitry to read at least one row or column of a given operand matrix from the matrix transpose circuitry and write that row or column to the operand register. This simplifies the coding of matrix processing instructions because any additional parameters used to select whether to read a column or a row from the matrix transpose circuitry (or to select which particular row or column should be read) can be encoded in the move instructions, resulting in less coding space required for such parameters in the matrix processing instructions.

[0055] However, another approach is to read operands from the matrix transpose circuit in response to matrix processing instructions and provide them directly to the circuit logic to perform matrix processing operations, without going through a set of operand registers.

[0056] Although the above does not explicitly describe such operand shifting circuitry in response to shift instructions, or the ability to read operands directly from matrix transpose circuitry, for the example using masking, these features are also provided in this example.

[0057] Furthermore, the masking function described in the preceding section can be combined with the position shifting function described above. Therefore, even in the position shifting example, a masking circuit that performs masking operations based on masking state data as described above can be provided.

[0058] In practice, combining on-load masking with position shifting (including predicates applied at the input of matrix processing operations) can be particularly useful. One might think that predicates are redundant when load masking is supported, but in reality, providing both can be quite useful. This is because on-load masking can be used to insert padding values ​​for 2D convolutions that support padding, even if the predicates applied at the input of the matrix processing operation are subsequently further masked to prevent certain rows from affecting the output (to handle the wraparound problem discussed above). This is because the location of rows affected by the wraparound problem can vary from kernel location to kernel location, so when using position shifting to allow computation of multiple kernel locations based on a set of data loaded for a single kernel location, predicates based on predicate values ​​can be used to select individual rows to suppress for each individual kernel location—a process that would be difficult to handle if such wraparounds were only processed when loading data from memory. However, masking methods can be used to provide padding values.

[0059] However, in the previously described example, if position shifting is not supported, then if a separate load is performed for each kernel position, the masking during the load operation is sufficient to handle the loopback problem, or alternatively, load masking may not be supported at all, and instead a mask / predicate can be applied when performing matrix processing operations.

[0060] Similarly, for the masking example, the resulting matrix generated for the matrix processing operation can be a two-dimensional result matrix generated from the first and second input operands in response to a single instruction, so there is no need to process each vector instruction that generates a one-dimensional vector result separately.

[0061] 2D convolution Figure 1 An example is shown where a 2D convolution operation is performed on an input matrix and a kernel matrix to generate an output matrix. In this example, the input matrix is ​​a 4×4 matrix, the kernel is a 3×3 matrix, and the output is a 2×2 matrix. It should be understood that the matrices involved are not necessarily square matrices with the same dimensions for the number of rows and columns, and... Figure 1 The set of specific matrix sizes shown is merely an example.

[0062] In 2D convolution operations, for each output element within the output matrix, the kernel generates an output element centered on the element in the input matrix at the position corresponding to the output element being generated. The value of the output element corresponds to the sum of the products of the corresponding kernel element and the input matrix element, with the input matrix element located at the corresponding position relative to the central kernel. For example, for the output matrix element F' corresponding to the input element F at position F, assuming the central kernel element K5 is located above the input element F corresponding to the output position F', the value of F' is generated by multiplying the corresponding input element and kernel element at the corresponding position. Therefore, F' = A*K1 + B*K2 + C*K3 + E*K4 + F*K5 + G*K6 + I*K7 + J*K8 + K*K9.

[0063] Similarly, for each other matrix element in the output matrix, the element is generated based on the sum of products, but its kernel is located above a different element of the input matrix. For example, for the output element G', the center element K5 of the kernel matrix is ​​located above the input matrix element G, meaning the sum of products is G' = B*K1 + C*K2 + D*K3 + F*K4 + G*K5 + H*K6 + J*K7 + K*K8 + L*K9. A similar operation is performed to generate the output elements J' and K'.

[0064] Figure 1The diagram illustrates a padding-free 2D convolution operation, meaning that the output elements F', G', J', K' are generated only for those input locations F, G, J, K, where the kernel can be concentrated at that input location, and no kernel element of the kernel matrix extends beyond the boundaries of the input matrix. For example, input elements A, B, C, D, E, H, I, L, N, M, O, P do not have corresponding elements in the output matrix because this would require a portion of the kernel to extend beyond the boundaries of the input matrix. Therefore, for padding-free 2D convolution, the output may often be smaller than the input.

[0065] like Figure 2 As shown, padded 2D convolutions can also be performed by providing padding values ​​(PVs) for element positions outside the boundaries of the input matrix (this would require applying a kernel centered near the edges of the input matrix), where the output matrix is ​​generated with the same dimensions as the input matrix. Figure 2 In the example, the input matrix and kernel may be related to Figure 1 The same as in the previous example, but this time the output matrix is ​​also a 4×4 matrix, except that it is the same as... Figure 1 In addition to the elements F', G', J', and K' calculated in the same way, the surrounding elements A' to P' are also included so that the output matrix is ​​on the same side as the input matrix.

[0066] For computations where the kernel is centered at one of these outer element positions, the kernel elements outside the input matrix are multiplied by the padding value (PV). For example, for computations used to generate the output element A', this would require the central kernel position K5 to be above element A in the input matrix, and therefore, when valid input values ​​exist for positions A, B, E, and F in the input matrix corresponding to kernel elements K5, K6, K8, and K9, the other kernel elements K1, K2, K3, K4, and K7 are multiplied by the padding value to generate new values ​​for the output matrix A' when generating the sum of the products.

[0067] Similarly, for other elements around the boundaries of the output matrix, padding values ​​will be located in different positions relative to the kernel, depending on the edges of the input matrix that the kernel overlaps with. For example, for output position L', the right columns of kernels K3, K6, and K9 will require padding values ​​because these positions are where the kernel would extend beyond the input matrix when centered at position L. Similarly, for output element N', kernel position K5 will be centered at position N, so this means the bottom rows of kernel positions K7, K8, and K9 extend beyond the input matrix and therefore require padding.

[0068] In one example, the padding value could simply be zero. However, some 2D convolution operations may require other types of padding values. For example, in some cases, a quantization scheme can be used where an offset is applied to the true value of the matrix when generating the stored value for each matrix element, so that "zero" can actually be represented by a non-zero value. In this case, the padding value could be a non-zero value representing the zero point. The padding value can be set based on the average of other elements within the input matrix. The precise rules used to set the padding value may depend on the specific application being performed. Therefore, the ability to support selection among several alternative types of padding values ​​(e.g., based on parameters specified by control registers and / or matrix processing instructions) can be useful.

[0069] Although Figure 1 and Figure 2 The example is not shown, but strided convolution can also be performed, where, with a given input element as the center, the kernel value is applied to adjacent input elements separated from the center input element by a constant stride interval (and...). Figure 1 and 2 The case with a stride length of 1 is the opposite; other examples may have a stride length of 2 or greater.

[0070] Padding-free and padding-free 2D convolution operations can be used in a range of processing applications. For example, 2D convolution can be used to apply filters to images, such as for blurring, sharpening, edge detection, etc. The applied kernel can be chosen based on the type of filter required and can have specific values ​​for kernel elements that will produce certain features, such as edges. Effectively, the kernel can slide across each consecutive image pixel and apply operations to generate new values ​​for the output pixel based on the number of that pixel and its surrounding pixels, using relationships defined by the kernel.

[0071] Another type of processing (which may include 2D convolution) is in the field of machine learning, such as in implementing neural networks. For example, a neural network trained to detect features within image data can be implemented using a set of kernels applied to image data in 2D convolution operations. More generally, feature maps representing some data to be processed can be processed by kernels to make inferences about the data.

[0072] like Figure 3As shown, for machine learning algorithms, supporting multi-channel input and output data and multiple sets of kernel weights can be useful in order to derive multiple different inferences from a set of data. Each input / output channel can comprise a two-dimensional matrix of elements. For example, the number of input channels can be IC, and the height and width of each input channel can be IH (input height) and IW (input width). The number of output channels is OC, and the height and width of each output channel can be OH (output height) and OW (output width). OC sets of kernel weights are provided, where OC matches the number of output channels. Each set of kernel weights includes KH*KW*IC weights (where KH and KW are the kernel height KH and kernel width KW, and IC is the number of input channels). By executing... Figure 1 or Figure 2 The basic 2D convolution operation IC instance of the type shown is used to generate a given output channel. Each instance combines a single input channel IC with a corresponding subset of KH*KW kernel weights and accumulates the results of the basic 2D convolution for each input channel to generate the corresponding output channel (or by performing other sequences of operations that give the same result, as described later). Other output channels are computed using similar operations, but with a different set of KH*KW*IC kernel weights for each output channel. Whether OH and OW are equal to or less than the input height IH and input width IW may depend on whether a padded or unpadded 2D convolution is being performed.

[0073] In this example, the number of output channels (OCs) equals the number of input channels (ICs), but this is not required. Other examples may have different numbers of ICs and OCs. Furthermore, Figure 3 The 2D convolution shown may simply be a step in a 2D convolution tree, so the input channels themselves can form the output of earlier convolutions, and Figure 3 The output channels themselves can be processed by additional convolutions.

[0074] When 2D convolution is to be applied to multiple input channels, there may be many options for the layout used to store the data of the input channels in memory. Figure 4A possible memory layout, called the NHWC memory layout, is shown, where C refers to the input channel, W to the width, H to the height, and N to the number of different objects represented by independent groups of IC input channels. The NHWC notation indicates that when reading data from consecutive addresses within a data structure in memory, the input channel identifier variable C is the fastest changing variable, while the object identifier variable N is the slowest changing variable. Therefore, in the NHWC layout, when traversing consecutively increasing addresses within a data structure in memory, the input matrix element at a given xy matrix position for each IC input channel is first stored in a consecutive address block in memory. Then, the elements in each input channel at the next position in the same row as the first matrix element are laid out, and so on for each other xy position. That is, the element first cycles through all input channels at an element position, then moves to the next element in the same row (because the width W is the fastest changing variable after the channel ID), and then once all positions in the same row (elements with the same y matrix coordinates) have been stored for all channels, the stored next element will be used for the next row at the next highest y position.

[0075] Therefore, when referring to Figure 3 hour, Figure 4 The first row of the memory layout shown corresponds to the elements within the crosshairs, which correspond to position A in each input channel. The next row corresponds to the elements shown in dashed shading, which correspond to position B in each input channel. This pattern continues for the remaining elements C, D, etc., in the first row. Once the end of a row is reached, the same operation is performed on the next row, starting with the element at position E in each input channel. When using a set of separate IC input channels to represent multiple objects to be processed (e.g., multiple individual images), all the data for one object (N=0) is stored in memory before the data for the next object (N=1).

[0076] It should be understood that, although for ease of understanding, Figure 4 The text shows the elements of a given input matrix position in all channels of a "row" in the address space, and then moves to... Figure 4 The 2D representation of the next "row" is used to store the element at the next input location B, but in reality, the address space is simply a monotonically increasing sequence of addresses, and there is no such thing as... Figure 4 The 2D layout of the address shown. Figure 4 The 2D representation shown is a graphical representation used for simplicity to adapt information onto the page. However, the information stored in memory represents multiple channels of matrices, which are two-dimensional structures logically arranged in rows and columns.

[0077] Figure 4The NHWC memory layout shown is one possible layout, but other specific implementations may store the matrix structure in different layouts. For example, if the NCHW memory layout is used, the layout can provide all X / Y values ​​for channel 0, then all X / Y values ​​for channel 1, and so on.

[0078] Regardless of the specific memory layout chosen for a given application, one problem with 2D convolution methods is that the elements required to combine with the kernel elements used to generate a given output element within the output matrix may not reside in contiguous memory addresses within the memory address space. For example, to compute... Figure 2 The top-left output position A' in the padding 2D convolution may require obtaining the input elements at positions A, B, E, and F from memory, but... Figure 4 As shown, when stored in the NHWC memory layout, these elements are not in a contiguous portion of the address space because they are separated by elements at input locations C and D. Each kernel location may require a different custom subset of elements extracted from the data structure defining the input matrix in memory.

[0079] Figure 5 A method for addressing this problem, called im2row, is illustrated. In im2row, before performing the 2D convolution operation itself, the input matrix structure representing the input channels is first rearranged to generate multiple row 2 data stored in the address space portion that differ from the original input data structure. Each row 2 corresponds to data that will be operated on by the kernel matrix for a specific output element position in the output matrix. For example, for output position A', the desired elements A, B, E, and F of the corresponding input channel can be grouped together and combined with appropriate padding so that they are in the correct positions corresponding to the kernel elements K1 through K9 in the correct order. This means that subsequent matrix processing operations can simply multiply each kernel element of the multiple kernel channels with the corresponding data at the matching position within row 2 and sum the resulting products to generate the data at that output position. Note that a given row 2 has the corresponding input values ​​of each input channel in the input channels IC that are positioned adjacent to each other, and these input values ​​will be operated on by the corresponding kernel values ​​at the same kernel position within different kernel channels.

[0080] Similarly, for each other output location within the output matrix, a distinct row 2 is generated by grouping together the corresponding input elements required to generate that output location. Therefore, this requires generating OH*OW rows 2 of additional data, where each row comprises KH*KW*IC elements. While this can incur significant overhead in extracting corresponding subgroups of elements from data stored in memory and copying them to other locations in memory to generate rows, it greatly simplifies subsequent 2D convolution operations, which can then simply apply the kernel values ​​directly to contiguous blocks of memory in matrix processing operations to generate the corresponding output matrix.

[0081] However, this approach has several problems. One problem is that the performance of matrix processing operations implemented in a given data processing system is constantly improving. As matrix processing performance improves, Amdahl's Law means that other operations performed along with the matrix processing operation itself have an increasingly significant impact on overall performance. Even if the matrix processing operation itself continues to improve performance, if other operations (such as...) Figure 5 The `im2row` operation shown does not exhibit the same performance improvement as matrix processing operations (because `im2row` is limited by memory bandwidth), thus failing to realize the full benefits of the performance improvements in matrix processing. Therefore, for some processing applications, performing operations such as... Figure 5 The overhead of im2row is becoming increasingly unacceptable. Another problem is that these remapping operations consume significant amounts of memory. For example, it should be noted that in... Figure 5 In the example, the input matrix element at position F is displayed across multiple rows 2. Therefore, this also wastes memory address space, as the copying of the input values ​​is merely to provide proper relative positioning of the input matrix with respect to the kernel matrix. For example, for some machine learning algorithms, im2row might require 8-9 times the memory of the original input data structure.

[0082] Another type of convolution operation is the 1×1 convolution operation, which is similar to the 2D convolution described above, but its kernel is a 1×1 matrix instead of having a 2-dimensional range. For a 1×1 kernel, the result of a 1×1 convolution operation is simply an output matrix, where each element corresponds to the result of multiplying the corresponding element of the input matrix by the same kernel element. For example... Figure 6 As shown, by using a series of 1×1 convolutions, the same result as 2D convolution can be generated by accumulating the results of multiple 1×1 convolutions with the result of a given 1×1 convolution operation and the result of a given 1×1 convolution operation at the position of the previous 1×1 convolution operation result by relative shift.

[0083] In the 2D convolution example shown above, the calculation of the sum of products has been shown for each position of the output matrix, with each set of products used for different input position / kernel position pairs, but the same output position.

[0084] However, multiplication can also be divided into different groups, treating the set of multiplications associated with a single kernel location as a group, where the group of multiplications generates one product of the required sum for each output location. Consider... Figure 2 For example, when considering a single kernel location (e.g., location K1), the kernel value K1 needs to be multiplied by the padding value when generating the output value A', by the input value G when generating the input value L', and by the input value I when generating the output element N'. Therefore, Figure 6 The top shows the relationship between the input elements to be multiplied by K1 to form a partial product used in the sum of each corresponding output element A' to P' in the output matrix.

[0085] Similarly, for each of the other kernel positions K2-K9, it can be determined which input element (or padding value) should be multiplied by that kernel element to generate another product in a product summed for each output position. Note that a given input matrix element affects different elements of the output matrix at each kernel position. For example, when considering input element F, this will affect output element K' when multiplied by kernel element K1, output element J' when multiplied by kernel element K2, output element I' when multiplied by kernel element K3, and so on, until F affects output element A' when multiplied by kernel element K9.

[0086] Therefore, between corresponding kernel element positions, there is a relative shift between the position of a given output element in the output matrix and the position of the corresponding input element that acts on that particular kernel element position. For example, the shift of the effective input matrix between K1 multiplication and K2 multiplication is a left shift of one column.

[0087] This means that by performing a series of 1×1 convolutions and accumulating the results of each 1×1 convolution into an accumulator matrix representing the total number of runs of the output matrix, the result is equivalent to the result of a 2D convolution operation performed on a kernel size larger than 1×1. For example, the result of each K2 multiplication in the K2 multiplication shown can be added to the corresponding element of the accumulator matrix produced by the K1 multiplication (e.g., the result of K2*B is added to the accumulator matrix element at position F' based on K1*A in the K1 1×1 convolution), and the result of each K3 multiplication in the K3 multiplication can then be added to the corresponding element of the accumulator matrix produced by the K1 and K2 multiplications (the result of K3*C is added to the accumulated value of the output element F', making F' now equal to K1*A+K2*B+K3*C). This is continuous for each consecutive kernel position, so at the end of the ninth 1×1 convolution operation, the output matrix has the same result as performing a 2D convolution operation using a 3×3 kernel matrix. It can be understood that it is not necessary to follow... Figure 6 The sequence K1, K2, K3, ..., K9 shown computes 1×1 convolutions, and kernel points can be used in any order. However, if a position-shifting example is used as described below, computing adjacent kernel positions consecutively can help improve performance because the shift between input positions for computing a given output position of consecutive 1×1 convolutions will be smaller. Therefore, when using the following text regarding... Figure 8 When using the variable position shift technique, this facilitates more frequent reuse of data loaded from memory across multiple 1×1 convolutions.

[0088] like Figure 7 As shown, using Figure 6 One advantage of the illustrated split 1×1 convolution method is that it means the multiplication required for a given kernel location Kn can be applied to data loaded from a memory block, which can be either a single contiguous block or several such contiguous blocks separated by regular stride intervals. This means the 1×1 convolution operation can be directly applied to data with a similar format to the data structure in memory, and does not require... Figure 5 The performance-intensive and memory-intensive im2row technologies are shown.

[0089] Figure 7 This demonstrates how to expand a 1×1 convolution to handle multiple input and output channels, similar to the previous example. Figure 7 The diagram illustrates the calculation of a single kernel location corresponding to the xy dimension (such as...). Figure 7 The matrix multiplication operation of the product set at kernel position K1 in the example. That is, Figure 7 Only those targeting Figure 6 The product calculation is performed at the top, but expanded to handle multiple input / output channels. It should be understood that a similar operation can then be performed for each other kernel location.

[0090] Figure 7 An example of a portion of a 2D convolution operation is shown, where there are crosses between the input channels to generate each output channel (i.e., the results of the 2D convolution applied to each pair of kernel / input channels are summed to give a matrix for a given output channel). This means that for a 1×1 convolution corresponding to a given kernel point K1, the value at a given position F' in a given output channel corresponds to the sum of the products. , where i increases across all input channels, and K1 i It is the kernel value at the corresponding position within each kernel channel, and A i This refers to the input element at the corresponding position within each input channel. Operations can be performed in parallel on multiple sets of different kernel channels (allowing for parallel detection of multiple features) to generate multiple output channels.

[0091] Therefore, as Figure 7 As shown, when evaluating across multiple input / output channels, a 1×1 convolution at a given kernel position K1 can be scaled up to a matrix multiplication operation. This operation multiplies a Z×IC input matrix 10 (which provides a set of Z input element values ​​A through K for each IC input channel in the IC input channels) with an IC×OC kernel matrix 11 (which provides a set of kernel values ​​for kernel position K1 for each IC input channel within each kernel channel of the OC group corresponding to the respective output channel). The result of the matrix multiplication is a Z×OC output matrix 12, which provides a set of Z output elements F' through P' for each output channel OC. Note that the Z dimension of the input / output matrices 10 and 12 will vary depending on the kernel position Kn being processed, because for K1, the range of required unpadded element positions extends from A to K, but for different element positions (e.g., K2), the range of unpadded elements can be larger (e.g., extending from A to L). Furthermore, if non-zero padding values ​​are used, additional matrix rows may be needed in the input / output matrices to accommodate the non-zero padding.

[0092] Input matrix 10 can be directly obtained from, for example Figure 4 The data structure shown is loaded from memory because each row of input matrix 10 includes a set of elements for a single xy position of each IC input channel across the IC input channels within the input matrix. For example, the top row of input matrix 10 provides the "A" element (e.g., x=0, y=0) for each different input channel, then the next row of input matrix 10 provides all the "B" elements (x=0, y=1), and so on. Therefore, if the data is loaded as follows... Figure 4 The NHWC layout shown is located in memory, so this input matrix 10 corresponds exactly to the data format stored in memory and can therefore be loaded as a single contiguous block of memory. Alternatively, if the number of input channels IC that the processing hardware can process in one operation is less than the actual number of channels C used in the matrix structure stored in memory... max Then the input matrix 10 can correspond to multiple discontinuous blocks separated by a constant step size, and is similar to... Figure 2 Loading 2D convolutions from memory is still much simpler than performing them in the manner shown, which would require many irregular memory access patterns, as illustrated in the im2row example. Therefore, the 1×1 convolution method means that the matrix structure stored in memory does not need to be remapped before performing the multiplications used to compute the 1×1 convolution.

[0093] Similarly, output matrix 12 has a layout corresponding to input matrix 10, so once all the 1×1 convolutions of the 2D convolutions have been accumulated, the result can be directly written back to... Figure 4 The matrix data structure in the memory layout shown.

[0094] like Figure 6 As shown at the top, when considering the top-left kernel weight K1, the relative shift between the input and output positions means that row A of the input matrix should be multiplied by the kernel weight K1 to produce row F of the output matrix, row B of the input matrix affects row G of the output matrix, and so on. This generally works for most rows because, for the K1 weight example, there is a constant shift down 5 rows between the input and output matrices. However, there are certain rows D and H in the input matrix. Multiplying these rows by the kernel weight and accumulating the results at the corresponding shift positions I' and M' of the output matrix will yield incorrect results because, as... Figure 6 As shown, this means that the leftmost element of the output matrix will be updated based on multiplication using the elements on the right edge of the input matrix, which is incorrect for 2D convolutions. This problem can be called the "wrap-back" problem. Although it can be solved by... Figure 7 The matrix multiplication between matrices 10 and 11 shown is broken down into multiple separate operations to avoid the loopback problem, where each operation corresponds to a block of input matrix 10, which consists only of blocks of rows AC (or EG or IK), all of which need to be applied to the output matrix, but this would require additional instructions and would degrade performance.

[0095] Therefore, to allow 1×1 convolutions to be applied over a large number of rows, it may be useful to support a masking operation, even if selected rows encounter wrapback problems. This masking operation allows certain rows of the input to be skipped when generating the output. This is indicated by the “X” marked on the path between input rows D, H and output rows I', M'. The masking operation can be controlled by masking state data, which defines the position of the masked rows (or, if the matrix is ​​arranged to have input elements extending within the same column for a given input channel position). An example of encoding the masking state data is described below. The masking operation can be implemented when data is loaded from memory into a register (so that instead of loading the actual data elements from memory, masking values ​​are loaded into the corresponding portion of the operand memory to store information used to form the input channel matrix 10). Alternatively, a masking operation can be performed during the matrix processing operation itself, such that when the matrix processing circuit reads operands for processing, a predicate is applied to mask the read rows of elements, ensuring that the matrix processing circuit treats these elements as masked values, rather than the actual values ​​stored in operand memory. The masked value can be zero or non-zero if non-zero values ​​are used to represent zeros. Either way, this means that wraparound problems can be prevented from causing errors, and it allows 1×1 convolutions to be performed in fewer instructions, as 1×1 convolutions can be applied to matrix sizes larger than contiguous blocks of rows that do not encounter wraparound problems.

[0096] For other kernel weight positions K2-K9, execution can be performed on K1. Figure 7 Similar matrix multiplication operations are shown, and the results are summed together.

[0097] Figure 8 Further observations are shown that can be used to improve performance by reducing the number of times data is loaded from the matrix data structure in memory to perform 1×1 convolutions over a range of kernel weight locations. From Figure 6 It can be observed that when evaluating the corresponding 1×1 convolutions at different kernel locations within the same row, the input matrices required for each of these kernel locations are very similar. For example, Figure 8 The input matrix 10 is shown for the left-center kernel position K4, the right-center kernel position K5, and the right-center kernel position K6, respectively. For the center kernel weight K5, the input matrix will be perfectly aligned with the output matrix because the kernel weight K5 is multiplied by position A when generating output A, multiplied by position B when generating output B, and so on for each other position in the input / output matrices 10, 12.

[0098] For the left-middle kernel position K4, when generating the output element B, K4 needs to be multiplied by the element A of the input matrix (because when the center of kernel K5 is on element B, K4 will be multiplied by A). Similarly, for each other position in the input / output matrices 10 and 12, there is one position shift between the input and output elements.

[0099] Similarly, for the right-middle kernel position, K6 needs to be multiplied by the input element B to generate the output element A, multiplied by the input element C to generate the output element B, and so on.

[0100] like Figure 8 As shown, for the left center and right center positions, due to the fact that... Figure 7 The described loopback problem involves skipping some lines, and the specific positions of the skipped lines vary depending on the kernel weight positions (e.g., for K4, the skipped input lines are lines D, H, and L, but for K6, the skipped input lines are lines E, I, and M, and for K5, there are no skipped input lines).

[0101] However, it can be seen that, generally speaking, for each of the three kernel weight positions K4, K5, and K6, the input data for row AP of input matrix 10 is essentially the same. The difference lies in that, relative to the center position K5, for the left-middle position K4, input matrix 10 is shifted down one row relative to the output, so that input row A is used to generate output row B, instead of generating row A as in the center position K5. Similarly, for the right-middle position, input matrix 10 is shifted up one row relative to output matrix 12, so that input row B is fed into output row A.

[0102] Therefore, it can be observed that by providing circuitry that performs a variable positional shift of the input relative to the output, it becomes possible to adjust which row of the output matrix is ​​updated based on a specific row of the input matrix, and multiple different alternative shift amounts can be selected. This allows matrix data blocks loaded from memory to be reused for 1×1 convolutions at multiple different kernel locations. This means that the memory bandwidth associated with loading the input row AP can be amortized across multiple different matrix processing operations, which significantly improves performance. If this positional shift is used, masking is required when reading previously loaded operands from registers or matrix transpose boxes because the position of the masking row used to handle the wraparound problem varies from kernel location.

[0103] Data processing devices that support matrix processing Figure 9 An example of a data processing apparatus 20 is illustrated schematically. The data processing apparatus has a processing pipeline 24 comprising multiple pipeline stages. In this example, the pipeline includes: a fetch stage 26 for fetching instructions from an instruction cache 28; a decode stage 30 for decoding the fetched program instructions to generate micro-operations to be processed by the remaining stages of the pipeline; an issue stage 32 for checking if the operands required for the micro-operation are available in a register file 34, and issuing the micro-operation for execution once the operands required for a given micro-operation are available; an execution stage 36 for performing a data processing operation corresponding to the micro-operation by processing the operands read from the register file 34 to generate a result value; and a write-back stage 38 for writing the processed result back to the register file 34. It should be understood that this is merely one example of a possible pipeline architecture, and other systems may have additional stages or different stage configurations. For example, in an out-of-order processor, a register renaming level may be included, which is used to map the architecture registers specified by program instructions or micro-operations to physical register specifiers in the identifier register file 34.

[0104] Execution level 36 includes multiple processing units for performing different types of processing operations. For example, the execution units may include a scalar arithmetic / logic unit (ALU) 40 for performing arithmetic or logical operations on scalar operands read from register 34; a floating-point unit 42 for operating on floating-point values; a branching unit 44 for evaluating the results of branching operations and adjusting the program counter representing the current execution point accordingly; a matrix processing unit 46 for matrix processing (which will be discussed in more detail below); and a load / store unit 48 for performing load / store operations to access data in memory systems 28, 50, 52, and 54.

[0105] In this example, the memory system includes a Level 1 data cache 50, a Level 1 instruction cache 28, a shared Level 2 cache 52, and main system memory 54. It should be understood that this is merely one example of a possible memory hierarchy, and other cache arrangements are also possible. The specific type of processing units 40 to 48 shown in execution phase 36 is only one example, and other implementations may have a different set of processing units or may include multiple instances of the same type of processing units, allowing multiple micro-operations of the same type to be processed in parallel. It should be understood that... Figure 1 This is merely a simplified representation of some components of a possible processor pipeline architecture, and the processor may include many other elements not shown for the sake of simplicity.

[0106] In some specific implementations, the data processing device 20 may be a multiprocessor device, which includes multiple CPUs (central processing units or processor cores) 60, each CPU having features specific to... Figure 9 The processing pipeline 24 is similar to the processing pipeline shown in one of the CPUs of CPU 60. The device 20 may also include at least one graphics processing unit (GPU) 62 and / or other master devices 64, which can communicate with each other and with the CPU via interconnects 66 for accessing memory 54.

[0107] One approach to supporting matrix processing operations is to break down the individual multiplications of a given matrix processing operation into separate integer or vector instructions, which can be processed on the processing pipeline 24 of a given CPU 60. However, this can be relatively slow.

[0108] Another approach to accelerate matrix processing could be to provide a hardware accelerator with dedicated hardware, designed for handling matrix operations, as one of the devices 64 connected to interconnect 66. To interact with such a hardware accelerator, CPU 24 would use load / store unit 48 to execute load / store instructions to write configuration data to the hardware accelerator. This configuration data defines the matrix operands to be read from memory by the hardware accelerator and the processing operations to be applied to the operands. The CPU could then read back the results of the matrix processing from the hardware accelerator using load instructions specifying addresses mapped to registers within the hardware accelerator. While this approach may be faster than using integer operations within the pipeline, there can still be overhead associated with transferring information between the general-purpose processor 60 and the hardware accelerator 64 using the load / store mechanism. The hardware accelerator approach can also present challenges when different virtual machines running on the same processing system need to share access to the hardware accelerator. Therefore, this approach may not scale well in virtualization implementations with multiple virtual machines.

[0109] Therefore, as Figure 9 As shown, matrix processing circuitry 46 can be provided within the regular processing pipeline 24 of a given CPU 60. This matrix processing circuitry can be controlled to perform matrix processing in response to matrix arithmetic program instructions decoded by the pipeline's decoding stage 30 (similar to using ALU 40 or floating-point unit 42 to control regular integer or floating-point arithmetic operations). This avoids the need to transfer data back and forth between the CPU 60 and the hardware accelerator, and makes it simpler to allow multiple different virtual machines to perform matrix operations.

[0110] Although Figure 9 A multiprocessor device 20 with several CPUs 60 is shown, but this is not necessary; the matrix processing circuitry 46 can also be implemented in a single-core system.

[0111] Figure 10A portion of the matrix processing circuitry 46 and associated registers supporting matrix processing are shown in more detail. The matrix processing circuitry 46 may include operand storage circuitry including an input operand register set 70, an output matrix register set 72, and a matrix transpose circuitry 74 (hereinafter referred to as the matrix transpose frame). Furthermore, the matrix processing circuitry includes: a matrix loading circuitry 80 for loading data from a matrix structure in memory into the operand storage circuits 70 and 74; an operand shifting circuitry 82 for shifting operand data between the matrix transpose frame 74 and the input operand register 70; and a matrix processing logic circuitry 84 for performing matrix processing operations on the input operands themselves stored in the input operand register 70 to generate a two-dimensional result matrix stored in the output matrix register 72.

[0112] The matrix transpose frame 74 includes multiple storage elements 88, each for storing a different matrix element of a given operand (input) matrix. The storage elements 88 are logically arranged in rows and columns such that they can be accessed as row groups 90, where all storage elements 88 corresponding to the same row of the input matrix are readable / writable, or as column groups 92, where all storage elements 88 corresponding to the same column of the input matrix are readable / writable. The physical arrangement of the storage elements 88 on the integrated circuit does not need to follow a logical arrangement in row and column form and any physical arrangement can be adopted. Alternatively, the ability to read or write elements 88 in the form of row groups 90 and column groups 92 is provided by providing read / write ports and multiplexing circuitry, making it possible to read relevant elements corresponding to a given row or a given column, regardless of their physical location on the chip.

[0113] This means that when loading data from a matrix data structure in memory, the matrix loading circuit 80 can select (in response to the row / column direction selection parameter 89) whether to load a portion of the matrix structure selected from memory based on the addressing information 94, either as a separate row group 90 or a separate column group 92 of the matrix transpose frame 74. A load instruction 98, decoded by the instruction decoder 30 to control the matrix loading circuit 80, can specify a row / column ID 99, which identifies which specific row or column to load. This instruction can specify the row / column ID 99 directly as an immediate parameter or indirectly by specifying a register containing the row / column ID 99.

[0114] The row / column selection parameter 89 can be explicitly encoded in the load instruction 98 using a field within the instruction encoding. This field selects whether row group 90 or column group 92 of the matrix transpose box 74 is loaded with data from memory. Alternatively, the row / column direction selection parameter can be implicitly encoded. For example, there may be a control parameter stored in a control register specifying whether the matrix load instruction 98 should currently select whether to load rows or columns of the matrix transpose box 74. The control parameter in the control register can toggle its state when a row / column direction switching instruction is executed. This avoids the need to specify an explicit row / column direction selection parameter for each matrix load instruction. Furthermore, a combination of parameters specified in the instruction encoding and parameters stored in the control register can be used, where a combination of control register bits and row / column selection bits in the instruction encoding selects which row / column direction to use. For example, control register bits can indicate whether a row / column is selected, but bits in the instruction encoding can select whether bits in the control register are inverted, for example: Of course, other encodings can also be used as alternatives; this is just one example.

[0115] Furthermore, the loading circuit 80 responds to the masking state information 96, 97 to select whether to replace the value loaded into the matrix transpose box 74 with a masked value instead of the value loaded from memory. In this example, the masking state information includes a first masking state signal 96 and a second masking state signal 97.

[0116] The first masking state information 96 is used to control the masking of certain row / column positions to prevent the corresponding row / column group of the matrix transpose frame 74 from being updated based on the corresponding value in memory. For each row / column position in the matrix transpose frame 74, the first masking state information 96 identifies whether the row / column position is a masked row / column position or an unmasked row / column position. That is, if the row / column selection parameter 89 indicates that the element will be written in the form of a row, the masking indication of the first masking state information corresponds to a different row position. If the row / column selection parameter 89 indicates that the element will be written in the form of a column, the masking indication of the first masking state information corresponds to a different column position.

[0117] If the first masking state information 96 specifies that the target row / column to be loaded is an unmasked row / column, then the second masking state information 98 can be used to identify which individual element positions in the target row / column are masked, and the matrix loading circuit 80 obtains the corresponding data from the matrix structure stored in memory and writes the unmasked elements of the target row / column into the corresponding elements 88 of the selected row / column group in the matrix transpose box 74 (while setting any masked elements in the selected row / column group to mask values). Therefore, the second masking state information 98 can provide a set of masking indications, where each masking indication corresponds to a different position extending in the opposite dimension to the position associated with the masking indication of the first masking state information. That is, if the row / column selection parameter 89 indicates that the element will be written in rows, then the masking indication of the second masking state information corresponds to a different column position. If the row / column selection parameter 89 indicates that the element will be written in columns in the matrix transpose box 74, then the masking indication of the second masking state information corresponds to a different row position.

[0118] The first masking state information 96 and the second masking state information 97 together represent two-dimensional masking state information because they indicate the position of the masking elements in the two dimensions of the matrix to be loaded into the matrix transpose frame 74. However, each individual instruction only uses the portion of the first masking state information corresponding to a single target row / column (the portions of the first masking state information related to other rows / columns are ignored). However, the first masking state information 96 and the second masking state information 97 together define the masking position on the entire 2D matrix transpose frame, making it unnecessary to change the masking state data between loading one row / column and the next.

[0119] On the other hand, if the selected row / column position is indicated as a masked row / column position by the first masking state information 96, instead of providing data loaded from memory, the masking value is written to each matrix element 88 in the selected row / column. Here, each element in the selected row / column may share the same item as the first masking state data 96, or all elements in the selected row / column may be marked as masked, or all matrix elements 88 in the selected row / column may be marked as unmasked. When a load instruction specifies a masked row / column; then, in response to the masking state information 96, the matrix loading circuit 80 alternatively writes the masking value to each element in the masked row / column.

[0120] Regardless of whether the masking value is provided to a specific element 88 of the matrix transpose box 74 due to masking of the entire row based on the first masking state data 96 or masking of a single element based on the second masking state data 97, the masking value can be a predetermined value such as zero, or it can be one of a plurality of alternative masking values ​​selectable based on masking selection information, which can be stored in a register or in parameters explicitly specified by the load instruction.

[0121] Addressing information 94 can be stored in general-purpose registers 34 of the CPU, which are also used for general integer operands, or in some examples, in some special matrix addressing information registers, which store information specific to identifying a portion of the matrix structure to be loaded from memory.

[0122] Figures 11 to 13 Examples of ways to encode masking state information and addressing information 94 are shown. Figure 11 In the example, addressing information 94 is specified in general-purpose register 34 and is also used for integer operands. In this case, before executing the matrix load instruction 98, earlier instructions may need to ensure that the referenced general-purpose registers include appropriate address operands to represent the addresses of the desired rows or columns of the matrix, and between the execution of consecutive load instructions 98 for different rows of the input matrix, these address operands need to be updated to point to the next row or column.

[0123] Also in Figure 11 In the example, the first masking state information (mask1) 96 is represented as a bitmap that includes multiple bit flag indicators 100, each bit flag indicator corresponding to a given row / column position within the matrix transpose box 74. The row / column number 99 specified by the load instruction 98 is used to select which bit flag indicator 100 in the masking bitmap 96 to read, and depending on the value of the read bit flag 100, this controls whether the corresponding row is masked (e.g., bit flag 1 may indicate an unmasked row / column, and bit flag 0 may indicate a masked row or column, and vice versa).

[0124] Similarly, the second masking state information (mask2) 97 is represented as a bitmap that includes a plurality of bit flag indicators 101, each bit flag indicator corresponding to a column / row position (the opposite dimension to the position indicated by each bit flag indicator 100 in the mask1 bitmap 96), such that mask2 indicates the position of each masking element within a target row / column having a row / column number 99 specified by the loading instruction 98, as described above.

[0125] The registers storing the first masking state information 96 and the second masking state information 97 can be dedicated registers used to store masking state information for masking matrix operands / processing (and not for other purposes), or they can have dual functions, allowing the same registers to be used for other information when processing instructions other than matrix processing-related instructions. For example, the masking state information 96 and 97 can be read from a predicate register, which can also be used to store vector predicates that control the masking of vector processing channels when vector instructions are executed.

[0126] Figure 12 The first masking state information 96 and the second masking state information 97 are shown again as... Figure 11 Another example of the same bitmap. However, in this case, the matrix processing circuitry can access a set of matrix addressing registers 102 (which at least specifies the base address 104 and the stride value 106), and optionally specify the in-row / in-column offset (sub-part selection information) 108. Using this method, the addressing information register 102 can be set before performing a set of loads for loading all rows or columns of a given input matrix, and it is not necessary to change the addressing information 102 between individual loads of different rows or columns in the same input matrix, because the matrix loading circuitry 80 is able to calculate the address of a single row / column based on the addressing information 102 and the row / column selection number 99 specified by the load instruction 98. Reference Figure 4 Comparing the illustrated memory layouts, base address 104 can be set to point to the beginning of a memory region corresponding to a portion of the matrix to be processed, and stride value 106 can be set to indicate the offset between the address indicating the beginning of a row of the marked matrix data structure and the address indicating the beginning of the next row (or, if a column-first layout is being used). In-row / in-column offset 108 can be used to select a single portion within a row of the entire matrix structure stored in memory, which is useful when the entire matrix structure in memory is larger than the maximum row / column length supported by the hardware within transpose box 74 and matrix processing logic 84. This allows the processing of large data structures in memory to be broken down into smaller blocks that can be processed multiple times by the hardware. Thus, in-row / in-column offsets can select a single portion within a “row” stored in memory. Supporting in-row / in-column offset value 108 is not necessary, as an alternative is to update base address 104 to point to the location of the next block between processing a block of a given row and processing the next block, rather than updating in-row / in-column offset value 108. Furthermore, offset value 108 can alternatively be provided within a general-purpose register referenced as a source register by load instructions.

[0127] Using this method, when processing a single load instruction 98, the matrix load circuit 80 can calculate the address of the data portion to be loaded into the selected row or column of the matrix transpose box 74 by adding the base address to the product of the step value 106 and the row / column number 99 specified by the instruction (optionally offsetting the in-row / column offset value 108 if necessary).

[0128] Figure 13 Another example representing addressing information 94 and masking state information 96, 97 is shown. In this example, addressing information 94 again includes a base address 104, but this time the addressing information also includes an offset data structure 110, which is stored in memory at a location identified by the offset structure base address 112. Here, the offset data structure 110 stored in memory serves both as part of the addressing information 94 and as the first masking state information 96. The second masking state information 97 can still be provided as a separate masking register "mask2", similar to... Figure 11 and Figure 12 Examples.

[0129] Offset data structure 110 defines an array of offset values, where each offset 114 corresponds to a specific row / column number that can be selected by a single matrix load instruction 98. When the load instruction specifies a given row / column number (e.g., column 2, as...), ... Figure 10 When the example shown is executed, the corresponding offset value 114-2 of that column will be selected, and the address of the corresponding row / column data in the matrix structure stored in memory can be derived by adding the selected offset value to the base address stored in the base address register 104. In most cases, if the selected row / column is indicated as an unmasked row or column, loading will proceed normally.

[0130] However, certain offset values ​​are reserved so that they cannot be used as valid offsets, but instead indicate the location of a masked row / column. For example, a reserved offset value could be -1 (i.e., a binary value with the most significant bit 1 and all other bits set to 0 to supplement the representation). Therefore, when calculating the address of a single load instruction, if it is determined that the selected offset value 114-2 of the selected row / column number has a reserved value, this is interpreted as the location of the masked row or column, and thus instead of performing an actual load from a portion of the matrix data structure stored in memory, each element in element 88 of the relevant row or column group 90, 92 of the matrix transpose box 74 is filled with a masked value (e.g., zero).

[0131] Therefore, by using this method, the offset of the corresponding row or column of the input matrix in memory from where it will be loaded into the matrix transpose box is also used as masking state information, which avoids the need for a separate register for masking state values.

[0132] The advantage of using array 110 with offset value 114 as part of the addressing information is that it requires far less storage capacity compared to an alternative method that stores an absolute address table indicating the addresses of the corresponding rows / columns of the matrix data in memory, because the offset can be indicated relative to a common base address, thus requiring fewer bits to represent. However, other specific implementations may omit this. Figure 13 In the example, the base address register 104 makes each offset actually an offset relative to 0, but this would require more bits for each offset value 114.

[0133] Furthermore, using a special reserved value in the offset field 110 to represent the masked row / column position may be more efficient than alternatively supporting padding by storing the padding value in the memory itself and representing the masked column / row by specifying an offset value pointing to the actual location in memory where the padding value is stored in the offset array 110 corresponding to the masked row / column. With this special reserved value method, there is no need to perform an actual load on the memory to obtain the padding value, as the padding value can instead be generated in real-time by the load circuit 80 based on the detected reserved offset value.

[0134] Although Figure 13 An example is shown where offset structure 110 is stored in the memory system at an address derived from offset structure base address 112. However, some microarchitectural designs may optionally provide an offset cache 116 in hardware that caches the values ​​of the offset structure so that the matrix loading circuitry 80 can access them more quickly, avoiding the need to fetch them from memory again in the future. It is recognized that the offset pattern to be applied may often be the same for multiple different locations within the matrix, making it efficient to retain the same offset structure as it can be reused. However, other specific implementations may provide an offset register required at the architecture level to store offset structure 110, making it unnecessary to allocate space for offset structure 110 in memory at all.

[0135] Regardless of how the specific masking state information 96, 97 and addressing information 94 are represented, this function enables the desired portion of the matrix stored in memory to be loaded into the matrix transpose frame 74, allowing the 1×1 convolution of the previously described operation to be applied to that portion of the matrix. Figure 7 As shown, masking allows skipping certain rows of the input to handle wraparound problems. Furthermore, by masking certain rows or columns of the intra-frame matrix, this is useful for providing padding values ​​to handle... Figure 2 Padding convolutions of the type shown can be useful. Furthermore, in some cases, 2D convolution operations can be applied to matrices whose width or height is smaller than the maximum width or height supported by the hardware, thus allowing the use of masking states to mask unused rows or columns at the end of the matrix.

[0136] After a row or column of a given operand matrix is ​​written into the matrix transpose box 74, the data can be read out by the operand move circuit 82 as a group of rows or columns and transferred to the input operand register 70 for matrix processing. The operand move circuit 82 is not limited to reading data from the matrix transpose box 74 in the same row / column direction as the matrix loading circuit 80. In practice, if the data structure for the input operands is stored in memory in a row-major / column-major format different from the output data structure, it may be useful for the operand move circuit 82 to read data in the opposite row / column direction to that used during loading. This real-time transposition of the matrix can be performed in hardware as it is loaded into the matrix transpose box 74 and read out for processing, which is far more efficient than remapping the data layout in memory. Therefore, this can significantly improve the performance of processing input matrices with potentially different memory layouts.

[0137] It is important to note that for any given memory layout of the matrix structure stored in memory, the same layout can be loaded column-by-column or row-by-row into the matrix transpose box 74. Therefore, whether the row / column selection parameter 89 specifies the row direction or the column direction can be chosen completely independently of the actual layout used in the underlying matrix structure in memory. This is because when transposing the matrix using the matrix transpose box, it is irrelevant whether the data is loaded column-by-column or read row-by-row, as both will yield the same result. In practice, when performing such real-time transposes, alternating between loading matrix data row-by-row and column-by-column is useful for better pipelined readings of earlier rows or columns of the matrix to be processed and loading of later rows or columns.

[0138] For example, consider a series of operations where a series of rows of a matrix structure in memory are loaded into rows 0 through 7 of matrix transpose frame 74, but then read out column by column because the output data structure combined with them has the opposite memory layout. In this case, after loading the last row 7 into the matrix transpose frame, operand move circuit 82 can read out columns one by one, starting from column 0 and ending with column 7. However, once the data for column 0 has been read out, the matrix loading circuit 80 can begin loading additional rows of the matrix structure from memory for the next block of the matrix to be processed while operand move circuit 82 continues to read out consecutive columns 1-7 for matrix processing object 84. Since matrix processing logic 84 may still need columns 1-7, it is more efficient to start loading those additional rows of the matrix columns into the corresponding columns 0, 1, 2, etc., as those columns become idle due to the operand move circuit reading them out for processing. Therefore, loading of the later portions of the matrix can be done into the early column positions 0 and 1 in the corresponding columns of the matrix transpose box 74, while the reading of the later columns associated with the previous matrix block is still in progress. For example, once the matrix moved by operand move circuit 82 has read data from a column (e.g., column 2), loading into that column can begin for the next read, thus achieving some performance improvements through pipelined processing. Then, once all columns have been loaded for the next block of the matrix to be processed in memory, the next set of operand move operations performed by operand move circuit 82 can be performed row by row, while loading continues just afterward to fill the row group 90 of the matrix transpose box that was just read by operand move circuit 82. Thus, it can be seen (when using dynamic transpose) that by alternating which direction is used for a set of loads, better performance can be provided than using the same row / column direction throughout the matrix.

[0139] Alternatively, if a specific set of operations is being performed where real-time changes to the matrix layout are not required (e.g., because the output data structure has the same layout in memory as the input data structure), a fixed row / column direction can be chosen for both the matrix load operation and the operand move operation. However, pipelining may still be possible, allowing operands to be read from some rows / columns for processing while the load is executed in other rows / columns.

[0140] exist Figure 10 In the example, to limit the hardware complexity of matrix processing logic 84 and the latency associated with a single instruction, matrix processing logic 84 does not support performing a full matrix multiplication operation on two two-dimensional matrix operands in a single instruction. Instead, it decomposes such a 2D matrix multiplication operation into multiple separate external accumulation operations, each performed on a one-to-one vector operand. Figure 7 The example is used to explain the outer product operation. Figure 7In the example, to generate output matrix 12 from input matrix 10 and kernel matrix 11, Figure 7 The example requires multiplying an 11×4 input matrix 10 by a 4×4 kernel matrix 11 to obtain an 11×4 output matrix 12. A full matrix multiplication operation would require a given output element of output matrix 12 (e.g., ...). Figure 7 The element marked as 200 at position F' should be generated based on the sum of the pairwise products of the corresponding element in the corresponding row 202 of the input matrix 10 and the corresponding element in the corresponding column 204 of the kernel matrix 11. Since matrix multiplication is performed as part of a series of 1×1 convolutions that are accumulated to generate an equivalent of a larger 2D convolution, the result of adding the pairwise products of row 202 and column 204 is added to the previous value of the output matrix 12 of element F' to generate the updated value of element F'.

[0141] However, for each output element position of output matrix 12, such a matrix multiplication operation would require calculating four separate products and then adding the five terms together (the four products and the previous value of the output element). This can be slow to implement and difficult to adapt to the pipeline timing of other operations.

[0142] In contrast, the outer product operation takes the first vector operand u = (u1, u2, ..., u...). m ) and the second vector operand v = (v1, v2, ..., v n Each vector operand consists of a one-dimensional array of elements, which are combined to form a two-dimensional result matrix W. Therefore, each element of the resulting matrix is ​​derived from a single product of a single element of the input vector operand and a single element of the second vector operand.

[0143] For the outer accumulation operation, each element of the updated result matrix W' also depends on the corresponding element in the previous value of the result matrix W: Therefore, even for external accumulation operations, each element only needs to have a single product calculated on an additional term. This allows for faster execution at a lower hardware cost.

[0144] The full matrix multiplication operation can be decomposed into separate outer product operations. For example, when taking... Figure 7 When the vector operand 206, corresponding to a column of the 11×4 input matrix, and the second vector operand 208, corresponding to a row of the kernel matrix 11, are shown, for each pair of column and row positions, each element of the first vector operand 206 is multiplied by the corresponding element of the second vector operand 208 to obtain a 2D array of intermediate results, where, for example, Figure 7The element 200 identified in the middle is derived by multiplying the element marked A in column 206 with the top-left K1 kernel wait in row 208 extracted from kernel matrix 11. By iterating through the outer accumulation operation on each corresponding combination of column positions in input matrix 10 and row positions in kernel matrix 11, after processing each combination of input columns and kernel rows, the result will be the same as performing a full matrix multiplication operation, but at a lower hardware cost.

[0145] Therefore, to support the outer product / accumulation operation performed by the matrix processing logic 84, the input operand register 70 stores one-dimensional vector operands, and the operand shift circuit 82 reads a portion of the input matrix from the matrix transpose box 74 one row or one column at a time. Thus, even if the underlying given operand matrix on which the operation is being performed is a two-dimensional matrix structure, it is treated as a series of one-dimensional vector operands when the matrix processing operation is applied. However, the matrix processing logic 84 can generate a result matrix as a two-dimensional matrix structure in a single instruction, corresponding to the result of applying an outer product / accumulation operation to a pair of vector operands. This means that the operation is still faster than processing individual vector processing instructions, each of which can only generate a single row / column of the result matrix at a time.

[0146] exist Figure 10 In the example, the input registers 70 of the matrix processing logic 84 include two input registers A0 and A1, each for storing the first vector operand, and two input registers B0 and B1, each for storing the second vector operand. Additionally, four result matrix registers C0 to C3 72 are provided, each capable of storing a result matrix in two dimensions (although...). Figure 10 The example shown is a square matrix of size N×N, but other examples may support different heights / widths of the resulting matrix. In some specific implementations, the matrix processing logic may be hardwired to which combination of input registers is used when generating the resulting matrix to be placed in a given result matrix register 72. For example, result matrix registers C0 through C3 may be generated based on the input operand pairs A0*B0; A0*B1; A1*B0; and A1*B1, respectively. This is recognized that, typically when performing matrix processing, it may be necessary to process the same set of rows or columns of an input matrix and the corresponding set of rows or columns of a second input matrix in different combinations. For example, for Figure 7In a 1×1 combination example, column 206 of input matrix 10 not only needs to be multiplied by the element in row 208 of kernel matrix 11 for the first outer product operation, but also needs to be multiplied by the corresponding element in the next row of kernel matrix 11 for subsequent outer product operations, and so on for the remaining rows. Similarly, kernel row 208 may need to be multiplied by multiple different columns 206 in the input matrix. By providing sufficient input register storage 70 to store multiple rows or columns at once, different combinations of rows or columns for operand A and rows or columns for operand B can be implemented by a single set of operand load / shift operations to fill register 70, and then multiple different matrix processing operations for multiple different combinations of operands can be applied to those operands without repeatedly loading / shifting for each individual matrix processing operation. Therefore, Figure 10 The method shown, using four output matrix registers, increases the number of matrix processing instructions processed per matrix load instruction. Other examples may provide additional input register 70 / output register 72, but the exact number of registers chosen can be a trade-off between hardware cost and performance.

[0147] Alternatively, other methods may provide sufficient input operand register storage 70 for only a single vector operand pair, in which case a new value needs to be loaded for the single vector register pair for each different combination of rows / columns of the corresponding input matrices to be multiplied.

[0148] Furthermore, it is not necessary to provide separate register sets for the two operands A and B. In another example, both operands A and B can be selected from the corresponding registers in a single combined register file.

[0149] like Figure 10 As shown, a single matrix processing instruction 240 can specify a given result destination register 72, a pair of input vector registers 70 for providing source operands for the operation, and control information including predicate (masking state) information 242 and shift selection information 244. As mentioned above, in some implementations, the selection of the result matrix register 72 to be used for a given operation may be implicit from the combination of selected source registers 70. Therefore, in this case, the instruction may not need to specify a separate destination register identifier, but providing an additional destination register specifier may be useful if a more arbitrary selection of the destination is allowed.

[0150] Figure 14 The matrix processing logic 84, including the use of predicate information 242 and shift selection information 244, is shown in more detail. Figure 14The diagram illustrates a vector outer product operation applied to a first vector operand 250 stored in an "A" input vector register 70 of the operand storage device and a second vector operand 252 stored in a "B" input vector register of the operand storage device. For example, in the convolution example above, the "A" register can be used for the input matrix 10, while the B register can be used for the kernel weights 11.

[0151] Matrix processing logic 84 includes position shifting circuitry 260, which applies a variable position shift between an element of one of the input operands 250 and the corresponding element position in the output matrix 270 generated in response to matrix processing instruction 240. Shift information 244 may be represented as an explicit parameter within the matrix processing instruction 240 or by control parameters stored in a control register. Shift parameter 244 specifies one of a plurality of variable shift amounts. Based on the selected shift amount, the position shifting circuitry activates a plurality of multiplexers to select which input elements from the first vector operand 250 are provided to each element position within the shift input operand 272. For example, if variable shift amount 0 is selected, each element of the input vector 250 is passed to the element at the corresponding position in the shift input vector 272, while if variable shift amount 1 is selected, the element at the given element position in the shift input vector 272 is set to the value of the element at the next highest element position in the original input vector 250. For the element at the highest position within the shift input vector 272, if a variable shift amount greater than 0 is chosen, a padding value 274 can be provided because there is no higher element position to inject within the original input vector. Similarly, for higher shift amounts, larger position shifts can be applied to adjust which position of the input vector 250 is provided in the shift input vector 272. No shift is applied to the second vector operand 252, which is used only at its original position.

[0152] Then, matrix processing logic 84 performs an outer product operation, such that, according to the expression... Each element C'[i,j] is generated, where i iterates over all rows of the resulting matrix C'[i, j], and j iterates over all columns of the resulting matrix C'[i, j]. Here, the predicate bit P[i] corresponding to a given row position i in the resulting matrix specifies whether the row is masked (inactive) or not masked (active). In this example, inactive rows of output matrix 270 are indicated by a predicate bit equal to 0, while active rows are indicated by a predicate bit 1. However, it should be understood that other examples may employ the inverse mapping of predicate values, such that predicate bit 1 can be used to identify inactive rows, and predicate bit 0 can be used to identify active rows. For inactive rows, in this example, it is assumed that the corresponding element of the shifted input vector 272 is replaced with a zero mask value, but other examples may use non-zero mask values.

[0153] Therefore, by using this method, the variable position shift provided by the position shifting circuit 260 helps to support Figure 8 The method shown, in which, after loading a specific vector 250 representing a given row or column of the input matrix into the input operand register 70, can execute multiple matrix processing instructions specifying different values ​​of a variable shift amount 244. These instructions act on the exact same contents of the input vector 250 in register 70 to interpret the relative position shifts between the input vector 250 and the output matrix 270, which are required to apply kernel weights for different kernel locations, such as... Figure 8 As shown. This avoids the need to reload vector operand register 250 for each kernel location. Furthermore, providing the predicate function using predicate value 242 helps handle the need to skip certain lines, such as... Figure 8 As shown, to resolve the issue regarding Figure 7 The discussion revolves around the problem. Predicates can also help handle situations where the number of rows in a column is insufficient to fill the entire vector supported by the hardware.

[0154] Although Figure 14 The diagram shows a position shift circuit 260 provided between reading the input vector operand 250 from the given input register 70 and providing the shifted operand to the matrix processing logic 84 to perform the outer product / accumulation operation. However, the position shift can also be applied between the matrix processing logic 84 that generates the result of the outer product / accumulation operation and the result that is written back to the result matrix register 72, although this approach is slightly more complex because if an accumulation operation is being performed, this also requires shifting a portion of the previous values ​​of the output matrix that are read as inputs to the outer product / accumulation operation (i.e., C[i,j] in the expression above).

[0155] Therefore, the above information is provided. Figures 10 to 14The features discussed help to handle matrix processing functions within the pipeline more efficiently, processing 2D convolution operations that are very common in machine learning. It is understandable that programmers may find other uses for the same functions, so these functions do not necessarily need to be specifically designed for such 2D convolution operations.

[0156] Although Figure 10 A matrix transpose frame 74 is shown. This frame can be used to allow the same set of instructions to process different layouts of matrix structures in memory, regardless of their memory layout. However, matrix transpose frame 74 is not required, and some implementations may omit it. In such cases, if there are differences in the memory layouts of the input and output matrices, any transpose needs to be handled separately by remapping the data stored in memory using load / store instructions before applying any matrix processing operation, or by generating the output and then converting its format before writing it back to the data structure in memory corresponding to the output. If matrix transpose frame 74 is not provided, matrix loading circuitry 80 may alternatively load rows or columns of the matrix structure in memory directly into the matrix processing logic-readable input register 70 when performing matrix processing operations.

[0157] Furthermore, in some specific implementations, it may not be necessary to provide an input operand register 70 at all, just as if a matrix transpose frame 74 were provided. In such cases, the matrix processing logic 84 could directly read its operands from the storage element 88 of the matrix transpose frame 74. Therefore, although it is generally possible to provide some operand storage circuitry for loading rows or columns of a matrix by the matrix loading circuitry 80, and the matrix processing logic 84 can obtain operands from these operand storage circuitry, it is not necessary to provide both the matrix transpose frame 74 and the input operand register 70 simultaneously. These two can be provided separately, or they can be provided as follows: Figure 10 The examples provide them in the same combination.

[0158] Although Figure 10 An example is shown that applies to a square matrix where the number of rows and columns in the matrix are equal, but this is not required, and other examples may support asymmetric numbers of rows and columns.

[0159] Providing both the row / column masking and position shifting functions would maximize performance, but this is not necessary, and some implementations may provide only one or the other of these functions.

[0160] Figure 15This is a flowchart illustrating a method for processing a matrix load instruction in one example where masking is applied during a load operation. When such an instruction is encountered at step 300, at step 302, the instruction decoder 30 decodes the load instruction to generate control signals that control the matrix load circuit 80 to obtain first masking state data 96 from internal registers within the CPU 60 (e.g., in register set 34 or in internal registers associated with the matrix load circuit 80), from data structure 110 in memory, or from offset cache 116. The first masking state data 96 is “whole row / column” masking state data, indicating whether an entire row / column is masked. Obtaining the entire first masking state data 96 is not required for the matrix load circuit 80; it is sufficient to refer to the masking indication 100 or 114 corresponding to the row / column number 99 of the target row / column to be loaded. Therefore, at step 304, the matrix load circuit determines, based on the obtained first masking state data 96, whether the row / column number 99 specified by the matrix load instruction corresponds to a masked row or column position within the input matrix being processed. If the specified row / column is a masked row / column, then at step 306, data with a masked value is loaded for the corresponding portion of the operand storage circuits 74, 70 corresponding to the target row / column, instead of actually loading the memory for the corresponding portion of the matrix data structure stored in memory. The masking value can be selected from multiple options based on a selection parameter encoded by the load instruction or specified elsewhere in the control register. Alternatively, some implementations may always use a fixed masking value, such as zero, by default.

[0161] On the other hand, if the target row or column position is not a masked row or column position, then at step 308, the matrix loading circuit 80 obtains second masking state data 97, which is per-element masking state data indicating the position of any individual masked column / row position within the target row / column. At step 310, the matrix loading circuit determines whether any active element exists within the target row / column (even if the first masking state data 96 indicates that the target row / column is not masked, the second masking state data 97 may have already set all elements in the target row / column to inactive). If at least one active element exists in the target row / column, then at step 312, the matrix loading circuit 80 triggers a loading operation to read a portion of the matrix data structure corresponding to the target row or column from memory. Figure 12In the example, for instance, the address from which data is loaded can be derived from the addressing information 94 by adding the base address 104 to a multiple of the row / column number and the specified stride 106. Then, after retrieving the relevant data block from memory, for any active element within that row or column, the loaded data is written to the corresponding storage element 88 of the matrix transpose frame 74, or directly loaded into the corresponding portion of the selected input operand register 70. In contrast, for any inactive element in the target row / column indicated by the second masking state data 97, a masking value is filled into the corresponding storage element 88 or a portion of the selected input operand register 70; this masking value can be zero or non-zero, and can be fixed or programmable.

[0162] If, at step 310, the matrix loading circuit 80 determines that all elements in the target row / column are indicated as inactive by the second masking state data 97, then at step 314, the loading operation is prevented from occurring, and a masking value is filled for each element in the target row / column in the operand storage circuit (i.e., the storage element 88 of the matrix transpose box 74 or the input operand register 70), without any loading from memory being performed at all.

[0163] Although Figure 15 Two separate steps 302, 308 are shown for obtaining the first masking state data 96 and the second masking state data 97, but other examples may obtain two masking state data 96, 97 at step 302 before checking whether the target row / column is masked by the first masking state 96.

[0164] Figure 16 A first example of processing matrix processing instruction 240 in an implementation supporting masking applied during matrix processing is shown. At step 320, the pipeline instruction decoder 30 identifies the instruction to be processed as a matrix processing instruction and generates control signals to control the matrix processing circuitry 46 to process the instruction. At step 322, in response to these control signals, the matrix processing logic 84 obtains a first operand and a second operand based on information stored in operand storage circuits 70, 74. As previously discussed, these operands may be obtained directly from the matrix transpose box 74 or from the input operand register 70. Furthermore, the matrix processing circuitry obtains masking state data 96 (e.g., predicate vector 242, such as...). Figure 14As shown (indicated), the masking state data indicates the masked row / column positions, for which input values ​​will be treated as if they represent masking values. At step 324, the matrix processing circuit 46 performs a matrix processing operation on the first and second operands to generate a two-dimensional result matrix, which can then be written back to a result matrix register in the result matrix register 72. For example, this operation could be an external accumulation operation as discussed above, where the first and second operands are vector operands. For any inactive row / column indicated as masked by the masking state data 96, the corresponding elements of the result matrix can retain their previous values, or alternatively, can be set to values ​​that would be generated if the corresponding input value were set to the masking value.

[0165] Figure 17 It shows the supporting reference Figure 8 and Figure 14 The described implementation of the variable position shift feature is a second example of processing matrix processing instructions. Steps 320, 322, and 324 are similar to... Figure 16 The corresponding steps ( Figure 17 The masking features are not explicitly shown, but may still be provided in some implementations. However, in Figure 17 China also supports Figure 14 The position shift function is shown. At step 326, the matrix processing circuit 46 selects one of a plurality of alternative shift values ​​based on the variable shift value 244 specified by the matrix processing instruction. Although Figure 14 It shows three different possible shift amounts to correspond to Figure 8 The examples shown are of the three options, but it should be understood that other specific implementations supporting larger kernel sizes may require more than three different shift amounts to choose from. Alternatively, to limit the complexity of the position shifting circuit 260, even if larger kernel sizes are supported, the position shift may be limited to a certain maximum size, and this may still be possible if additional loading is required to support larger kernel sizes.

[0166] Therefore, at step 328, the position shifting circuit 260 applies a variable position shift based on the shift amount selected at step 326, such that which row or column of the 2D result matrix 270 is updated is changed based on a given element of one of the input operands 250. Figure 17 At step 324, matrix processing operations are then applied based on the variable position shift to generate the result matrix 270.

[0167] Therefore, in summary, these ideas help support more efficient hardware to support the processing of 2D convolution operations, which are common operations in the fields of machine learning and image processing.

[0168] Other examples are illustrated in the following clauses: (1) An apparatus comprising: A matrix processing circuit is used to perform matrix processing operations on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix. Operand storage circuitry, the operand storage circuitry being used to store information about the first input operand and the second input operand used to form the matrix processing circuitry; and A position shifting circuit is used to apply a variable position shift to change which row or column of the result matrix is ​​updated during a given matrix processing operation, based on a given element of one of the first input operands and the second input operand stored in the operand storage circuit. The variable position shift is based on one of a plurality of alternative shift amounts selectable for the given matrix processing operation, each alternative shift amount corresponding to a different number of rows or columns of the first input operand and the second input operand shifted relative to the result matrix.

[0169] (2) The apparatus according to clause (1), wherein the first input operand and the second input operand comprise one-dimensional vector operands.

[0170] (3) The apparatus according to clause (2), wherein the matrix processing operation includes an outer product operation applied to the first input operand and the second input operand to generate the result matrix.

[0171] (4) The apparatus according to clause (3), wherein the outer product operation includes an outer accumulation operation, and for the outer accumulation operation, the result matrix includes an updated value of a corresponding element of an accumulator matrix, wherein the updated value of a given element of the accumulator matrix corresponds to the result of adding the previous value of the given element of the accumulator matrix to the corresponding element of the outer product result matrix, the corresponding element corresponding to the result of performing the outer product operation on the first input operand and the second input operand.

[0172] (5) The apparatus according to any of the preceding clauses, wherein the position shifting circuit is configured to select the one of the plurality of alternative shift amounts based on parameters specified by a matrix processing instruction, the matrix processing instruction being used to control the matrix processing circuit to perform the matrix processing operation.

[0173] (6) The apparatus according to any of the preceding clauses, wherein when a given row or column of the result matrix corresponds to an active row or column position indicated by predicate information accessible to the matrix processing circuit, the matrix processing circuit is configured to generate elements of the given row or column of the result matrix, the elements having values ​​of the corresponding row or column depending on one of the first input operand and the second input operand, the corresponding row or column being selected according to one of a plurality of alternative shift amounts selected for the given matrix processing operation; and when the given row or column corresponds to an inactive row or column position indicated by the predicate information, the matrix processing circuit is configured to generate elements of the given row or column of the result matrix value, the elements having values ​​of the corresponding row or column independent of one of the first input operand and the second input operand.

[0174] (7) The apparatus according to any of the preceding clauses, wherein the operand storage circuit includes a matrix transpose circuit including a plurality of storage cells for storing corresponding matrix elements of a given operand matrix, wherein the storage cells of the matrix transpose circuit are readable in row groups corresponding to rows of the given operand matrix and in column groups corresponding to columns of the given operand matrix.

[0175] (8) The apparatus according to clause (7), wherein: when the given operand matrix is ​​written to the matrix transpose circuit in the form of row groups, the matrix transpose circuit is configured to support reading the given operand matrix from the matrix transpose circuit in the form of column groups; and when the given operand matrix is ​​written to the matrix transpose circuit in the form of column groups, the matrix transpose circuit is configured to support reading the given operand matrix from the matrix transpose circuit in the form of row groups.

[0176] (9) The apparatus according to any one of clauses (7) and (8), wherein the operand storage circuitry includes an operand register for storing the first input operand and the second input operand for the matrix processing operation; and The device includes an operand shift circuit that, in response to a shift command, reads at least one row or column of the given operand matrix from the matrix transpose circuit and writes the at least one row or column into the operand register.

[0177] (10) The apparatus according to any one of clauses (7) to (9), wherein the apparatus includes an operand shifting circuit that, in response to a matrix processing instruction, reads at least one row or column of the given operand matrix from the matrix transpose circuit and provides the at least one row or column as one of the first input operand and the second input operand to the matrix processing circuit.

[0178] (11) An apparatus according to any of the preceding clauses, the apparatus comprising a loading circuit that, in response to a loading instruction, loads information corresponding to a target row or column of a given operand matrix into an operand storage circuit based on a portion of a matrix data structure stored in memory; wherein: in response to the loading instruction, the loading circuit is configured to obtain masking state data indicating the positions of one or more masked rows or columns within the given operand matrix, and when the target row or column corresponds to the masked row or column position indicated by the masking state data, the loading circuit is configured to load data with masking values ​​from a portion of the operand storage circuit corresponding to the target row or column, rather than based on data from that portion of the matrix data structure stored in memory.

[0179] (12) The apparatus according to any of the preceding clauses, wherein the matrix processing circuit is configured to generate the result matrix from the first input operand and the second input operand in response to a single instruction.

[0180] (13) An apparatus comprising: means for performing a matrix processing operation on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix; means for storing information forming the first input operand and the second input operand for the means for execution; and means for applying a variable position shift to change which row or column of the result matrix is ​​updated during a given matrix processing operation based on a given element of one of the first input operand and the second input operand stored in the means for storage, the variable position shifting based on one of a plurality of alternative shift amounts selectable for the given matrix processing operation, each alternative shift amount corresponding to a shift of the position of one of the first input operand and the second input operand relative to a different number of rows or rows and columns of the result matrix.

[0181] (14) A data processing method comprising: performing a matrix processing operation on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix and the first input operand and the second input operand depend on information stored in an operand storage circuit; and during a given matrix processing operation, applying a variable position shift to change which row or column of the result matrix is ​​updated based on a given element of one of the first input operand and the second input operand stored in the operand storage circuit, the variable position shifting based on one of a plurality of alternative shift amounts selectable for the given matrix processing operation, each alternative shift amount corresponding to a different row number or row / column position shift of the first input operand and the second input operand relative to the result matrix.

[0182] In this application, the phrase "configured as..." is used to mean that the elements of the device are configured to perform the defined operations. In this context, "configuration" means the arrangement or manner of interconnection of hardware or software. For example, the device may have dedicated hardware that provides the defined operations, or a processor or other processing device may be programmed to perform the function. "Configured as" does not mean that the elements of the device need to be changed in any way to provide the defined operations.

[0183] While exemplary embodiments of the invention have been described in detail herein with reference to the accompanying drawings, it should be understood that the invention is not limited to those precise embodiments, and various changes and modifications can be made therein by those skilled in the art without departing from the scope of the invention as defined by the appended claims.

Claims

1. An apparatus comprising: A matrix processing circuit, wherein the matrix processing circuit is used to perform matrix processing operations on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix; An operand storage circuit is provided for storing information about the first input operand and the second input operand used to form the matrix processing circuit. and A masking circuit for performing a masking operation to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuit based on masking state data, the masking state data indicating one or more masked row or column positions of one of the first input operands and the second input operands, the input values ​​of the one or more masked row or column positions being treated as representing masking values.

2. The apparatus of claim 1, wherein the masking value is zero.

3. The apparatus according to claim 1 or 2, wherein the masking value is selected from a plurality of masking values ​​according to at least one of the following: Masking value selection parameters, which are specified by instructions that enable the masking operation to be performed; Control values, which are stored in a control register; and A masking vector that specifies individual masking values ​​for multiple elements of a masked row or column.

4. The apparatus of claim 1 or 2, wherein the masking state data has an encoding that identifies elements within a two-dimensional array of elements that will be considered to represent the masking value.

5. The apparatus of claim 4, wherein the masking state data specifies: First masking state data, indicating one or more masked row or column positions, wherein for each of the one or more masked row or column positions, all elements in that masked row or column position are considered to represent the masking value; and The second masking state data indicates whether the positions of individual elements within a given row or column should be masked.

6. The apparatus of claim 1 or 2, wherein the masking state data has an encoding capable of indicating at least two non-adjacent row or column positions separated by at least one unmasked row or column position as masked row or column positions.

7. The apparatus of claim 1 or 2, wherein the operand storage circuit includes a matrix transpose circuit, the matrix transpose circuit including a plurality of storage units for storing corresponding matrix elements of a given operand matrix, wherein the storage units of the matrix transpose circuit are readable in row groups corresponding to rows of the given operand matrix and are also readable in column groups corresponding to columns of the given operand matrix.

8. The apparatus according to claim 7, wherein: When the given operand matrix is ​​written to the matrix transpose circuit in the form of row groups, the matrix transpose circuit is configured to support reading the given operand matrix from the matrix transpose circuit in the form of column groups; and When the given operand matrix is ​​written to the matrix transpose circuit in column group form, the matrix transpose circuit is configured to support reading the given operand matrix from the matrix transpose circuit in row group form.

9. The apparatus according to claim 1 or 2, wherein: The matrix processing circuit includes the masking circuit and performs the matrix processing operation in response to the masking information, wherein a portion of the row or column position of one of the first input operands and the second input operands corresponding to the one or more masks is regarded as representing the masking value, rather than the actual value of the portion of the first input operand and the second input operand stored in the operand storage circuit.

10. The apparatus of claim 1, further comprising a loading circuit that, in response to a loading instruction, loads information corresponding to a target row or column of a given operand matrix into the operand storage circuit based on a portion of a matrix data structure stored in memory; wherein: The loading circuit includes the masking circuit, and when the target row or column corresponds to the masked row or column position indicated by the masking state data, the loading circuit is configured to load data with the masking value from a portion of the operand storage circuit corresponding to the target row or column, rather than data based on the portion of the matrix data structure stored in memory.

11. The apparatus of claim 10, wherein, in response to the loading instruction, when the masking state data corresponding to the target row or column indicates that the target row or column corresponds to a masked row or column position, the loading circuit is configured to determine whether each of the plurality of matrix elements of the target row or column should be masked based on a shared item of the masking state data shared among the plurality of matrix elements of the target row or column.

12. The apparatus of any one of claims 10 and 11, wherein the masking state data comprises a plurality of offset values, each offset value corresponding to a corresponding row or column position of the given operand matrix, and indicating the offset of the address of the corresponding portion of the matrix data structure in memory relative to a base address; and The masked row or column position is indicated by the offset value of the masked row or column position having a predetermined reserved offset value.

13. The apparatus of any one of claims 10 and 11, wherein the loading circuit is configured to obtain the masking state data from a memory based on masking state addressing information stored in at least one masking state addressing register.

14. The apparatus of claim 11, wherein the loading circuit is configured to determine, based on addressing information, the target address in memory of the portion of the matrix data structure.

15. The apparatus of claim 14, wherein the addressing information comprises a plurality of address pointers, each address pointer indicating an address in the matrix data structure corresponding to a portion of a corresponding row or column position of the given operand matrix.

16. The apparatus of claim 14, wherein the addressing information includes: The base address of the matrix data structure; as well as A stride value, which indicates the difference between the address of a portion of the matrix data structure corresponding to a row or column of the given operand matrix and the address of the portion of the matrix data structure corresponding to the next row or column of the given operand matrix.

17. The apparatus of claim 14, wherein the addressing information includes: The base address of the matrix data structure; as well as Offset information, which includes one of the following: Multiple offset values, each offset value corresponding to a specific row or column position of the given operand matrix, and indicating the offset of the corresponding portion of the matrix data structure in memory relative to the base address; as well as Offset data structure address, which indicates the address of the data structure in memory that provides the plurality of offset values.

18. The apparatus according to any one of claims 14 to 17, wherein the addressing information further includes sub-part selection information for selecting which sub-part of the portion of the matrix data structure identified by the addressing information in the memory will be loaded into the operand storage circuit.

19. The apparatus according to any one of claims 14 to 17, the apparatus comprising: At least one addressing register, wherein the at least one addressing register is used to store the addressing information; as well as A prefetch circuit, the prefetch circuit being configured to generate a prefetch request for prefetching a portion of the given operand matrix from memory, based on the addressing information stored in the at least one addressing register.

20. The apparatus according to any one of claims 1-2, 10-11 and 14-17, wherein the first input operand and the second input operand are one-dimensional vector operands.

21. The apparatus according to any one of claims 1-2, 10-11 and 14-17, wherein the matrix processing operation includes an outer product operation applied to the first input operand and the second input operand to generate the result matrix.

22. The apparatus of claim 21, wherein the outer product operation includes an outer accumulation operation, and for the outer accumulation operation, the result matrix includes updated values ​​of corresponding elements of an accumulator matrix, wherein the updated value of a given element of the accumulator matrix corresponds to the result of adding a previous value of the given element of the accumulator matrix to a corresponding element of the outer product result matrix, the corresponding element corresponding to the result of performing the outer product operation on the first input operand and the second input operand.

23. The apparatus according to any one of claims 1-2, 10-11 and 14-17, wherein the matrix processing circuitry is configured to generate the result matrix from the first input operand and the second input operand in response to a single instruction.

24. An apparatus comprising: A component for performing matrix processing operations on a first input operand and a second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix; A component for storing information forming the first input operand and the second input operand for the component to be executed; as well as A component for performing a masking operation to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuit based on masking state data, the masking state data indicating one or more masked row or column positions of one of the first input operands and the second input operand, the input values ​​of the one or more masked row or column positions being treated as representing masking values.

25. A data processing method, comprising: The operand storage circuit stores information used to form the first and second input operands for matrix processing operations; as well as Perform matrix processing operations on the first input operand and the second input operand to generate a result matrix, wherein the result matrix is ​​a two-dimensional matrix; as well as A masking operation is performed to mask at least a portion of the matrix processing operation or the information stored in the operand storage circuit based on masking state data, wherein the masking state data indicates one or more masked row or column positions of one or more of the first input operands and the second input operands, and the input values ​​of the one or more masked row or column positions are treated as representing masking values.

Images