Method and device for performing tile traversal on tensor for convolution operation

Abstract

Methods and apparatus for tile traversal of tensors for convolution operations are disclosed. An apparatus for performing convolution on an input tensor includes: a parameter generator for generating a horizontal hardware execution parameter for a horizontal dimension of the input tensor based on a kernel parameter or a layer parameter, and generating a vertical hardware execution parameter for a vertical dimension of the input tensor based on the kernel parameter or the layer parameter; an accelerator interface to configure a hardware accelerator circuit based on the horizontal hardware execution parameters and the vertical hardware execution parameters; a horizontal iterator controller to determine when the hardware accelerator circuit completes a first horizontal iteration of the convolution; and a vertical iterator controller to determine when the hardware accelerator circuit completes a first vertical iteration of the convolution.

Description

[0001] Divisional Application Instructions

[0002] This application is a divisional application of a Chinese invention patent application with an application date of June 24, 2020, application number 202010583620.8, and titled “Method and device for performing block traversal on a tensor to perform convolution operations”. Technical Field

[0003] The present disclosure relates generally to computer processing, and more particularly, to tile walking tensors to perform convolution operations. Background Art

[0004] Artificial intelligence (AI), machine learning (ML), and deep learning (DL) models use neural networks to convolve data to identify features on the input data. The input data can be a large data set called a tensor. Tensors store data units that represent the characteristics of objects (e.g., images, videos, audio, radar imaging, etc.) analyzed by AI, ML, and / or DL ​​models (sometimes referred to herein as AI / ML / DL models). In this way, the AI / ML / DL model uses convolution to perform a feature recognition process to identify features in different types of data in the input tensor. Through such convolution, the neural network can generate one or more probability values ​​that indicate the possibility of one or more features being present in the input data. In this way, a device or computer can use the probability values ​​to confirm the presence of one or more features with one or more highest probabilities in the input data. Summary of the invention

[0005] According to an embodiment of the present disclosure, a device for performing convolution on an input tensor is provided, the device comprising: a parameter generator, for performing the following operations: generating horizontal hardware execution parameters for the horizontal dimension of the input tensor based on at least one of kernel parameters or layer parameters, and generating vertical hardware execution parameters for the vertical dimension of the input tensor based on at least one of the kernel parameters or the layer parameters; an accelerator interface, for configuring a hardware accelerator circuit based on the horizontal hardware execution parameters and the vertical hardware execution parameters; a horizontal iterator controller, for determining when the hardware accelerator circuit completes a first horizontal iteration of the convolution; and a vertical iterator controller, for determining when the hardware accelerator circuit completes a first vertical iteration of the convolution.

[0006] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium having instructions is provided, which, when executed by a controller, causes the controller to perform at least the following operations: generate horizontal hardware execution parameters for the horizontal dimension of an input tensor based on at least one of kernel parameters or layer parameters; generate vertical hardware execution parameters for the vertical dimension of the input tensor based on at least one of the kernel parameters or the layer parameters; configure a hardware accelerator circuit based on the horizontal hardware execution parameters and the vertical hardware execution parameters; determine when the hardware accelerator circuit completes the first horizontal iteration of the convolution; and determine when the hardware accelerator circuit completes the first vertical iteration of the convolution.

[0007] According to an embodiment of the present disclosure, a method for performing convolution on an input tensor is provided, the method comprising: generating horizontal hardware execution parameters for a horizontal dimension of the input tensor based on at least one of kernel parameters or layer parameters; generating vertical hardware execution parameters for a vertical dimension of the input tensor based on at least one of the kernel parameters or the layer parameters; configuring a hardware accelerator circuit based on the horizontal hardware execution parameters and the vertical hardware execution parameters; determining when the hardware accelerator circuit completes a first horizontal iteration of the convolution; and determining when the hardware accelerator circuit completes a first vertical iteration of the convolution.

[0008] According to an embodiment of the present disclosure, a device for performing convolution on an input tensor is provided, the device comprising: a parameter generating device for performing the following operations: generating horizontal hardware execution parameters for the horizontal dimension of the input tensor based on at least one of kernel parameters or layer parameters; and generating vertical hardware execution parameters for the vertical dimension of the input tensor based on at least one of the kernel parameters or the layer parameters; a configuring device for configuring a convolution execution device based on the horizontal hardware execution parameters and the vertical hardware execution parameters; a horizontal iterator control device for determining when the convolution execution device completes a first horizontal iteration of the convolution; and a vertical iterator control device for determining when the convolution execution device completes a first vertical iteration of the convolution. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 An example acceleration manager and an example hardware accelerator are shown that perform convolutions of input tensors based on a hybrid firmware-hardware tile traversal.

[0010] Figure 2 yes Figure 1 Figure 2. Example input tensor for , showing the horizontal, vertical, and depth dimensions of a neural network tile traversal.

[0011] Figure 3is an example flow of stages of a compiler, firmware, and hardware accelerator to perform convolution of an input tensor based on a hybrid firmware-hardware tile traversal according to examples disclosed herein.

[0012] Figure 4 shows how hardware execution parameters are defined and Figure 1 and Figure 2 An example input tensor associated with a tile edge.

[0013] Figure 5 shows how the hardware execution parameters are defined Figure 1 , Figure 2 and Figure 4 Input pointer to the tile of sample input tensors.

[0014] Figure 6 An example output tensor with an output size defined by the hardware parameters is shown.

[0015] Figure 7 is an example iteration order for a neural network to perform convolution of an input tensor based on a hybrid firmware-hardware tile traversal according to examples disclosed herein.

[0016] Figures 8A-8H Included are example lines of code (LOC) representing machine readable instructions that may be executed to perform convolution of an input tensor based on a hybrid firmware-hardware tile traversal according to examples disclosed herein.

[0017] Fig.9A and Fig. 9B An example flow diagram representing example machine readable instructions that may be executed to perform convolution of an input tensor based on a hybrid firmware-hardware tile traversal according to examples disclosed herein is shown.

[0018] Fig.10 is a block diagram of an example processor platform according to examples disclosed herein, the processor platform being configured to perform Figure 7 , Figures 8A-8H , and / or Fig.9A and Fig. 9B The instructions represented in Figure 1 An example graph compiler, acceleration manager, and / or hardware accelerator is provided to perform convolution of an input tensor based on a hybrid firmware-hardware tile traversal.

[0019] Wherever possible, the same reference numerals are used throughout the drawings (one or more) and the accompanying written description to refer to the same or similar parts. The connecting lines or connectors shown in the various figures presented are intended to represent example functional relationships and / or physical or logical couplings between the various elements.

[0020] In this article, when identifying multiple elements or components that can be mentioned separately, the descriptors "first", "second", "third", etc. are used. Unless otherwise specified or otherwise understood based on the context of its use, such descriptors are not intended to impart any meaning of priority, physical order, or arrangement in a list or temporal order, but are simply used as labels for separately referring to multiple elements or components to facilitate understanding of the disclosed examples. In some examples, the descriptor "first" can be used to refer to a certain element in a specific embodiment, while the same element can be referred to in a claim with different descriptors, such as "second" or "third". In this case, it should be understood that such descriptors are used only for easy reference to multiple elements or components. DETAILED DESCRIPTION

[0021] Artificial intelligence (AI), including machine learning (ML), deep learning (DL), and / or other artificial machine-driven logic, enables machines (e.g., computers, logic circuits, etc.) to process input data using a model to generate outputs based on patterns and / or associations previously learned by the model through a training process. For example, the model can be trained to recognize patterns and / or associations using data during the training phase. In the recognition phase, the model follows such patterns and / or associations when processing input data so that other (one or more) inputs produce (one or more) outputs consistent with the patterns and / or associations recognized during the training phase. Applications that employ AI, ML, DL, and / or other artificial machine-driven logic involve many computational processes. Examples of such computational processes include convolution operations on input tensors to numerically identify and detect features in the input data. For example, the input data to the AI / ML / DL model may be an image (or any other data) represented by a multidimensional tensor using cell values. Such cell values ​​represent features of the input data that will be analyzed for possible recognition by the AI / ML / DL model. As part of such analysis, the AI / ML / DL model may use filters, some of which are implemented by a convolution engine. During the training phase, different weights are selected for different nodes of the filter by learning which weight values ​​and / or combinations of weight values ​​produce acceptable results through the AI / ML / DL model. During the recognition phase, these weight values ​​are convolved with the cell values ​​of the input tensor so that the convolution engine can emphasize and / or filter different characteristics about the input data based on the training of the AI / ML / DL model during training. However, for large input tensors, performing such convolutions only in software or firmware may be resource-limiting due to the number of execution cycles and therefore the time required to process the entire input tensor.

[0022] The examples disclosed herein provide a hybrid firmware-hardware technique for performing convolution operations on an input tensor. For example, the techniques disclosed herein partition an input tensor into multiple tiles, and use a combination of firmware and hardware acceleration to perform multiple per-tile convolution operations by tile traversal of the tiles of the input tensor. The examples disclosed herein can be used to implement hardware acceleration of DL algorithms for AI applications. The examples disclosed herein can be implemented using a ternary neural network (TNN) model or architecture (e.g., a hardware accelerator for implementing convolutions on binary and ternary data precision values). In the examples disclosed herein, the TNN is a convolutional neural network (CNN) model or architecture constructed to analyze input data (e.g., an input tensor). The examples disclosed herein can be implemented using Nervana TM The examples disclosed herein are implemented using an inference neural network processor (NNP-I), which includes 12 inference compute engines (ICE) low-precision deep learning accelerators, one or more of which can implement TNN. Additionally or alternatively, the examples disclosed herein can be implemented using any other appropriate type of AI / ML / DL model and / or AI / ML / DL architecture.

[0023] A tensor is a data structure for organizing data in an arrangement representing one or more dimensions. For example, an array can be a one-dimensional tensor (e.g., a vector), and a two-dimensional matrix can be a two-dimensional tensor (e.g., (x, y)). However, when used to organize data in three or more dimensions to represent, for example, real-world characteristics as input data for a deep learning model, tensors become more complex and useful in the fields of AI and deep learning. When performing tensor mathematical operations, a tensor can be divided into multiple smaller tiles, and the smaller tiles can be processed separately to achieve processing of the entire tensor. As used herein, a tile is a subset of the cell values ​​of a corresponding larger tensor. For example, a tile can have a three-dimensional structure substantially similar to a corresponding larger tensor, but the tile has a smaller dimension size (e.g., a smaller horizontal (X) dimension, a vertical (Y) dimension, and / or a depth (Z) dimension) and only a subset of the cell values ​​of the larger tensor. In addition, as described below, the tiles can be divided into smaller micro-tiles, which can have a three-dimensional structure substantially similar to the corresponding larger tiles, but the micro-tiles have smaller dimensional sizes (e.g., smaller horizontal (X) dimensions, vertical (Y) dimensions, and / or depth (Z) dimensions) and only a subset of the cell values ​​of the larger tiles. In this way, the tensor includes multiple tiles, and each tile includes multiple micro-tiles. Processing a tensor as smaller tiles and / or micro-tiles as disclosed herein is referred to as tile traversal in this article. In order to tile traverse a tensor, multiple nested loops can be used to iteratively process multiple tiles of the tensor by accessing tensor data for each tile from memory and performing tensor operations tile by tile until the tile traversal processes all tiles of the tensor. In some examples, tiles can be processed in parallel to process the entire tensor faster.

[0024] The examples disclosed herein perform a two-stage data tile traversal to perform convolutions on tiles of an input tensor, such that the complex portion of the tile traversal is implemented on firmware that can be programmed and / or compiled to run on a scalar digital signal processor (DSP) (e.g., an application-specific instruction set processor (ASIP)) that communicates with a hardware accelerator that implements a convolution engine. The examples disclosed herein employ a convolution engine in a hardware accelerator to perform a more detailed, cell-value-based tile traversal on tiles of an input tensor. This enables the firmware to flexibly configure / manage complex tile traversals for different scenarios (e.g., different input tensor dimensions, different data types, different types of convolutions, etc.), while providing efficient hardware execution to perform numerical processing intensive operations when convolving cell values ​​of tiles with different weight values. In the examples disclosed herein, the input tensor may be divided into smaller tiles, and the tiles may be divided into micro-tiles or execution blocks (also referred to herein as blocks) (e.g., 2×2 micro-tiles, 4×4 micro-tiles, etc.), and the hardware accelerator may manage the smaller tiles under the guidance or specific definition provided by the firmware through multiple parameters. In the examples disclosed herein, the firmware "understands" the larger picture of multiple tiles that form the entire input tensor. In the examples disclosed herein, the firmware "understands" the larger picture because the firmware is provided with the dimensions and / or characteristics of the entire input tensor, the firmware divides the input tensor into smaller tiles and / or micro-tiles, and the firmware manages the processing of the tiles / micro-tiles by coordinating how the hardware accelerator performs convolution on each tile / micro-tile, so that the final result is a convolution of the entire tensor managed by the firmware. When managing such convolutions of the input tensor, the firmware can support complex functions, such as padding tile data, specifying different traversal modes based on the input tensor size, specifying different convolution spans based on the input tensor size, etc.

[0025] Figure 1 An exemplary system 100 includes an exemplary deep learning accelerator 101 including an exemplary acceleration manager 102 and an exemplary hardware accelerator 104 to perform convolution of an input tensor (e.g., input tensor 106) based on a hybrid firmware-hardware tile traversal. Figure 1 In the example of , the example acceleration manager 102 is implemented by the example programmable controller 108. The example programmable controller 108 can be implemented using a DSP, an ASIP, and / or any other suitable programmable device (e.g., a controller or a processor) capable of executing firmware instructions according to the teachings of the present disclosure. The example deep learning accelerator 101 also includes a local memory 110 (e.g., a static random access memory (SRAM), a dynamic random access memory (DRAM), etc.), which can be implemented as an on-chip memory or a memory in the same chip package as the deep learning accelerator 101. In addition, in Figure 1In the example of FIG. 1 , system 100 includes an example host processor 112 and an example system memory 114. Figure 1 As shown, the deep learning accelerator 101 and the host processor 112 communicate with the system memory 114. The example host processor 112 includes an example graph compiler 116, which can be implemented as software executed by the host processor 112. The example graph compiler 116 compiles a graphical representation of software code, parameters, and / or data for use by the acceleration manager 102 and / or the hardware accelerator 104 to implement the AI / ML / DL model. For example, as described in more detail below, the graph compiler 116 interprets input parameters corresponding to input tensors and generates additional parameters for use by the acceleration manager 102 to manage hybrid firmware-hardware tile traversals processed by the acceleration manager 102 and the hardware accelerator 104. In some examples, the graph compiler 116 can perform its processing in advance, possibly on a different system separate from the system 100. In such an example, the result of the graph compiler 116 is a compilation workload of parameters (e.g., layer parameters 132 and / or kernel parameters 134) and / or compiler decisions (e.g., compiler decisions 142) that can be loaded by the host processor 112 for access by the deep learning accelerator 101. For example, the host processor 112 can load the compilation workload of parameters and / or compiler decisions to the system memory 114 for access by the deep learning accelerator 101, and / or can provide the compilation workload directly to the deep learning accelerator 101.

[0026] The example hardware accelerator 104 is a circuit (e.g., a hardware accelerator circuit) that implements an accelerator for deep learning convolution operations. For example, the hardware accelerator 104 may be implemented using logic circuits (e.g., an integrated circuit (IC), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), etc.). Figure 1 Although a single hardware accelerator 104 is shown in FIG, the example deep learning accelerator 101 may be provided with multiple hardware accelerators substantially similar or identical to the hardware accelerator 104. In such an example, multiple hardware accelerators 104 may be used to perform convolution operations on multiple tiles / micro-tiles in parallel. To implement the convolution operations, the hardware accelerator 104 is provided with an example convolution engine 118. In addition, the hardware accelerator 104 is provided with example parameter configuration registers 122 in which parameters from the acceleration manager 102 are stored to configure the convolution engine 118 to perform convolution operations on tiles from the input tensor 106. Although in FIG. Figure 1 A single convolution engine 118 is shown in FIG, but in other examples, the hardware accelerator 104 may be provided with multiple convolution engines 118 to perform convolution operations on multiple tiles / micro-tiles in parallel. Figure 1In the illustrated example of FIG, to program the parameter configuration register 122, the acceleration manager 102 communicates with the hardware accelerator 104 via an example message bus 124 (e.g., a control bus). In the example shown, the message bus 124 is 96 bits wide. However, the example message bus 124 can have any other suitable bit width. Figure 1 In the illustrated example of , the acceleration manager 102 programs and controls the operation of the hardware accelerator 104 via the example message bus 124. For example, the acceleration manager 102 transmits a command message to the hardware accelerator 104. The command message defines the neural network computation structure parameters for programming and storage in the parameter configuration register 122. In the examples disclosed herein, the command message includes parameters such as an address pointer from the acceleration manager 102 to inform the hardware accelerator 104 of the memory address where the tile data (e.g., cell value) is located in the local memory 110. In this way, the hardware accelerator 104 can access the tile data from the local memory 110 so that the convolution engine 118 can traverse the tile to perform a convolution operation, thereby convolving the input tensor 106 (or a portion thereof) with one or more weight values. Other example parameters transmitted by the acceleration manager 102 to the hardware accelerator 104 are described below. By providing the hardware accelerator 104 (e.g., the convolution engine 118 and the parameter configuration registers 122) as a hardware circuit implemented separately from the acceleration manager 102, the acceleration manager 102 can be exempted from performing complex convolution operations, and instead offloads them to the hardware accelerator 104, which is configured to perform such convolution operations in an efficient manner. In addition, by providing the deep learning accelerator 101 (e.g., the acceleration manager 102 and the hardware accelerator 104) separately from the host processor 112, the host processor 112 is exempted from managing convolutions and / or performing convolution operations, and instead offloads them to the deep learning accelerator 101 (e.g., managing convolutions performed by the acceleration manager 102 and convolution operations performed by the hardware accelerator 104).

[0027] exist Figure 1In the example of , the system 100 includes an example direct memory access (DMA) controller 126. The example DMA controller 126 can be used to provide direct memory access to the deep learning accelerator 101 to copy tile data directly from a memory location in the system memory 114 to a memory location of the local memory 110 of the deep learning accelerator 101. In this way, the deep learning accelerator 101 can retrieve tile data directly from the system memory 114 (e.g., using the DMA controller 126 without relaying or accessing such data through the example programmable controller 108 and / or the example host processor 112). In this way, data access speed is improved by using the DMA controller 126. In some examples, the memory address location of the tile data can be provided to the DMA controller 126 by the acceleration manager 102 and / or the host processor 112. Additionally or alternatively, such a memory address location of the tile data can be provided to the DMA controller 126 by the hardware accelerator 104 based on an address pointer programmed into the parameter configuration register 122. In the examples disclosed herein, the DMA controller 126 is managed by a DMA program that is part of a compiled workload generated by the graphics compiler 116 (e.g., the DMA program can be part of the compiler decision 142). In this manner, by generating a DMA program, the graphics compiler 116 can control a specific pass to be implemented by the hardware accelerator 104 to perform convolution on the input tensor 106. Based on the DMA program, the DMA controller 126 autonomously obtains tiles from the input tensor 106 stored in the system memory 114, loads the tile data into the local memory 110, and signals the acceleration manager 102 (e.g., firmware) that the input tile data is available in the local memory 110. In the examples disclosed herein, the DMA controller 126 also copies kernel data (e.g., weight data) used for the convolution from the system memory 114 to the local memory 110 based on the DMA program for access by the hardware accelerator 104. Additionally, in examples disclosed herein, the DMA controller 126 copies output data generated by the hardware accelerator 104 from the local memory 110 to the system memory 114 to store the output tensors in the system memory 114. In other examples, the DMA controller 126 may be omitted, and the example programmable controller 108 and / or the example host processor 112 may perform data retrieval operations from the system memory 114 and store the retrieved data in the local memory 110 and / or provide the retrieved data to the hardware accelerator 104. In such examples, the example programmable controller 108 and / or the example host processor 112 also copies the output data from the local memory 110 to the system memory 114.

[0028] Briefly turn to Figure 2 , showing in more detail Figure 1 106 to emphasize the horizontal (X), vertical (Y), and depth (Z) dimensions of a neural network tile traversal. The example tile traversals disclosed herein are performed using a multi-channel two-dimensional convolutional layer (e.g., a multi-channel Conv2D). In such an example, the output data (e.g., an output feature map (OFM)) is composed of Figure 1 The acceleration manager 102 and the hardware accelerator 104 calculate a 3D structure based on convolving three-dimensional (3D) input data (e.g., an input feature map (IFM)) with four-dimensional (4D) weight values. The example input tensor 106 may be very large. By dividing the input tensor 106 into multiple smaller tiles, the smaller tiles can be copied to the example cache 128 in the hardware accelerator 104 to perform the convolution operation. In the examples disclosed herein, the convolution is performed by convolving a layer of cell values ​​of the tile with the weight values. For example, this can be accomplished by performing multiple iterations of the convolution layer by layer until the hardware accelerator 104 completes the convolution of all layers of the tile. Figure 2 In the example of , the layers of the input tensor 106 are defined as unit values ​​extending along the horizontal (X) dimension and the vertical (Y) dimension. In such a tile structure configuration, multiple layers are arranged along the depth (Z) direction. In order to reuse the locally cached tile data as much as possible during the convolution operation, the acceleration manager 102 uses different types of tile traversal structures to process tensors of different dimensions, such as those best suited for higher cache data reuse. Examples of different tile traversal structures include a batch traversal structure, an input reuse traversal structure, and a weight reuse traversal structure. In the example batch traversal structure, an additional dimension is added to the weight value of the convolution so that the convolution can simultaneously convolve two layers of the tile with their respective weight values. The example input reuse traversal structure involves reusing previous input data (e.g., IFM data) of a tile layer (e.g., as shown in Figure 1) when calculating output data (e.g., OFM data) along the depth (Z) dimension. Figure 2 An example weight reuse traversal structure involves reusing weight values ​​of multiple tile layers (e.g., as shown in FIG. 1 ) when computing output data (e.g., OFM data) in horizontal (X) and vertical (Y) dimensions. Figure 2 In other examples, any other suitable combination of traversal structures may be used, and may be based on the dimensions and edges of the tiles.

[0029] Figure 2 The example of shows that the input tensor 106 is divided into a plurality of smaller 3D tiles and micro-tiles (e.g., Figure 2 4×4 micro-tiles in the example). Example convolution engine 118 ( Figure 1) traverses smaller tiles (e.g., processes smaller tiles stored in local memory 110 rather than trying to process the entire tensor at once) to more efficiently process the larger input tensor 106 stored in system memory 114. The acceleration manager 102 ( Figure 1 ) considers multiple parameters to divide the responsibility of tile traversal between the hardware accelerator 104 and the firmware implementing the acceleration manager 102. That is, the complexity of dividing the responsibility of tile traversal involves dividing the traversal into two levels of tile traversal, including a firmware-based traversal portion (e.g., the first level in the two-level tile traversal) and a hardware-based traversal portion (e.g., the second level in the two-level tile traversal). In this way, the examples disclosed herein implement hardware-assisted firmware to perform hybrid firmware-hardware convolution operations. To improve performance, the acceleration manager 102 handles the complex management portion of the traversal in the firmware to generate parameters and commands to control the hardware accelerator 104 to perform the recursive and mathematically intensive convolution portion of the traversal. In the examples disclosed herein, the firmware-based acceleration manager 102 provides commands to configure the hardware accelerator 104 to convolve a sufficient amount of input data (e.g., a tile size or number of tiles) so that the hardware accelerator 104 does not end and stall prematurely before the firmware-based acceleration manager 102 can provide the next input data for convolution. In this manner, the performance of the example hybrid firmware-hardware tile traversal techniques disclosed herein may be improved by increasing (e.g., maximizing) the throughput of both firmware-based operations and hardware-based operations. Increasing (e.g., maximizing) such throughput involves reducing (e.g., minimizing) stalls of the firmware-based acceleration manager 102 between hardware-based convolution operations, and reducing (e.g., minimizing) stalls of the hardware accelerator 104 between firmware-based management operations. For example, to increase the likelihood that the hardware accelerator 104 runs convolution operations in a substantially continuous manner, the acceleration manager 102 calculates the next operation to be performed by the hardware accelerator 104 while the hardware accelerator 104 is performing the current convolution operation. The acceleration manager 102 then instructs the hardware accelerator 104 regarding the next operation by sending a command message to the hardware accelerator 104 via the message bus 124 (e.g., in a first-in, first-out (FIFO) manner). Thus, in the steady state of the system 100, detailed instructions for the next operation are ready for the hardware accelerator 104 to continuously process the next micro-tile waiting in, for example, a FIFO buffer in the local memory 110. In this way, the high utilization rate of the hardware accelerator 104 is utilized by keeping the hardware accelerator 104 busy with pipeline commands to process incoming micro-tiles, which provides high processing performance for the entire system 100.

[0030] return Figure 1, the example system memory 114 stores example layer parameters 132, example kernel parameters 134, and example hardware execution parameters 136 (e.g., hardware parameters 136). The example layer parameters 132 define operational characteristics for processing the input tensor 106, as well as structural characteristics of the input tensor 106. In the examples disclosed herein, the layer parameters are defined or set based on the input tensor 106 and apply to the entire tensor convolution to be performed on the input tensor 106. The example layer parameters 132 include one or more of the following: example operation parameters, example input dimension size parameters, example output dimension size parameters, example data type parameters, example kernel (filter) size parameters, example padding mode parameters, example IFM stride parameters, example pooling mode / size / stride parameters, and example quantization and activation parameters. The example operation parameters define the type of computational operation to be performed on the input tensor 106, which may include, for example, convolution operations, pooling operations, fully connected operations, etc. The example input and output dimension size parameters define the horizontal (X), vertical (Y), and depth (Z) dimension sizes of the input tensor to be convolved and the output tensor to be generated by the convolution process.

[0031] Example data type parameters define the data types of input tensor data (e.g., IFM data), weight values, and output tensor data (e.g., OFM). The data type indicates how many bits are used per cell to represent the cell value of the input tensor. Example data types for input tensor data, weight values, and output tensor data are INT1 (e.g., one bit per cell), INT2 (e.g., two bits per cell), INT4 (e.g., four bits per cell), and INT8 (e.g., eight bits per cell). In some examples, the data type is a signed integer value (e.g., eight bits represent a numeric range of -128 to 127). In addition, an example data type for output tensor data is INT32 (e.g., 32 bits per cell). The INT32 data type for output tensor data can be used in situations where multiplying input data with weights during convolution produces a larger output value that needs to be represented by more bits (e.g., 32-bit integer values) than the number of bits per cell value representing the input data. However, in some examples, a quantization operation can be performed on the output data to reduce the number of bits used to represent the resulting output data (e.g., by truncating the least significant bit). Through quantization, the output data bit value per unit can be kept in the lower bit range of INT1, INT2, INT4, and / or INT8, but the precision is lower than that of using more bits per unit. Therefore, using more bits per unit value can provide higher precision, but also requires more memory capacity. Alternatively, using fewer bits per unit value reduces precision, but requires less memory capacity to store tile data.

[0032] The example kernel (filter) size parameter defines the number of taps in the horizontal (X) dimension and the number of taps in the vertical (Y) dimension to be configured into the convolution engine 118 to perform the convolution operation. The example padding mode parameter is used to specify how much (if any) padding data needs to be added in any dimension to generate the output tensor. For example, padding data can be used when one or more dimensions of the input tensor are smaller than the expected dimension (or dimensions) of the output tensor. The example IFM stride parameter specifies whether to perform a computation operation (e.g., convolution) by processing any input value from the IFM, or whether to skip two or more elements between convolution operations. The acceleration manager 102 can use this parameter to manage the order in which tiles of the input tensor 106 are provided to the hardware accelerator 104 for processing. The example pooling mode / size / stride parameter can be used to define how to configure the pooling layer of the neural network in the convolution engine 118 to reduce or downsample the output data represented by fewer unit values ​​(e.g., smaller horizontal (X) and vertical (Y) dimensions). Example quantization and activation parameters are used to configure the accumulators of the convolution engine 118 to output fewer bits per cell value based on the desired precision of the cell values ​​of the output tensor.

[0033] The example kernel parameters 134 include layer parameters 132 and additional parameters that define weight data and structural characteristics of the convolution to be performed on the input tensor 106, and where to access the tile data of the input tensor 106 in the system memory 114. For example, the kernel parameters 134 define the dimensional characteristics of the input tile data and the output tile data of the convolution to be performed, and the weight values ​​to be convolved with the input tile data to generate the output tile data. The example kernel parameters 134 include example tile dimension size parameters and example buffer pointer parameters. The example tile dimension size parameters define the size of the horizontal (X), vertical (Y), and depth (Z) dimensions based on the number of cells per dimension of the input tile data and the weight values ​​to be used to perform the convolution on the input tile data. The example buffer pointer parameters define address pointers to memory (e.g., system memory 114) where input data (e.g., input tensor 106, input tiles, and / or input microtiles) are stored for retrieval by the hardware accelerator 104, where output data (e.g., output microtiles, output tiles, and / or output tensors) are to be stored by the hardware accelerator 104, and where weight values ​​are stored for retrieval by the hardware accelerator 104 to perform the convolution.

[0034] The example hardware execution parameters 136 represent interfaces (e.g., knobs) exposed by the hardware accelerator 104 to the acceleration manager 102 (e.g., firmware) so that the acceleration manager 102 can configure the hardware accelerator 104 to perform convolution. In the examples disclosed herein, a set of hardware execution parameters 136 is generated for a corresponding tile layer. When a tile layer is provided to the hardware accelerator 104, a corresponding set of hardware execution parameters 136 for the layer is programmed into the parameter configuration registers 122 of the hardware accelerator 104 to configure how the convolution engine 118 performs convolution on the layer. The example hardware execution parameters 136 include an example input depth (Z) size parameter, an example output size parameter, an example edge parameter, an example start row parameter, an example buffer pointer parameter, an example bias setting parameter, and an example eviction parameter.

[0035] The example input depth (Z) size parameter defines the size of the depth (Z) dimension of the input micro-tile. In the examples disclosed herein, the depth (Z) size of the input micro-tile data is the only explicit input data dimension required by the hardware accelerator 104, because the hardware accelerator 104 can derive the input size of the horizontal (X) dimension and vertical (Y) dimension of the micro-tile based on the horizontal (X) dimension and vertical (Y) dimension defined in the output dimension parameter and / or based on the fill mode. The example output size parameter defines the horizontal (X) dimension, vertical (Y) dimension, and depth (Z) dimension of the output data (e.g., output micro-tile) to be generated by the hardware accelerator 104. Figure 6 The output size is described in more detail. Example edge parameters define whether an edge (e.g., left edge, right edge, top edge, bottom edge) of an input micro-tile is an internal edge (e.g., adjacent to another tile) or an external edge (e.g., the outer boundary of the input tensor 106). Figure 4 The tile edges are described in more detail. The example start row parameter defines the row of the tile of the input tensor being processed. For example, if the input tensor 106 has four rows of tiles (e.g., along the vertical (Y) dimension), and the current row of tiles being processed is the second row, then the example start row parameter is set to two to specify the second row as the current row for the convolution operation. The example buffer pointer parameter specifies an address pointer to a local memory (e.g., local memory 110) where the input data of the tile is stored for retrieval by the hardware accelerator 104. The example buffer pointer parameter includes multiple pointers that define limits or boundaries on how the input data is stored based on the structural horizontal (X) dimension, vertical (Y) dimension, and depth (Z) dimension of smaller sized tiles of data (referred to herein as microtiles, which form larger tiles). The following is combined with Figure 4 and Figure 5 Describe the example micro-tile. Figure 5Describes an example input pointer that may be represented in the buffer pointer parameter. An example bias setting parameter sets an initial bias value in an accumulator of the convolution engine 118 that the convolution engine 118 adds to the accumulator when performing a convolution as a tile traversal is performed on multiple tiles of the input tensor 106. In some examples, the accumulator is reset to the initial bias value when the accumulator is reset to begin the convolution of the next row and / or column. An example eviction parameter specifies when the last tile of a tile traversal is reached. For example, the eviction parameter is set to false when the current tile is not the last tile of the input tensor 106 to be processed, and the eviction parameter is set to true when the current tile is the last tile of the input tensor 106 to be processed.

[0036] Figure 3 The generation and / or use of example layer parameters 132, example kernel parameters 134, and / or hardware execution parameters 136 associated with an example flow of stages including an example graphics compiler stage 302, an example firmware stage 304, and an example hardware accelerator stage 306 are shown. In the example graphics compiler stage 302, the graphics compiler 116 ( Figure 1 ) will input tensor 106 ( Figure 1 ) into smaller tiles (e.g., Figure 4 and Figure 5 The graphics compiler 116 may further include a graphics compiler 116 that defines a buffer pointer table to point to different memory locations where cell values ​​of the tiles are stored in the system memory 114. In the illustrated example of the graphics compiler stage 302, the graphics compiler 116 receives the layer parameters 132 as user input and generates the kernel parameters 134 based on the layer parameters 132.

[0037] exist Figure 3 In the illustrated example of FIG. 1 , the graphics compiler 116 also generates a graph based on the layer parameters 132 and / or tiles (eg, Figure 4 and Figure 5 The example tile partitioning and example buffer allocation information are shown in FIG. Figure 114 as a compiler decision 142 stored in system memory 114. The buffer allocation information specifies an amount of memory space to be allocated in system memory 114 to store data of input tensors 106 and / or output tensors to be generated based on convolution operations of hybrid firmware-hardware tile traversals. For example, the graph compiler 116 may generate the buffer allocation information based on example input dimension size parameters, example output dimension size parameters, example data type parameters, example fill mode parameters, and / or example quantization and activation parameters. In the example shown, the host processor 112 ( Figure 1 ) can use the buffer allocation information of the compiler decision 142 to make a memory allocation request to allocate memory space in the system memory 114 to store the input tensor 106. Figure 3 In the example graph compiler stage 302 of FIG. 1 , the graph compiler 116 may also generate and store Figure 1 Examples of compiler decisions 142 include other types of compiler-generated information.

[0038] exist Figure 3 In the example firmware stage 304 of , the acceleration manager 102 ( Figure 1 ) establishes a convolution operation for the input tensor 106 based on the tile traversal to be performed by the hardware accelerator 104. For example, the acceleration manager 102 accesses the system memory 114 ( Figure 1 ) and uses the layer parameters 132 and / or kernel parameters 134 to generate state information and uses the hardware execution parameters 136 ( Figure 1 ) to configure the hardware accelerator 104 to perform tile-based convolution. In the example shown, the state information includes a pointer to a tile of the input tensor 106 (e.g., a tile of a surface). The pointer to the tile of the input tensor 106 specifies the local memory 110 ( Figure 1 ) at an address location where the DMA controller 126 has stored the data of the input tile (e.g., copied from the system memory 114 to the local memory 110). In this way, the acceleration manager 102 uses the pointer to the tile to identify the location in the local memory 110 to access the data of the input tile. In this way, the acceleration manager 102 can manage or coordinate multiple tile-based convolutions of multiple input tiles and reconstruct an output tensor based on multiple output tiles produced by the convolution operations. In the example firmware stage 304, the acceleration manager 102 also generates iterators to traverse multiple tiles of the input tensor 106 in the horizontal (X) dimension, the vertical (Y) dimension, and the depth (Z) dimension. The acceleration manager 102 uses the multi-dimensional iterator to control the hardware accelerator 104 to implement tile traversal across all tiles of the input tensor 106, thereby performing multiple iterations of convolution along multiple dimensions of the input tensor 106.

[0039] exist Figure 3 In the example accelerator stage 306 of FIG. 1 , the layer parameters 132 and the hardware execution parameters 136 are provided to the hardware accelerator 104 ( Figure 1 In some examples, the acceleration manager 102 generates one or more commands to provide the layer parameters 132 and the hardware execution parameters 136 to the hardware accelerator 104 and transmits the commands to the hardware accelerator 104 via the message bus 124 ( Figure 1 ) sends one or more commands to the hardware accelerator 104. Additionally or alternatively, the hardware accelerator 104 accesses the layer parameters 132 and the hardware execution parameters 136 from the system memory 114 based on the one or more commands received from the acceleration manager 102 via the message bus 124. In any case, the hardware accelerator 104 programs the layer parameters 132 and the hardware execution parameters 136 in the parameter configuration registers 122 ( Figure 1 ) to configure how the convolution engine 118 performs convolution on the input tile data to calculate output data (e.g., OFM data for the convolved tile).

[0040] although Figure 3 The example stages of the graphics compiler 102, 304, 306 are shown as accessing and / or generating parameters as input and / or output, but it should be understood that only some of the above-mentioned layer parameters 132 (and / or other layer parameters) can be used to generate kernel parameters 134 and / or compiler decisions 142 at the graphics compiler stage 302. Similarly, only some of the above-mentioned layer parameters 132 and / or kernel parameters 134 (and / or other layer parameters and / or kernel parameters) can be used to generate hardware execution parameters 136 and / or pointers to tiles within a surface at the firmware stage 304. In addition, only some of the above-mentioned layer parameters 132 and / or hardware execution parameters 136 (and / or other layer parameters and / or hardware execution parameters) can be provided to configure the hardware accelerator 104.

[0041] Figure 4 Example hardware execution parameters 136 are shown ( Figure 1 )How to define the tile edges associated with the example input tensor 106. Figure 4 The example of FIG. 400 shows an example tile 402, which is the current tile to be processed. The acceleration manager 102 decomposes the example tile 402 into smaller micro tiles. For example, the parameter generator 154 ( Figure 1) generates hardware execution parameters 136 (e.g., example input depth (Z) size parameters, example output size parameters, example edge parameters, example start row parameters, example buffer pointer parameters, etc.) that define a smaller micro-tile structure to divide the input tensor 106 into smaller micro-tiles. As used herein, an edge of a tile or micro-tile refers to a two-dimensional side of the tile or micro-tile. An example micro-tile 404 is shown in an enlarged view to emphasize the edges shown as an example left edge 408, an example right edge 410, an example top edge 412, and an example bottom edge 414. In the examples disclosed herein, the example left edge 408, the example right edge 410, the example top edge 412, and the example bottom edge 414 are represented in the hardware execution parameters 136 as example edge parameters. In the illustrated example, the hardware accelerator 104 performs a tile-based convolution on the tile 402 by iteratively convolving smaller micro-tiles such as the micro-tile 404 along multiple dimensions of the input tensor 106. To perform such iterative convolutions, the example left edge 408, the example right edge 410, the example top edge 412, and the example bottom edge 414 of each micro-tile are programmed into the parameter configuration register 122 to configure the convolution engine 118 for each corresponding convolution operation for that tile. Although the micro-tile has six edges (e.g., six sides), the examples disclosed herein set per-tile convolution operations based on four edges of the micro-tile because the convolution operations performed are 2D convolutions along the horizontal (X) dimension and the vertical (Y) dimension.

[0042] In the examples disclosed herein, the acceleration manager 102 configures the hardware accelerator 104 to know when it is processing an outer micro-tile (e.g., a micro-tile located at the outer surface edge of the tile 402) so that the hardware accelerator 104 knows when to reset its horizontal (X) and / or vertical (Y) accumulators for convolution operations in the horizontal (X) and vertical (Y) dimensions. Figure 4 , micro-tile 404 is a corner micro-tile that places its two edges on the outer surfaces of tile 402 and input tensor 106. Specifically, left edge 408 is set equal to zero (e.g., left edge=0) because left edge 408 is located at the outer surface edge of input tensor 106, right edge 410 is set equal to a null value (e.g., right edge=no edge) because right edge 410 is located at the inner surface edge of input tensor 106, top edge 412 is set equal to zero (e.g., top edge=0) because top edge 412 is located at the outer surface edge of input tensor 106, and bottom edge 414 is set equal to a null value (e.g., bottom edge=no edge) because the bottom edge is located at the inner surface edge of input tensor 106. In some examples, the inner surface edge is represented by a value of 1 (e.g., right edge=1, bottom edge=1) or any other value other than null.

[0043] Figure 5Example hardware execution parameters 136 are shown ( Figure 1 ) How to define an example input pointer 502 for a tile of an example input tensor 106. Figure 5 In the example of , the input pointer 502 points to the internal micro-tile 504 of the tile 402. The example input pointer 502 can be represented by the buffer pointer parameter of the hardware execution parameter 136 to specify the address in the local memory 110 where the input data of the micro-tile 504 is stored for retrieval by the hardware accelerator 104 ( Figure 1 ). To perform a tile traversal across tiles 402, the acceleration manager 102 iteratively updates the input pointer 502 along the micro-tiles of the tile 402 in the horizontal (X), vertical (Y), and depth (Z) dimensions to provide the hardware accelerator 104 with access to input data for subsequent micro-tiles to be convolved. During the tile traversal, the acceleration manager 102 also updates the weight pointer so that the hardware accelerator 104 knows the location of the weight values ​​stored in the local memory 110 for performing the convolution on each micro-tile. In addition, the acceleration manager 102 also updates the output pointer so that the hardware accelerator 104 knows the location of the local memory 110 to store the output data generated by the convolution operation for each micro-tile.

[0044] Figure 6 An example output tensor 600 is shown having example hardware execution parameters 136 ( Figure 1 ). The example output size parameters define the horizontal (X) dimension, the vertical (Y) dimension, and the depth (Z) dimension of the output micro-tile 602 to be generated by the hardware accelerator 104. That is, the acceleration manager 102 stores the output size parameters in the parameter configuration registers 122 to configure the hardware accelerator 104 with respect to the size of the output micro-tile 602 to be calculated by the convolution engine 118. In the examples disclosed herein, the output size parameters define fixed sizes for the horizontal (X), vertical (Y), and depth (Z) dimensions that apply to all output micro-tiles to be generated for the input tensor. However, in some cases, one or more output micro-tiles at one or more edges of the output tensor 600 need to be smaller than the fixed size defined by the output size parameters set in the parameter configuration registers 122. This may occur when the horizontal (X) dimension size, the vertical (Y) dimension size, and / or the depth (Z) dimension size cannot be evenly divided by the fixed output size of the output micro-tile. In such an example, the acceleration manager 102 may update the output size parameters set in the parameter configuration registers 122 on a per micro-tile basis. Figure 6 In the example of , this smaller output micro-tile calculation is shown as an oversized excess portion 604 that is excluded from the output micro-tile 602 calculated by the convolution engine 118.

[0045] return Figure 1 , in order to implement convolution management in firmware (e.g., in Figure 3 ), the example acceleration manager 102 includes an example memory interface 152, an example parameter generator 154, an example horizontal iterator controller 156, an example vertical iterator controller 158, an example depth iterator controller 160, an example command generator 162, and an example accelerator interface 164. The example memory interface 152 accesses the system memory 114 to retrieve and / or store multiple of the following items: example layer parameters 132, example kernel parameters 134, example hardware execution parameters 136, tensor data, and / or example compiler decisions 142. The example parameter generator 154 generates example hardware execution parameters 136, and / or a pointer to an address in the local memory 110 to copy a tile from the input surface 106 to the system memory 114. In some examples, the parameter generator 154 may also generate one or more example kernel parameters 134. The example horizontal iterator controller 156 generates a horizontal (X) dimension iterator counter to iterate over tiles of the input tensor 106 in the horizontal (X) dimension. The acceleration manager 102 uses a horizontal (X) dimension iterator counter to control the hardware accelerator 104 to perform tile traversal across tiles of the input tensor 106 along the horizontal (X) dimension. The example vertical iterator controller 158 generates a vertical (Y) dimension iterator counter to iterate over tiles of the input tensor 106 in the vertical (Y) dimension. The acceleration manager 102 uses a vertical (Y) dimension iterator counter to control the hardware accelerator 104 to perform tile traversal across tiles of the input tensor 106 along the vertical (Y) dimension. The example depth iterator controller 160 generates a depth (Z) dimension iterator counter to iterate over tiles of the input tensor 106 in the depth (Z) dimension. The acceleration manager 102 uses a depth (Z) dimension iterator counter to control the hardware accelerator 104 to perform tile traversal across tiles of the input tensor 106 along the depth (Z) dimension. The following is combined with Figure 7 An example iteration order that may be implemented by the horizontal iterator controller 156, the vertical iterator controller 158, and the depth iterator controller 160 is described.

[0046] The example command generator 162 generates commands to program multiple parameters in the layer parameters 132 and / or the hardware execution parameters 136 in the parameter configuration register 122 to set the hardware accelerator 104 to perform convolution of the tile in a tile traversal manner. In some examples, the command may include the value of the layer parameters 132 and / or the hardware execution parameters 136 in one or more fields of a message structure sent by the acceleration manager 102 to the hardware accelerator 104. In other examples, the command instructs the hardware accelerator 104 to retrieve multiple parameters in the layer parameters 132 and / or the hardware execution parameters 136 from the system memory 114 to be programmed into the parameter configuration register 122. The example accelerator interface 164 sends or transmits the command from the command generator 162 to the hardware accelerator 104 via the message bus 124.

[0047] Although in Figure 1 An example manner of implementing the acceleration manager 102, the hardware accelerator 104, and the graphics compiler 116 is shown in FIG. Figure 1 One or more of the elements, processes, and / or devices shown in the drawings may be combined, divided, rearranged, omitted, eliminated, and / or implemented in any other manner. Figure 1The example graphics compiler 116, the example memory interface 152, the example parameter generator 154, the example horizontal iterator controller 156, the example vertical iterator controller 158, the example depth iterator controller 160, the example command generator 162, the example accelerator interface 164, the example convolution engine 118, and / or more generally the example acceleration manager 102 and / or the hardware accelerator 104 may be implemented by hardware, software, firmware, and / or any combination of hardware, software, and / or firmware. Thus, for example, any of the example graphics compiler 116, the example memory interface 152, the example parameter generator 154, the example horizontal iterator controller 156, the example vertical iterator controller 158, the example depth iterator controller 160, the example command generator 162, the example accelerator interface 164, the example convolution engine 118, and / or more generally the example acceleration manager 102 and / or the hardware accelerator 104 may be implemented by one or more analog or digital circuits, logic circuits, (one or more) programmable processors, (one or more) programmable controllers, (one or more) graphics processing units (GPUs), (one or more) digital signal processors (DSPs), (one or more) application specific integrated circuits (ASICs), (one or more) programmable logic devices (PLDs), and / or, (one or more) field programmable logic devices (FPLDs). When any apparatus or system claims of this patent are read to cover pure software and / or firmware implementations, at least one of the example graph compiler 116, the example memory interface 152, the example parameter generator 154, the example horizontal iterator controller 156, the example vertical iterator controller 158, the example depth iterator controller 160, the example command generator 162, the example accelerator interface 164, and / or the example convolution engine 118 is expressly defined as including a non-transitory computer-readable storage device or storage disk, such as a memory, a digital versatile disk (DVD), a compact disk (CD), a Blu-ray disk, etc., including software and / or firmware. Further, Figure 1 The example graphics compiler 116, the example acceleration manager 102, and / or the example hardware accelerator 104 may include, in addition to Figure 1 In addition to or in place of the components shown in Figure 1 The present invention may include one or more elements, processes and / or devices of the elements shown in, and / or may include more than one of any or all of the elements, processes and / or devices shown. As used herein, the phrase "communicating with..." (including its variants) covers direct communication and / or indirect communication through one or more intermediate components, and does not require direct physical (e.g., wired) communication and / or constant communication, but additionally includes selective communication at regular intervals, scheduled intervals, non-periodic intervals, and / or one-time events.

[0048] In the examples disclosed herein, the device for generating parameters can be implemented by the example parameter generator 154. In the examples disclosed herein, the device for controlling the horizontal iterator can be implemented by the example horizontal iterator controller 156. In the examples disclosed herein, the device for controlling the vertical iterator can be implemented by the example vertical iterator controller 158. In the examples disclosed herein, the device for controlling the depth iterator can be implemented by the example depth iterator controller 160. In the examples disclosed herein, the device for generating commands can be implemented by the example command generator 162. In the examples disclosed herein, the device for configuring the hardware accelerator (e.g., a device for performing convolution) can be implemented by the example accelerator interface 164. In the examples disclosed herein, the device for generating kernel parameters can be implemented by the example graphics compiler 116. In the examples disclosed herein, the device for performing convolution can be implemented by the hardware accelerator 104.

[0049] Figure 7 Example iteration sequence of a neural network, Figures 8A-8H Sample code lines (LOC), and Fig.9A and Fig. 9B The example flowcharts of the example hardware logic, machine readable instructions, hardware-implemented state machines, and / or any combination thereof for implementing the example graphics compiler 116, the example acceleration manager 102, and / or the example hardware accelerator 104. The machine readable instructions may be one or more executable programs or (one or more) portions of executable programs for use by, for example, the following in conjunction with Fig.10 The program(s) may be embodied in software stored in a non-transitory computer-readable storage medium (e.g., a CD-ROM, floppy disk, hard drive, DVD, Blu-ray disk, or memory associated with the processor 1012), but the entire program and / or portions thereof may alternatively be executed by a device other than the processor 1012, and / or by a device embodied in firmware or dedicated hardware. Furthermore, although reference is made to Fig.9A and 9B The flowchart shown, Figure 7 An example iteration sequence of a neural network, and / or Figures 8A-8HThe example lines of code (LOC) of the example program(s) are described, but many other methods of implementing the example graphics compiler 116, the example acceleration manager 102, and / or the example hardware accelerator 104 may be used instead. For example, the order of execution of the blocks may be changed, and / or some of the blocks described may be changed, eliminated, or combined. Additionally or alternatively, any or all of the blocks may be implemented by one or more hardware circuits (e.g., discrete and / or integrated analog and / or digital circuits, FPGAs, ASICs, comparators, operational amplifiers (op-amps), logic circuits, etc.) that are constructed to perform the corresponding operations without executing software or firmware.

[0050] The machine-readable instructions described herein may be stored in one or more of a compressed format, an encrypted format, a segmented format, a compiled format, an executable format, an encapsulated format, and the like. The machine-readable instructions described herein may be stored as data (e.g., portions of instructions, codes, representations of codes, and the like) that can be used to create, manufacture, and / or generate machine-executable instructions. For example, the machine-readable instructions may be segmented and stored on one or more storage devices and / or computing devices (e.g., servers). The machine-readable instructions may require one or more of the following: installation, modification, adaptation, updating, merging, supplementation, configuration, decryption, decompression, decapsulation, distribution, redistribution, compilation, and the like, so that they are directly readable, interpretable, and / or executable by a computing device and / or other machine. For example, the machine-readable instructions may be stored in a plurality of parts that are compressed, encrypted, and stored on separate computing devices, respectively, wherein the parts, when decrypted, decompressed, and combined, form a set of executable instructions that implement a program such as described herein.

[0051] In another example, the machine-readable instructions may be stored in a state that is readable by a computer, but require the addition of a library (e.g., a dynamic link library (DLL)), a software development kit (SDK), an application programming interface (API), etc., in order to execute the instructions on a particular computing device or other device. In another example, the machine-readable instructions and / or corresponding program(s) may need to be configured (e.g., stored settings, data inputs, recorded network addresses, etc.) before the machine-readable instructions and / or corresponding program(s) can be executed in whole or in part. Therefore, the disclosed machine-readable instructions and / or corresponding program(s) are intended to cover such machine-readable instructions and / or program(s) regardless of the particular format or state in which the machine-readable instructions and / or program(s) are stored or otherwise at rest or transported.

[0052] The machine-readable instructions described herein may be represented by any past, present, or future instruction language, scripting language, programming language, etc. For example, any of the following languages ​​may be used to represent machine-readable instructions: C, C++, Java, C#, Perl, Python, JavaScript, Hypertext Markup Language (HTML), Structured Query Language (SQL), Swift, etc.

[0053] As mentioned above, Fig.9A and 9B Example process, Figure 7 The order of iterations of the neural network, and / or Figures 8A-8H The LOC may be implemented using executable instructions (e.g., computer and / or machine readable instructions) stored on a non-transitory computer and / or machine readable medium, such as a hard drive, flash memory, read-only memory, compact disk, digital versatile disk, cache, random access memory, and / or any other storage device or storage disk in which information is stored for any period of time (e.g., for an extended period of time, permanently, transiently, temporarily buffered, and / or cached information). As used herein, the term non-transitory computer readable medium is expressly defined to include any type of computer readable storage device and / or storage disk, and does not include propagating signals and does not include transmission media.

[0054] "Include" and "comprising" (and all forms and tenses thereof) are used herein as open-ended terms. Thus, whenever a claim adopts any form of "include" or "comprising" (e.g., includes, comprises, includes, contains, has, etc.) as a preamble or in any type of claim statement, it should be understood that additional elements, terms, etc. may be present without exceeding the scope of the corresponding claim or statement. As used herein, when the phrase "at least" is used as a transitional term, such as in the preamble of a claim, it is open-ended in the same manner as the terms "include" and "comprising" are open-ended. The term "and / or", when used, for example, in a form such as A, B, and / or C, refers to any combination or subset of A, B, and C, such as (1) only A, (2) only B, (3) only C, (4) A and B, (5) A and C, (6) B and C, and (7) A and B and C. As used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A and B" is intended to refer to embodiments that include: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing structures, components, items, objects, and / or things, the phrase "at least one of A or B" is intended to refer to embodiments that include: 1) at least one A, (2) at least one B, and (3) at least one A and at least one B. As used herein in the context of describing execution or operation procedures, instructions, actions, activities, and / or steps, the phrase "at least one of A and B" is intended to refer to embodiments that include: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B. Similarly, as used herein in the context of describing execution or operation procedures, instructions, actions, activities, and / or steps, the phrase "at least one of A or B" is intended to refer to embodiments that include: (1) at least one A, (2) at least one B, and (3) at least one A and at least one B.

[0055] As used herein, singular references (e.g., "a," "an," "first," "second," etc.) do not exclude a plurality. As used herein, the term "a" or "an" entity refers to one or more of the entity. The terms "a" (or "an"), "one or more," and "at least one" are used interchangeably herein. In addition, although listed separately, multiple devices, elements, or method actions may be implemented by, for example, a single unit or processor. In addition, although individual features may be included in different examples or claims, these features may be combined, and inclusion in different examples or claims does not mean that the combination of features is not feasible and / or disadvantageous.

[0056] Figure 7700 is an example iteration sequence 700 of a neural network (e.g., CNN, TNN, etc.) for performing convolution of an input tensor based on a hybrid firmware-hardware tile traversal. The example iteration sequence 700 represents machine-readable instructions executable by a processor to generate and manage horizontal (X) dimension iterators, vertical (Y) dimension iterators, and depth (Z) dimension iterators to perform a hybrid firmware-hardware tile traversal as disclosed herein. The example iteration sequence 700 includes an example back-end surface iteration portion 702, an example firmware-based iteration portion 704, and an example hardware-based iteration portion 706. The example back-end surface iteration portion 702 may be performed by, for example, a host processor 112 ( Figure 1 ) is implemented by the graphics compiler 116 executed to determine the example kernel parameters 134 ( Figure 1 ), determine buffer allocation requirements, and / or control the movement of tensor data between system memory 114 and local memory 110 ( Figure 1 ). For example, the back-end surface iteration portion 702 uses a vertical (Y) dimension iteration, a horizontal (X) dimension iteration, and a depth (Z) dimension iteration to retrieve input tile data (e.g., IFM) and output / store output tile data. In some examples, the DMA controller 126 ( Figure 1 ) to perform the movement of tensor data in the back-end surface iteration portion 702. In some examples, as part of the back-end surface iteration portion 702, the host processor 112 also controls the DMA controller 126 to move tile data from the system memory 114 to the local memory 110 ( Figure 1 ). For example, graphics compiler 116 (executed by host processor 112) may generate a DMA program to control the data copying / moving performed by DMA controller 126. In this way, hardware accelerator 104 may copy tile data from local memory 110 to its cache 128. For example, a tile (e.g., Figure 4 and Figure 5 402) and / or one or more micro-tiles (eg, micro-tile 404 ( Figure 4 ) and / or 504( Figure 5 )) data can be stored in each cache line of cache 128.

[0057] The example firmware-based iteration portion 704 may be implemented by, for example, the acceleration manager 102 ( Figure 1 ) to determine the hardware execution parameters 136 ( Figure 1 ), and generates and manages iterators (e.g., vertical (Y) dimension iterators, horizontal (X) dimension iterators, and depth (Z) dimension iterators) to control the hardware accelerator 104 ( Figure 1 ) performs convolution iterations of the tile traversal. Figure 7In the example of , micro-tiles are referred to as blocks (e.g., multiple blocks per tile). The example hardware-based iteration portion 706 can be implemented by, for example, the hardware accelerator 104 to perform convolution for each micro-tile or block as directed by the acceleration manager 102. For example, the hardware-based iteration portion 706 refers to tiles as kernels and includes a kernel area loop (kernel_area) to convolve a 16×16 vector of weight values ​​(ker_vec

[16] ) with a 4×4 block (e.g., micro-tile) of input tile data (ifm_vec in block area (block_area)(4×4)). In addition, in the example hardware-based iteration portion 706, the convolution operation is performed using a multiply-accumulate (MAC) operation represented as MAC(ifm_vec, ker_vec[i]).

[0058] Figures 8A-8H includes example lines of code (LOC) representing machine readable instructions that may be executed to implement Figure 1 The acceleration manager 102 performs convolution of the input tensor based on the hybrid firmware-hardware tile traversal disclosed herein. Figures 8A-8H An example LOC can be used in Figure 3 The example firmware is implemented during stage 304 Figure 7 The example is based on the iterative portion 704 of the firmware. Figures 8A-8H The example LOC of is used to generate an iterator and hardware execution parameters 136, and calculate the hardware execution parameters 136 in multiple iterations tracked using the iterator. In this way, the acceleration manager 102 can configure and control the convolution operation of the hardware accelerator 104 to implement a tile traversal of the input tensor 106 ( Figure 1 , Figure 2 , Figure 4 , Figure 5 ).exist Figures 8A-8H In the example LOC, the MIN and MINU annotations are the minimum operation and the unsigned minimum operation, and the MAX annotation represents the maximum operation.

[0059] Fig. 8A An example LOC 802 is shown, which is executed once per layer (e.g., once per input tensor 106) to determine kernel state (e.g., structural characteristics of weight data to be convolved with input tensor 106), Figure 1 The example LOC 802 determines the memory allocation requirements in the system memory 114, local memory 110, and / or cache 128. As part of the kernel state calculation, the example LOC 802 determines the height (X), width (Y), and depth (Z) dimensions of the input tensor 106 and the output tensor to be generated. The example LOC 802 also accesses the weight values ​​to be used for the convolution. Figure 8BAn example LOC 804 is shown for setting the storage of weight values ​​(w_block, w_lines) and input tile data (IFM) to be stored in cache lines of cache 128. The example LOC 804 also sets the initial convolution center of the first TNN block on the input data (IFM surface). In the examples disclosed herein, Fig. 8A LOC802 and Figure 8B The LOC can be executed to achieve Figure 1 The parameter generator 154 is used to generate hardware execution parameters 136 based on the kernel parameters 134 and / or the layer parameters 132.

[0060] Figure 8C An example LOC 806 is shown for generating additional hardware execution parameters 136 based on layer parameters 132 and kernel parameters 134. The example LOC 806 prepares hardware execution parameters 136 for the horizontal (X) dimension, which exploits symmetry along the input tile during traversal of the tile from top to bottom. Fig.8D An example LOC 808 is shown that may be executed to generate an output data dimension size that is larger than the input tile data by continuing the convolution along non-existent data (e.g., non-real delta-x) until the end of the output tile dimension is reached to generate data padding for the output data dimension size. Fig. 8E An example LOC 810 is shown for controlling the construction and storage of output tile data (OFM) by looping over the output tile data based on the input tile data (IFM) and weight values ​​(W) to be used in a convolution.

[0061] Fig.8F An example LOC 812 is shown for iterating in the horizontal (X) dimension by sending hardware execution parameters 136 from the acceleration manager 102 to the hardware accelerator 104 for a plurality of micro-tiles of a tile of an input tensor. Figure 8G An example LOC 814 is shown for completing iteration of input data for a tile along the horizontal (X) dimension and incrementing to the next starting point in the vertical (Y) dimension of the tile. Figure 8H An example LOC 816 is shown for managing convolution iteration operations in the vertical (Y) dimension of a tile.

[0062] Fig.9A and Fig. 9B An example flow chart representing example machine readable instructions that may be executed to perform a hybrid firmware-hardware tile traversal based on an input tensor (eg, Figure 1 , Figure 2 , Figure 4 and Figure 5 The convolution of the input tensor 106). Fig.9A and Fig. 9B The example instructions represented in may be implemented in firmware and / or software executed by the host processor 112 to implement Figure 1 The graphics compiler 116 and / or may be Figure 1 The programmable controller 108 is implemented in firmware and / or software to implement Figure 1 The acceleration manager 102.

[0063] Fig.9A and Fig. 9B The program(s) in Fig.9A Beginning at block 902 of , the example graph compiler 116 generates a graph based on one or more layer parameters 132 ( Figure 1 ) generates one or more kernel parameters 134 ( Figure 1 ). For example, the graphics compiler 116 may access one or more layer parameters 132 in the system memory 114 based on user input to generate (one or more) kernel parameters 134. The example graphics compiler 116 may also generate (one or more) kernel parameters 134 based on the example compiler decisions 142 in the system memory 114. Example accelerator interface 164 ( Figure 1 ) sends the layer parameter(s) 132 and / or the kernel parameter(s) 134 to the example hardware accelerator 104 (block 904). The example memory interface 152 ( Figure 1 ) waits for the tile data of the input tensor 106 to be copied to the local memory 110 (block 906). When the tile data of the input tensor 106 is available in the local memory 110, the horizontal iterator controller 156 ( Figure 1 ) generates a horizontal (X) iterator for calculating one or more horizontal hardware execution parameters of the tile data of the input tensor 106 (block 908). For example, the horizontal (X) iterator can be implemented as a counter or a programming loop (e.g., a for loop, a while loop, etc.) for analyzing the tile data of the input tensor 106 to generate a first row of micro-tiles (e.g., including) for convolving along the horizontal (X) dimension. Figure 4 (one or more) corresponding horizontal hardware execution parameters of the rows of micro-tiles 404).

[0064] Example parameter generator 154 ( Figure 1) generates one or more horizontal hardware execution parameters for a first row of micro-tiles along a horizontal (X) dimension of the tile data of the input tensor 106 (block 910). For example, the (one or more) horizontal hardware execution parameters may be one or more of the hardware execution parameters 136 generated based on the (one or more) kernel parameters 134 and / or the (one or more) layer parameters 132. The (one or more) horizontal hardware execution parameters are used to configure the hardware accelerator 104 to perform horizontal iterations of the convolution. For example, the horizontal hardware execution parameters generated at block 910 may be the horizontal (X) dimension size of the micro-tile, edge parameters of the micro-tile, etc. Example vertical iterator controller 158 ( Figure 1 ) generates a vertical (Y) iterator for performing convolution on tile data of the input tensor 106 (block 912). For example, the vertical (Y) iterator can be implemented as a counter or a programming loop (e.g., a for loop, a while loop, etc.) that the vertical iterator controller 158 can use to track when the vertical iteration of the convolution is completed. The example parameter generator 154 generates one or more vertical hardware execution parameters for the vertical (Y) dimension (block 914). For example, the (one or more) vertical hardware execution parameters can be one or more of the hardware execution parameters 136 generated based on the (one or more) kernel parameters 134 and / or the (one or more) layer parameters 132. The (one or more) vertical hardware execution parameters are used to configure the hardware accelerator 104 to perform the vertical iteration of the convolution. For example, the vertical hardware execution parameters generated at block 914 can be the vertical (Y) dimension size of the micro-tile, the edge parameters of the micro-tile, the starting row parameters, the bias setting parameters, etc. The example horizontal (X) iterator controller 156 generates a horizontal (X) iterator for performing convolution on the tile data of the input tensor 106 (block 916). For example, the horizontal (X) iterator may be implemented as a counter or a programming loop (e.g., a for loop, a while loop, etc.) that the horizontal iterator controller 156 may use to track when the horizontal iteration of the convolution is complete. The example vertical iterator controller 158 generates a depth (Z) iterator for performing the convolution on the tile data of the input tensor 106 (block 918). For example, the depth (Z) iterator may be implemented as a counter or a programming loop (e.g., a for loop, a while loop, etc.) that the vertical iterator controller 158 may use to track when the depth iteration of the convolution is complete.

[0065] The example parameter generator 154 generates one or more depth hardware execution parameters for the depth (Z) dimension (block 920). For example, the (one or more) depth hardware execution parameters may be one or more of the hardware execution parameters 136 generated based on the (one or more) kernel parameters 134 and / or the layer parameters 132. The (one or more) depth hardware execution parameters are used to configure the hardware accelerator 104 to perform depth iterations of the convolution. For example, the (one or more) depth hardware execution parameters generated at block 920 may be an input depth (Z) size of a micro-tile, an edge parameter of a micro-tile, an eviction parameter, etc. The example accelerator interface 164 sends one or more of the (one or more) hardware execution parameters 136 to the hardware accelerator 104 (block 922) to configure the hardware accelerator 104 based on the (one or more) hardware execution parameters 136. For example, the accelerator interface 164 may send one or more commands to the hardware accelerator 104 via the message bus 124 in a pipelined manner. The command generator 162 ( Figure 1 ) may generate (one or more) commands including one or more hardware execution parameters 136 to provide a specific / detailed operation definition for performing a convolution operation on a tile and / or micro-tile (e.g., a 2×2 micro-tile, a 4×4 micro-tile, etc.). In this manner, the acceleration manager 102 informs the hardware accelerator 104 how to traverse the tiles / micro-tiles of the input tensor 106 based on the one or more hardware execution parameters. The hardware accelerator 104 may execute one or more commands from the message bus 124 one after another in a stateless manner to program one or more horizontal hardware execution parameters, one or more vertical hardware execution parameters, and / or one or more depth hardware execution parameters in the parameter configuration register 122.

[0066] Go to Fig. 9B As the convolution engine 118 of the hardware accelerator 104 performs convolution based on the hardware execution parameter(s) 136 programmed into the parameter configuration register 122, the acceleration manager 102 monitors the completion of horizontal iterations, vertical iterations, and depth iterations to advance the convolution operation of the convolution engine 118 along the horizontal (X) dimension, the vertical (Y) dimension, and the depth (Z) dimension until the convolution of the entire input tensor 106 is completed. Fig. 9BAt block 924 of , the horizontal iterator controller 156 determines when the hardware accelerator 104 has completed a horizontal iteration of the convolution. For example, the horizontal iterator controller 156 may increment a horizontal (X) iterator (e.g., the horizontal (X) iterator generated at block 916) for each microtile provided to the hardware accelerator 104 and monitor the completion of the horizontal iteration based on the horizontal (X) iterator (e.g., all microtiles in a horizontal row have been convolved). When the hardware accelerator 104 completes the horizontal iteration of the convolution, the example parameter generator 154 generates one or more updated horizontal hardware execution parameters (block 926). The example accelerator interface 164 sends the one or more updated horizontal hardware execution parameters to the hardware accelerator 104 (block 928) to configure the hardware accelerator 104 to perform another horizontal iteration of the convolution based on the one or more updated horizontal hardware parameters.

[0067] The vertical iterator controller 158 determines when the hardware accelerator 104 completes a vertical iteration of the convolution (block 930). For example, the vertical iterator controller 158 may increment a vertical (Y) iterator (e.g., the vertical (Y) iterator generated at block 912) for each horizontal row of micro-tiles processed by the hardware accelerator 104, and monitor the completion of the vertical iteration based on the vertical (Y) iterator (e.g., all rows of micro-tiles in the horizontal-vertical layer have been convolved). When the hardware accelerator 104 completes a vertical iteration of the convolution, the example parameter generator 154 generates one or more updated vertical hardware execution parameters (block 932). The example accelerator interface 164 sends the one or more updated vertical hardware execution parameters to the hardware accelerator 104 (block 932) to configure the hardware accelerator 104 to perform another vertical iteration of the convolution based on the one or more updated vertical hardware execution parameters.

[0068] The depth iterator controller 160 determines when the hardware accelerator 104 completes the depth iteration of the convolution (block 936). For example, the vertical iterator controller 158 may increment a depth (Z) iterator (e.g., the depth (Z) iterator generated at block 918) for each layer processed by the hardware accelerator 104 and monitor the completion of the depth iteration based on the depth (Z) iterator (e.g., all micro-tiles of all layers in the tile have been convolved). When the hardware accelerator 104 completes the depth iteration of the convolution, the example parameter generator 154 generates one or more updated depth hardware execution parameters (block 938). The example accelerator interface 164 sends the one or more updated depth hardware execution parameters to the hardware accelerator 104 (block 940) to configure the hardware accelerator 104 to perform another depth iteration of the convolution based on the one or more updated depth hardware execution parameters.

[0069] The acceleration manager 102 can manage these special cases by updating one or more hardware execution parameters 136 at blocks 926, 932, and 938 according to the special cases. For example, when the output tensor reaches the end of the output depth and additional input tensor data and / or weight values ​​are to be convolved, the parameter generator 154 can update the corresponding hardware execution parameters 136 to truncate the additional convolution data to maintain the smaller size defined for the output tensor. Additionally or alternatively, when the input tensor 106 reaches the end of the input width and does not reach the specified size of the output tensor, the parameter generator 154 can update the corresponding hardware execution parameters 136 to fill the additional unit values ​​of the output tensor with padding data to meet the defined size of the output tensor. Fig.9A and Fig. 9B The example program(s) then ends.

[0070] Fig.10 is a block diagram of an example processor platform 1000 that is configured to perform Figure 7 , Figures 8A-8H , and / or Fig.9A and Fig. 9B In order to achieve the instructions Figure 1 The example graph compiler 116, the example acceleration manager 102, and / or the example hardware accelerator 104 of the example graphics processor 1000 are used to perform convolution of input tensors based on a hybrid firmware-hardware graph tile traversal according to the examples disclosed herein. The processor platform 1000 can be, for example, a server, a personal computer, a workstation, a self-learning machine (e.g., a neural network), a mobile device (e.g., a cellular phone, a smart phone, a tablet computer, such as an iPad TM ), a personal digital assistant (PDA), an Internet appliance, a DVD player, a digital video recorder, a Blu-ray player, a game console, a personal video recorder, a set-top box, a head-mounted or other wearable device, or any other type of computing device.

[0071] The processor platform 1000 of the illustrated example includes a processor 1012. The processor 1012 of the illustrated example is hardware. For example, the processor 1012 can be implemented by one or more integrated circuits, logic circuits, microprocessors, GPUs, DSPs, or controllers of any desired series or manufacturer. The hardware processor can be a semiconductor-based (e.g., silicon-based) device. In this example, the processor implements an example graphics compiler 116, an example acceleration manager 102, an example memory interface 152, an example parameter generator 154, an example horizontal iterator controller 156, an example vertical iterator controller 158, an example depth iterator controller 160, an example command generator 162, and an example accelerator interface 164. In some examples, the processor 1012 also implements a hardware accelerator 104, including a convolution engine 118 and parameter configuration registers 122. In other examples, the hardware accelerator 104 is implemented in a separate dedicated hardware logic circuit (e.g., ASIC).

[0072] The processor 1012 of the illustrated example includes a local memory 1013 (e.g., cache). The processor 1012 of the illustrated example communicates with a main memory including a volatile memory 1014 and a non-volatile memory 1016 via a bus 1018. The volatile memory 1014 may be implemented by a synchronous dynamic random access memory (SDRAM), a dynamic random access memory (DRAM), Dynamic Random Access Memory And / or any other type of random access memory device. The non-volatile memory 1016 may be implemented by flash memory and / or any other desired type of memory device. Access to the main memory 1014, 1016 is controlled by a memory controller.

[0073] The processor platform 1000 of the illustrated example also includes an interface circuit 1020. The interface circuit 1020 may be implemented by any type of interface standard, for example, an Ethernet interface, a universal serial bus (USB), interface, near field communication (NFC) interface, and / or PCI Express interface.

[0074] In the example shown, one or more input devices 1022 are connected to the interface circuit 1020. The input device(s) 1022 allow a user to enter data and / or commands into the processor 1012. The input device(s) may be implemented by, for example, an audio sensor, microphone, camera (still or video), keyboard, button, mouse, touch screen, trackpad, trackball, isopoint, and / or voice recognition system.

[0075] One or more output devices 1024 are also connected to the interface circuit 1020 of the illustrated example. The output device 1024 can be implemented, for example, by a display device (e.g., a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), a cathode ray tube display (CRT), an in-place switch (IPS) display, a touch screen, etc.), a tactile output device, a printer, and / or a speaker. Therefore, the interface circuit 1020 of the illustrated example typically includes a graphics driver card, a graphics driver chip, and / or a graphics driver processor.

[0076] The interface circuitry 1020 of the illustrated example also includes communication devices, such as transmitters, receivers, transceivers, modems, residential gateways, wireless access points, and / or network interfaces to facilitate data exchange with external machines (e.g., any kind of computing device) via the network 1026. Communications may be through, for example, an Ethernet connection, a digital subscriber line (DSL) connection, a telephone line connection, a coaxial cable system, a satellite system, a field line wireless system, a cellular telephone system, etc.

[0077] The processor platform 1000 of the illustrated example also includes one or more mass storage devices 1028 for storing software and / or data. Examples of such mass storage devices 1028 include floppy disk drives, hard disk drives, optical disk drives, Blu-ray disk drives, redundant array of independent disks (RAID) systems, and digital versatile disk (DVD) drives.

[0078] Figure 7 , Figures 8A-8H , Fig.9A and / or Fig. 9B The machine-executable instructions 1032 represented in may be stored in the mass storage device 1028, in the volatile memory 1014, in the non-volatile memory 1016, and / or on a removable, non-transitory computer-readable storage medium (e.g., a CD or DVD).

[0079] Based on the foregoing, it will be appreciated that example methods, apparatus, and articles of manufacture have been disclosed for performing hybrid firmware-hardware tile traversals to perform convolution operations on tensors using hardware acceleration managed by firmware. The disclosed methods, apparatus, and articles of manufacture improve the efficiency of using a computing device by using software and / or firmware to decompose an input tensor into multiple smaller tiles and / or micro-tiles, using firmware to generate hardware execution parameters to control convolution operations on input data, and configuring a hardware accelerator (e.g., an accelerator circuit) to perform processing-intensive convolution operations on input data corresponding to the smaller tiles and / or micro-tiles based on the hardware execution parameters. By using a hardware accelerator, software / firmware processing can offload processing-intensive recursive convolution operations from a host processor and / or controller to a hardware accelerator, which can be configured to perform each convolution iteration according to a configuration provided by the software / firmware. This frees up resources of a host processor and / or programmable controller (e.g., a DSP, ASIP, etc.) to execute software / firmware to perform other processing while the hardware accelerator performs digitally intensive convolution operations. In addition, the hardware accelerator can complete the convolution operation faster than the host processor and / or programmable controller executing the software / firmware. In addition, the host processor and / or programmable controller executing the software / firmware can offload the large amount of memory access used to access input tensor data and store output tensor data by configuring the hardware accelerator to directly perform memory access (e.g., through a DMA controller) as part of the convolution operation. Other advantages of the present technology are also apparent from the examples disclosed herein. Therefore, the disclosed methods, apparatus, and articles of manufacture relate to one or more improvements in computer functionality.

[0080] Example methods, apparatus, systems, and articles of manufacture for performing tile traversal of a tensor to perform a convolution operation are disclosed herein. Other examples and combinations thereof include the following:

[0081] Example 1 includes an apparatus for performing convolution on an input tensor, the apparatus comprising: a parameter generator for generating horizontal hardware execution parameters for a horizontal dimension of the input tensor based on at least one of kernel parameters or layer parameters, and generating vertical hardware execution parameters for a vertical dimension of the input tensor based on at least one of kernel parameters or layer parameters; an accelerator interface for configuring a hardware accelerator circuit based on the horizontal hardware execution parameters and the vertical hardware execution parameters; a horizontal iterator controller for determining when the hardware accelerator circuit completes a first horizontal iteration of the convolution; and a vertical iterator controller for determining when the hardware accelerator circuit completes a first vertical iteration of the convolution.

[0082] Example 2 includes the apparatus of Example 1, wherein the layer parameters include input dimension size parameters of the input tensor, the kernel parameters include tile dimension size parameters of tiles of the input tensor, and the hardware execution parameters include input depth size parameters of micro-tiles of the tile.

[0083] Example 3 includes an apparatus of any one or more of Examples 1-2, wherein the horizontal hardware execution parameters are used to configure the hardware accelerator circuit to perform a first horizontal iteration of the convolution, and the vertical hardware execution parameters are used to configure the hardware accelerator circuit to perform a first vertical iteration of the convolution.

[0084] Example 4 includes an apparatus of any one or more of Examples 1-3, wherein the parameter generator is used to generate a depth hardware execution parameter for a depth dimension of a micro-tile of an input tensor based on at least one of a kernel parameter or a layer parameter, and the accelerator interface is used to configure the hardware accelerator circuit to perform convolution based on the depth hardware execution parameter.

[0085] Example 5 includes an apparatus of any one or more of Examples 1-4, further comprising a depth iterator controller for determining when the hardware accelerator circuit completes a first depth iteration of the convolution, a parameter generator for generating updated depth hardware execution parameters when the hardware accelerator circuit completes the first depth iteration of the convolution, the updated depth hardware execution parameters being used to configure the hardware accelerator circuit to perform a second depth iteration of the convolution.

[0086] Example 6 includes an apparatus of any one or more of Examples 1-5, wherein the parameter generator is further used to generate updated horizontal hardware execution parameters when the hardware accelerator circuit completes a horizontal iteration, and to generate updated vertical hardware execution parameters when the hardware accelerator circuit completes a vertical iteration of convolution, and the accelerator interface is further used to configure the updated horizontal hardware execution parameters in the hardware accelerator circuit to perform a second horizontal iteration of convolution, and to configure the updated vertical hardware execution parameters in the hardware accelerator circuit to perform a second vertical iteration of convolution.

[0087] Example 7 includes the apparatus of any one or more of Examples 1-6, further comprising a graph compiler for generating kernel parameters based on layer parameters of an input tensor.

[0088] Example 8 includes the apparatus of any one or more of Examples 1-7, further comprising a hardware accelerator circuit and a programmable controller in communication with the hardware accelerator, the programmable controller comprising a parameter generator, an accelerator interface, a horizontal iterator controller, and a vertical iterator controller.

[0089] Example 9 includes a non-transitory computer-readable storage medium having instructions that, when executed by a controller, cause the controller to perform at least the following operations: generate horizontal hardware execution parameters for a horizontal dimension of an input tensor based on at least one of kernel parameters or layer parameters, generate vertical hardware execution parameters for a vertical dimension of the input tensor based on at least one of kernel parameters or layer parameters, configure a hardware accelerator circuit based on the horizontal hardware execution parameters and the vertical hardware execution parameters, determine when the hardware accelerator circuit completes a first horizontal iteration of a convolution, and determine when the hardware accelerator circuit completes a first vertical iteration of a convolution.

[0090] Example 10 includes the non-transitory computer-readable storage medium defined in Example 9, wherein the layer parameters include input dimension size parameters of the input tensor, the kernel parameters include tile dimension size parameters of the tiles of the input tensor, and the hardware execution parameters include input depth size parameters of micro-tiles of the tiles.

[0091] Example 11 includes a non-transitory computer-readable storage medium defined in any one or more of Examples 9-10, wherein the horizontal hardware execution parameters are used to configure the hardware accelerator circuit to perform a first horizontal iteration of the convolution, and the vertical hardware execution parameters are used to configure the hardware accelerator circuit to perform a first vertical iteration of the convolution.

[0092] Example 12 includes a non-transitory computer-readable storage medium defined in any one or more of Examples 9-11, wherein the instructions are further used to cause the controller to generate a depth hardware execution parameter for a depth dimension of a micro-tile of an input tensor based on at least one of the kernel parameters or the layer parameters, and configure the hardware accelerator circuit to perform convolution based on the depth hardware execution parameter.

[0093] Example 13 includes a non-transitory computer-readable storage medium as defined in any one or more of Examples 9-12, wherein the instructions are further used to cause the controller to determine when the hardware accelerator circuit completes a first depth iteration of the convolution, and to generate an updated depth hardware execution parameter when the hardware accelerator circuit completes the first depth iteration of the convolution, the updated depth hardware execution parameter being used to configure the hardware accelerator circuit to perform a second depth iteration of the convolution.

[0094] Example 14 includes a non-transitory computer-readable storage medium as defined in any one or more of Examples 9-13, wherein the instructions are further used to cause the controller to perform the following operations: generate updated horizontal hardware execution parameters when the hardware accelerator circuit completes a horizontal iteration, generate updated vertical hardware execution parameters when the hardware accelerator circuit completes a vertical iteration of the convolution, configure the updated horizontal hardware execution parameters in the hardware accelerator circuit to perform a second horizontal iteration of the convolution, and configure the updated vertical hardware execution parameters in the hardware accelerator circuit to perform a second vertical iteration of the convolution.

[0095] Example 15 includes a non-transitory computer-readable storage medium as defined in any one or more of Examples 9-14, wherein the instructions are further configured to cause the controller to generate kernel parameters based on layer parameters of the input tensor.

[0096] Example 16 includes a method for performing convolution on an input tensor, the method comprising: generating horizontal hardware execution parameters for a horizontal dimension of the input tensor based on at least one of kernel parameters or layer parameters, generating vertical hardware execution parameters for a vertical dimension of the input tensor based on at least one of kernel parameters or layer parameters, configuring a hardware accelerator circuit based on the horizontal hardware execution parameters and the vertical hardware execution parameters, determining when the hardware accelerator circuit completes a first horizontal iteration of the convolution, and determining when the hardware accelerator circuit completes a first vertical iteration of the convolution.

[0097] Example 17 includes the method defined in Example 16, wherein the layer parameters include input dimension size parameters of the input tensor, the kernel parameters include tile dimension size parameters of the tiles of the input tensor, and the hardware execution parameters include input depth size parameters of micro-tiles of the tiles.

[0098] Example 18 includes the method defined in any one or more of Examples 16-17, wherein the horizontal hardware execution parameters are used to configure the hardware accelerator circuit to perform a first horizontal iteration of the convolution, and the vertical hardware execution parameters are used to configure the hardware accelerator circuit to perform a first vertical iteration of the convolution.

[0099] Example 19 includes the method defined by any one or more of Examples 16-18, further comprising: generating a depth hardware execution parameter for the depth dimension of the micro-tile of the input tensor based on at least one of the kernel parameters or the layer parameters, and configuring the hardware accelerator circuit to perform convolution based on the depth hardware execution parameter.

[0100] Example 20 includes the method defined in any one or more of Examples 16-19, further comprising: determining when the hardware accelerator circuit completes a first depth iteration of the convolution, and generating updated depth hardware execution parameters when the hardware accelerator circuit completes the first depth iteration of the convolution, the updated depth hardware execution parameters being used to configure the hardware accelerator circuit to perform a second depth iteration of the convolution.

[0101] Example 21 includes the method defined in any one or more of Examples 16-20, further comprising: generating updated horizontal hardware execution parameters when the hardware accelerator circuit completes the horizontal iteration, generating updated vertical hardware execution parameters when the hardware accelerator circuit completes the vertical iteration of the convolution, configuring the updated horizontal hardware execution parameters in the hardware accelerator circuit to perform a second horizontal iteration of the convolution, and configuring the updated vertical hardware execution parameters in the hardware accelerator circuit to perform a second vertical iteration of the convolution.

[0102] Example 22 includes the method defined in any one or more of Examples 16-21, further comprising generating kernel parameters based on layer parameters of the input tensor.

[0103] Example 23 includes a device for performing convolution on an input tensor, the device comprising: a parameter generation device for generating horizontal hardware execution parameters for a horizontal dimension of the input tensor based on at least one of kernel parameters or layer parameters, and generating vertical hardware execution parameters for a vertical dimension of the input tensor based on at least one of kernel parameters or layer parameters; a configuration device for configuring the convolution execution device based on the horizontal hardware execution parameters and the vertical hardware execution parameters; a horizontal iterator control device for determining when the convolution execution device completes a first horizontal iteration of the convolution; and a vertical iterator control device for determining when the convolution execution device completes a first vertical iteration of the convolution.

[0104] Example 24 includes the device of Example 23, wherein the layer parameters include input dimension size parameters of the input tensor, the kernel parameters include tile dimension size parameters of tiles of the input tensor, and the hardware execution parameters include input depth size parameters of micro-tiles of the tile.

[0105] Example 25 includes an apparatus of any one or more of Examples 23-24, wherein the horizontal hardware execution parameters are used to configure the convolution execution device to perform a first horizontal iteration of the convolution, and the vertical hardware execution parameters are used to configure the convolution execution device to perform a first vertical iteration of the convolution.

[0106] Example 26 includes an apparatus of any one or more of Examples 23-25, wherein the parameter generation device is used to generate a depth hardware execution parameter for the depth dimension of the micro-tile of the input tensor based on at least one of the kernel parameters or the layer parameters, and the configuration device for configuring the convolution execution device is used to configure the convolution execution device to perform the convolution based on the depth hardware execution parameter.

[0107] Example 27 includes an apparatus of any one or more of Examples 23-26, further comprising: a depth iterator control device for determining when the convolution execution device completes a first depth iteration of the convolution, a parameter generation device for generating updated depth hardware execution parameters when the convolution execution device completes the first depth iteration of the convolution, and the updated depth hardware execution parameters are used to configure the convolution execution device to perform a second depth iteration of the convolution.

[0108] Example 28 includes an apparatus of any one or more of Examples 23-27, wherein the parameter generation device is further used to generate updated horizontal hardware execution parameters when the convolution execution device completes a horizontal iteration, and to generate updated vertical hardware execution parameters when the convolution execution device completes a vertical iteration of the convolution, and the configuration device for configuring the convolution execution device is used to configure the updated horizontal hardware execution parameters in the convolution execution device to perform a second horizontal iteration of the convolution, and to configure the updated vertical hardware execution parameters in the convolution execution device to perform a second vertical iteration of the convolution.

[0109] Example 29 includes the apparatus of one or more of Examples 23-28, further comprising means for generating kernel parameters based on layer parameters of an input tensor.

[0110] Although certain example methods, apparatus, and articles of manufacture have been disclosed herein, the scope of coverage of this patent is not limited thereto. On the contrary, this patent covers all methods, apparatus, and articles of manufacture that fully fall within the scope of the claims of this patent.

Claims

1. An apparatus for running a convolutional layer of a neural network, comprising: A plurality of memory units are configured to store an input tensor of a convolutional layer of the neural network, wherein the input tensor is divided into a plurality of tiles, and different memory units are configured to store different tiles in the input tensor of the convolutional layer; a plurality of convolution engines configured to operate in parallel to run the convolution layer, the convolution engines configured to perform a plurality of multiplication-accumulation (MAC) operations on tiles in the input tensor of the convolution layer, and different convolution engines configured to perform MAC operations in parallel on different tiles in the input tensor of the convolution layer; as well as A controller is configured to control multiple iterations of the convolution layer on the multiple tiles based on a first iterator and a second iterator, the iterations of the convolution layer including the multiple MAC operations performed by the convolution engine on the tiles in the input tensor of the convolution layer, different iterations are performed in parallel by different convolution engines on different tiles in the multiple tiles, the first iterator is configured to count the tiles processed by the multiple convolution engines along a first dimension of the input tensor, and the second iterator is configured to count the tiles processed by the multiple convolution engines along a second dimension of the input tensor.

2. The device according to claim 1, wherein The controller is configured to control iteration of the convolutional layer by incrementing the first iterator or the second iterator for a tile provided to at least one convolution engine.

3. The device according to claim 1, wherein The input tensor includes an image that is input to the neural network.

4. The device according to claim 1, wherein The controller is configured to further control the iteration of the convolutional layer based on a third iterator configured to count tiles along a third dimension, The third dimension is different from the first dimension and the second dimension.

5. The device according to claim 4, wherein The first dimension, the second dimension, or the third dimension is selected from the group consisting of a width of the input tensor, a height of the input tensor, and a depth of the input tensor.

6. The apparatus according to claim 1, further comprising: a parameter generator configured to generate parameters to be used to configure the convolution engine to perform iterations, The parameter indicates the size of the input tensor, the location of the edge of a tile in the input tensor, the size of a convolution kernel, or the number of channels in the input tensor.

7. A method for running a convolutional layer of a neural network, comprising: Receiving a plurality of tiles in an input tensor of the convolutional layer, wherein each tile is a portion of the input tensor; storing the plurality of tiles in one or more memory units, each memory unit storing one or more of the plurality of tiles; Providing the plurality of tiles to a plurality of convolution engines for running the convolution layer, the convolution engines being configured to perform a plurality of multiply-accumulate (MAC) operations on the tiles in the input tensor of the convolution layer, different convolution engines being configured to perform the MAC operations on different tiles in the input tensor of the convolution layer in parallel; and Based on a first iterator and a second iterator, multiple iterations of the convolution layer on the multiple tiles are controlled, the iterations of the convolution layer include the multiple MAC operations performed by the convolution engine on the tiles in the input tensor of the convolution layer, different iterations are performed in parallel by different convolution engines on different tiles in the multiple tiles, the first iterator is configured to count the tiles that have been processed by the multiple convolution engines along a first dimension of the input tensor, and the second iterator is configured to count the tiles that have been processed by the multiple convolution engines along a second dimension of the input tensor.

8. The method according to claim 7, further comprising: further controlling a plurality of iterations of the convolutional layer based on a third iterator, the third iterator configured to count tiles along a third dimension, The third dimension is different from the first dimension and the second dimension.

9. The method according to claim 8, wherein The first dimension, the second dimension, or the third dimension is selected from the group consisting of a width of the input tensor, a height of the input tensor, and a depth of the input tensor.

10. The method according to claim 7, wherein: Controlling multiple iterations of the convolutional layer includes: The first iterator or the second iterator is incremented for a tile provided to at least one convolution engine.

11. The method according to claim 7, wherein: The input tensor includes an image that is input to the neural network.

12. The method according to claim 7, wherein: The first iterator and the second iterator have different hierarchical structures.

13. A system for running a convolutional layer of a neural network, comprising: a memory configured to store an input tensor of one or more convolutions, the input tensor comprising a plurality of tiles; as well as an accelerator associated with the memory, the accelerator comprising: a plurality of memory units for storing the plurality of tiles; a plurality of convolution engines configured to operate in parallel to run the convolution layer, the convolution engines configured to perform a plurality of multiply-accumulate (MAC) operations on tiles in the input tensor of the convolution layer, different convolution engines configured to perform MAC operations in parallel on different tiles in the input tensor of the convolution layer; and A controller is configured to control multiple iterations of the convolution layer on the multiple tiles based on a first iterator and a second iterator, the iterations of the convolution layer including the multiple MAC operations performed by the convolution engine on the tiles in the input tensor of the convolution layer, different iterations are performed in parallel by different convolution engines on different tiles in the multiple tiles, the first iterator is configured to count the tiles processed by the multiple convolution engines along a first dimension of the input tensor, and the second iterator is configured to count the tiles processed by the multiple convolution engines along a second dimension of the input tensor.

14. The system of claim 13, further comprising: A compiler is configured to partition the input tensor into the plurality of tiles.

15. The system according to claim 13, wherein: The controller is configured to: further control a plurality of iterations of the convolutional layer based on a third iterator, the third iterator being configured to count tiles along a third dimension, The third dimension is different from the first dimension and the second dimension.

16. The system according to claim 15, wherein: The first dimension, the second dimension, or the third dimension is selected from the group consisting of a width of the input tensor, a height of the input tensor, and a depth of the input tensor.

17. The system of claim 13, wherein: The controller is configured to control iteration of the convolutional layer by incrementing the first iterator or the second iterator for a tile provided to at least one convolution engine.

18. The system of claim 13, wherein: The input tensor includes an image that is input to the neural network.

19. A computer readable medium comprising instructions which, when executed by at least one processor, cause the processor to perform the method according to any one of claims 7 to 12.

20. An apparatus comprising means for performing the method according to any one of claims 7 to 12.

21. A computer program product comprising instructions which, when executed by at least one processor, cause the processor to perform the method according to any one of claims 7 to 12.

Images