General padding support for convolution on systolic arrays

Abstract

Methods and systems, including computer programs encoded on computer storage media. In one aspect, a method includes the following acts: receiving a request to perform a convolution computation for a neural network on a hardware circuit having a matrix computation unit, the request specifying a convolution computation to be performed on a feature tensor and a filter and padding to be applied to the feature tensor prior to performing the convolution computation; and generating instructions that, when executed by the hardware circuit, cause the hardware circuit to perform operations comprising: transferring feature tensor data from a main memory of the hardware circuit to a scratchpad memory of the hardware circuit; and repeatedly performing the following operations: identifying a current subset of the feature tensor; and determining whether a memory view into the scratchpad memory for the current subset is consistent with a memory view of the current subset in the main memory.

Description

Technical Field

[0001] This specification relates to implementing neural networks on pulsating arrays. Background Technology

[0002] Neural networks use one or more layers to generate outputs from received inputs, such as classification machine learning models. In addition to the output layer, some neural networks also include one or more hidden layers. The output of each hidden layer is used as input to another layer in the network, such as the next hidden layer or the output layer. Each layer of the network generates its output from the received inputs based on the current values ​​of its corresponding parameter set. A systolic array is an array of data processing units. Such arrays can be used to efficiently perform neural network operations. Summary of the Invention

[0003] This specification describes techniques for providing general padding support for convolution computations on systolic arrays in convolution processing systems. The system creates a view of the unpadded input data in a note-based memory, mirroring the view in main memory. Figure 1 The system coordinates the memory views in note-based memory and main memory. It dynamically calculates whether a portion of the transferred data in note-based memory is in the unfilled view of main memory and applies padding values ​​to the unfilled data view where convolution calculations are being performed.

[0004] More specifically, the technology involves dynamically traversing input data from main memory to sticky memory based on the portion of the input data being transferred. This traversal of input data can be accomplished by accessing the sticky memory and varying the DMA size for data transfer between memories. Furthermore, the coordination of memory views involves aligning the top left corner of the memory view in the sticky memory with the unfilled real memory view in main memory to determine if they are within each other's boundaries. If the memory view data in the sticky memory is outside the boundaries, the system replaces the value loaded from the sticky memory with a fill value.

[0005] Compared to the padding input data transfer from main memory to the matrix multiplication unit, the system utilizes a faster data transfer from notepad memory to the matrix multiplication unit, thereby improving system performance when performing convolution computations during convolutional neural network (CNN) operations. The technology further improves the use of notepad memory to support convolution computations within the matrix multiplication unit, reducing memory consumption in main memory used to store input data along with various machine learning models, and making it easier to avoid using the matrix multiplication unit on data fully padded with padding. Eliminating the need for a matrix multiplication unit on padding-only data prevents unnecessary computations in matrix multiplication and reduces the time required to perform convolutions with padding and the amount of multiplication unit usage. The system provides general padding support for convolution computations across multiple frameworks.

[0006] Details of one or more embodiments of the subject matter of this specification are set forth in the following drawings and description. Other features, aspects, and advantages of the subject matter will become apparent from the description, drawings, and claims. Attached Figure Description

[0007] Figure 1 An example convolutional neural network processing system is shown.

[0008] Figure 2 This is a flowchart illustrating an example process for performing computations on the convolutional layers of a neural network.

[0009] Figure 3 An example neural network processing system is shown.

[0010] Figure 4 This is a flowchart of an example process for performing a data transfer across main memory and notepad memory.

[0011] Figure 5 This is an example of data transfer from main memory to note-based memory with a consistent memory view.

[0012] Figure 6 This is a flowchart of an example method for performing padding on sticky memory data to perform convolution calculations.

[0013] Figure 7 This is an example of note-style memory data padding used to perform convolution operations on neural network layers.

[0014] Similar reference numerals and names in the various figures indicate similar elements. Detailed Implementation

[0015] Convolutional neural networks (CNNs) are a type of feedforward neural network that consists of one or more CNN layers and is adapted to manipulate and compute two-dimensional data such as images. A CNN layer is similar to a regular (e.g., fully connected) neural network layer and consists of neurons with weights and biases. Neurons in a CNN layer receive input and perform a dot product that follows a non-linear operation.

[0016] Convolutional neural networks (CNNs) typically begin with a convolutional layer and include multiple intermediate convolutional layers. In a CNN, convolution is a linear operation involving multiplying a set of weights by the input feature data. For example, for two-dimensional input feature data, multiplication is performed between an array of input data and an array of two-dimensional weights (filters). Filters can be smaller than the input data, and the type of multiplication applied between the filter and a patch of filter-sized values ​​is a dot product. In some implementations, having filters smaller than the input allows the same filter (weight set) to be multiplied by the input array multiple times at different points on the input. Specifically, filters are systematically applied to each overlapping portion or patch of filter-sized values ​​of the input data, for example, from left to right and from top to bottom. In a convolutional layer, the output from multiplying the filters by the input feature data is a single value. Because filters are applied multiple times to the input feature data, the result is a two-dimensional array representing the output values ​​of the filtered input feature data.

[0017] In some implementations, the layers of a neural network are arranged sequentially. In other implementations, the layers are arranged as a directed graph. That is, any particular layer can receive multiple inputs, multiple outputs, or both. The layers of a neural network can also be arranged such that the output of one layer can be sent back as input to the previous layer.

[0018] Hyperparameters of a convolutional layer include kernel size, stride, and padding. Kernel size represents the receptive field of the convolutional operation performed by that layer. For example, in 2D convolution, the kernel size can be defined as 2x2. Stride is the number of pixels shifted over the input tensor and controls how the kernel convolves around the input data. When the stride is set to 1, the kernel convolves around the amount of input data by shifting one unit at a time. Padding is the process applied to the boundaries of the input data to help control the spatial size of the output and preserve information about the input data boundaries. For example, applying a 5x5 filter to a 32x32 input with a stride of 1 will result in an output of 28x28 due to the reduced spatial dimension after the convolution computation. In the early layers of a neural network and many other cases, it is preferable to preserve as much information as possible about the original input to enable the extraction of low-level features. For this purpose, zero-padding can be applied to the aforementioned 32x32 input with a size of 2. In this case, zero-padding fills the input with zeros around the boundaries of the input data, such as the input tensor, and produces a 36x36 input. The padded input, convolved with a 5x5 filter with a stride of 1, produces a 31x31 output, while exhibiting less output dimensionality shrinkage and information loss at the corners of the input data. Typically, a tensor is an ordered set of numerical values ​​with a rank that defines the number of dimensions in the tensor. For example, scalar values ​​are rank zero tensors, vectors are rank one tensors, matrices are rank two tensors, and so on.

[0019] Machine learning models implemented in different frameworks (such as PyTorch, TensorFlow, or Caffe) have different default padding configurations. For example, when a single primitive or column padding is required, the default padding configuration in Caffe is to add zeros to the left or top of the tensor. In contrast, the default padding configuration in TensorFlow is to add zeros to the right or bottom of the tensor. Furthermore, the default padding configuration in PyTorch is to add zeros at all boundaries of the input tensor. In another example, memory loading from main memory to memory cells and padding support can also vary in matrix multiplication engines. The aforementioned various padding configurations used for convolution computations in different frameworks can lead to different system performance. Additionally, domain-specific compilers can be configured to support all types of frameworks with near-peak performance on a variety of hardware architectures. As a result, general padding support for convolution computations is necessary for performance portability to support peak system performance across various frameworks and hardware architectures.

[0020] Figure 1 An example convolutional neural network processing system 100 is shown. The convolutional neural network processing system 100 is an example of a system implemented as one or more computers in one or more locations, wherein the following systems, components and technologies are capable of being implemented.

[0021] The convolutional neural network processing system 100 is a system that uses dedicated hardware circuitry 110 to perform neural network computations. Dedicated hardware circuitry 110 is an integrated circuit for performing neural network computations and includes a matrix computation unit 120 that performs matrix multiplication in hardware. Dedicated hardware circuitry 110 also includes a vector computation unit 130 that performs vector multiplication in hardware. (See the following references...) Figure 3 The example dedicated hardware circuit 110 is described in more detail.

[0022] like Figure 1 As shown, the dedicated hardware circuit 110 includes a main memory 140 and a notepad memory 150. The main memory 140 stores the input data to the neural network. The main memory 140 can also transfer input feature data from outside the dedicated hardware circuit 110, for example, from a data bus line or an off-chip computer storage medium. In addition, the main memory 140 is typically configured to have a larger storage capacity than the notepad memory 150 and to transfer data to and from the notepad memory 150.

[0023] The note-based memory 150 is a high-speed internal memory used for temporary storage of ongoing calculations, data, and other tasks. In the convolutional neural network processing system 100, the note-based memory 150 is configured to store small amounts of data for rapid retrieval by the matrix computation unit 120 and the vector computation unit 130 during neural network computations. The note-based memory 150 can be considered the closest memory to the matrix and vector computation units, and direct memory access (DMA)-based data transfer is often used to move data from memory to the computation units on the hardware circuitry 110.

[0024] The note-based memory 150 may include an SRAM meta-array and a portion of the global memory address space of dedicated hardware circuitry 110 for the note-based memory 150. In neural network computation, any address falling within the dedicated address space can be directly indexed into the note-based memory 150 to access the corresponding data. Therefore, the note-based memory 150 is energy efficient compared to a cache and has been configured for use in accelerator hardware designs due to its energy efficiency, timing predictability, and scalability in neural network computation.

[0025] In some implementations, the convolutional neural network processing system 100 receives a request to implement a neural network on a dedicated hardware circuit 110, implements the neural network on the dedicated hardware circuit 110, and, once the given neural network is implemented, uses the dedicated hardware circuit 110 to process the inputs to the neural network to generate neural network inference.

[0026] In other words, the convolutional neural network processing system 100 can receive requests to specify the neural network architecture for the neural network that will be used to process the input. The neural network architecture defines the number and configuration of layers in the neural network, as well as the values ​​of the parameters for each layer, which has parameters.

[0027] To implement the neural network on dedicated hardware circuitry 110, the convolutional neural network processing system 100 includes a neural network implementation engine 160 implemented as one or more computer programs on one or more computers at one or more physical locations.

[0028] The neural network implementation engine 160 generates instructions that, when run by the dedicated hardware circuit 110, cause the dedicated hardware circuit 110 to perform operations specified by the neural network to generate a neural network output from the received neural network input.

[0029] Once the instructions have been generated by the neural network implementation engine 160 and provided to the dedicated hardware circuit 110, the convolutional neural network processing system 100 is able to receive neural network inputs and use the neural network to process the neural network inputs by causing the dedicated hardware circuit 110 to run the generated instructions.

[0030] As described above, a convolutional layer in a neural network specifies a convolutional computation on a neural network matrix (e.g., input features) that has a filter matrix that produces feature outputs. Some neural networks specify two types of results for the convolutional computation. In the first type, where no padding is applied to the input features, the convolutional features are reduced in dimension compared to the input. In the second type, where padding is applied to the input features, the convolutional features are either increased in dimension or remain unchanged compared to the input. For example, the input features can be padded with zeros within the boundaries of elements that are not present in the input features by the filters. If the boundary pixels are not black, this padding can, for example, result in the detection of edge elements of the input features at the boundaries of the input features.

[0031] The main hardware units performing matrix operations on the dedicated hardware circuit 110 are the matrix computation unit 120 and the vector computation unit 130. To implement a neural network that specifies convolution computations on matrices, the neural network implementation engine 160 generates instructions that, when run by the dedicated hardware circuit 110 during processing of neural network inputs, cause the dedicated hardware circuit 110 to use the matrix computation unit 120 and the vector computation unit 130 to perform convolution operations on the matrices. In this example, the system 100 utilizes a notepad memory 150 to efficiently illustrate various padding schemes that can be adopted to fill input features in different frames. Specifically, this example includes aligning the top left corner of a portion of the notepad memory data with memory view data in main memory 140, determining whether all portions of the notepad memory data are within the boundaries of the memory view data in main memory 140, and then applying padding values ​​to the notepad memory data. See the following references... Figure 4-7 These instructions and operations will be described in more detail.

[0032] Figure 2 This is a flowchart of an example process 200 for performing computations on convolutional layers of a neural network using dedicated hardware circuitry 110. For convenience, process 200 will be described relative to a system having one or more circuits that perform the execution process 200. Process 200 can be executed for each convolutional layer of the neural network to compute inference or training from received input.

[0033] The convolutional neural network processing system 100 receives an input feature set, such as an image (step 202), and a kernel input set, such as a linear image filter for a given convolutional layer (step 204). The input feature set and kernel input set can be received and stored in main memory 140 and note-taking memory 150 on dedicated hardware circuitry 110, respectively. In some implementations, both the weight input set and the activation input set can be received and stored in main memory 140.

[0034] The convolutional neural network processing system 100 uses the matrix computation unit 120 of dedicated hardware circuitry 110 to generate accumulated values ​​from input features and input filters (step 206). In some implementations, dedicated hardware circuitry 110 selects input feature data and performs element-wise multiplication with the corresponding filter data, then sums the results to a single output element. Dedicated hardware circuitry 110 repeats this process for each position it traverses on the input features, then delivers the output element to vector computation unit 130.

[0035] The convolutional neural network processing system 100 can use the vector computation unit 130 of the dedicated hardware circuit 110 to generate layer outputs from accumulated values ​​(step 208). The layer outputs can be stored in the note-taking memory 150 for use as input to subsequent layers in the neural network or for determining inference, i.e., the final output of the neural network. The system completes the processing of the neural network when the received inputs have been processed through each layer of the neural network to generate inference or training outputs for the received inputs.

[0036] Figure 3 An example neural network processing system 300 is shown. Specifically, Figure 3 An example of a dedicated hardware system 300 for performing padding for convolutional computations of a neural network is shown. System 300 includes a host interface 302. Host interface 302 receives and delivers instructions including parameters for neural network computations. Parameters may include one or more of the following: the number of neural network layers to be processed, the corresponding weight input set for each layer of the neural network, the initial input feature set (i.e., the input of the neural network to which the convolution is to be computed), the corresponding input and output sizes for each convolutional layer, the stride value for any given convolutional layer, and the type of layer to be processed, such as a convolutional layer or a fully connected layer.

[0037] The host interface 302 can send instructions to the sequencer 306, which translates the instructions into low-level control signals for the control circuitry to perform neural network calculations. In some implementations, the control signals regulate the data flow in the circuitry, such as how feature input sets and filter input sets flow through the circuitry. The sequencer 306 can send control signals to the notepad memory 150, matrix computation unit 120, main memory 140, and vector computation unit 130. In some implementations, the sequencer 306 is a processor that generates the control signals. The sequencer 306 can use the timing of the control signals to send control signals to each component of the dedicated hardware system 300 at appropriate times. In some implementations, the sequencer 306 can issue comparison instructions to both the notepad memory 150 and the main memory 140 and compare the stored information. In other implementations, the host interface 302 receives control signals from an external processor.

[0038] The host interface 302 can send feature input sets and filter input sets to the direct memory access (DMA) engine 304. The DMA engine 304 can store the feature and filter input sets at notepad memory 150. In some implementations, direct memory access stores the filter input sets in main memory 140. In some implementations, main memory 140 is located outside of dedicated hardware circuitry 110.

[0039] Notepad memory 150 is a high-speed internal memory used for temporary storage of ongoing computations, data, and other tasks. In dedicated hardware system 300, notepad memory 150 can be used to store a portion of the feature and filter input set from DMA engine 304 and the output of vector computation unit 314. Notepad memory is typically the only memory that can be directly utilized by matrix computation unit and vector computation unit. To access data in main memory or off-chip memory, the processor must issue an explicit DMA request. DMA is configured to transfer large amounts of data between memory locations without processor intervention, thus providing an additional level of parallelism through overlapping computation and data prefetching. See below for reference. Figure 4 and Figure 5 The data transfer between main memory 140 and notepad memory 150 is described in more detail. DMA engine 304 is also capable of reading the output of vector computing unit 314 from notepad memory 150.

[0040] In some implementations, the notepad memory 150 and the main memory 140 are capable of sending the input feature set and the filter input set to the matrix computation unit 120, respectively. In some implementations, the matrix computation unit 120 is a two-dimensional systolic array. The matrix computation unit 120 can also be a one-dimensional systolic array or other circuitry capable of performing mathematical operations (e.g., multiplication and addition). In some implementations, the matrix computation unit 120 is a general-purpose matrix processor. The dedicated hardware system 300 can use the matrix computation unit 120 to perform matrix multiplication operations for convolution calculations.

[0041] Matrix computation unit 120 is capable of processing input features and filter inputs, and provides the convolution computation result as output to notepad memory 150. In some implementations, the convolution output can be transferred to notepad memory 150 via vector computation unit 130 with additional features (i.e., activation functions). In some implementations, the convolution output (i.e., a two-dimensional matrix) is stored in notepad memory 150 as a vector set. The vector set can be generated along one dimension of matrix computation unit 120 (i.e., the rows of matrix computation unit 120). In some implementations, the convolution output can be configured as feature input to matrix computation unit 120, for example, for use in subsequent convolutional layers in a neural network.

[0042] In this example, the input feature data is stored in main memory 140, and the neural network processing system 300 creates a view in note-style memory 150 of the unfilled input feature memory in main memory 140. Figure 1The memory views in the note-based memory 150 include at least a portion (e.g., a contiguous region) of the memory address space of the note-based memory 150, and the memory views in the main memory include at least a portion (e.g., a contiguous region) of the memory address space of the main memory 140. The sequencer 306 coordinates the memory views in the main memory 140 and the note-based memory 150 to dynamically calculate whether a portion of the data in the note-based memory 150 is in the memory view. The neural network processing system 300 then applies padding values ​​to the unpadded memory views in the note-based memory 150 and transfers the padded memory views to the matrix computation unit 120 for convolution computation.

[0043] Figure 4 This is a flowchart of an example process 400 for performing a data transfer across main memory and notepad memory. Specifically, Figure 4 This illustrates the method for creating an unfilled view in the notepad memory 150 compared to the main memory 140. Figure 1 Example process 400 is a memory view. For convenience, process 400 will be described with respect to a system having one or more circuits that execute process 400. Process 400 can be executed on each convolutional layer of a neural network to compute inference or training from received input.

[0044] System 100 receives instructions from host interface 302 to perform convolution computation on a specific convolutional layer in the convolutional neural network processing system 100. Specifically, host interface 302 sends operation instructions to create a memory view in note-taking memory 150 consistent with the unfilled input feature data (i.e., unfilled kernel input in a two-dimensional matrix) in main memory 140 (step 402). Main memory 140 stores the input features and filter inputs for the convolution computation in dedicated hardware circuitry 110. In some implementations, the input features may be an image pixel dataset and the filter inputs may be a set of filter matrices.

[0045] System 100 identifies the access address of the input feature data on main memory 140, the step length of the shifted data transfer from main memory 140, the DMA size, and the access address of the notepad memory 150 used to store the transferred input feature data (step 404). In this example, the transfer of input feature data from main memory 140 to notepad memory 150 is configured to be consistent. However, the data storage configuration may differ between main memory 140 and notepad memory 150. As a result, the addresses of the sub-blocks storing the input feature data in notepad memory 150 need to be updated to maintain the same memory view of the input feature data between main memory 140 and notepad memory 150. Consequently, the access address of notepad memory 150 is updated to the input feature data on main memory 140 after each step, such that the memory data view in the transferred notepad memory 150 matches the unfilled memory view in main memory 140. In some implementations, during the initial cycle of a memory view data transfer, the access address of main memory 140 is the starting storage address of the input features on main memory 140. During the memory view transfer, the access address of main memory 140 can be updated based on the stride length of shifting the memory view from the current block of input feature data to the next block of input feature data. In some implementations, the DMA size is configured to match the filter size used for convolution computation. In some implementations, the DMA size is optimized to minimize the number of data transfers from main memory 140 to notepad memory 150. For example, for filter data with 1050 pixels, the DMA size can be configured to 525 pixels to complete the data transfer in two cycles. In another example, for filter data with 1050 pixels, the DMA size can be configured to 350 pixels to complete the data transfer in three cycles.

[0046] System 100 transfers memory view data from main memory 140 to notepad memory 150 in DMA (step 406). In some implementations, matrix calculation unit 120 may be configured as a two-dimensional systolic array, and data transfer from main memory 140 to notepad memory 150 may require the use of block-based DMA. In some implementations, input features (e.g., an image pixel matrix) may be segmented into a number of contiguous sub-matrix blocks and then transferred to notepad memory 150.

[0047] To transfer or copy data from main memory 140 to notepad memory 150, host interface 302 issues instructions to DMA engine 304 responsible for the data transfer. These instructions may include the source data read address in main memory 140, the destination data store address in notepad memory 150, and the block size. Upon receiving the instruction, the data transfer begins and the input features are divided into smaller sub-matrix blocks, which are transferred from main memory 140 to notepad memory 150 via on-board interconnects. The duration of the data transfer phase is proportional to the amount of data and is limited by the bandwidth of the on-board interconnects.

[0048] In some implementations, host interface 302 sends instructions to the DMA to perform a non-contiguous but regularly structured block data transfer from main memory 140 to notepad memory 150. The non-contiguous block data transfer can be performed using a cross-DMA instruction, which specifies, in addition to specifying the addresses of the main memory and notepad memory, the step size as the offset to access the next contiguous block in main memory 140. Non-contiguous block data transfer can benefit convolution computation efficiency and the performance of the convolutional neural network processing system 100.

[0049] In subsequent iterations, system 100 targets unfilled memory view data transferred from main memory 140 to note-memory memory 150 across memory addresses on note-memory 150 (step 408). The system repeats steps 404, 406, 412, and 408 to transfer all unfilled input feature data from main memory 140 to note-memory 150. The system repeats steps 404, 406, 412, 408, and 410 to transfer all input feature data from main memory 140 to note-memory 150 for convolutional neural network computation.

[0050] In this example, input feature data padding is performed in sticky memory 150. Once an unpadded memory view of the input feature data is transferred from main memory 140 to sticky memory 150, sequencer 306 can issue coordination instructions to both sticky memory 150 and main memory 140 and compare the stored input feature data. Based on the comparison result, system 100 performs padding on the input feature data stored in sticky memory 150. Padding configured for sticky memory 150 reduces main memory consumption because the input feature data is only written to and transferred from main memory 140 to sticky memory 150 once. A consistent memory view of the input feature data stored in sticky memory 150 can be re-accessed as the input kernel for convolution calculations in matrix computation unit 120. Re-accessing the memory view data in sticky memory 150 improves system 100 performance because its communication with matrix computation unit 120 is faster than communication with main memory 140. Furthermore, main memory consumption can be reduced by avoiding the materialization of a padding memory view of the data in main memory 140. See below for reference Figures 6-7 These instructions and operations will be described in more detail.

[0051] Figure 5 This is an example of data transfer from main memory 140 to note-style memory 150 with a consistent memory view. This example can be used... Figure 4 Example process 400 and Figure 3 It is executed by a dedicated hardware system 300.

[0052] In this example, the input features are a 4x4 matrix stored in example main memory 510. Host interface 302 sends a view to be created in note-style memory 150, corresponding to the unfilled input feature data storage in main memory 140. Figure 1 The DMA engine 304 identifies the main memory access address as the address of the first input feature element on main memory 140. The DMA engine 304 also identifies the size of the data to be transferred to note-memory memory 150 in each iteration of the operation, and the initial access address of note-memory memory 150 for storing the transferred data. In this example, the input features are divided into smaller blocks of 2x2 matrices for consistent memory view data transfer.

[0053] exist Figure 5 In stage (a), the dedicated hardware circuit 110 forms the current subdivision of the 4x4 matrix by dividing the entire input feature matrix into two subdivision matrices. The first subdivision matrix... twenty three 6 6

[0056] The subdivision matrix is ​​transferred to note-memory 150 with the same memory view without any padding. The first transferred subdivision matrix is ​​stored at a pre-identified access address in note-memory 150. In some implementations, the elements of the first row of the subdivision matrix are transferred first and stored contiguously at access addresses in note-memory 150. In this example, the input feature data elements [2,3] of the first row of the subdivision matrix are transferred and stored contiguously at the initial access address in note-memory 150. To maintain a consistent memory view of the input feature data on main memory 140 and note-memory 150, memory accesses on note-memory 150 are skipped before the second row of the subdivision matrix data is transferred from main memory 140 to note-memory 150. In this example, the second row elements [6,6] of the input feature data on the subdivision matrix are stored at the skipped address in note-memory 150, as shown in note-memory 520a. After two iterations of sending line data, the first subdivision matrix is ​​transferred to a notepad memory 150 which has the same memory view as the memory view in main memory 140.

[0057] Figure 5 Phase (b) is depicted in a consistent memory view of the second subdivision matrix. 7 4 9 8

[0060] The data is transferred from main memory 140 to notepad memory 150. System 100 updates the access addresses of main memory 140 to the addresses of the leading data elements of the second subdivision matrix. The system also updates the access addresses of notepad memory 150 to mirror the memory view of the second subdivision matrix in notepad memory 150. The DMA engine 340 identifies the same data size used for the transfer as in the previous stage (a). In some implementations, the first row of data elements [7,4] of the second subdivision matrix is ​​transferred to notepad memory 150 first. Once the first raw data is stored in notepad memory 320b, and before the transfer of the second row of data elements begins, the access address is crossed. In the next step, the second row of data elements [9,8] of the subdivision matrix is ​​transferred to the crossed access address in notepad memory 150, and then the transfer of the second subdivision matrix data with a consistent memory view between main memory 140 and notepad memory 150 is completed.

[0061] Similar to stages (a) and (b), system 100 sequentially transfers the third subdivision matrix in stage (c) and the fourth subdivision matrix in stage (d) to the note-style memory as shown on 520c and 520d, respectively, to complete the entire input feature data transfer from main memory 140 to note-style memory 150 in a consistent memory view.

[0062] In this example, the input feature data is transferred from main memory 140 to notepad memory 150 in a consistent memory view. This is accomplished by memory access across notepad memory 150 and by changing the size of the DMA during the memory data transfer. The consistent memory view in notepad memory 150 enables the padding configuration of the input feature data to be used for convolution computation.

[0063] Figure 6 This is a flowchart of an example process 600 for performing padding of note-based memory data to perform convolution calculations in a convolutional neural network processing system 100. Typically, process 600 comprises dedicated hardware circuitry (e.g., Figure 1 One or more computer systems execute dedicated hardware circuits 110.

[0064] Dedicated hardware circuitry 110 receives from host interface 302 an instruction to load kernel filter data (e.g., a two-dimensional matrix) into matrix computation unit (step 602). Specifically, the instruction includes location information of the kernel data stored in matrix computation unit 120.

[0065] Dedicated hardware circuitry 110 identifies the type of convolution computation in a specific convolutional neural network layer (step 604). In some implementations, the convolution computation is focused on higher levels of the input feature characters and performs efficient convolution without using any padding of the input features. In some implementations, the input feature data has additional padding values ​​at the boundaries to generate an output with the same dimensions as the input features for the convolution computation. This type of convolution is defined as "identical convolution" and always requires padding of the input data.

[0066] Dedicated hardware circuit 110 determines the convolution parameters (step 606). The convolution parameters include, for example, the number of padding elements, the input size, the output size, the kernel size used in the convolutional layer, the number of kernels, and the stride length of the convolution operation.

[0067] Dedicated hardware circuitry 110 instructs the DMA engine to fill input feature data, such as a two-dimensional systolic array, transferred from notepad memory 150 to matrix computation unit 120 (step 608). The input feature data can be stored in the matrix computation unit in a manner that matches pre-stored filter data for convolution calculations. For example, a two-dimensional filter data matrix can be pre-loaded into the processing unit of matrix computation unit 120. The input feature data, such as the filled sub-matrix blocks of input feature data stored in notepad memory 150, can be transferred to the same processing unit of matrix computation unit 120 as the filter data processing unit for convolution calculations.

[0068] Dedicated hardware circuitry 110 supports general padding for convolution calculations within convolutional neural network layers. Sequencer 306 coordinates the comparison (610) of the top left corner of the note-like data (e.g., a two-dimensional matrix loaded in matrix computation unit 120 for convolution calculations) with a view of the data in main memory.

[0069] The convolutional neural network processing system 100 determines whether each position of the input feature data in the note-like view is within the bounds of the unfilled input feature view in the main memory 140 (612). When the top left corner of the note-like data is bound to the unfilled data view in the main memory 140, the system 100 commands the matrix calculation unit 120 to perform matrix calculation, and then performs vector accumulation in the vector calculation unit 130 to complete the convolution calculation (step 616).

[0070] When the top left corner of the notepad memory data is not bound to the unfilled input feature data view in the main memory 140, the sequencer core and vector calculation unit are instructed by the system to replace the unbounded value on the notepad memory 150 with a filled value (step 614).

[0071] Various methods exist for padding values ​​into the input feature data at matrix computation unit 120. One example padding technique is zero-padding, which means filling the boundaries of the input features with zero values. In another example, existing values ​​pre-stored in the note-like memory 150 and near the outer boundaries of the input features can be selected as padding values. In some other examples, values ​​stored in the opposite position of the input feature matrix (e.g., data stored in the bottom right of a two-dimensional matrix) can be used as padding values ​​at the top left of the two-dimensional matrix. The padded input feature data is updated in the matrix computation unit for convolution calculation.

[0072] Figure 7 This is an example of input feature data padding used in the notepad memory 150 for performing convolution calculations on convolutional neural network layers. This example can be used... Figure 6 The process and Figure 3 It is executed by a dedicated hardware system 300.

[0073] In the first stage of this example, similar to Figure 5 The aforementioned input feature data transition involves transferring the input feature data from main memory 140 to notepad memory 150. In this example, the input feature data is transferred and stored in notepad memory 150 as a 4x4 matrix 510. The boundary values ​​marked "x" and surrounding the edges of the 4x4 matrix are existing data pre-stored in notepad memory 150 and may be unrelated to the input feature data.

[0074] In the next stage of this example, the type and parameters of the convolution are determined by dedicated hardware circuitry 110. In this example, the convolution is the same as the previous one and the input feature data needs to be padded for the convolution calculation. Specifically, the padding is set to 1, the stride is set to 1, and the size of the padded data and the filter data is set to a 3x3 matrix.

[0075] The padding input feature data on the notepad memory 150 is shown on 720a-720d. As previously illustrated, the padding is determined to be 1, therefore values ​​existing outside the boundaries of the 4x4 matrix are selected as the padding values ​​for the input feature data. Existing padding values ​​are labeled "x" and can be different.

[0076] In the next stage of this example, dedicated hardware circuitry 110 instructs DMA engine 304 to load the selected input feature data (e.g., the top left 3x3 matrix data in 720a) from note memory 150 into matrix computation unit 120 for convolution computation.

[0077] The dedicated hardware circuit 110 utilizes the sequencer 306 to reconcile and compare the top left corner of the selected notepad memory data with the data view in main memory 140. In this example, the top left corner of the selected 3x3 matrix data in 720a does not match the top left corner of the corresponding data view in main memory 140. The top left corner data in the selected notepad memory 150 is determined to be outside the boundaries of the actual memory image and will be replaced with padding values. In some implementations, zero padding is applied to the convolution calculation, and the dedicated hardware circuit 110 operates the sequencer 306 and the vector calculation unit 130 to insert zeros into the padding data view. In some other implementations, the dedicated hardware circuit 110 may utilize existing values ​​x as padding values, but without performing convolution calculations on these padding values.

[0078] In the padded data matrix 730a of this example, the top row and left column data have been determined as an unpadded real data view not bound to main memory 140 and replaced with padded values ​​marked "p". The padded data matrix 730a is then convolved with the filter matrix in the matrix calculation unit 120.

[0079] Dedicated hardware circuitry 110 controls the DMA engine 304 as it crosses the sticky data and repeatedly compares the top left corner of the filled sticky memory data view with the real image data stored in main memory 140. As shown in sticky memory view 720b, the sticky access is shifted by one column and a second 3x3 matrix is ​​being transferred from the sticky memory to the matrix computation unit. The top left corner data element of the sticky view data is outside the boundary of the top left corner data of the real image data in main memory 140, and is therefore replaced with a padding value. In the filled view 430b, the top row data is replaced with a padding value for convolution calculation in matrix computation unit 120. Similarly, the padding data matrix described in 730a can be an existing value x from sticky memory 150, or zeros inserted from sequencer 306 and vector computation unit 130.

[0080] The dedicated hardware circuit 110 repeatedly accesses and fills the sticky memory data, compares the sticky memory data with the real image data in the main memory 140, and updates the fill values ​​in the sticky data until all input feature data is transferred from the sticky memory 150 to the matrix calculation unit 120 for convolution calculation.

[0081] The embodiments of the subject matter and functional operation described in this specification can be implemented using digital electronic circuit systems, tangibly embodied computer software or firmware, computer hardware (including the structures disclosed in this specification and their equivalents), or combinations thereof. Embodiments of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible, non-transitory program carrier for execution by or control of a data processing apparatus. Alternatively or additionally, program instructions can be encoded on artificially generated propagation signals (e.g., machine-generated electrical, optical, or electromagnetic signals) generated to encode information for transmission to a suitable receiver device for execution by the data processing apparatus. The computer storage medium can be a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or combinations thereof.

[0082] The term "data processing apparatus" encompasses all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus can include dedicated logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). In addition to hardware, the apparatus can also include code that creates the runtime environment for the computer program, such as code constituting processor firmware, protocol stacks, database management systems, operating systems, or combinations thereof.

[0083] A computer program (which may also be referred to or described as a program, software, software application, module, software module, script, or code) can be written in any form of programming language, including compiled or interpreted languages ​​or declarative or procedural languages, and can be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but does not necessarily, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to said program, or in multiple coordinating files (e.g., files storing portions of one or more modules, subroutines, or code). A computer program can be deployed to run on a single computer or on multiple computers located at a single site or distributed across multiple sites and interconnected via a communication network.

[0084] The processes and logic flows described in this specification can be executed by one or more programmable computers running one or more computer programs to perform functions by manipulating input data and generating output. The processes and logic flows can also be executed by dedicated logic circuit systems such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits), and the apparatus can also be implemented as a dedicated logic circuit system.

[0085] As an example, a computer suitable for running computer programs can be based on a general-purpose microprocessor or a special-purpose microprocessor, or both, or any other type of central processing unit. Typically, the central processing unit receives instructions and data from read-only memory or random access memory, or both. The essential components of a computer are the central processing unit for executing or running instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receiving data from, transferring data to, or both from one or more mass storage devices (e.g., disks, magneto-optical disks, or optical disks) for storing data. However, a computer does not need to have such devices. Furthermore, a computer can be embedded in another device such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device, such as a universal serial bus (USB) flash drive, to name a few.

[0086] Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory can be supplemented by or incorporated into dedicated logic circuitry systems.

[0087] To invoke interaction with the user, embodiments of the subject matter described herein can be implemented on a computer having a display device for displaying information to the user, such as a CRT (cathode ray tube) or LCD (liquid crystal display) monitor, and a keyboard and pointing device, such as a mouse or trackball, through which the user can send input to the computer. Other types of devices can also be used to invoke interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, voice, or tactile input. Additionally, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a web page to a web browser on the user's client device in response to a request received from a web browser.

[0088] Embodiments of the subject matter described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or middleware components (e.g., an application server), or front-end components (e.g., a client computer having a graphical user interface or web browser that a user can interact with in a manner that allows them to interact with an implementation of the subject matter described herein), or any combination of one or more such back-end, middleware, or front-end components. The components of the system can be interconnected via digital data communication (e.g., a communication network) of any form or medium. Examples of communication networks include local area networks (LANs) and wide area networks (WANs), such as the Internet.

[0089] A computing system can include clients and servers. Clients and servers are typically geographically separated and usually interact through a communication network. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other.

[0090] While this specification contains numerous details of specific implementations, these should not be construed as limiting the scope of any invention or the scope of possible claims, but rather as descriptions of features that may be specific to particular embodiments of a particular invention. Certain features described in the context of a single embodiment can also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment can also be implemented individually in multiple embodiments or in any suitable sub-combination. Furthermore, although features may be described above as acting in certain combinations and even initially claimed in this way, it is possible in some cases to remove one or more features from a claimed combination, and the claimed combination may be for sub-combinations or variations thereof.

[0091] Similarly, although operations are depicted in a specific order in the accompanying drawings, this should not be construed as requiring such operations to be performed in the specific order shown or in sequential order, or requiring the execution of all illustrated operations to achieve the desired result. In some cases, multitasking and parallel processing can be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0092] Specific embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions described in the claims can be performed in different orders and still achieve the desired result. As an example, the processes depicted in the drawings do not necessarily require a specific order or sequence to achieve the desired result. In some implementations, multitasking and parallel processing can be advantageous.

Claims

1. A method for providing universal padding support, the method comprising: A request is received to perform a convolution computation on a neural network on a hardware circuit having a matrix computation unit, the request specifying (i) the convolution computation to be performed on a feature tensor and a filter and (ii) padding to be applied to the feature tensor before performing the convolution computation; as well as Generate instructions that, when executed by the hardware circuit, cause the hardware circuit to perform operations, the operations including: The feature tensor data is transferred from the main memory of the hardware circuit to the notepad memory of the hardware circuit. Repeat the following operations: Identify the current subset of the feature tensor; Determining that the memory view of the current subset entering the note-based memory is inconsistent with the memory view of the current subset in the main memory, wherein the determination includes calculating whether a portion of the data in the memory view of the note-based memory is in the memory view of the current subset in the main memory, wherein the portion of the data in the memory view of the note-based memory is determined based on padding to be applied; and In response to determining that the memory view of the current subset in the note-taking memory is inconsistent with the memory view of the current subset in the main memory: The current subset of the feature tensor is loaded from the note-based memory into the matrix computation unit, and during the loading process, a general fill operation is performed on the loaded subset of the feature tensor; and The matrix computation unit is used to compute the matrix computation between the current subset of the feature tensor and the filter.

2. The method according to claim 1, wherein, The determination of whether the memory view of the current subset entering the note-based memory is consistent with the memory view of the current subset in the main memory includes: Coordinate the corner portion of the memory view in the notepad storage with the memory view in the main storage; and Determine whether all portions of the memory view in the note-taking memory are within the boundaries of the memory view in the main memory.

3. The method according to claim 1, wherein, The transition of the feature tensor data includes: The access address of the main memory is identified, the step length of shifting the access address in the main memory, the direct memory access size, and the access address of the note-like memory used to store the transferred feature tensor. Repeat the following operations: In a consistent memory view, a portion of the feature tensor data is transferred from the main memory to the note-style memory; The access address spanning the notepad memory; and In a consistent memory view, the next block of the feature tensor data is transferred from the main memory to the note-style memory.

4. The method according to claim 1, wherein, The general filler includes: Replace one or more values ​​loaded from the note-style memory with fill values; and Use one or more values ​​that are pre-stored in the note-style memory and are adjacent to the memory view in the note-style memory as fill values.

5. The method according to claim 1, wherein, The universal filler also includes: Identifies the type of convolution computation in a specific convolutional neural network layer; Determine the convolution parameters; and The feature tensor data is loaded from the note-style memory into the matrix calculation unit.

6. The method according to claim 1, wherein, The filter is loaded from the note-memory or the main memory into the matrix computation unit before the feature tensor data is loaded from the note-memory to the matrix computation unit.

7. The method according to claim 1, wherein, The universal filler also includes: The convolution parameters are determined, including the padding amount, the size of the input data, the size of the output data, the size of the filter data, the number of filters, and the stride of the convolution operation.

8. The method according to claim 2, wherein, The coordination between the corner portion of the memory view in the note-taking memory and the memory view in the main memory is performed by the sequencer of the hardware circuit.

9. The method according to claim 1, wherein, The general filling process is performed by the sequencer and vector calculation unit of the hardware circuit.

10. The method according to any one of claims 1-9, wherein, Using the matrix calculation unit to perform matrix calculations includes a series of vector multiplication operations.

11. A system for providing universal padding support, the system comprising one or more computers and a storage device for storing one or more instructions, the instructions being operable, when executed by the one or more computers, to cause the one or more computers to perform operations, the operations including: A request is received to perform a convolution computation on a neural network on a hardware circuit having a matrix computation unit, the request specifying (i) the convolution computation to be performed on a feature tensor and a filter and (ii) padding to be applied to the feature tensor before performing the convolution computation; as well as Generate instructions that, when executed by the hardware circuit, cause the hardware circuit to perform operations, the operations including: The feature tensor data is transferred from the main memory of the hardware circuit to the notepad memory of the hardware circuit. Repeat the following operations: Identify the current subset of the feature tensor; Determining that the memory view of the current subset entering the note-based memory is inconsistent with the memory view of the current subset in the main memory, wherein the determination includes calculating whether a portion of the data in the memory view of the note-based memory is in the memory view of the current subset in the main memory, wherein the portion of the data in the memory view of the note-based memory is determined based on padding to be applied; and In response to determining that the memory view of the current subset entering the note-style memory is inconsistent with the memory view of the current subset: The current subset of the feature tensor is loaded from the note-based memory into the matrix computation unit, and during the loading process, a general fill operation is performed on the loaded subset of the feature tensor; and The matrix computation unit is used to compute the matrix computation between the current subset of the feature tensor and the filter.

12. The system according to claim 11, wherein, The determination of whether the memory view of the current subset entering the note-based memory is consistent with the memory view of the current subset in the main memory includes: Coordinate the corner portion of the memory view in the notepad storage with the memory view in the main storage; and Determine whether all portions of the memory view in the note-taking memory are within the boundaries of the memory view in the main memory.

13. The system according to claim 11, wherein, The transition of the feature tensor data includes: The access address of the main memory is identified, the step length of shifting the access address in the main memory, the direct memory access size, and the access address of the note-like memory used to store the transferred feature tensor. Repeat the following operations: In a consistent memory view, transfer partial blocks of the feature tensor data from the main memory to the notepad memory; and The access address spanning the notepad memory; and In a consistent memory view, the next block of the feature tensor data is transferred from the main memory to the note-style memory.

14. The system according to claim 11, wherein, The general filler includes: Replace one or more values ​​loaded from the note-style memory with fill values; and Use one or more values ​​that are pre-stored in the note-style memory and are adjacent to the memory view in the note-style memory as fill values.

15. The system according to claim 11, wherein, The universal filler also includes: Identifies the type of convolution computation in a specific convolutional neural network layer; Determine the convolution parameters; and The feature tensor data is loaded from the note-style memory into the matrix calculation unit.

16. The system according to claim 11, wherein, The filter is loaded from the note-memory or the main memory into the matrix computation unit before the feature tensor data is loaded from the note-memory to the matrix computation unit.

17. The system according to claim 11, wherein, The universal filler also includes: The convolution parameters are determined, including the padding amount, the size of the input data, the size of the output data, the size of the filter data, the number of filters, and the stride of the convolution operation.

18. The system according to claim 12, wherein, The coordination between the corner portion of the memory view in the note-taking memory and the memory view in the main memory is performed by the sequencer of the hardware circuit.

19. The system according to claim 11, wherein, The general filling process is performed by the sequencer and vector calculation unit of the hardware circuit.

20. The system according to any one of claims 11 to 19, wherein, Using the matrix calculation unit to perform matrix calculations includes a series of vector multiplication operations.

Images