Memory Bandwidth Reduction Techniques for Low-Power Convolutional Neural Network Inference Applications

Abstract

To provide systems, apparatuses, and methods for implementing memory bandwidth reduction techniques for low power convolutional neural network inference applications.SOLUTION: A system includes at least a processing unit and an external memory coupled to the processing unit. A method implementing a convolution layer includes: detecting a request to perform a convolution operation on input data from a plurality of channels; partitioning, in response to detecting the request, the input data from the plurality of channels into 3D blocks so as to minimize external memory bandwidth utilization for the convolution operation performed; loading a selected 3D block from an external memory into an internal memory and generating convolution output data for the selected 3D block for one or more features; and adding, for each feature, convolution output data together across channels prior to writing the convolution output data to the external memory.SELECTED DRAWING: Figure 10

Description

[Background technology]

[0001] Machine learning is an emerging technology field, and convolutional neural networks are a type of machine learning model. Convolutional neural networks have demonstrated excellent performance in tasks such as handwritten digit classification and face detection. Convolutional neural networks have also shown promise for performing well in other, more challenging visual classification tasks. Machine learning models utilize significant amounts of processing resources and external memory bandwidth. Therefore, techniques for improving the performance and / or reducing memory bandwidth utilization of machine learning models are desirable.

[0002] The advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of one embodiment of a computing system. [Figure 2] FIG. 2 is a block diagram of another embodiment of a computing system. [Figure 3] FIG. 1 is a block diagram of one embodiment of a neural network implementation. [Figure 4] FIG. 2 is a block diagram of an embodiment of an inference engine. [Figure 5] FIG. 2 is a block diagram of another embodiment of an inference engine. [Figure 6] FIG. 1 is a block diagram of an embodiment of dividing channel data into 3D blocks. [Figure 7] FIG. 2 is a block diagram of another embodiment of an inference engine. [Figure 8] FIG. 1 is a block diagram of an embodiment that uses a vertical order for processing memory blocks to reduce external memory bandwidth utilization in neural network implementations. [Figure 9]FIG. 1 is a block diagram of one embodiment of block sizes for implementing different layers of a neural network. [Figure 10] 1 is a generalized flowchart illustrating one embodiment of a method for implementing a convolutional layer. [Figure 11] 1 is a generalized flowchart illustrating another embodiment of a method for selecting a technique for reducing external memory bandwidth. [Figure 12] FIG. 10 is a block diagram of one embodiment of a 3D block during a convolution operation. [Figure 13] FIG. 1 is a block diagram of one embodiment of dynamically reshaping 3D blocks during a convolution operation. DETAILED DESCRIPTION OF THE INVENTION

[0004] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art will recognize that various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements.

[0005] Various systems, apparatuses, methods, and computer-readable media that implement memory bandwidth reduction techniques for low-power convolutional neural network inference applications are disclosed herein. In one embodiment, a system includes at least a processing unit and an external memory connected to the processing unit. In one embodiment, the processing unit is a graphics processing unit (GPU). In other embodiments, the processing unit may be other types of processing units (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), or a central processing unit (CPU)). In one embodiment, the processing unit includes at least a command processor, one or more calculation units, and one or more caches. The one or more caches may also be referred to as internal memories of the processing unit.

[0006] In one embodiment, the system detects a request to perform a convolution operation on input data from multiple channels. In one embodiment, the convolution operation on the input data from the multiple channels is performed as part of a convolution layer of a neural network. In response to detecting the request, the system divides the input data from the multiple channels into three-dimensional (3D) blocks based on one or more factors. The one or more factors include determining how to minimize external memory bandwidth utilization for the convolution operation being performed. In another embodiment, the one or more factors include a size of internal memory, a size of one or more features, and / or a size of the convolution.

[0007] The system divides input data from multiple channels into multiple 3D blocks, selects a first 3D block to process, and loads the selected 3D block from external memory into the internal memory of the processing unit. The system then generates convolution output data for the selected 3D block for one or more features. The system then adds together the convolution output data across the first multiple channels of the 3D block for each one or more features before writing the convolution output data to external memory. The system then writes each sum of the convolution output data across the first multiple channels generated for the corresponding feature to external memory. The system then selects the next 3D block to process. In one embodiment, the system follows a specific pattern for processing 3D blocks to minimize external memory bandwidth usage. In one embodiment, the system starts by selecting the top-left 3D block and then proceeds vertically downward until it reaches the bottom boundary of the entire input data. The system then moves to the next column to the right and processes each 3D block, starting at the top of the column and moving down the column. This pattern of processing 3D blocks can continue for the remaining 3D blocks.

[0008] 1, a block diagram of one embodiment of a computing system 100 is shown. In one embodiment, computing system 100 includes at least an inference engine 105, one or more processors 110, an input / output (I / O) interface 120, a bus 125, and one or more memory devices 130. Note that inference engine 105 may also be referred to as a convolutional neural network 105 or a neural network 105. In other embodiments, computing system 100 may include other components and / or may be arranged differently.

[0009] In one embodiment, inference engine 105 is implemented on a processing unit (e.g., a graphics processing unit (GPU)). In another embodiment, inference engine 105 is implemented on programmable logic (e.g., a field programmable gate array (FPGA)). In a further embodiment, inference engine 105 is implemented on dedicated logic (e.g., an application specific integrated circuit (ASIC)). In other embodiments, inference engine 105 may be implemented on other types of components, other types of logic, and / or a combination of multiple different types of components or processing units.

[0010] In one embodiment, the inference engine 105 implements one or more layers of a convolutional neural network. For example, in one embodiment, the inference engine implements one or more convolutional layers, one or more pooling layers, one or more normalization layers, and / or one or more fully connected layers. The stack of layers may be repeated multiple times, with the type of stack (e.g., convolution-normalization-pooling) and the layers within the stack varying from one embodiment to another.

[0011] Generally speaking, an “inference engine” is defined as hardware and / or software that receives image data and generates one or more label probabilities for the image data. An “inference engine” may also be referred to as a “classification engine” or a “classifier.” In one embodiment, the inference engine 105 is a trained neural network. For example, in this embodiment, the inference engine 105 is configured to analyze video frames and generate one or more label probabilities for the video frames. For example, potential use cases include at least eye tracking, object recognition, point cloud estimation, ray tracing, light field modeling, depth tracking, etc. In an eye tracking use case, the inference engine may be based on learning patterns, dwell, transition angles, blinks, etc. In other embodiments, the inference engine can be customized for other types of use cases. In one embodiment, the inference engine is approximated by integer or low-precision weight parameters. These approximations can approach optimal accuracy, allowing for significant speedup and power reduction of the inference engine. Also, in one embodiment, the inference engine is implemented using multiple internal channel processing engines to reduce memory bandwidth utilization when implementing a convolutional neural network. In other embodiments, the inference engine may implement other types of machine learning models.

[0012] Processor 110 represents any number and type of processing unit (e.g., central processing unit (CPU), GPU, digital signal processor (DSP), FPGA, ASIC). In one embodiment, some of the processing associated with inference engine 105 is performed by processor 110. Memory device 130 represents any number and type of memory device. For example, the type of memory in memory device 130 may include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc. Memory device 130 is accessible by inference engine 105 and processor 110. I / O interface 120 represents any number and type of I / O interface (e.g., peripheral component interconnect (PCI) bus, PCI-Express (PCI-X), PCI Express (PCI-E) bus, Gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices may be connected to I / O interface 120. Such peripherals include, but are not limited to, displays, keyboards, mice, printers, scanners, joysticks, other types of game controllers, media recording devices, external storage devices, network interface cards, and the like.

[0013] In various embodiments, computing system 100 may be a computer, a laptop, a mobile device, a game console, a server, a streaming device, a wearable device, or any of a variety of other types of computing systems or devices. It should be noted that the number of components of computing system 100 may vary from embodiment to embodiment. Each component / subcomponent may include more or fewer components than those shown in FIG. 1. It should also be noted that computing system 100 may include other components not shown in FIG. 1. In other embodiments, computing system 100 may be configured in a manner other than that shown in FIG. 1.

[0014] Referring now to FIG. 2, a block diagram of another embodiment of a computing system 200 is shown. In one embodiment, the system 200 includes a GPU 205, a host CPU 210, a system memory 225, and a local memory 230. The system 200 also includes other components not shown to avoid obscuring the diagram. Applications run on the host CPU 210 and interface with a driver 212 to send program commands and data to the GPU 205. The host CPU 210 is connected to a memory controller 220 and the system memory 225. The GPU 205 includes at least a command processor 240, a dispatch unit 250, computation units 255A-255N, the memory controller 220, a global data share 270, a level 1 (L1) cache 265, and a level 2 (L2) cache 260. In other embodiments, GPU 205 may include other components, may omit one or more of the illustrated components, and / or may be organized in other suitable ways.

[0015] Certain types of machine learning and neural network applications have high bandwidth requirements, and if the external memory bandwidth can be reduced, improved performance and faster processing times can be achieved. In some cases, external memory bandwidth can be a bottleneck when implementing neural networks. Therefore, reducing the external memory bandwidth can significantly improve the execution speed of neural networks on system 200.

[0016] Global data share 270, L1 cache 265, and L2 cache 260 of GPU 205 represent the term "internal memory" as used herein. Internal memory is distinguished from memory external to GPU 205, and system memory 225 and local memory 230 represent the term "external memory" as used herein. In executing various neural network applications, GPU 205 is configured to minimize external memory bandwidth for reading and writing data from system memory 225 and local memory 230.

[0017] In various embodiments, computing system 200 is configured to implement any of various types of machine learning algorithms. For example, in one embodiment, computing system 200 is configured to implement a neural network (e.g., a perceptron, a convolutional neural network). Applications executing on host CPU 210 utilize driver 212 to send data and commands to GPU 205. Command processor 240 is configured to receive commands from host CPU 210 and utilize dispatch unit 250 to issue commands to compute units 255A-255N. Computing units 255A-255N are configured to read and write data from global data share 270, L1 cache 265, and L2 cache 260 within GPU 205 using internal memory bandwidth. Although not shown in FIG. 2 , computing units 255A-255N may include one or more caches and / or local memories within each computing unit 255A-255N. These caches and / or local memories may be included in the expression "internal memory" of GPU 205. Additionally, compute units 255A-255N are configured to utilize external memory bandwidth to read and write data from system memory 225 and / or local memory 230 via memory controller 220.

[0018] The data splitting unit 245 may be implemented using any suitable combination of hardware and / or software. In one embodiment, the command processor 240 includes the channel data splitting unit 245, which determines how to split the channel data for one or more layers of the implemented neural network. In another embodiment, the channel data splitting unit 245 may be included as part of the driver 212. In further embodiments, the channel data splitting unit 245 may be split between the driver 212, the command processor 240, and / or one or more other locations. In one embodiment, the channel data splitting unit 245 determines how to split the channel data for each neural network layer to minimize external memory bandwidth usage for a given layer of the neural network. In some embodiments, one or more layers of a neural network are merged. In these embodiments, the splitting unit 245 determines how to split the channel data for the merged layer to minimize external memory bandwidth usage for the merged layer. Techniques for splitting the channel data for layers of a neural network to minimize external memory bandwidth usage are described in more detail in the remainder of this disclosure.

[0019] 3, there is shown a block diagram of one embodiment of an implementation of a neural network 300. Neural network 300 includes a convolutional layer 302, a subsampling layer 304, a convolutional layer 306, a subsampling layer 308, and a fully connected layer 310. In other embodiments, neural network 300 may include other numbers of layers and arrangements of layers.

[0020] When implementing neural network 300 on a computing system (e.g., system 100 of FIG. 1), memory bandwidth requirements can be significant. When implementing neural network 300 on a computing system, one or more techniques may be utilized to reduce the amount of external memory bandwidth consumed. For example, a first technique involves reading an input channel once rather than multiple times and performing a convolution of the input channel for all features. In the first technique, the read memory bandwidth of the input channel is reduced by the number of channels. A second technique involves generating multiple channels of convolution output data for one or more features and combining the convolution output data across the multiple channels before writing the convolution output data back to external memory. A third technique involves combining two or more layers of neural network 300 and performing the two or more layers inline before writing the output data back to external memory. In various embodiments, other techniques may be utilized and / or combined with one or more of these three techniques.

[0021] Referring now to FIG. 4, a block diagram of one embodiment of an inference engine 400 is shown. Depending on the embodiment, the inference engine 400 may be implemented on system 100 (of FIG. 1), system 200 (of FIG. 2), or another computing system. The inference engine 400 includes multiple channel processing engines 402A-402N. Note that the inference engine 400 may also be referred to as an inference accelerator. The number of channel processing engines 402A-402N may vary from embodiment to embodiment. The channel processing engine 402A is expanded to show the components of a channel processing engine according to one embodiment. The other channel processing engines 402B-402N may include a similar arrangement of components.

[0022] The implementation of the inference engine 400 illustrates a first technique for reducing external memory bandwidth utilization. Using the first technique illustrated in FIG. 4 , input channel data for each channel is read once, rather than multiple times. Convolution is then performed on the input channel data for all features. For example, in one embodiment, input channel data 404 is read for channel 0, and then convolution is performed on all “M” features 410A-410M. The number of “M” features may vary depending on the layer and embodiment. The results of the convolution of the input data with each feature are shown as output data blocks 415A-415M, and the output data 415A-415M are written to external memory represented by blocks 420A-420M. After convolution of all M features is performed for channel 0, convolution of all M features is performed for channel 1, and so on. Blocks 402B-402N show convolutions being performed on other channels, with the resulting output data being written to external memory. The number of channels "N" may vary according to the layers of the neural network and according to the embodiment.

[0023] Adders 430A-430M represent sums calculated for each feature of all output channel data generated from all convolutions of the N channels. The sums generated by adders 430A-430M are shown as blocks 440A-440M. These sums may be written to external memory after being calculated by adders 430A-430M. After the sums are written to external memory, one or more other layers of the neural network may be executed.

[0024] 5, a block diagram of another embodiment of an inference engine 500 is shown. Inference engine 500 illustrates a second technique for reducing external memory bandwidth utilization. As shown in inference engine 500, data for multiple channels is loaded into the internal memory of a processing unit (e.g., GPU 205 of FIG. 2). The number of channels loaded into the internal memory is limited by the size of the internal memory of the processing unit.

[0025] In one embodiment, the channel blob shape representing the input data is divided into three-dimensional (3D) blocks based on the availability of internal memory size. Next, an "N" channel convolution is performed on the first feature of each channel in convolution calculation blocks 502A-502N. The convolution output data of the N channels and feature 0 are summed together by adder 520A for feature 0 before writing any of the N channel convolution output data for feature 0 to external memory. Also, in parallel with the N channel convolution of feature 0, N channel convolutions of the other M-1 features are also performed. Each of the M features has another adder 520A-520M that sums the N channel convolution output data of each feature before writing the convolution output data to external memory. Next, after adders 520A-520M generate sums for different features 0-M, the resulting sums are written to external memory as shown in blocks 530A-530M.

[0026] Referring now to FIG. 6, a block diagram of one embodiment for dividing channel data into three-dimensional (3D) blocks is shown. Channel data 605 is shown as a 3D blob of data on the left side of FIG. 6. In one embodiment, a channel data division unit (e.g., channel data division unit 245 of FIG. 2) determines how to divide channel data 605 into blocks 610A-610H (and other unlabeled blocks behind the 3D blob). For purposes of this description, assume that the entire channel data 605 is divided into a total of 16 blocks, each of size N×N in the X and Y directions and size M / 2 in the Z (or channel) direction. It should be understood that this division is merely indicative of one particular embodiment. In other embodiments, the entire channel data 605 may be divided into other numbers and sizes of blocks.

[0027] In one embodiment, blocks 610A-610H of channel data 605 are loaded one block at a time from external memory into the internal memory of a processing unit (e.g., GPU 205). For example, block 610A is loaded from external memory into internal memory, then a convolution operation and optionally one or more operations are performed on block 610A for a given layer or layers of the neural network, and the output data is written back to external memory. Next, block 610B is loaded and the same operation is performed before writing the output data back to external memory, then block 610B is processed, and so on. In another embodiment, the pattern of processing the 3D blocks may vary, such as block 610A being processed first, then block 610E, then block 610B, then block 610F, etc.

[0028] Block 610A is expanded on the right side of FIG. 6 to show how convolution is performed in one embodiment after block 610A is loaded from external memory into the processing unit's internal memory. In one embodiment, the first convolution step (step 1) is performed on the leftmost N / 4 pixels of channels 0 through 3. Next, a second convolution step is performed on the leftmost N / 4 pixels of channels 4 through 7, a third convolution step is performed on the leftmost N / 4 pixels of channels 8 through 11, and so on, until step M / 8 is performed on the convolution of the leftmost N / 4 pixels of channels M / 2-4 through M / 2-1. After step M / 8, the next step M / 8+1 can return to the beginning of block 610A, starting with the second column from the left and proceeding toward the end of block 610A. This pattern of processing the convolutions of block 610A with all features can continue until the column after the rightmost column of block 610A is reached. Note that the convolutions of different portions of block 610A are performed without utilizing external memory bandwidth, which helps reduce the overall external memory bandwidth utilization for implementing one or more layers of the neural network.

[0029] Referring now to FIG. 7, a block diagram of another embodiment of an inference engine 700 is shown. The inference engine 700 illustrates a third technique for reducing external memory bandwidth utilization. The third technique involves executing multiple layers on groups of channel data after reading the groups of channel data from external memory to internal memory and before writing the output data to external memory. A traditional approach to executing layers of a neural network performs a single layer of processing on input data and, after the single layer is complete, writes the processed data back to external memory. The processed data is then read back from external memory, and the next layer of the neural network is executed. This approach utilizes a large amount of external memory bandwidth.

[0030] To execute multiple layers of the neural network, groups of channel data 705 are loaded from external memory into the internal memory of the processing unit implementing the inference engine 700. In one embodiment, a convolutional layer 710 is executed as the first layer after loading a block of channel data from external memory.

[0031] Rather than writing the output of the convolutional layer 710 back to external memory, the output of the convolutional layer 710 is processed by the next layer in the neural network. As shown in FIG. 7, the next layer in the neural network is a pooling layer 715. In other embodiments, other layers may follow the convolutional layer 710. In one embodiment, a normalization layer 720 follows the pooling layer 715, with the resulting data 725 written back to external memory. It should be understood that the example of combining convolutional layers with pooling and normalization layers in the inference engine 700 is merely illustrative of one embodiment. In other embodiments, other numbers and / or types of layers may be combined and performed inline on groups of channel data 705 before writing the output resulting data 725 back to external memory.

[0032] Referring now to Figure 8, a block diagram of one embodiment is shown using a vertical order for processing memory blocks to reduce external memory bandwidth utilization in neural network implementations. A group of channel data 800 is shown on the left side of Figure 8. The group of channel data 800 is divided into multiple 3D blocks 805A-805P. In one embodiment, the channel data 800 is divided into multiple 3D blocks 805A-805P by a channel data division unit (e.g., channel data division unit 245 of Figure 2).

[0033] In one embodiment, after splitting, the top left 3D block 805A is processed by an inference engine (e.g., inference engine 700 of FIG. 7). Then, moving down the group of channel data 800, 3D block 805B is processed, then 3D block 805C, then 3D block 805D, then 3D block 805E, and so on. Each vertical column is processed from top to bottom, and the next right column is processed from top to bottom, until all columns of the group of channel data 800 have been processed.

[0034] The right side of Figure 8 shows source pixels 810 at the boundary between 3D blocks 805A and 805E. To generate convolution output pixel 815 on the boundary, source pixels 810 from both 3D blocks 805A and 805E are utilized. Pooled output pixels 820 from the pooling layer are shown in the bottom row of the figure. Pooled output pixels 820 near the boundary use source pixels 810 from both 3D blocks 805A and 805E. Additionally, the top and bottom pixels of each 3D block 805A-805P require pixels from adjacent blocks. Fetching extra lines from adjacent blocks results in extra fetches of data from external memory. Therefore, an efficient technique for processing 3D blocks 805A-805P is to use the processing pattern shown on the left side of Figure 8.

[0035] Referring now to FIG. 9, a block diagram of one embodiment of block sizes for implementing different layers of a neural network is shown. The description of FIG. 9 is intended to be a continuation of the description of FIG. 8. Blocks 805A, 805B, and 805E in FIG. 9 are intended to represent the same blocks 805A, 805B, and 805E of the group of channel data 800 in FIG. 8. To implement a convolutional layer, additional pixels from adjacent blocks are used to perform the convolutional layer processing. These additional pixels are shown in block 910, which includes additional pixels on each side of block 805A. Also, to implement a pooling layer, additional pixels from adjacent blocks are used to perform the pooling layer processing. These additional pixels are shown in block 920, which includes even more pixels on each side of block 805A (compared to block 910).

[0036] For purposes of this description, it is assumed that, in one embodiment, each block 805A, 805B, and 805E includes 15 lines of pixels. In other embodiments, each block 805A, 805B, and 805E may include other numbers of lines of pixels. In an embodiment in which blocks 805A, 805B, and 805E each include 15 lines of pixels, 15 lines of pixels are fetched from external memory and two additional lines of pixels are added to perform the convolutional and pooling layers in the processing of block 805A. In other embodiments, other numbers of additional lines besides two may be required to perform the convolutional and pooling layers (and any additional layers that are performed). Next, moving vertically to block 805B, the bottom four lines of pixels are retained in internal memory from previous processing operations of block 805A. The next 15 lines of pixels are fetched, and these 15 lines and the retained four lines are processed together to perform the convolutional and pooling layers for block 805B. The bottom four lines of pixels are retained from the processing of block 805A, avoiding refetching these boundary lines, which helps reduce external memory bandwidth usage when performing convolutional and pooling layers as processing moves between vertically adjacent blocks.

[0037] Referring now to FIG. 10, one embodiment of a method 1000 for executing a convolutional layer is shown. For purposes of explanation, the steps in this embodiment and the embodiment of FIG. 11 are shown sequentially. However, it should be noted that in various embodiments of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as desired. Any of the various systems or devices described herein may be configured to perform method 1000.

[0038] The system detects a request to perform a convolution operation on input data from multiple channels (block 1005). For example, in one embodiment, the request may specify that a convolution layer of a neural network be executed to process input data from multiple channels. In one embodiment, the system includes at least a processing unit (e.g., a GPU) connected to external memory. The processing unit includes internal memory and various computation units, execution units, multiply-accumulate units (MACs), and / or other logic.

[0039] In response to detecting the request, the system divides the input data of multiple channels into multiple three-dimensional (3D) blocks based on one or more factors (block 1010). Two of the three dimensions (of the 3D blocks) correspond to the x, y spatial dimensions of the original input (e.g., image, video frame), and the third dimension (i.e., z) corresponds to the multiple channels. In one embodiment, the system divides the input data into 3D blocks to minimize external memory bandwidth utilization for a given layer of the neural network. In another embodiment, the system divides the input data into 3D blocks based on the size of internal memory, the size of one or more features, and the size of the convolutions. In other embodiments, the system can divide the input data into 3D blocks based on one or more other factors and / or to achieve one or more other goals.

[0040] Next, the system selects a first 3D block from the divided 3D blocks for processing (block 1015). The system then loads the selected 3D block from external memory into the processing unit's internal memory (block 1020). Next, the system generates convolution output data for the selected 3D block for one or more features (block 1025). Next, for each of the one or more features, the system adds together the convolution output data across the first plurality of channels of the 3D block before writing the convolution output data to external memory (block 1030). Next, the system writes each sum of the convolution output data across the first plurality of channels generated for the corresponding feature to external memory (block 1035). Next, the system determines whether other 3D blocks remain to be processed (conditional block 1040). If other 3D blocks remain to be processed (conditional block 1040: Yes), the system selects the next 3D block to be processed (block 1045), and method 1000 returns to block 1020. Note that the system may follow a particular pattern for processing 3D blocks to minimize external memory bandwidth utilization. An example pattern is described in connection with Figure 8. If all 3D blocks have been processed (conditional block 1040: No), method 1000 ends.

[0041] 11 , one embodiment of a method 1100 for selecting an external memory bandwidth utilization technique to be implemented for an inference engine is illustrated. A request to execute one or more layers of a neural network is detected (block 1105). In one embodiment, the processing unit is a GPU. In other embodiments, the processing unit may be other types of processing units. In response to detecting the request, the processing unit determines the number of channels, the amount of channel data, and the type of processing associated with the one or more layers (block 1110). The processing unit may also determine other factors (e.g., number of features, feature size, stride) associated with executing one or more layers of the neural network in block 1110.

[0042] The processing unit then determines which one or more of a plurality of external memory bandwidth reduction techniques to use based on the factors determined in block 1110 (block 1115). For example, the processing unit may calculate the amount of external memory bandwidth used for each technique. The processing unit may also calculate the amount of external memory bandwidth used when two or more techniques are combined. The processing unit may then select one or more techniques that minimize external memory bandwidth usage. The processing unit then partitions the channel data according to the selected one or more techniques and based on the amount of internal memory available to the processing unit (block 1120). The processing unit then runs one or more layers of a neural network on each portion of the partitioned channel data (block 1125). The processing unit then writes the processed channel data to external memory for each portion (block 1130). After block 1130, method 1100 ends.

[0043] Referring now to FIG. 12 , a block diagram of one embodiment of a 3D block during a convolution operation is shown. The input channel blob 1205 is shown on the left side of FIG. 12 . It should be understood that the dimensions of the channel blob 1205 and blocks 1210A-1210H illustrate one particular embodiment. In other embodiments, the dimensions of the channel blob 1205 and blocks 1210A-1210H may differ from those shown in FIG. 12 . In one embodiment, the 3D block dimensions of blocks 1210A-1210H are selected based on internal memory / cache size, the input 3D blob shape (channel × blob width × blob height), and the convolution kernel size. Reshaping the 3D block in the Z direction reduces partial accumulator storage requirements and external memory bandwidth usage. In one embodiment, dynamic resizing of the 3D block size can be implemented in firmware and / or software before initiating the convolution process of the input channel blob 1205.

[0044] Once block 1210A is processed, the processed block is retrieved from input channel storage 1235 in external memory 1220 and stored in internal memory 1215. The size of internal memory 1215 is limited, and therefore the dimensions of blocks 1210A-1210H can be selected so that a single block fits into internal memory 1215.

[0045] A partial accumulator storage 1230 in the external memory 1220 is used to store all features for the partially convolved accumulated values ​​for the first set of channels 0 through 127. These partial values ​​are written to the external memory 1220 for all 3D blocks in the X and Y directions. The convolution process then moves in the Z direction to process the rear 3D blocks (3D blocks for channels 128 through 255). While processing the rear 3D blocks, the partially convolved accumulated values ​​1230 for the previous blocks are read from the external memory 1220, and these values ​​are combined with the partially convolved accumulated values ​​from the rear blocks to produce the final convolution output, which is stored in the output feature storage area 1225 of the external memory 1220. This convolution process to produce the final convolution output requires additional read and write memory bandwidth associated with the partial accumulator storage 1230 as it moves from the front to the rear of the channel blob 1205.

[0046] Referring now to FIG. 13, a block diagram of one embodiment for dynamically reshaping 3D blocks during a convolution operation is shown. As shown on the left side of FIG. 13, an input channel blob 1305 is divided into multiple 3D blocks 1310A-1310S. The dimensions of the 3D blocks 1310A-1310S are selected so that all of channels 0 through 255 are contained within a single block. The dimensions of the 3D blocks are also selected so that the entire 3D block fits within internal memory 1315 for all channels from channel 0 through channel 255. It should be understood that the example of an input channel blob 1305 having 256 channels is merely illustrative of one embodiment. In other embodiments, the input channel blob 1305 may have other numbers of channels.

[0047] By reshaping blocks 1310A-1310S in the Z direction, all input channels can be stored in internal memory 1315 to process the convolution of all input channels of input channel blob 1305. This helps minimize the external memory bandwidth associated with partial accumulator storage in external memory 1320. Compared to the exemplary 3D block sizing shown in FIG. 12, only reading input channels from input channel storage 1335 and writing output features ultimately uses external memory bandwidth. Reducing external memory bandwidth usage allows for reduced system power consumption.

[0048] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general-purpose processor or a special-purpose processor are contemplated. In various embodiments, such program instructions may be expressed in a high-level programming language. In other embodiments, the program instructions may be compiled from the high-level programming language into binary, intermediate, or other format. Alternatively, program instructions may be written that describe the operation or design of hardware. Such program instructions may be expressed in a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog may be used. In various embodiments, the program instructions are stored on any of a variety of non-transitory computer-readable storage media. The storage medium is accessible by a computing system during use and provides the program instructions to the computing system for program execution. Generally, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0049] It should be emphasized that the above-described embodiments are merely non-limiting examples of implementations. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to embrace all such variations and modifications.

Claims

1. 1. An apparatus comprising: a plurality of channel processing engines, each of the plurality of channel processing engines configured to perform a convolution operation on any one of a plurality of channels; a first channel processing engine of the plurality of channel processing engines; reading first data corresponding to a first channel among the plurality of channels; performing a calculation on a plurality of features using the first data before performing a read of second data for a second channel of the plurality of channels; configured to: Device.

2. the first channel processing engine is configured to generate first output data responsive to the operation.

10. The apparatus of claim 1.

3. a second channel processing engine of the plurality of channel processing engines, after operations on the plurality of features, reading second data corresponding to a second channel among the plurality of channels; performing an operation on the plurality of features using the second data before performing a read of data for a third channel of the plurality of channels; configured to:

3. The apparatus of claim 2.

4. the second channel processing engine is configured to generate second output data responsive to the operation using the second data.

4. The apparatus of claim 3.

5. further comprising a plurality of adders, each of the plurality of adders corresponding to one of the plurality of features; 5. The apparatus of claim 4.

6. a first adder of the plurality of adders configured to generate a sum of the first output data and the second output data; 6. The apparatus of claim 5.

7. the device is configured to read the first data corresponding to the first channel only once to generate both the first output data and the second output data.

5. The apparatus of claim 4.

8. the first channel processing engine is configured to store the first output data in an external memory.

3. The apparatus of claim 2.

9. 1. A method comprising: a first channel processing engine among the plurality of channel processing engines reads first data corresponding to a first channel among the plurality of channels, and each of the plurality of channel processing engines is configured to perform a convolution operation on any one of the plurality of channels; the first channel processing engine performing operations on a plurality of features using the first data before performing a read of second data for a second channel of the plurality of channels; method.

10. the first channel processing engine generating first output data responsive to the operation; 10. The method of claim 9.

11. a second channel processing engine of the plurality of channel processing engines, after operations on the plurality of features, reading second data corresponding to a second channel among the plurality of channels; performing an operation on the plurality of features using the second data before performing a read of data for a third channel of the plurality of channels; The method of claim 10.

12. the second channel processing engine generating second output data in response to the operation using the second data. The method of claim 11.

13. any one of a plurality of adders generating a sum of the first output data and the second output data; 13. The method of claim 12.

14. reading the first data corresponding to the first channel only once to generate both the first output data and the second output data.

14. The method of claim 13.

15. the first channel processing engine storing the first output data in an external memory; The method of claim 10.

16. 1. A system comprising: a processing unit including an internal memory; an external memory connected to the processing unit; an interface engine; The interface engine a plurality of channel processing engines, each of the plurality of channel processing engines configured to perform a convolution operation on any one of a plurality of channels; a first channel processing engine of the plurality of channel processing engines; reading first data corresponding to a first channel of the plurality of channels from the external memory to the internal memory; performing a calculation on a plurality of features using the first data before performing a read of second data for a second channel of the plurality of channels; configured to: system.

17. the first channel processing engine is configured to generate first output data responsive to the operation.

17. The system of claim 16.

18. the first channel processing engine is configured to store the first output data in the external memory.

18. The system of claim 17.

19. a second channel processing engine of the plurality of channel processing engines, after operations on the plurality of features, reading second data corresponding to a second channel of the plurality of channels from the external memory to the internal memory; performing an operation on the plurality of features using the second data before performing a read of data for a third channel of the plurality of channels; configured to:

17. The system of claim 16.

20. the second channel processing engine is configured to generate second output data responsive to the operation using the second data.

20. The system of claim 19.

Images