Memory Bandwidth Reduction Techniques for Low-Power Convolutional Neural Network Inference Applications

Abstract

Systems, devices, and methods for implementing memory bandwidth reduction techniques for low-power convolutional neural network inference applications are disclosed. A system includes at least a processing unit and an external memory connected to the processing unit. The system detects a request to perform a convolution operation on input data from multiple channels. In response to detecting the request, the system partitions the input data from the multiple channels into 3D blocks so as to minimize external memory bandwidth utilization for performing the convolution operation. Next, the system loads the selected 3D blocks from the external memory into an internal memory, and then generates convolution output data for one or more features for the selected 3D blocks. Then, for each feature, the system adds the convolution output data together across channels before writing the convolution output data to the external memory.

Description

Background Art

[0001] Related technical description

[0002] One emerging technology area is machine learning, where convolutional neural networks are a type of machine learning model. Convolutional neural networks have shown excellent performance on tasks such as handwritten digit classification and face detection. In addition, convolutional neural networks have shown the potential to perform well in other more challenging visual classification tasks. Machine learning models utilize a significant amount of processing resources and external memory bandwidth. Therefore, techniques for improving the performance of machine learning models and / or reducing their memory bandwidth utilization are desired. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings, in which:

[0004] Figure 1 is a block diagram of one embodiment of a computing system.

[0005] Figure 2 is a block diagram of another embodiment of a computing system.

[0006] Figure 3 is a block diagram of one embodiment of an implementation of a neural network.

[0007] Figure 4 is a block diagram of one implementation of an inference engine.

[0008] Figure 5 is a block diagram of another embodiment of an inference engine.

[0009] Figure 6 is a block diagram of one implementation of partitioning channel data into 3D blocks.

[0010] Figure 7 is a block diagram of another embodiment of an inference engine.

[0011] Figure 8 is a block diagram of one embodiment of using vertical sequential processing memory blocks to reduce external memory bandwidth utilization in neural network implementations.

[0012] Fig. 9 is a block diagram of one embodiment of block sizes for implementing different layers of a neural network.

[0013] Fig.10 is a generalized flow chart illustrating one embodiment of a method for implementing a convolutional layer.

[0014] Fig.11is a generalized flow chart illustrating another embodiment of a method for selecting a technique for external memory bandwidth reduction.

[0015] Fig.12 is a block diagram of one implementation of a 3D block during a convolution operation.

[0016] Fig.13 is a block diagram of one implementation for dynamically reshaping 3D blocks during convolution operations. DETAILED DESCRIPTION

[0017] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms proposed herein. However, it will be appreciated by those of ordinary skill in the art that various embodiments may be practiced without these specific details. In some cases, well-known structures, components, signals, computer program instructions, and techniques are not shown in detail in order to avoid obscuring the methods described herein. It should be understood that for clarity of illustration, the elements shown in the drawings are not necessarily drawn to scale. For example, the size of some elements may be enlarged relative to other elements.

[0018] Disclosed herein are various systems, devices, methods, and computer-readable media for implementing memory bandwidth reduction techniques for low-power convolutional neural network inference applications. In one embodiment, a system includes at least a processing unit and an external memory connected to the processing unit. In one embodiment, the processing unit is a graphics processing unit (GPU). In other embodiments, the processing unit can be other types of processing units (e.g., application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), digital signal processors (DSPs), central processing units (CPUs)). In one embodiment, the processing unit includes at least a command processor, one or more computing units, and one or more caches. The one or more caches may also be referred to as the internal memory of the processing unit.

[0019] In one embodiment, the system detects a request to perform a convolution operation on input data from a plurality of channels. In one embodiment, the convolution operation on the input data from the plurality of channels is implemented as part of a convolution layer of a neural network. In response to detecting the request, the system partitions the input data from the plurality of channels into three-dimensional (3D) blocks based on one or more factors. The one or more factors include determining how to minimize external memory bandwidth utilization for the convolution operation being performed. In another embodiment, the one or more factors include the size of internal memory, the size of one or more features, and / or the size of the convolution.

[0020] After the input data from the multiple channels is partitioned into multiple 3D blocks, the system selects a first 3D block for processing and loads the selected 3D block from the external memory into the internal memory of the processing unit. Then, the system generates convolution output data for one or more features for the selected 3D block. Then, for each feature in the one or more features, the system adds the convolution output data together across the first multiple channels of the 3D block before writing the convolution output data to the external memory. Then, the system writes each and of the convolution output data across the first multiple channels generated for the corresponding feature to the external memory. Then, the system selects the next 3D block to be processed. In one embodiment, the system follows a specific mode for processing 3D blocks in order to minimize external memory bandwidth utilization. In one embodiment, the system starts by selecting the top 3D block on the left, and then moves downward in the vertical direction until the bottom boundary of the entire input data is reached. Then, the system moves one column to the right, and starts from the top of the column and moves down along the column to process each 3D block. This pattern of processing the 3D block may continue for the remainder of the 3D block.

[0021] Reference now Figure 1 , a block diagram of one embodiment of a computing system 100 is shown. In one embodiment, computing system 100 includes at least an inference engine 105, one or more processors 110, an input / output (I / O) interface 120, a bus 125, and one or more memory devices 130. It should be noted that inference engine 105 may also be referred to as convolutional neural network 105 or neural network 105. In other embodiments, computing system 100 may include other components and / or computing system 100 may be arranged in different ways.

[0022] In one embodiment, the inference engine 105 is implemented on a processing unit, such as a graphics processing unit (GPU). In another embodiment, the inference engine 105 is implemented on programmable logic, such as a field programmable gate array (FPGA). In another embodiment, the inference engine 105 is implemented on dedicated logic, such as an application specific integrated circuit (ASIC). In other embodiments, the inference engine 105 may be implemented on other types of components, other types of logic, and / or any combination of multiple different types of components or processing units.

[0023] In one embodiment, the inference engine 105 implements one or more layers of a convolutional neural network. For example, in one embodiment, the inference engine implements one or more convolutional layers, one or more pooling layers, one or more normalization layers, and / or one or more fully connected layers. The layer stack may be repeated multiple times, where the type of stack (e.g., convolution-normalization-pooling) and the layers within the stack vary from embodiment to embodiment.

[0024] In general, an "inference engine" is defined as hardware and / or software that receives image data and generates one or more label probabilities for the image data. An "inference engine" may also be referred to as a "classification engine" or a "classifier". In one embodiment, the inference engine 105 is a trained neural network. For example, in this embodiment, the inference engine 105 is configured to analyze video frames to generate one or more label probabilities for video frames. For example, possible use cases include at least eye tracking, object recognition, point cloud estimation, ray tracing, light field modeling, depth tracking, etc. For eye tracking use cases, the inference engine may be based on learned patterns, stays, angles of transition, blinks, etc. In other embodiments, the inference engine may be customized for other types of use cases. In one embodiment, the inference engine is approximated by integer or reduced precision weighted parameters. These approximations can be close to optimal accuracy and achieve substantial speed-up and power reduction of the inference engine. In addition, in one embodiment, the inference engine is implemented with multiple internal channel processing engines to reduce memory bandwidth utilization when implementing a convolutional neural network. In other embodiments, the inference engine may implement other types of machine learning models.

[0025] The one or more processors 110 represent any number and type of processing units (e.g., central processing unit (CPU), GPU, digital signal processor (DSP), FPGA, ASIC). In one embodiment, some of the processing associated with the reasoning engine 105 is performed by the one or more processors 110. The one or more memory devices 130 represent any number and type of memory devices. For example, the type of memory in the one or more memory devices 130 may include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc. The one or more memory devices 130 can be accessed by the reasoning engine 105 and the one or more processors 110. The I / O interface 120 represents any number and type of I / O interface (e.g., peripheral component interconnect (PCI) bus, PCI extension (PCI-X), PCIE (PCI Express) bus, Gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices can be connected to the I / O interface 120. Such peripheral devices include, but are not limited to, displays, keyboards, mice, printers, scanners, joysticks or other types of game controllers, media recording devices, external storage devices, network interface cards, and the like.

[0026] In various embodiments, computing system 100 may be any of a computer, a laptop, a mobile device, a game console, a server, a streaming device, a wearable device, or various other types of computing systems or devices. It should be noted that the number of components of computing system 100 may vary from embodiment to embodiment. Figure 1 Each component / subcomponent may be present in greater or lesser quantities than those shown. It should also be noted that the computing system 100 may include Figure 1 Other components not shown. In addition, in other embodiments, the computing system 100 may be connected to Figure 1 The different ways of structuring are shown.

[0027] Now go to Figure 2, a block diagram of another embodiment of a computing system 200 is shown. In one embodiment, the system 200 includes a GPU 205, a host CPU 210, a system memory 225, and a local memory 230. The system 200 also includes other components, which are not shown to avoid obscuring the drawings. The application is executed on the host CPU 210 and interfaces with the driver 212 to send program commands and data to the GPU 205. The host CPU 210 is connected to the memory controller 220 and the system memory 225. The GPU 205 includes at least a command processor 240, a dispatch unit 250, computing units 255A-255N, a memory controller 220, a global data sharing 270, a first level (L1) cache 265, and a second level (L2) cache 260. In other embodiments, the GPU 205 may include other components, omit one or more of the illustrated components, and / or be organized in other suitable ways.

[0028] Certain types of machine learning and neural network applications have high bandwidth requirements, and if the external memory bandwidth can be reduced, performance enhancements and reduced processing time can be achieved. In some cases, external memory bandwidth can become a bottleneck when implementing a neural network. Therefore, any reduction in external memory bandwidth can result in a significant speedup in executing a neural network on the system 200.

[0029] The global data share 270, L1 cache 265, and L2 cache 260 of GPU 205 represent "internal memory" as the term is used herein. Internal memory is distinguished from external memory to GPU 205, where system memory 225 and local memory 230 represent "external memory" as the term is used herein. When executing various neural network applications, GPU 205 is configured to minimize the external memory bandwidth for reading and writing data to system memory 225 and local memory 230.

[0030] In various embodiments, computing system 200 is configured to implement any of various types of machine learning algorithms. For example, in one embodiment, computing system 200 is configured to implement a neural network (e.g., a perceptron, a convolutional neural network). Applications executing on host CPU 210 utilize driver 212 to send data and commands to GPU 205. Command processor 240 is configured to receive commands from host CPU 210 and utilize dispatch unit 250 to issue commands to computing units 255A-255N. Computing units 255A-255N are configured to read and write data to global data share 270, L1 cache 265, and L2 cache 260 within GPU 205 using internal memory bandwidth. Although Figure 2Although not shown in the drawings, the compute units 255A-255N may also include one or more caches and / or local memories within each compute unit 255A-255N. These caches and / or local memories may also be included in the representation of “internal memory” of the GPU 205. The compute units 255A-255N are also configured to read and write data to the system memory 225 and / or the local memory 230 via the memory controller 220 using external memory bandwidth.

[0031] The data partitioning unit 245 may be implemented using any suitable combination of hardware and / or software. In one embodiment, the command processor 240 includes a channel data partitioning unit 245 to determine how to partition channel data for one or more layers of the neural network being implemented. In another embodiment, the channel data partitioning unit 245 may be included as part of the driver 212. In another embodiment, the channel data partitioning unit 245 may be split between the driver 212, the command processor 240, and / or one or more other locations. In one embodiment, the channel data partitioning unit 245 determines how to partition the channel data for each neural network layer so as to minimize external memory bandwidth utilization for a given layer of the neural network. In some embodiments, one or more layers of the neural network are merged. In these embodiments, the partitioning unit 245 determines how to partition the channel data for the merged layers so as to minimize external memory bandwidth utilization for the merged layers. The remainder of this disclosure describes in more detail techniques for partitioning channel data for layers of a neural network so as to minimize external memory bandwidth utilization.

[0032] Now go to Figure 3 , a block diagram of one embodiment of an implementation of a neural network 300 is shown. The neural network 300 includes a convolutional layer 302, a downsampling layer 304, a convolutional layer 306, a downsampling layer 308, and a fully connected layer 310. In other embodiments, the neural network 300 may include other numbers and arrangements of layers.

[0033] When in a computing system (e.g., Figure 1When neural network 300 is implemented on a computing system 100), memory bandwidth requirements may be substantial. One or more techniques may be utilized when implementing neural network 300 on a computing system to reduce the amount of external memory bandwidth consumed. For example, a first technique involves reading an input channel once instead of multiple times, and then performing convolutions for all features for the input channel. For the first technique, input channel read memory bandwidth is reduced by a number of channels. A second technique involves generating convolution output data for one or more features for multiple channels, and then combining the convolution output data across multiple channels before writing the convolution output data back to external memory. A third technique involves combining two or more layers of neural network 300 and sequentially executing the two or more layers before writing the output data back to external memory. In various embodiments, other techniques may be utilized and / or may be combined with one or more of these three techniques.

[0034] Now go to Figure 4 , a block diagram of an embodiment of the inference engine 400. According to the embodiment, the inference engine 400 can be implemented in ( Figure 1 ) system 100, ( Figure 2 ) system 200 or another computing system. Inference engine 400 includes multiple channel processing engines 402A-402N. It should be noted that inference engine 400 may also be referred to as an inference accelerator. The number of channel processing engines 402A-402N may vary from implementation to implementation. Channel processing engine 402A is enlarged to illustrate the components of a channel processing engine according to one implementation. Other channel processing engines 402B-402N may include similar component arrangements.

[0035] The implementation of inference engine 400 illustrates a first technique for reducing external memory bandwidth utilization. Figure 4 In the first technique shown, the input channel data for each channel is read once instead of being read multiple times. Then, convolution is performed for all features for the input channel data. For example, in one embodiment, the input channel data 404 is read in for channel 0, and then convolution is performed for all "M" features 410A-410M. The number of "M" features may vary according to the layer and according to the embodiment. The result of the convolution of the input data with each feature is shown as output data blocks 415A-415M, and the output data 415A-415M is written to the external memory represented by blocks 420A-420M. After convolution is performed for all M features for channel 0, convolution is then performed for all M features for channel 1, convolution is performed for all M features for channel 2, and so on. Blocks 402B-402N illustrate the convolution performed for other channels, wherein the resulting output data is written to the external memory. The number of channels "N" may vary according to the layer of the neural network and according to the embodiment.

[0036] Adders 430A-430M illustrate sums calculated for each feature for all output channel data generated from the convolution of all N channels. The sums generated by adders 430A-430M are shown as blocks 440A-440M. These sums can then be written to an external memory after being calculated by adders 430A-430M. After the sums are written to the external memory, one or more additional layers of the neural network can be implemented.

[0037] Reference now Figure 5 , a block diagram of another embodiment of an inference engine 500 is shown. Inference engine 500 illustrates a second technique for reducing external memory bandwidth utilization. As shown in inference engine 500, data from multiple channels is loaded into a processing unit (e.g., Figure 2 The number of channels loaded into the internal memory is limited by the size of the internal memory of the processing unit.

[0038] In one embodiment, the channel binary large object shape representing input data is split into three-dimensional (3D) blocks based on the internal memory size availability. Then, in the convolution calculation blocks 502A-502N, the convolution of "N" channels is performed for each of the channels for the first feature. Before any one of the convolution output data of the N channels of feature 0 is written to the external memory, the convolution output data of N channels and feature 0 are added together by feature 0 adder 520A. In addition, the convolution of N channels for other M-1 features is also performed in parallel with the convolution of N channels and feature 0. Each of the M features has a separate adder 520A-520M, which adds the convolution output data of N channels for the corresponding feature before the convolution output data is written to the external memory. Then, after adders 520A-520M have generated and for different features 0 to M-1, the resulting sum is written to the external memory as illustrated by blocks 530A-530M.

[0039] Now go to Figure 6 , a block diagram showing one embodiment of partitioning channel data into three-dimensional (3D) blocks. The channel data 605 is Figure 6 The left side of is shown as a 3D binary large object of data. In one embodiment, the channel data partition unit (e.g., Figure 2The channel data partitioning unit 245 of the 3D blob determines how to partition the channel data 605 into blocks 610A-610H (and other untagged blocks on the back side of the 3D blob). For the purposes of this discussion, it is assumed that the overall channel data 605 is partitioned into a total of 16 blocks, where each block has a size of NxN in the XY direction and a size of M / 2 in the Z (or channel) direction. It should be understood that this partitioning is indicative of only one specific embodiment. In other embodiments, the overall channel data 605 may be partitioned into other numbers and sizes of blocks.

[0040] In one embodiment, blocks 610A-610H of channel data 605 are loaded from external memory into internal memory of a processing unit (e.g., GPU 205) one block at a time. For example, block 610A is loaded from external memory into internal memory, then a convolution operation is performed on block 610A for one or more given layers of a neural network and optionally one or more operations are performed, and then the output data is written back to the external memory. Block 610B is then loaded and the same one or more operations are performed, followed by writing the output data back to the external memory, and then block 610C is processed, and so on. In another embodiment, the pattern of processing 3D blocks can vary, where block 610A is processed first, followed by block 610E, then block 610B, block 610F, and so on.

[0041] Block 610A Figure 6 The right side of is enlarged to illustrate the manner in which convolution is performed in one embodiment after block 610A is loaded from external memory into the internal memory of the processing unit. In one embodiment, a first convolution step (step 1) is performed for the leftmost N / 4 pixels of channels 0-3. Then, a second convolution step is performed for the leftmost N / 4 pixels of channels 4-7, and then a third convolution step is performed for the leftmost N / 4 pixels of channels 8-11, and so on, until step M / 8 is performed for the convolution of the leftmost N / 4 pixels of channels M / 2-4 to M / 2–1. After step M / 8, the next step M / 8+1 may return to the front of block 610A, starting from the second leftmost column and proceeding toward the back of block 610A. This pattern of processing the convolution of block 610A with all features may continue until the rightmost rear column of block 610A is reached. It should be noted that the convolution of different parts of block 610A is performed without utilizing any external memory bandwidth. This helps to reduce the overall external memory bandwidth utilization for implementing one or more layers of the neural network.

[0042] Reference now Figure 7, a block diagram of another embodiment of an inference engine 700 is shown. Inference engine 700 illustrates a third technique for reducing external memory bandwidth utilization. The third technique involves implementing multiple layers on a set of channel data after reading the set of channel data from the external memory into the internal memory and before writing the output data to the external memory. A traditional method for implementing layers of a neural network is to perform a single layer of processing on the input data and then write the processed data back to the external memory after the single layer has been completed. The processed data is then read back from the external memory to implement the next layer of the neural network. This approach utilizes a large amount of external memory bandwidth.

[0043] To implement multiple layers of a neural network, a set of channel data 705 is loaded from external memory into the internal memory of a processing unit implementing the inference engine 700. In one embodiment, a convolutional layer 710 is implemented as the first layer after loading blocks of channel data from external memory.

[0044] Instead of writing the output of convolutional layer 710 back to external memory, the output of convolutional layer 710 is processed by the next layer of the neural network. Figure 7 As shown, the next layer of the neural network is a pooling layer 715. In other embodiments, other layers may follow the convolutional layer 710. In one embodiment, the pooling layer 715 is followed by a normalization layer 720, and then the result data 725 is written back to the external memory. It should be understood that the example of combining the convolutional layer with the pooling layer and the normalization layer in the inference engine 700 is only illustrative of one embodiment. In other embodiments, other numbers and / or types of layers may be combined and implemented on a set of channel data 705 in a sequential operation before the output result data 725 is written back to the external memory.

[0045] Now go to Figure 8 , a block diagram showing one embodiment of using vertical sequential processing memory blocks to reduce external memory bandwidth utilization in a neural network implementation. Figure 8 The left side of FIG. 8 shows a set of channel data 800. The set of channel data 800 is partitioned into a plurality of 3D blocks 805A-805P. In one embodiment, the channel data partition unit (e.g., Figure 2 The channel data partitioning unit 245) partitions the channel data 800 into a plurality of 3D blocks 805A-805P.

[0046] After partitioning, in one embodiment, an inference engine (e.g., Figure 7805A. The set of channel data 800 is then moved downward to process 3D block 805B, then 3D block 805C, then 3D block 805D, then 3D block 805E, and so on. Each vertical column is processed from top to bottom, and then the next column to the right is processed from top to bottom until all columns of the set of channel data 800 have been processed.

[0047] exist Figure 8 On the right side of the diagram, source pixels 810 at the boundary between 3D blocks 805A and 805E are shown. To generate convolution output pixels 815 at the boundary, source pixels 810 from both 3D block 805A and 3D block 805E are utilized. In addition, pooled output pixels 820 of the pooling layer are shown on the bottom row of the illustration, and pooled output pixels 820 near the boundary use source pixels 810 from both 3D block 805A and 3D block 805E. In addition, pixels at the top and bottom of each 3D block 805A-805P require pixels from neighboring blocks. These fetches of additional rows from neighboring blocks result in additional fetches of data from external memory. Therefore, an efficient technique for processing 3D blocks 805A-805P is to use Figure 8 The processing mode is shown on the left.

[0048] Reference now Fig. 9 , a block diagram of one embodiment for implementing block sizes for different layers of a neural network. Fig. 9 The purpose of the discussion is Figure 8 The discussion continues. Fig. 9 Blocks 805A, 805B, and 805E in are intended to represent Figure 8 800 of the same block 805A, 805B and 805E of the set of channel data 800. To implement the convolutional layer, the processing of the convolutional layer is performed using the extra pixels from the adjacent blocks. These extra pixels are shown in block 910, which includes the extra pixels on each side of block 805A. In addition, to implement the pooling layer, the processing of the pooling layer is performed using the extra pixels from the adjacent blocks. These extra pixels are shown in block 920, which includes the additional extra pixels on each side of block 805A (compared to block 910).

[0049] For the purpose of this discussion, it will be assumed that in one embodiment, each piece 805A, 805B and 805E comprises 15 pixel lines. In other embodiments, each piece 805A, 805B and 805E may comprise the pixel lines of other quantities. In the described embodiment, when pieces 805A, 805B and 805E comprise 15 pixel lines separately, when processing piece 805A, 15 pixel lines will be extracted from external memory to add two extra pixel lines to realize convolution and pooling layer. In other embodiments, the extra lines of other quantities except two may be needed to realize convolution layer and pooling layer (and any other layer being realized). Then, when moving to piece 805B in the vertical direction, the bottom four pixel lines just remain in the internal memory from the previous processing operation of piece 805A. Extract the next 15 pixel lines, then process these 15 pixel lines and four lines retained together for realizing convolution layer and pooling layer on piece 805B. Since the bottom four pixel lines are retained from the start of processing of block 805A, re-extraction of these boundary lines is avoided. This helps to reduce external memory bandwidth utilization when implementing convolutional and pooling layers as processing moves between adjacent blocks in the vertical direction.

[0050] Now go to Fig.10 , shows one embodiment of a method 1000 for implementing a convolutional layer. For discussion purposes, the steps and Fig.11 However, it should be noted that in various embodiments of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as needed. Any of the various systems or devices described herein is configured to implement method 1000.

[0051] The system detects a request to perform a convolution operation on input data from a plurality of channels (block 1005). For example, in one embodiment, the request may specify that a convolution layer of a neural network should be implemented to process input data from a plurality of channels. In one embodiment, the system includes at least a processing unit (e.g., a GPU) coupled to an external memory. The processing unit includes internal memory and various computational units, execution units, multiply-accumulate units (MACs), and / or other logic.

[0052] In response to detecting the request, the system partitions the input data of the multiple channels into a plurality of three-dimensional (3D) blocks based on one or more factors (block 1010). Two of the three dimensions (of the 3D blocks) correspond to the x, y spatial dimensions of the original input (e.g., image, video frame), and the third (or z) dimension corresponds to the multiple channels. In one embodiment, the system partitions the input data into 3D blocks in order to minimize external memory bandwidth utilization for a given layer of the neural network. In another embodiment, the system partitions the input data into 3D blocks based on the size of the internal memory, the size of one or more features, and the size of the convolution. In other embodiments, the system may partition the input data into 3D blocks based on one or more other factors and / or to achieve one or more goals.

[0053] Next, the system selects a first 3D block from the partitioned 3D blocks for processing (block 1015). Then, the system loads the selected 3D block from the external memory into the internal memory of the processing unit (block 1020). Next, the system generates convolution output data for one or more features for the selected 3D block (block 1025). Then, for each of the one or more features, the system adds the convolution output data together across a first plurality of channels of the 3D block before writing the convolution output data to the external memory (block 1030). Next, the system writes each sum of the convolution output data generated across the first plurality of channels for the corresponding feature to the external memory (block 1035). Then, the system determines whether there are any other 3D blocks left to process (conditional block 1040). If there are 3D blocks left to process (conditional block 1040, "yes" branch), the system selects the next 3D block to be processed (block 1045), and then method 1000 returns to block 1020. It should be noted that the system may follow a specific pattern for processing 3D blocks in order to minimize external memory bandwidth utilization. An example of a pattern is Figure 8 If all 3D blocks have been processed (conditional block 1040, "no" branch), method 1000 ends.

[0054] Reference now Fig.11 , shows one embodiment of a method 1100 for selecting an external memory bandwidth utilization technique to be implemented for an inference engine. A request to implement one or more layers of a neural network is detected (block 1105). In one embodiment, the processing unit is a GPU. In other embodiments, the processing unit can be other types of processing units. In response to detecting the request, the processing unit determines the number of channels, the amount of channel data, and the type of processing associated with the one or more layers (block 1110). In block 1110, the processing unit may also determine other factors related to implementing the one or more layers of the neural network (e.g., the number of features, the size of the features, the stride).

[0055] Next, based on the factors determined in block 1110, the processing unit determines which one or more of the plurality of external memory bandwidth reduction techniques to utilize (block 1115). For example, the processing unit may calculate the amount of external memory bandwidth that will be utilized for each technique. The processing unit may also calculate the amount of external memory bandwidth that will be utilized if two or more of the techniques are combined. The processing unit may then select one or more techniques that minimize the external memory bandwidth utilization. The processing unit then partitions the channel data according to the one or more selected techniques and based on the amount of internal memory available to the processing unit (block 1120). Next, the processing unit implements one or more layers of a neural network for each portion of the partitioned channel data (block 1125). Then, for each portion, the processing unit writes the processed channel data to external memory (block 1130). After block 1130, the method 1100 ends.

[0056] Now go to Fig.12 , a block diagram of one implementation of a 3D block during a convolution operation is shown. Fig.12 The input channel blob 1205 is shown on the left side of FIG. It should be understood that the dimensions of the channel blob 1205 and blocks 1210A-1210H are indicative of one particular embodiment. In other embodiments, the dimensions of the channel blob 1205 and blocks 1210A-1210H may be different from FIG. Fig.12 1205. The dimensions shown are different. In one embodiment, the 3D block dimensions of blocks 1210A-1210H are selected based on the internal memory / cache size, the input 3D blob shape (channel x blob width x blob height), and the convolution kernel size. By reshaping the 3D blocks in the Z direction, the partial accumulator storage requirements and external memory bandwidth usage are reduced. In one embodiment, dynamic resizing of the 3D block size can be implemented in firmware and / or software before starting the convolution processing of the input channel blob 1205.

[0057] When block 1210A is processed, the processed block is retrieved from input channel storage area 1235 in external memory 1220 and then stored in internal memory 1215. The size of internal memory 1215 is limited, and therefore the size of blocks 1210A-1210H can be chosen so that a single block will fit within internal memory 1215.

[0058] The partial accumulator storage area 1230 in the external memory 1220 is used to store the partial convolution convolution values ​​on the first set of channels 0-127 for all features. These partial values ​​are written to the external memory 1220 in the XY direction for all 3D blocks. The convolution process then moves in the Z direction to process the back-side 3D blocks (3D blocks for channels 128-255). During the processing of the back-side 3D blocks, the partial convolution convolution values ​​1230 of the previous blocks are read from the external memory 1220, and then these values ​​are combined with the partial convolution convolution values ​​from the back-side blocks to obtain the final convolution output, which is stored in the output feature storage area 1225 of the external memory 1220. This convolution process for generating the final convolution output requires additional read and write memory bandwidth associated with the partial accumulator storage area 1230 when moving from the front side to the back side of the channel blob 1205.

[0059] Reference now Fig.13 , a block diagram showing an embodiment of dynamically reshaping a 3D block during a convolution operation. Fig.13 As shown on the left side of , the input channel blob 1305 is partitioned into a plurality of 3D blocks 1310A-1310S. The size of the 3D blocks 1310A-1310S is selected so that all channels 0-255 are included in a single block. In addition, the size of the 3D block is selected so that the entire 3D block can fit in the internal memory 1315 for all channels from channel 0 to channel 255. It should be understood that the example of the input channel blob 1305 with 256 channels only illustrates one embodiment. In other embodiments, the input channel blob 1305 can have other numbers of channels.

[0060] By reshaping the blocks 1310A-1310S in the block Z direction, all input channels can be stored in the internal memory 1315 to perform convolution on all input channels of the input channel blob 1305. This helps minimize the external memory bandwidth associated with the partial accumulator storage area in the external memory 1320. Fig.12 Compared to the exemplary 3D block size setting shown, only the input channels read from the input channel storage area 1335, and the output feature writes end up using external memory bandwidth. The reduction in external memory bandwidth usage allows for a reduction in system power consumption.

[0061] In various embodiments, the program instructions of the software application are used to implement the methods and / or mechanisms described herein. For example, it is envisioned that the program instructions that can be executed by a general-purpose processor or a special-purpose processor. In various embodiments, such program instructions can be represented by a high-level programming language. In other embodiments, the program instructions can be compiled into a binary form, an intermediate form, or other forms from a high-level programming language. Alternatively, program instructions describing the behavior or design of hardware can be written. Such program instructions can be represented by a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog can be used. In various embodiments, program instructions are stored on any of a variety of non-temporary computer-readable storage media. The storage medium can be accessed by a computing system during use to provide program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute program instructions.

[0062] It should be emphasized that the above embodiments are only non-limiting examples of implementations. Once the above disclosure is fully understood, various changes and modifications will become apparent to those skilled in the art. The appended claims are intended to be interpreted as covering all such changes and modifications.

Claims

1. A system comprising: A processing unit, the processing unit comprising an internal memory and a plurality of computing blocks; as well as an external memory coupled to the processing unit; wherein the processing unit comprises a circuit configured to: detecting a request to perform a convolution operation on input data from a plurality of channels; In response to detecting the request: partitioning the input data from the plurality of channels into a plurality of N-dimensional blocks; loading input data representing different channels of a plurality of channels of a first N-dimensional block of the N-dimensional blocks from the external memory into the internal memory and each computation block; generating, by each computational block for each channel of the first N-dimensional block, convolution output data for each of the one or more features; as well as Before generating convolution output data for a second N-dimensional block among the multiple N-dimensional blocks, the convolution output data generated by each computing block for the first N-dimensional block are added together to form corresponding convolution output data, and the corresponding convolution output data are written to the external memory.

2. The system of claim 1, wherein N is three.

3. The system of claim 1, wherein after processing the first N-dimensional block, the system is configured to process subsequent N-dimensional blocks from the input data in a vertical direction from top to bottom and then from left to right.

4. The system of claim 3, wherein when moving from top to bottom in a vertical direction, one or more lines of pixels are retained in an internal memory from processing a previous block and used to process a next block, wherein the one or more lines correspond to boundary lines.

5. The system of claim 4, wherein the system is further configured to determine how to partition the input data from the plurality of channels into three-dimensional blocks based on a size of the internal memory, a size of the one or more features, and a size of the convolution operation.

6. The system of claim 1, wherein the convolution operation corresponds to a convolutional layer, wherein the convolutional layer is one of a plurality of layers of a neural network being implemented on the system.

7. A method comprising: Detecting, by a processor (205) including an internal memory, a request to perform a convolution operation on input data from a plurality of channels; In response to detecting the request, the processor: partitioning the input data from the plurality of channels into a plurality of N-dimensional blocks; loading input data representing different channels of a plurality of channels corresponding to a first N-dimensional block of the N-dimensional blocks from an external memory into the internal memory and each of the plurality of computational blocks; generating, by each computational block for each channel of the first N-dimensional block, convolution output data for each of the one or more features (1025); and Before generating convolution output data for a second N-dimensional block among the multiple N-dimensional blocks, a separate adder is used for each feature to add together the convolution output data generated by each computing block for the first N-dimensional block to form corresponding convolution output data, and the corresponding convolution output data is written to the external memory.

8. The method of claim 7, wherein N is three.

9. The method of claim 8, wherein the method further comprises: After processing the first three-dimensional block, subsequent three-dimensional blocks from the input data are processed in a vertical direction from top to bottom and then from left to right.

10. The method of claim 9, wherein when moving from top to bottom in a vertical direction, one or more lines of pixels are retained in an internal memory from processing a previous block and used to process a next block, wherein the one or more lines correspond to boundary lines.

11. The method of claim 10, further comprising partitioning the input data from the plurality of channels into three-dimensional blocks based on a size of the internal memory, a size of the one or more features, and a size of the convolution operation.

12. The method of claim 7, further comprising: The convolution output data is added together across the plurality of channels for each of the plurality of features, wherein the adding of the convolution output data is performed in parallel for each of the plurality of features.

Images