Automatic generation and tuning tool for convolution kernels

Abstract

Systems, apparatuses, and methods for implementing an automated generation and tuning tool for convolution kernels are disclosed. A processor performs a plurality of tuning runs of a given layer of a neural network while using a different set of operational parameter values for each tuning run. The operational parameters can include one or more of an input dataset fetch group size, an output channel group size, and other parameters. The processor captures performance data for each tuning run, and then, after all of the tuning runs are complete, the processor determines which set of operational parameter values resulted in better performance for the given neural network layer. The processor uses these operational parameter values for subsequent iterations of the given layer. The processor also performs the same technique for other layers to determine which set of operational parameter values to use for each layer in order to maximize performance of the neural network.

Description

BACKGROUND

[0001] A graphics processing unit (GPU) is a complex integrated circuit that performs graphics processing tasks. For example, a GPU performs graphics processing tasks required by end-user applications such as video game applications. GPUs are also increasingly being used to perform other tasks that are not image related. In some implementations, a GPU is a discrete device or is included in the same device as another processor such as a central processing unit (CPU).

[0002] In many applications, such as graphics processing applications performed by a GPU, a sequence of processing work items (which can also be referred to as threads) are processed in order to output a final result. In one implementation, each processing element executes a respective instantiation of a particular work item to process input data. A work item is one of a set of parallel executions of a kernel invoked on a compute unit. A work item is distinguished from other executions within the set by a global ID and a local ID. As used herein, the term "compute unit" is defined as a collection of processing elements (e.g., single instruction multiple data (SIMD) units) that perform a synchronized execution of multiple work items. The number of processing elements per compute unit can vary from implementation to implementation. A subset of work items in a work group that are executed together on a compute unit can be referred to as a wavefront, warp, or vector. The width of a wavefront is a property of the hardware of the compute unit. As used herein, a set of wavefronts is referred to as a "work group." Also, a "wavefront" can also be referred to as a "wave" herein.

[0003] An emerging field of technology is machine learning, where a neural network is one type of machine learning model. Implementing a neural network on a GPU or other parallel processing unit (e.g., digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC)) involves loading and processing large amounts of data. A convolutional neural network typically includes many layers, such as convolution layers, pooling layers, normalization layers, and so on. A convolution layer filters an input data set (e.g., an image) using a convolution kernel. The kernel is typically a small matrix, and the kernel can also be referred to as a convolution matrix. The kernel is used for blurring, sharpening, embossing, edge detection, and so on. The convolution layer performs a convolution between the kernel and the image.

[0004] Neural networks are used in a wide variety of applications (e.g., handwritten digit classification, face detection). Neural networks have demonstrated superior performance on tasks such as handwritten digit classification and face detection. In addition, neural networks also show promise in performing well on other more challenging visual classification tasks. Other applications of neural networks include speech recognition, language models, sentiment analysis, text prediction, and other applications. However, neural networks often use large amounts of processing and memory resources. BRIEF DESCRIPTION OF DRAWINGS

[0005] The advantages of the methods and mechanisms described herein can be better understood by referring to the following description in conjunction with the accompanying drawings in which:

[0006] Figure 1 is a block diagram of one implementation of a computing system.

[0007] Figure 2 is a block diagram of another implementation of a computing system.

[0008] Figure 3 is a block diagram of one implementation of a neural network.

[0009] Figure 4 is a block diagram of another implementation of a neural network.

[0010] Figure 5 is a block diagram of one implementation of an auto-generation and tuning tool for convolutional kernels.

[0011] Figure 6 is a block diagram of one implementation of a data path of an auto-generation and tuning tool for convolutional kernels.

[0012] Figure 7 is a block diagram of one implementation of a computing unit generating an output dataset.

[0013] Figure 8 is a generalized flow diagram illustrating one implementation of a method for implementing an auto-generation and tuning tool for convolutional kernels.

[0014] Figure 9 is a generalized flow diagram illustrating one implementation of a method for automatically tuning kernels for optimizing performance when implementing a neural network. DETAILED DESCRIPTION

[0015] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, one of ordinary skill in the art will recognize that the various implementations can be practiced without these specific details. In some instances, descriptions of well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail in order to avoid obscuring the approach presented herein. It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some of the elements can be exaggerated relative to other elements.

[0016] Disclosed herein are various systems, devices, and methods for automatically generating convolution kernels with a tuning tool. One system includes at least a processor and a memory storing input datasets and weights for various layers of a neural network. The system determines which operation parameters to use for each layer of the neural network when executing the neural network to maximize performance of the system. In one implementation, the system tries multiple sets of operation parameters for each layer during multiple tuning runs. Each tuning run uses a different set of operation parameters. The system captures performance data during each tuning run, and after performing multiple tuning runs for a given layer of the neural network, the system determines which tuning run resulted in better (i.e., improved) performance of the system compared to the performance of the other tuning runs. In some implementations, the best-performing tuning run of the tuning runs is determined. The system identifies and stores the set of operation parameters from the tuning run that resulted in better performance of the system. Subsequently, the system uses this set of operation parameters for subsequent iterations of the given layer. Also, the system repeats this technique for each layer of the neural network in order to find better operation parameters for all layers of the entire neural network. Thus, during actual implementation of the neural network, the system can change the operation parameters when moving from one layer to the next.

[0017] Referring now to Figure 1 , a block diagram showing one implementation of a computing system 100 is shown. In one implementation, the computing system 100 includes at least processors 105A-105N, an input / output (I / O) interface 120, a bus 125, a memory controller(s) 130, a network interface 135, a memory device(s) 140, a display controller 150, and a display 155. In other implementations, the computing system 100 includes other components, omits one or more of the illustrated components, and / or is arranged differently.

[0018] In one implementation, the processor 105A is a general-purpose processor, such as a central processing unit (CPU). In one implementation, the processor 105N is a data parallel processor with a highly parallel architecture. Data parallel processors include graphics processing units (GPUs), digital signal processors (DSPs), field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), and the like. In some implementations, the processors 105A-105N include multiple data parallel processors. In one implementation, the processor 105N is a GPU that provides pixels to the display controller 150 for driving to the display 155.

[0019] The memory controller(s) 130 is representative of any number and type of memory controllers that are accessible by the processors 105A through 105N. The memory controller(s) 130 is coupled to any number and type of memory device(s) 140. The memory device(s) 140 is representative of any number and type of memory devices. For example, the memory types in the memory device(s) 140 include dynamic random access memory (DRAM), static random access memory (SRAM), NAND-type flash memory, NOR-type flash memory, ferroelectric random access memory (FeRAM), or other memory.

[0020] The I / O interface 120 is representative of any number and type of I / O interfaces (e.g., peripheral component interconnect (PCI) bus, PCI-extended (PCI-X), PCIE (PCI Express) bus, gigabit Ethernet (GBE) bus, universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to the I / O interface 120. Such peripheral devices include, but are not limited to, a display, a keyboard, a mouse, a printer, a scanner, a joystick or other type of game controller, a media recording device, an external storage device, a network interface card, and so forth. The network interface 135 is used to receive and send network messages over a network.

[0021] In various implementations, the computing system 100 is a computer, a laptop computer, a mobile device, a game console, a server, a streaming device, a wearable device, or any of various other types of computing systems or devices. It should be noted that the number of components of the computing system 100 varies from implementation to implementation. For example, in other implementations, there are more or fewer of each component than shown. It should also be noted that in other implementations, the computing system 100 includes other components not shown in FIG. 1. Additionally, in other implementations, the computing system 100 is structured in other ways than shown in FIG. 1. Figure 1 Figure 1 Figure 1

[0022] Turning now to Figure 2 , a block diagram of another implementation of a computing system 200 is shown. In one implementation, the system 200 includes a GPU 205, a system memory 225, and a local memory 230. The system 200 also includes other components not shown to avoid obscuring the drawing. The GPU 205 includes at least a command processor 235, control logic 240, a dispatch unit 250, compute units 255A through 255N, a memory controller 220, a global data share 270, a level one (LI) cache 265, and a level two (L2) cache 260. In other implementations, the GPU 205 includes other components, omits one or more of the components shown, although​​​Figure 2 only one instance is shown but multiple instances of the component are included, and / or organized in other suitable manners. In one implementation, the circuitry of GPU 205 is included in processor 105N Figure 1 ).

[0023] In various implementations, computing system 200 executes any of various types of software applications. As part of executing a given software application, the host CPU (not shown) of computing system 200 initiates a kernel to be executed on GPU 205. Command processor 235 receives the content from the host CPU and issues corresponding wavefronts to compute units 255A-255N using dispatch unit 250. In one implementation, each compute unit 255A-255N includes a matrix operation unit. For example, in one implementation, the matrix operation unit performs a matrix multiplication operation to implement a convolutional layer of a neural network. Additionally, in various implementations, the matrix operation unit performs other types of matrix operations. The wavefronts executed on compute units 255A-255N read data and write data to global data share 270, LI cache 265, and L2 cache 260 within GPU 205. Although not shown in FIG. 2, in one implementation, compute units 255A-255N also include one or more caches and / or local memories within each compute unit 255A-255N. Figure 2

[0024] In one implementation, computing system 200 implements a tuning run of one or more layers of the neural network on compute units 255A-255N. During each tuning run, for each layer, a different set of operation parameters is used while capturing performance parameters. For example, in one implementation, for a convolutional layer, the operation parameters include the amount of data fetched from global memory and / or cache at each fetch and the number of output channels computed in one loop. In a pooling layer, other operation parameters (e.g., workgroup size) are included in the set of operation parameters adjusted for each tuning run.

[0025] ​For example, for a convolutional layer having 64 output channels, in a first tuning run, a program is generated in which 2 output channels are computed per loop (for a total of 32 loops). In a second tuning run, a program is generated in which 4 output channels are computed in one loop, and the program will loop 16 times. Any number of other programs are executed in which other numbers of output channels are computed per loop. Also, the number of input dataset elements fetched per fetch can vary in each program generated in each tuning run. For example, 256 dataset elements are fetched per fetch in a first program, 512 input dataset elements are fetched per fetch in a second program, and so on. Additionally, in one implementation, the number of filter weights fetched per fetch also varies per program. Other parameters can also vary in different programs, and the number of programs executed can be large enough to try a variety of different combinations of parameter values.

[0026] Performance data is captured for each program, and based on the captured performance data, a program with better performance is determined. In one implementation, the execution time of each program is measured, and the program with the shortest execution time is considered to have better performance. In other implementations, other performance data is captured. For example, power consumption, memory usage, memory bandwidth usage, register usage, and / or other performance metrics are captured. Subsequently, after the preferred operation parameters for the first layer are determined, the process is repeated for the second layer. For the second layer, the operation parameters can be different depending on the type of layer (e.g., pooling, normalization, convolution). Eventually, all layers of the neural network are processed in this manner.

[0027] Turning now to Figure 3 , a block diagram of one implementation of a neural network 300 is shown. The neural network 300 includes a convolutional layer 302, a subsampling layer 304, a convolutional layer 306, a subsampling layer 308, and a fully connected layer 310. In other embodiments, the neural network 300 can include other numbers and arrangements of layers. When the neural network 300 is implemented on a computing system (e.g., the system 100 of Figure 1 , the system 200 of Figure 1 , the performance of the system can vary widely depending on the particular program parameters selected for each layer. Accordingly, in one implementation, the system executes multiple programs (i.e., tuning runs) to determine the preferred operation parameters for each layer of the neural network 300 in order to optimize performance. Subsequently, during subsequent iterations of the neural network 300, the system uses the preferred parameters to optimize the performance of each layer.

[0028] Turning now to Figure 4 , a block diagram of another embodiment of a neural network 400 is shown. The neural network 400 illustrates that the neural network 300 can be implemented on a computing system (e.g., the system 100 of Figure 1 , the system 200 of Figure 2 , the system 300 ofFigure 2 Another example of a neural network implemented on a system 200) is shown. The neural network 400 includes at least a convolutional layer 410, a pooling layer 415, a normalization layer 420, a convolutional layer 430, a pooling layer 435, a normalization layer 440, and any number of other layers. The neural network 400 processes a channel data set 405 to generate a result data 445. In one implementation, the channel data set 405 is an image. In other implementations, the channel data set 405 is any of a variety of other types of input data sets.

[0029] During execution of the neural network 400, a number of tuning runs are performed to determine improved operational parameters for each layer in order to have optimized performance. Subsequently, during subsequent operations of the neural network 400, these improved operational parameters are used to implement each layer. In one implementation, performance is optimized when execution time is minimized. In another implementation, performance is optimized when power consumption is minimized. In other implementations, other ways of defining what constitutes improved performance can be utilized. For example, improved performance can not be when any one parameter is minimized or maximized. Rather, the improved performance can be determined based on a combination of factors, where some factors are not at a minimum or maximum. For example, improved performance can mean reducing power consumption as much as possible while maintaining a given level of performance. Or, improved performance can be based at least in part on a given level of memory consumption or bandwidth. These and other embodiments are possible and contemplated.

[0030] Referring now to Figure 5 , a block diagram of one implementation of an auto-generation and tuning tool for convolutional kernels is shown. On the left side of Figure 5 , an input data set 505 is represented by a three-dimensional (3D) arrangement of cubes. For the input data set 505, dimensions C, H, W, and N are shown according to one implementation. In one implementation, the input data set 505 is an image. The dimension "C" is data for input channels, "W" is the width of the image, "H" is the height of the image, and "N" is the number of batches of input data. In one implementation, C equals 192, W equals 28, H equals 28, and N equals 16. It should be understood that in other implementations, one or more of these dimensions can equal other values.

[0031] In one implementation, the processor changes one or more parameters used to perform a convolution operation between the input data set 505 and the weights 510 to generate the output data set 515. In one implementation, the processor changes the number of input data elements taken from the input data set 505 in a single group and filtered through the weights 510. For example, each group (0), group (1), and so on, is processed by the processor in a sequential manner in order to generate the output data set 515. The number of input data elements taken from the input data set 505 in a single group will determine how many loops are needed to process the entire input data set 505. The processor collects performance data for each of a total number of programs executed, where each program uses a different set of parameters. After performing a total number of iterations and collecting the performance data, the processor determines which parameters result in better performance for the current layer of the neural network. The processor then uses these parameters when performing this layer of the neural network on other input data sets. The processor repeats this process for other layers of the neural network. Thus, separate from other layers of the multi-layer neural network, the processor determines which parameters result in better performance for each layer of the neural network.

[0032] Turning now to Figure 6FIG. 6 shows a block diagram of one implementation of a data path for an auto-generation and tuning tool for convolution kernels. An input data set 605 is represented as a 3D arrangement of cubes to depict input channel data processed by a given layer of a neural network. In one implementation, the input data set 605 is divided into a plurality of groups that are processed sequentially. In one implementation, the plurality of groups are processed by a plurality of compute units of a parallel processor (e.g., a GPU). In one implementation, a convolution operation is implemented to process each group in a single loop within a compute unit. The convolution operation is performed for each group of input data set 605 and for a corresponding set of filter weights 620. In one implementation, the set of filter weights used to implement the convolution filter operation are loaded from memory into a sequencer cache (SQC) 615 and the input data set is loaded directly to the compute unit through the LI cache 265. In one implementation, the filter weights are pre-fetched into the SQC 615 prior to performing the convolution filter operation so that lookups for the weights will hit in the SQC 615 during execution of the convolution filter code. In one implementation, the SQC 615 is shared between compute units (CUs) 610A and 610B. In other implementations, the SQC 615 is shared between other numbers of compute units. The CUs 610A-B perform the convolution operation between elements of the input data set group 605 and the set of filter elements 620. After performing the convolution operation for a given group, the CUs 610A-B load another input data set group 605 and another set of filter elements and repeat the convolution operation until all of the input data set groups 605 are processed.

[0033] In one implementation, the processor performs a plurality of tuning runs of a convolution layer of a neural network on an input data set 605. Each tuning run uses a different size of group for processing a portion of the input data set 605 in a single loop. As used herein, the term "loop" is defined as a group of instructions for processing a portion of an input data set according to a specific type of operation to generate an output data. In one implementation, the specific type of operation is a convolution filter operation. The processor also optionally varies one or more other operation parameters during each tuning run. During each tuning run, the processor captures various performance metrics. After the performance metrics are captured for different sizes of groups, the processor determines which tuning run results in better performance for the given layer. The group size and any other operation parameters from this tuning run are then utilized when processing subsequent data sets through the given layer of the neural network.

[0034] Reference is now made to Figure 7FIG. 7B shows a block diagram illustrating one implementation of generating an output dataset 710 by the compute units 705A-B. In one implementation, the compute units 705A-B implement a given layer of a neural network by performing an operation on an input dataset (not shown) and filter weights (not shown) one group at a time to generate the output dataset 710. Depending on the implementation, the given layer is a convolutional layer, a pooling layer, a normalization layer, or other type of layer. In one implementation, the compute units 705A-B perform a convolution operation on the input dataset and filter weights one group at a time to generate the output dataset 710. It should be understood that although two compute units 705A-B are shown in FIG. 7B, other numbers of compute units can be used to generate the output dataset 710 in other implementations. Figure 7

[0035] In one implementation, the group size of the input dataset processed by the compute units 705A-B in a single cycle is adjusted during different tuning runs. The compute units 705A-B capture performance data during these tuning runs. The group size that results in better performance will be selected for use in subsequent iterations of the particular layer of the neural network. It should be understood that the size of the output dataset 710, as well as the number of pixels and channels per group and wave, can vary depending on the layer, the neural network, and the particular implementation.

[0036] Turning now to Figure 8 FIG. 8 shows one implementation of a method 800 for implementing an auto-generation and tuning tool for convolutional kernels. For purposes of discussion, the steps and those steps of this implementation are shown in sequential order. However, it is to be understood that some of the steps illustrated and certain elements of the described methods can occur or be performed concurrently, in a different order than illustrated, or be omitted entirely. Other additional steps as desired can also be executed. Any of the various systems or devices described herein are configured to implement the method 800. Figure 9

[0037] The processor (e.g., GPU) receives a request to tune a neural network having multiple layers (block 805). In response to receiving the request, the processor selects a first layer of the neural network (block 810). Subsequently, the processor selects a first set of values for operational parameters of the selected layer of the neural network (block 815). For example, in one implementation, the first set of values specifies an amount of data fetched from a global memory and / or cache in each fetch and a number of output channels computed in one cycle. In other implementations, the first set of values specifies values for other operational parameters (e.g., convolution kernel size) that dictate how the processor performs the selected layer of the neural network.

[0038] ​​Next, the processor captures one or more performance metrics while executing the selected layer of the neural network using the selected set of values for the operational parameters (block 820). Depending on the implementation, the one or more performance metrics include execution time for the selected layer, power consumed during execution of the selected layer, memory usage, memory bandwidth usage, register usage, and / or other metrics. In some implementations, the performance metrics utilized are user configurable, selected by a software application, or programmable by other sources or can be programmable based on various detected conditions.

[0039] Next, the processor determines whether other sets of values for the operational parameters should be tested for the selected layer of the neural network (conditional block 825). For example, in one implementation, the processor tries N different sets of operational parameters for N different executions of the selected layer, where N is a positive integer greater than one. For example, in one implementation, N equals 100. In other implementations, N equals other values.

[0040] If the processor determines that other sets of operational parameters should be tested for the selected layer of the neural network (conditional block 825, "yes" branch), the processor selects a different set of values for the operational parameters for the selected layer (block 830). After block 830, the method 800 returns to block 820. If the processor determines that a sufficient number of sets of values for the operational parameters have been tested for the selected layer of the neural network (conditional block 825, "no" branch), the processor compares the captured performance metrics from the tests using the different sets of values for the operational parameters to find a better performance metric for the selected layer (block 835). In one implementation, the processor generates a performance score from the multiple captured performance metrics that are captured and then compares the performance scores in block 835 against the different tests to find the highest performance score. For example, in one implementation, a performance score is generated by adding each performance metric multiplied by a different weight together. In this implementation, the performance score equals a first metric multiplied by a first percentage plus a second metric multiplied by a second percentage plus a third metric multiplied by a third percentage, and so on. The values of the metrics can be normalized to some range that is consistent across all metrics. For example, in one implementation, each metric is converted to a value in the range from 0.0 to 1.0. It should be understood that other techniques for calculating a performance score are possible and contemplated. Next, the memory stores the set of values for the operational parameters that correspond to the better performance metric for the selected layer (block 840).

[0041] After block 840, the processor determines whether more layers of the neural network are to be tuned (conditional block 845). If more layers of the neural network are to be tuned (yes leg of conditional block 845), the processor selects the next layer of the neural network (block 850), and then the method 800 returns to block 815. If no more layers of the neural network are to be tuned (no leg of conditional block 845), the processor uses the stored set of operational parameter values for the plurality of layers for subsequent iterations of the neural network (block 855). After block 855, the method 800 ends.

[0042] Referring now to Figure 9 , one implementation of a method 900 for automatically tuning kernels for optimizing performance when implementing a neural network is shown. The processor selects a set of operational parameter values for each layer of the neural network, with the goal of optimizing the performance of each layer (block 905). In one implementation, the processor implements an auto-generation and tuning tool for generating kernels, as described in method 800 Figure 8 ) to determine which set of operational parameter values to use for each layer of the neural network. It should be noted that the definition of performance can vary from implementation to implementation. For example, in one implementation, performance is optimized when execution time is minimized. In another implementation, performance is optimized when power consumption is minimized. In other implementations, other ways of defining performance that constitute an improvement can be utilized.

[0043] Next, the processor loads an input dataset (e.g., an image) to be processed by the neural network (block 910). It should be noted that in some implementations, the processor loads the input dataset in portions one at a time, where the size of the portions is specified by one of the operational parameter values. Subsequently, the processor implements each layer of the neural network with the selected set of operational parameter values to process the input dataset (block 915). Subsequently, the processor generates a result from the implementation of the neural network (block 920). In one implementation, the result is data identifying a classification of the input dataset into a given class from a plurality of classes ("classification data"). For example, in one implementation, the input dataset is an image, and the classification identifies a given class to which the image belongs. The data identifying the classification is then stored for further processing. In some embodiments, the classification data is used to alter or otherwise affect the display data. For example, in some embodiments, the classification data is used to identify a person or object appearing in a video frame, and then a certain type of indication of this classification is presented on a video display. In other embodiments, the classification data is used for further processing to determine how to process the identified person or object for further processing. For example, if the classification data indicates that the object is a car, then further processing can use this data to predict the location of the car in later video frames. These and other embodiments are possible and contemplated. In another implementation, the input dataset is a video, and the classification assigns the video to a given class. In other implementations, the input dataset includes other types of data. In other implementations, other types of results are generated. After block 920, the method 900 ends.

[0044] In various implementations, the methods and / or mechanisms described herein are implemented using program instructions for a software application. For example, program instructions executable by a general or special purpose processor are contemplated. In various implementations, such program instructions are represented by a high-level programming language. In other implementations, the program instructions are compiled from a high-level programming language to binary, intermediate, or other form. Alternatively, the program instructions are written in a hardware description language (HDL), such as Verilog. In various implementations, the program instructions are stored on any of a variety of non-transitory computer readable storage media. The storage media is accessed by a computing system during use to provide the program instructions to the computing system for program execution. Generally, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0045] It should be emphasized that the above-described embodiments are merely non-limiting examples of implementations. Once fully understood, those skilled in the art will appreciate that many changes and modifications can be made. The claims are intended to cover all such changes and modifications.

Claims

1. A system comprising: A processor, comprising circuitry configured to: Multiple iterations of a given layer of a neural network comprising multiple layers are performed to process a first dataset, wherein at least two of the multiple iterations of the given layer are performed before another layer of the neural network is performed and each of the at least two iterations uses a different set of computational parameter values; Performance data is captured during each of the at least two iterations, wherein the performance data is based on hardware resource usage metrics measured during execution, and the performance data includes one or more of the following: execution time, power consumption, memory usage, memory bandwidth usage, register usage, or other performance metrics; During the iteration of at least two of the plurality of iterations, one of the different sets of computational parameter values ​​corresponding to the improved performance is stored for use in future executions of the given layer; as well as The given layer of the neural network is executed using one of the different sets of computational parameter values ​​corresponding to the improved performance to process the second dataset.

2. The system of claim 1, wherein the processor is configured to execute the plurality of iterations of the given layer before executing another layer of the neural network.

3. The system as described in claim 1, wherein, The processor is also configured to capture performance data corresponding to each of the plurality of iterations, wherein the performance data includes one or more of the following: execution time, power consumption, memory usage, memory bandwidth usage, and / or register usage.

4. The system of claim 1, wherein the iteration with the lowest execution time among the plurality of iterations has the improved performance.

5. The system of claim 1, wherein the processor is further configured to store one of the different sets of operational parameter values ​​in response to one of the different sets of operational parameter values ​​corresponding to the improved performance.

6. The system of claim 1, wherein the processor is further configured to execute the given layer of the neural network using the different sets of computational parameter values ​​corresponding to the improved performance, to process multiple other datasets.

7. The system of claim 1, wherein the different set of operational parameter values ​​includes one or more of the following: image size, number of channels, batch size, and number of output feature maps.

8. A method comprising: The circuitry of the processor executes multiple iterations of a given layer of a neural network comprising multiple layers to process a first dataset, wherein at least two of the multiple iterations of the given layer are executed before another layer of the neural network is executed and each of the at least two iterations uses a different set of computational parameter values; Performance data is captured during each of the at least two iterations, wherein the performance data is based on hardware resource usage metrics measured during execution, and the performance data includes one or more of the following: execution time, power consumption, memory usage, memory bandwidth usage, register usage, or other performance metrics; During at least two of the plurality of iterations, the processor stores one of a different set of computational parameter values ​​corresponding to the improved performance for use in future executions of the given layer; as well as The given layer of the neural network is executed using one of the different sets of computational parameter values ​​corresponding to the improved performance to process the second dataset.

9. The method of claim 8, further comprising performing the plurality of iterations of the given layer before performing another layer of the neural network.

10. The method of claim 8, further comprising capturing performance data corresponding to each of the plurality of iterations, wherein the performance data includes one or more of the following: execution time, power consumption, memory usage, memory bandwidth usage, and / or register usage.

11. The method of claim 8, wherein the iteration with the lowest execution time among the plurality of iterations has the improved performance.

12. The method of claim 8, further comprising: For each of the multiple layers of the neural network, in response to one of the different sets of operational parameter values, one of the different sets of operational parameter values ​​is stored.

13. The method of claim 8, further comprising executing the given layer of the neural network using the different sets of computational parameter values ​​corresponding to the improved performance to process multiple other datasets.

14. The method of claim 8, wherein the different set of computational parameter values ​​includes one or more of the following: image size, number of channels, batch size, and number of output feature maps.

15. An apparatus comprising: A memory that stores the first dataset and the second dataset; as well as Multiple computing units coupled to the memory, wherein the multiple computing units are configured to: Multiple iterations of a given layer of a neural network comprising multiple layers are performed to process a first dataset, wherein at least two of the multiple iterations of the given layer are performed before another layer of the neural network is performed and each of the at least two iterations uses a different set of computational parameter values; Performance data is captured during each of the at least two iterations, wherein the performance data is based on hardware resource usage metrics measured during execution, and the performance data includes one or more of the following: execution time, power consumption, memory usage, memory bandwidth usage, register usage, or other performance metrics; During at least two of the plurality of iterations, one of the different sets of computational parameter values ​​corresponding to the improved performance is stored for use in future executions of the given layer; as well as The given layer of the neural network is executed using one of the different sets of computational parameter values ​​corresponding to the improved performance to process the second dataset.

16. The device of claim 15, wherein the iteration corresponds to a layer tuning process, and the plurality of computing units are configured to perform the layer tuning process at least on other layers of the neural network.

17. The device of claim 15, wherein the device is further configured to capture performance data corresponding to each of the plurality of iterations, wherein the performance data includes one or more of the following: execution time, power consumption, memory usage, memory bandwidth usage, and / or register usage.

18. The device of claim 15, wherein the computational parameter values ​​include one or more of the following: image size, number of channels, batch size, and number of output feature maps.

19. The device of claim 15, wherein, in response to one of the different sets of computational parameter values ​​corresponding to the improved performance, one of the different sets of computational parameter values ​​is stored.

20. The device of claim 15, wherein the first dataset and the second dataset are images.

Images