Methods and systems for opportunistic load balancing in neural networks using metadata

Abstract

Methods and systems for opportunistic load balancing in deep neural networks (DNNs) using metadata. Representative computation costs are captured, obtained, or determined for a given architecture, function, or computation aspect of a DNN system. The representative computation costs are implemented as metadata for the given architecture, function, or computation aspect of the DNN system. In one implementation, the computed computation costs are implemented as the metadata. A scheduler detects whether neurons are ready for execution in subsequent layers. The scheduler uses the metadata and neuron availability to schedule and load balance across computing resources and available resources.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of U.S. nonprovisional patent application No. 16 / 019,374, filed June 26, 2018, the contents of which are incorporated herein by reference. Background Technology

[0003] Deep neural networks (DNNs) are used in many artificial intelligence and machine learning applications. These DNNs typically include multiple hidden layers between the input and output layers. Recently, DNNs have begun to use increasingly more layers, which provides greater capacity and accuracy for various prediction problems in image, video, and speech recognition processing and analysis. However, deeper DNNs also lead to increasingly greater performance challenges.

[0004] For example, DNNs are computationally extremely expensive. It's not uncommon for training a neural network to take days, weeks, or even months. These DNNs are massively parallel and require high-throughput devices to function efficiently. However, most current DNNs are serialized between layers. Many DNNs have more than 100 such layers, which severely limits their ability to improve parallelism.

[0005] A common technique known as "layer fusion" allows more than one layer to be exposed to concurrent computation, but this requires explicit programming. The programmer exposes the compiler to the entire computation across all layers. However, layer fusion is not strict but opportunistic; currently, no compiler is capable of recognizing that subsequent layers may have sufficient computational demand to guarantee early commencement of computation for them. Furthermore, this approach is difficult. The programmer must expose the compiler to the entire computation across all layers. This is considered extremely challenging to program and unintuitive (i.e., the abstract concept of layers may be lost in the process). Aside from specialized implementations of layer fusion in highly optimized libraries, layers are implicitly serialized.

[0006] Even with optimization, this network cannot be transferred to many other networks with different connectivity, architectures, and hardware capabilities. Furthermore, the optimization process is error-prone, time-consuming, and costly to implement. Additionally, even with the fusion of several layers, only the fused layers are exposed, not the entire computational graph. Finally, this approach cannot take advantage of dynamic edits to the neural network during runtime, such as pruning. This is important because pruning alters the computational intensity of the pruned layers. Attached Figure Description

[0007] A more detailed understanding can be obtained from the following description, given by way of example in conjunction with the accompanying drawings, in which:

[0008] Figure 1 It is a block diagram of an example device based on certain implementation methods;

[0009] Figure 2 It depends on certain implementation methods. Figure 1 A block diagram of the device;

[0010] Figure 3 It is a block diagram of a heterogeneous system architecture (HSA) platform based on certain implementation methods;

[0011] Figure 4 This is a block diagram of an example system based on certain implementation methods, illustrating the queue structure;

[0012] Figure 5A This is an example block diagram for grouping commands based on certain implementation methods;

[0013] Figure 5B Example elements of command groups, including command groups and indirect buffer (IB) command groups, are shown according to certain implementations;

[0014] Figure 5C It is an example indirect buffer based on certain implementation methods;

[0015] Figure 6 Two representative layers of a deep neural network (DNN) are shown;

[0016] Figure 7 This is an example block diagram illustrating how computational costs are determined using computational kernels in a DNN system, based on certain implementation methods; and

[0017] Figure 8 This is a flowchart illustrating a method for opportunistic load balancing in a DNN using metadata, based on certain implementations. Detailed Implementation

[0018] This paper describes a method and system for opportunistic load balancing in deep neural networks (DNNs) using metadata. The parallelism of DNN computation can be fully utilized by exposing the entire computation graph, or at least a portion thereof, to a hardware scheduler, compiler, dispatcher, or operating system (O / S) scheduler (collectively, “scheduler”). In one implementation, metadata is used to label the computational kernels, neurons, layers or other architectures, functions or computational aspects, parts, characteristics, and / or features of the DNN, allowing the scheduler to more effectively and intelligently predict computational complexity and perform load balancing across existing resources. This metadata provides fundamental information about the computational complexity of the computational kernels, thus allowing for accurate load balancing. For example, convolutional neural networks exhibiting repetitive computations with a certain regularity and frequency are particularly well-suited for improved load balancing and job scheduling. However, this method is applicable to other types of networks with regular computational patterns. In one implementation, the method is suitable for dataflow-like architectures where explicitly exposing the entire computation graph allows for full utilization of the inherent parallelism of DNNs.

[0019] Figure 1 This is a block diagram of an example device 100 in which one or more features of this disclosure can be implemented. Device 100 includes, for example, a computer, gaming device, handheld device, set-top box, television, mobile phone, or tablet computer. Device 100 includes a processor 102, memory 104, storage device 106, one or more input devices 108, and one or more output devices 110. Device 100 may also optionally include an input driver 112 and an output driver 114. It should be understood that device 100 includes... Figure 1 Additional components not shown.

[0020] In various alternatives, processor 102 includes a central processing unit (CPU), a graphics processing unit (GPU), a CPU and GPU located on the same die, or one or more processor cores, where each processor core can be a CPU or a GPU. In various alternatives, memory 104 is located on the same die as processor 102, or is located separately from processor 102. Memory 104 includes volatile or non-volatile memory, such as random access memory (RAM), dynamic RAM, or cache.

[0021] Storage device 106 includes fixed or removable storage devices, such as hard disk drives, solid-state drives, optical disks, or flash drives. Input device 108 includes, but is not limited to, a keyboard, keypad, touchscreen, touchpad, detector, microphone, accelerometer, gyroscope, biometric scanner, or network connection (e.g., a wireless LAN card for transmitting and / or receiving wireless IEEE 802 signals). Output device 110 includes, but is not limited to, a display, speaker, printer, haptic feedback device, one or more lights, antenna, or network connection (e.g., a wireless LAN card for transmitting and / or receiving wireless IEEE 802 signals).

[0022] Input driver 112 communicates with processor 102 and input device 108, and allows processor 102 to receive input from input device 108. Output driver 114 communicates with processor 102 and output device 110, and allows processor 102 to send output to output device 110. It should be noted that input driver 112 and output driver 114 are optional components, and device 100 will operate in the same manner if input driver 112 and output driver 114 are not present. Output driver 116 includes an accelerated processing unit (“APD”) 116 coupled to display device 118. APD receives calculation commands and graphics rendering commands from processor 102, processes these calculation and graphics rendering commands, and provides pixel output to display device 118 for display.

[0023] As described in further detail below, APD 116 includes one or more parallel processing units to perform computations according to the Single Instruction Multiple Data (“SIMD”) paradigm. Therefore, although various functions are described herein as being performed by or in combination with APD 116, in various alternatives, the functions described as being performed by APD 116 may additionally or alternatively be performed by other computing devices with similar capabilities, which in some cases are not driven by a host processor (e.g., processor 102) and, in some implementations, provide (graphics) output to display device 118. For example, it is conceivable that any processing system that performs processing tasks according to the SIMD paradigm can perform the functions described herein. Alternatively, it is conceivable that a computing system that does not perform processing tasks according to the SIMD paradigm can perform the functions described herein.

[0024] Figure 2This is a block diagram of device 100, illustrating additional details related to the execution of processing tasks on APD 116. Processor 102 maintains one or more control logic modules in system memory 104 for execution by processor 102. The control logic modules include operating system 120, kernel-mode driver 122, and application program 126. These control logic modules control various features of the operation of processor 102 and APD 116. For example, operating system 120 communicates directly with the hardware and provides an interface to the hardware for other software executing on processor 102. Kernel-mode driver 122 controls the operation of APD 116 by providing, for example, an application programming interface (“API”) to software executing on processor 102 (e.g., application program 126) to access various functions of APD 116. Kernel-mode driver 122 also includes a just-in-time (JIT) compiler that compiles programs for execution by the processing units of APD 116 (such as SIMD unit 138, discussed further in detail below).

[0025] APD 116 executes commands and procedures for selected functions, such as graphics operations and non-graphics operations suitable for parallel and / or non-ordered processing. APD 116 is used to perform graphics pipeline operations such as pixel manipulation and geometric calculations, and to render images to display device 118 based on commands received from processor 102. APD 116 also performs computational processing operations not directly related to graphics operations based on commands received from processor 102, such as operations related to video, physics simulations, computational fluid dynamics, or other tasks.

[0026] APD 116 includes computation units 132, which include one or more SIMD units 138 that execute operations in parallel at the request of processor 102 according to a SIMD paradigm. A SIMD paradigm is one in which multiple processing elements share a single program control flow unit and program counter, and thus execute the same program, but can execute that program using different data. In one example, each SIMD unit 138 includes sixteen channels, where each channel executes the same instruction simultaneously with other channels in SIMD unit 138, but executes that instruction using different data. If not all channels need to execute a given instruction, assertions can be used to shut down channels. Assertions can also be used to execute programs with different control flows. More specifically, for programs with conditional branches or other instructions where the control flow is based on computations executed by individual channels, assertions on channels corresponding to control flow paths that are not currently being executed, and the serial execution of different control flow paths, allow arbitrary control flows. In one implementation, each computation unit 132 may have a local L1 cache. In one implementation, multiple computation units 132 share an L2 cache.

[0027] The basic unit of execution in computing unit 132 is a work item. Each work item represents a single instance of a program that will be executed in parallel on a specific channel. Work items can be executed simultaneously on a single SIMD processing unit 138 as “wavefronts.” One or more wavefronts are included in a “workgroup,” which comprises a set of work items designated for executing the same program. Workgroups are executed by executing the individual wavefronts that make up the workgroup. In various alternatives, wavefronts are executed sequentially on a single SIMD unit 138, or partially or entirely in parallel on different SIMD units 138. A wavefront can be considered as the largest set of work items that can be executed simultaneously on a single SIMD unit 138. Therefore, if a command received from processor 102 indicates that a particular program is to be parallelized to the extent that the program cannot be executed simultaneously on a single SIMD unit 138, the program is divided into two or more wavefronts, which are parallelized on two or more SIMD units 138 or serialized on the same SIMD unit 138 (or both parallelized and serialized as needed). Scheduler 136 performs operations related to scheduling individual wavefronts on different computing units 132 and SIMD units 138.

[0028] The parallelism provided by computing unit 132 is suitable for graphics-related operations, such as pixel value calculation, vertex transformation, and other graphics operations. Therefore, in some instances, graphics pipeline 134, which receives graphics processing commands from processor 102, provides computational tasks to computing unit 132 for parallel execution.

[0029] The computing unit 132 is also used to perform computational tasks unrelated to graphics or not as part of the "normal" operation of the graphics pipeline 134 (e.g., custom operations executed to supplement the operation of the graphics pipeline 134). The application 126 or other software executing on the processor 102 transmits programs defining such computational tasks to the APD 116 for execution.

[0030] Figure 3 It shows part based on Figure 1 and Figure 2 The device uses a heterogeneous system architecture (HSA) platform 300. The HSA platform 300 includes an HSA accelerated processing unit (APU) 310 that is connected to or communicates with system memory 350 (collectively, the “connection”). The HSA APU 310 includes: a multi-core CPU 320, a GPU 330 with multiple HSA compute units (H-CUs) 332, 334, and 336, and an HSA memory management unit (HMMU or HSA MMU) 340. The CPU 320 includes any number of cores... Figure 3The diagram shows cores 322, 324, 326, and 328. The GPU 330 includes any number of H-CUs, although... Figure 3 Three are shown. Although HSA is discussed and presented in the described implementation, the systems and methods of the present invention can be utilized on both homogeneous and heterogeneous systems. System memory 350 includes one or both of a consistent system memory 352 and a non-consistent system memory 357.

[0031] The HSA 300 provides a unified view of basic computing elements. The HSA 300 allows programmers to write programs that seamlessly integrate the CPU 320 (also known as the latency computing unit) with the GPU 330 (also known as the throughput computing unit), while benefiting from their respective optimal properties. The HSA 300 allows programmers to utilize the parallel processors in the GPU 330 as peers of the traditional multi-threaded CPU 320. A peer device is defined as an HSA device that shares the same memory coherence domain with another device.

[0032] Devices in the HSA 300 communicate with each other using queues, as shown in reference... Figure 4-6 To explain further, queues are an integral part of the HSA architecture. A queue is a physical memory area in which producers issue requests or tasks to consumers. Depending on the complexity of the HSA hardware, queues may be managed by any combination of software or hardware. In a sense, hardware-managed queues offer significant performance advantages because applications running on latency processors (such as the CPU 320) directly queue jobs to throughput processors (such as the GPU 330) without any intermediate operating system calls. This allows for very low-latency communication between devices in the HSA 300.

[0033] Figure 4 This is a block diagram of example system 400, illustrating the queue structure. System 400 includes a CPU 405, system memory 415, a driver 410, a graphics processing unit (GPU) 420, and communication infrastructure or bus 425. Those skilled in the art will understand that system 400 includes, in addition to... Figure 4 Software, hardware, and firmware components that are other than or different from the components shown. It should be understood that system 400 includes... Figure 4 Additional components not shown.

[0034] CPU 405, GPU 420, and system memory 415 can be configured according to the following specifications: Figure 1-3This is implemented as described. CPU 405 executes an operating system (not shown) and one or more applications, and is the control processor of system 400. The operating system executing on CPU 405 controls, facilitates access to, and coordinates the completion of tasks for system 400. Driver 410 (e.g., a graphics driver) includes software, firmware, hardware, or any combination thereof. In one implementation, driver 410 is implemented entirely in software. Driver 410 provides an interface and / or application programming interface (API) for CPU 405 and applications executing on CPU 405 to access GPU 420. Bus 425 provides coupling between the components of system 400 and includes one or more communication buses, such as Peripheral Component Interconnect (PCI), Advanced Graphics Port (AGP), etc.

[0035] GPU 420 provides system 400 with graphics acceleration and other computing capabilities as described herein. GPU 420 includes multiple command processors (CPs) – CP 1, ..., CP n 430, and multiple engines – engine 1, ..., engine n 435, such as a 3D engine, a unified video decoder (UVD) engine, a digital rights management (DRM) direct memory access (DMA) engine, and so on.

[0036] CP 1, ..., CP n 430 control processing within GPU 420 and are connected to Engine 1, ..., Engine n 435. Each CP 1, ..., CP n 430 is associated with Engine 1, ..., Engine n 435, and each pair is an Engine Block (EB) – EB 1, ..., EB n 437. In another implementation, CP 1, ..., CP n 430 is a single instruction processor. Generally, CP 1, ..., CP n 430 receives instructions to be executed from CPU 405 and coordinates the execution of these instructions on Engine 1, ..., Engine n 435 within GPU 420. In some cases, CP 1, ..., CP n 430 generates one or more commands to be executed in GPU 420, corresponding to each command received from CPU 405. The logical instructions that implement the functionality of CP1, ..., CP n 430 are implemented in hardware, firmware, or software, or a combination thereof.

[0037] Memory 415 includes one or more memory devices, and may be Dynamic Random Access Memory (DRAM) or similar memory devices for non-persistent storage of data. Memory 415 includes one or more memory buffers 445 through which CPU 405 transmits commands to GPU 420. Where appropriate, memory buffer 445 corresponds to engine 435 or engine block 437. Memory buffer 445 is implemented as a queue, circular buffer, or other data structure suitable for efficient queuing of work items or command groups. In the case of a queue, command groups are placed into and removed from memory buffer 445 in a circular manner. For illustrative purposes, memory buffer 445 is referred to herein as queue 1, ..., queue n 445.

[0038] Memory 415 includes indirect buffers 455. Indirect buffers 455 hold actual commands (e.g., instructions, data, pointers, and non-pointers). For example, when CPU 405 transmits a command packet to GPU 420, the command packet is stored in indirect buffer 455, and a pointer to that indirect buffer 455 is inserted into queue 1, ..., queue n 445. As described below, some indirect buffers 455 hold neuron data. That is, multiple indirect buffers are used for different purposes. CPU 405 acts as a writer of commands for queues 1, ..., queue n 445 via driver 410, and GPU 420 acts as a reader of such commands, coordinating write and read pointers that respectively indicate the last added and last read items in queues 1, ..., queue n 445.

[0039] Figure 5A This is an example block diagram of command grouping processing between GPU 500, driver 510, queue 515, and indirect buffer 535. GPU 500 includes GPU memory 502, register 504, command processor 505, and engine 508. Register 504 includes read pointer 512 and write pointer 514. Queue 515 includes elements 520, 522, 524, and free space 530. Each element, such as elements 520, 522, and 524, stores queue groups. Figure 5B Example element 570 is shown, which includes command group 572 and indirect buffer (IB) command group 576 pointing to indirect buffer 535. Figure 5CAs shown, the indirect buffer 535 includes command packets 540 that instruct the GPU 500 to perform operations. For example, a kernel dispatch packet in the HSA (an example of command packet 540) includes information such as how the computation kernel should start threads (grid dimension, workgroup size), the required size of dedicated and group memory allocations, handles to objects in memory containing executable ISA images for the computation kernel, and additional control and synchronization information. Generally, in DNNs, computation kernels are typically implemented using high-level libraries such as MIOpen and rocBLAS, performing convolutions, matrix multiplications, Fast Fourier Transform (FFT), pooling, and activations.

[0040] The architecture described above provides unidirectional communication from the host processor (the writer, represented by driver 510) to the GPU 500 (the reader, represented by command processor 505). Initially, read pointer 512 and write pointer 514 point to the same location, indicating that queue 515 is empty. Queue 515 has free space 530, into which driver 510 writes command packets corresponding to tasks. Then, driver 510 updates write pointer 514 to a location after the last command packet or to the first available space. Now, write pointer 514 and read pointer 512 point to different locations. Command processor 505 retrieves a command packet at the read pointer 512 location and moves read pointer 512 until it equals write pointer 514.

[0041] Convolution in neural networks is a local operation because only the output of a small number of neurons is necessary to compute some neurons in subsequent layers. As a result, computation in subsequent layers can be performed in parallel without waiting for all neurons in the current layer to complete their computation. Figure 6 Two representative layers from DNN 600 are shown in the diagram as a directed acyclic graph (DAG): layer 1 (605) and layer 2 (610). Layer 1 (605) includes, for example, neurons 620, 622, and 624, and layer 2 (610) includes, for example, neurons 630, 632, 634, 636, and 638. In some cases, some neurons in the second layer will be able to function before the entire first layer is evaluated. This is in... Figure 6 The middle section is indicated by darker and thicker lines, which allow two neurons (e.g., neurons 630 and 632) in layer 2 610 to be evaluated before the last neuron (e.g., neuron 624) in layer 1 605 is computed.

[0042] This is beneficial in two situations. First, performing a naive sequential execution of the layers may leave some computational resources idle. In this case, those idle resources can be deployed to the next layer. Second, if the second layer requires substantially more work than the first layer, it may be more efficient to start computation from that layer earlier, utilizing several available computational resources. Modern DNNs have many hidden layers (hundreds or even thousands), so it may be advantageous to start computation from many deep layers before completing computation in a single layer. However, to accurately predict which computations should be prioritized, a scheduler (such as...) Figure 2 The scheduler (136) in the algorithm needs some knowledge about what the expected cost of any computation is.

[0043] The methods and systems described in this paper use metadata to label or associate certain architectural, functional, or computational aspects, parts, properties, or characteristics (collectively, "(one or more) aspects") of a DNN system to allow for more accurate job scheduling and load balancing, where the metadata provides a representative computational cost associated with or related to a particular DNN aspect. For example, these specific aspects of a DNN system can be computational kernels, neurons, layers, or other similar aspects. In an illustrative example, the metadata might be kernel or filter size, dropout layers, the number of neurons present in a layer, activation functions, and other similar items that can be used to determine computational costs. The metadata can be used to label aspects of a DNN system automatically, for example, by an application or user, through the framework described herein.

[0044] Metadata can be stored in various locations, including but not limited to instructions, scalar registers, L1 / L2 caches, tables, etc. In one implementation, the location depends on the size of the metadata, its adaptability to compression, where the metadata is captured, and which scheduler is using the metadata.

[0045] In an illustrative example, some metadata, such as activation functions and the number of neurons per layer, is most easily captured by software (e.g., programmers, compilers, etc.) and stored in job requests or instructions, while other metadata needs to be measured in hardware runtime (e.g., kernel runtime) and stored in hardware tables.

[0046] In another illustrative example, the location depends on which scheduler is using the metadata. If the metadata is utilized at the O / S level, such as... Figure 2 In O / S 120, metadata can be embedded in a job request message, which is sent to the OS scheduler. If the metadata is utilized at the hardware scheduler or dispatcher level, such as... Figure 2 Scheduler 136 or Figure 7In the hardware dispatcher 720, metadata can be stored in a hardware table such as hardware table 710, which is located in the same place as scheduler 136 or hardware dispatcher 720. If the hardware scheduler also utilizes metadata captured by software (e.g., activation functions), the metadata can be passed from software to job request messages, then to the OS, and finally to the hardware table.

[0047] Several techniques can be used to measure computational cost. In one implementation, the relationship between kernel size and evaluation time is used to determine computational cost. In an illustrative implementation, different heuristics are used, which identify that larger filter sizes correspond to more work.

[0048] In another illustrative implementation, kernel runtime provides an indication of relative speed to determine computational cost. This requires the dispatcher to perform a table lookup of kernel-sized runtimes to allocate work. The estimated runtime is then combined with resource availability to efficiently dispatch a workgroup while balancing the load across computational resources.

[0049] In one implementation, such as Tensorflow ® Frameworks like (a trademark of Google LLC) or MIOpen provide metadata indicating filter sizes or other similar information for specific layers. In an illustrative implementation, this is determined at the start of runtime because the DNN does not change on a substantial basis. This allows for dynamic configuration of the kernel, since timing the kernel only after the first iteration does not incur significant cost for the entire runtime.

[0050] In addition to the computational cost of using kernels, the scheduler also needs to detect whether some neurons in subsequent layers are ready to execute. In one implementation, this is achieved by building a Directed Acyclic Graph (DAG) of the computations being performed in the DNN. A DAG allows the scheduler to allocate or deallocate locks for specific neurons, rather than restricting computation to the entire layer. That is, the scheduler can expose specific neurons in a layer for use when no other neurons are needed. Once the scheduler has the entire graph of the expected computations and the metadata, it uses heuristics to schedule runs across available computational resources to ensure efficient processing.

[0051] Figure 7This is an example block diagram for using computational kernels in a DNN system 700 to determine computational costs to allow for more accurate job scheduling and load balancing. In this implementation, computational cost is kernel runtime stored as metadata in a table. The DNN system 700 includes at least a processor, such as a GPU 705, that communicates with or connects to the table, where the table is, for example, a hardware table 710. This table can be implemented, for example, in memory, registers, cache, or other similar space. The GPU 705 includes at least a dispatcher, such as a hardware dispatcher 720 that communicates with multiple compute units (CUs)—CU 0 722, CU 1 724, ..., CU N 726. The hardware table 710 associates a kernel ID 730 with the measured kernel runtime 732. In one implementation, due to the regularity of certain types of DNNs (such as convolutional neural networks (CNNs)), kernel runtime measurements may only need to be performed once.

[0052] When determining job assignments, the hardware dispatcher 720 considers the metadata associated with each core. In this context, the metadata is the measured kernel runtime. For example, the hardware dispatcher 720 considers the measured kernel runtime 732 for each core by looking up the kernel ID 730 and retrieving the associated measured kernel runtime 732. In one implementation, the hardware dispatcher 720 combines the measured kernel runtime 732 with processor or compute unit resources to efficiently assign workgroups while balancing the load across processor or compute unit resources.

[0053] In one implementation, metadata is applied to the dynamic pruning and sparsity of the DNN. In these cases, individual neurons are randomly pruned or removed from the DNN. A dynamic approach to evaluating and distributing work would allow load balancing across iterations. One way to achieve this is, for example, to allow the scheduler or a separate helper thread to check the readiness state of the neurons to be computed, where the readiness state refers to or describes the dependencies of each neuron (which can be pre-computed). This information (i.e., the readiness state) would be tagged as metadata for that neuron. For example, when a neuron is pruned, the readiness states of all dependent neurons are updated and used by the scheduler for load balancing across iterations.

[0054] In one implementation, metadata is used between multiple devices, where metadata allows overlapping computation and communication (whether between devices or between CUs) by initiating computation on subsequent layers without waiting for local storage data.

[0055] Figure 8This is a flowchart of a method 800 for opportunistic load balancing in a DNN using metadata. A representative computational cost is captured, obtained, or determined for a given architecture, function, or computational aspect of the DNN system (step 805). The representative computational cost is implemented as metadata for the given architecture, function, or computational aspect of the DNN system (step 810). In one implementation, the computed computational cost is implemented as metadata. The scheduler detects whether there are neurons ready to execute in subsequent layers (step 815). The scheduler uses the metadata and neuron availability to schedule and load balance across computational and available resources (step 820).

[0056] It should be understood that many variations are possible based on the disclosure herein. Although features and elements are described above in specific combinations, each feature or element can be used alone without other features and elements, or in various combinations with or without other features and elements.

[0057] The provided methods can be implemented in general-purpose computers, processors, or processor cores. Suitable processors include, for example, general-purpose processors, special-purpose processors, common processors, digital signal processors (DSPs), multiple microprocessors, one or more microprocessors associated with a DSP core, controllers, microcontrollers, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), any other type of integrated circuit (IC), and / or state machines. Such processors can be manufactured by configuring the manufacturing process using the results of processed hardware description language (HDL) instructions and other intermediate data, including netlists (which can be stored on a computer-readable medium). The result of such processing can be a mask, which is then used in the semiconductor manufacturing process to manufacture processors that implement various aspects of the embodiments.

[0058] The methods or flowcharts provided herein can be implemented in a computer program, software, or firmware incorporated in a non-transitory computer-readable storage medium for execution by a general-purpose computer or processor. Examples of non-transitory computer-readable storage media include read-only memory (ROM), random access memory (RAM), registers, cache memory, semiconductor memory devices, magnetic media (such as internal hard disks and removable disks), magneto-optical media, and optical media such as CD-ROMs and digital versatile optical discs (DVDs).

Claims

1. A method for load balancing in a neural network system, the method comprising: Metadata is used to label portions of the neural network system, the portions including any one or more kernels, one or more neurons, and one or more layers of the neural network system or a combination thereof, the metadata including representative computational costs of the portions, and metadata including ready states representing dependencies of each neuron is used to label neurons of the neural network system. During execution, the scheduler updates the ready state of the neuron; The scheduler detects whether there are neurons available for execution in the layers following the currently being processed layer; as well as The scheduler uses the metadata, including the representative computational cost, and the detected neuron availability to schedule and load balance computations performed in layers following the current layer being processed across computational resources.

2. The method of claim 1, wherein the metadata includes at least one of kernel size, filter size, dropout layer, number of neurons in the layer, and activation function.

3. The method of claim 1, wherein the neuron readiness state is updated after the neuron associated with the neuron readiness state is pruned.

4. The method of claim 1, wherein the metadata is tagged to the portion of the neural network system by at least one of an application, a framework, or a user.

5. The method of claim 1, wherein the metadata is stored in at least one of instructions, scalar registers, memory, and hardware tables.

6. The method of claim 1, wherein the location where the metadata is stored depends on at least one of the following: metadata size, compression capability, where the metadata is captured, and the type of scheduler using the metadata.

7. A system for load balancing in a neural network system, comprising: The portion of the neural network system tagged with metadata includes one or more kernels, one or more neurons, and one or more layers or a combination thereof. The metadata includes representative computational costs associated with the portion, and neurons of the neural network tagged with metadata including a ready state representing the dependencies of each neuron. as well as A scheduler connected to said portion of the neural network system, wherein the scheduler is configured to: The ready state of the neuron is updated during execution; The system detects neurons in layers following the currently being processed that can be used for execution. as well as The metadata, including the representative computational cost, and the detected neuron availability are used to schedule and load balance computations performed in layers following the current layer being processed across computational resources.

8. The system of claim 7, wherein the metadata includes at least one of kernel size, filter size, dropout layer, number of neurons in the layer, and activation function.

9. The system of claim 7, wherein the neuron readiness state is updated after the neuron associated with the neuron readiness state is pruned.

10. The system of claim 7, wherein the metadata is tagged to the portion of the neural network system by at least one of an application, framework, or user.

11. The system of claim 7, wherein the metadata is stored in at least one of instructions, scalar registers, memory, and hardware tables.

12. The system of claim 7, wherein the location where the metadata is stored depends on at least one of the following: metadata size, compression capability, where the metadata is captured, and the type of scheduler using the metadata.

Images