Computing device, resource allocation method and multipath neural network analyzer

Abstract

The present invention provides a multipath neural network, a method for resource allocation, and a multipath neural network analyzer. The method for resource allocation can use a multipath neural network analyzer including an interface and a processing device to optimally allocate for a multipath neural network. The interface receives the multipath neural network. The processing device generates a multipath neural network to include one or more layers of a critical path through the multipath neural network, and the one or more layers are allocated with a first computing resource allocation that can be used to execute the multipath neural network. The critical path limits the throughput of the multipath neural network. The first computing resource allocation reduces the execution time of the multipath neural network to less than the baseline execution time of the second computing resource allocation for the multipath neural network. The first computing resource allocation for the first layer of the critical path is different from the second computing resource allocation for the first layer of the critical path.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. Provisional Application No. 62 / 796,071 filed on January 23, 2019, the disclosure of which is incorporated herein by reference in its entirety. In addition, this application is filed concurrently with U.S. Patent Application Serial No. (Attorney Docket No. 1535-479), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The subject matter disclosed herein relates to neural networks. More specifically, the subject matter disclosed herein relates to systems and methods for allocating computational resources to different layers in different paths of a multipath neural network. Background Art

[0004] Typical graphics processing unit (GPU) platforms do not allow for parallel execution of GPU operations. To launch multiple operations in parallel on a typical GPU, each operation must be assigned to a separate executor, that is, a separate stream in the Compute Unified Device Architecture (CUDA) programming model. However, it is almost impossible to run two or more GPU kernels in parallel for most popular GPU kernels. Popular kernels occupy all GPU core resources (including registers and shared memory) and do not allow the GPU scheduler to execute blocks from another kernel on the same core. Current GPU platforms rely on multiple blocks for scheduling, and parallel execution of another GPU operation may only be possible when the number of blocks is low. Since most GPU kernels typically have enough blocks to occupy all available GPU cores, the execution of another GPU kernel is postponed until after the first kernel is completed, resulting in sequential execution of the two operations. Therefore, current GPU platforms utilize 100% of the device (e.g., GPU) cores for individual layers (kernels), thereby preventing parallel execution of independent kernels.

[0005] Another weak link associated with current GPU platforms is inefficient resource utilization. Although GPU cores can be optimized for performance on specific inputs and GPU microarchitectures, computational efficiency and DRAM utilization are not fully optimized for a variety of GPU algorithms for some inputs. For example, the execution time of some convolutions (mainly General Matrix Multiplication (GEMM) class) is saturated with a limited number of cores, and performing convolutions on additional cores causes underutilization of the cores. By running multiple operations in parallel, underutilized resources in one GPU operation can be used for another GPU operation taking into account the algorithm, input data, data layout, and microarchitecture.

[0006] Another weakness associated with current GPU platforms is that existing GPU platforms execute operations sequentially in the order in which the GPU device receives the operations, prioritizing this to reduce the latency of earlier requested operations. This scheduling method can increase the average latency of GPU operations for multiple applications. For example, if a GPU operation that takes a long time to complete arrives at the GPU device before another GPU operation that takes less time to complete arrives, then the latency (or response time) of the second GPU operation becomes unnecessarily long.

[0007] Another weak link associated with current GPU platforms is that current GPU platforms are not optimized for workloads with multiple paths. GPU workloads may form a directed acyclic graph (DAG), where each node represents an operation and a directed edge from node A to node B represents an input-output dependency between node A and node B; that is, node B must be executed after node A. In recent years, GPU workloads with multipath computation graphs have become common. Multipath workloads may emerge from different GPU applications launched by a single user, by multiple users sharing a GPU platform, or by a single GPU application with multiple paths. A multipath neural network is an example of a single GPU application with multiple paths.

[0008] Multipath neural networks have gained a lot of attention in the field of machine learning (ML) as a better representation of real-world ML problems. Typically, for parallel neural network paths, there will be one path that can be called a critical path that limits the throughput of the overall neural network. The flow of signals through a neural network can be characterized as the flow of signals through a series of dependent layers. Typically, when executing a neural network, the layers are executed in a sequential iteration, where the input of a given layer depends on the output of the immediately previous layer in the sequence.

[0009] Current deep learning frameworks are not configured to optimize the execution of multi-path neural networks based on resource allocation between kernels in different paths or different paths of execution. If the overall workspace memory required to execute the kernels is larger than the available memory on the device executing the neural network, inefficient memory allocation may prevent parallel execution of kernels of the multi-path neural network. Inefficient computational resource allocation in the critical path of the multi-path neural network may adversely affect the overall neural network throughput. Summary of the invention

[0010] Example embodiments provide a multipath neural network that may include two or more than two paths through the multipath neural network, where a first path may include one or more layers. The first path may be a critical path through the multipath neural network that limits the throughput of the multipath neural network. A first layer of the critical path may be assigned a first computing resource allocation capable of executing the multipath neural network. The first computing resource allocation reduces the execution time of the multipath neural network to less than a baseline execution time for a second computing resource allocation for the multipath neural network. The first computing resource allocation for the first layer may be different from the second computing resource allocation for the first layer. In one embodiment, the first computing resource allocation for the first layer may be assigned prior to executing the multipath neural network. In another embodiment, the first computing resource allocation for the first layer may be assigned during executing the multipath neural network.

[0011] Example embodiments provide a method of allocating resources to layers of a multipath neural network, the method may include: determining a baseline execution time for a multipath neural network based on allocating computing resources capable of executing the multipath neural network to the multipath neural network, the multipath neural network including two or more paths through the multipath neural network; iteratively determining execution times of the two or more paths through one or more second computing resource allocations for the two or more paths, each second computing resource allocation being different from the first computing resource allocation; and identifying a second computing resource allocation for a first layer in the first path, the second computing resource allocation providing an overall execution time for the multipath neural network that is less than the overall baseline execution time for the multipath neural network.

[0012] Example embodiments provide a multipath neural network analyzer that may include an interface and a processing device. The interface may receive a multipath neural network that may include two or more paths through the multipath neural network and a first path of the two or more paths may include one or more layers. The processing device may generate the multipath neural network to include one or more layers of a critical path through the multipath neural network, the one or more layers being allocated a first computing resource allocation capable of executing the multipath neural network. The critical path may limit the throughput of the multipath neural network. The first computing resource allocation may reduce the execution time of the multipath neural network to less than a baseline execution time of a second computing resource allocation for the multipath neural network. The first computing resource allocation for a first layer of the critical path may be different from the second computing resource allocation for the first layer of the critical path. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In the following sections, aspects of the subject matter disclosed herein will be described with reference to exemplary embodiments shown in the figures, in which:

[0014] Figure 1 A functional block diagram depicting a multipath neural network analyzer that optimizes the allocation or partitioning of computational resources to different layers in different paths of a multipath neural network in accordance with the subject matter disclosed herein.

[0015] Figure 2A Depicted is a processing unit, such as a GPU, having four cores on which multiple GPU operations have been scheduled in parallel, for example using modified kernel launch functions, in accordance with the subject matter disclosed herein.

[0016] Figure 2B Depicts a GPU after a second kernel has been launched using a modified kernel launch function in accordance with the subject matter disclosed herein.

[0017] Figure 3 Depicts an example multipath neural network with two paths between the input and the decision block.

[0018] Figure 4A Depicting the relative baseline execution times and sequences of portions of two example paths through a multipath neural network.

[0019] Figure 4B Depicts the relative execution time and sequence of two paths for a convolutional layer that has been allocated 4.5 gigabytes (GB).

[0020] Figure 4C Plotting the relative execution time and sequence of the two paths for a convolutional layer that has been allocated 4.0GB of memory.

[0021] Figure 5 Depicted is an example of window class resource allocation analysis for a layer in accordance with the subject matter disclosed herein.

[0022] Figure 6 Describes a method for implementing a method according to the subject matter disclosed herein. Figure 1 Block diagram of an exemplary architecture of a data processing system of a multipath neural network analyzer.

[0023] Description of Figure Numbers

[0024] 100: Multipath neural network analyzer;

[0025] 101: Multipath Neural Network;

[0026] 102, 103, 104, 106, 109: steps;

[0027] 105: Device resource constraints;

[0028] 107: Cost model;

[0029] 108: Scheduler;

[0030] 201, 202: kernel;

[0031] 300: Multipath neural network;

[0032] 301, 302, 503, 504: path;

[0033] 303: input terminal;

[0034] 304: Decision block;

[0035] 401, 402: data streaming;

[0036] 500: layer;

[0037] 501, 502: window;

[0038] 600: Demonstration structure;

[0039] 601: processor;

[0040] 602: storage element;

[0041] 603: system bus;

[0042] 604: local storage;

[0043] 605: mass storage device;

[0044] 606: keyboard;

[0045] 607: display device;

[0046] 608: pointing device;

[0047] 609: Network adapter;

[0048] 610: Operating system;

[0049] 611: Application;

[0050] 612: Platform;

[0051] 613: Communication link;

[0052] 614: Neural network accelerator. DETAILED DESCRIPTION

[0053] In the following detailed description, many specific details are set forth in order to provide a thorough understanding of the present disclosure. However, those skilled in the art will appreciate that the disclosed aspects may be practiced without these specific details. In other cases, well-known methods, steps, components, and circuits are not described in detail to avoid obscuring the subject matter disclosed herein.

[0054] Reference to "one embodiment" or "an embodiment" of this specification means that the specific features, structures or characteristics described in conjunction with the embodiment may be included in at least one embodiment disclosed herein. Therefore, the phrases "in one embodiment" or "in an embodiment" or "according to an embodiment" (or other phrases with similar meanings) appearing in different places throughout this specification may not necessarily all refer to the same embodiment. In addition, in one or more embodiments, the specific features, structures or characteristics may be combined in any suitable manner. In this regard, as used herein, the word "exemplary" means "used as an embodiment, example or illustration". Any embodiment described herein as "exemplary" should not be understood as necessarily being preferred or advantageous over other embodiments. In addition, in one or more embodiments, the specific features, structures or characteristics may be combined in any suitable manner. In addition, depending on the context of the discussion in this article, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. Similarly, hyphenated terms (e.g., "two-dimensional", "pre-determined", "pixel-specific", etc.) may be used occasionally interchangeably with corresponding unhyphenated versions (e.g., "two dimensional", "predetermined", "pixel-specific", etc.), and capitalized terms (e.g., "Counter Clock", "Row Select", "PIXOUT", etc.) may be used interchangeably with corresponding non-capitalized versions (e.g., "counter clock", "row select", "pixout", etc.). Such accidental interchangeable usage should not be considered inconsistent with each other.

[0055] Furthermore, depending on the context of the discussion herein, singular terms may include corresponding plural forms, and plural terms may include corresponding singular forms. It should also be noted that the various figures (including component diagrams) depicted and discussed herein are for illustrative purposes only and are not drawn to scale. Similarly, various waveforms and timing diagrams are depicted for illustrative purposes only. For example, the size of some elements may be enlarged relative to other elements for clarity. Furthermore, if deemed appropriate, reference numerals are repeated in the drawings to indicate corresponding and / or similar elements.

[0056] The terms used herein are only for the purpose of describing some example embodiments and are not intended to limit the subject matter claimed. As used herein, unless the context clearly indicates otherwise, the singular forms "a / an" and "the" are also intended to include the plural forms. It should be further understood that the term "comprises and / or comprising" when used in this specification specifies the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. As used herein, unless so clearly defined, the terms "first", "second", etc. are used as noun labels before them and do not imply any type of order (e.g., space, time, logic, etc.). In addition, the same figure mark may be used on two or more figures to refer to parts, components, blocks, circuits, units or modules with the same or similar functions. However, such use is only for brevity and ease of discussion of the description; it does not imply that the construction or architectural details of such components or units are the same in all embodiments, or that such commonly referenced components / modules are the only way to implement some of the example embodiments disclosed herein.

[0057] It should be understood that when an element or layer is referred to as being "on", "connected to" or "coupled to" another element or layer, the element or layer may be directly on, directly connected to or coupled to another element or layer, or there may be intermediate elements or layers. In contrast, when an element is referred to as being "directly" "on", "directly connected to" or "directly coupled to" another element or layer, there are no intermediate elements or layers. Throughout the text, the same reference numerals represent the same elements. As used herein, the term "and / or" includes any and all combinations of one or more associated listed items.

[0058] As used herein, unless explicitly defined as such, the terms "first", "second", etc. are used as noun labels preceding them and do not imply any type of order (e.g., spatial, temporal, logical, etc.). In addition, the same reference numerals may be used in two or more figures to refer to parts, components, blocks, circuits, units, or modules having the same or similar functions. However, such use is only for brevity of illustration and ease of discussion; it does not imply that the construction or architectural details of such components or units are the same in all embodiments, or that such commonly referenced components / modules are the only way to implement some of the example embodiments disclosed herein.

[0059] Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by those skilled in the art to which the disclosed subject matter belongs. It is further understood that terms (such as those defined in commonly used dictionaries) should be interpreted as having a meaning consistent with their meaning in the context of the relevant technology, and unless explicitly defined as such, will not be interpreted in an idealized or overly formal sense.

[0060] As used herein, the term "module" refers to any combination of software, firmware, and / or hardware that is configured to provide the functionality described herein in conjunction with the module. Software may be implemented as a software package, code, and / or instruction set or instructions, and the term "hardware" as used in any implementation described herein may, for example, include fixed-line circuits, programmable circuits, state machine circuits, and / or firmware storing instructions executed by programmable circuits, alone or in any combination. Modules may be implemented together or individually as circuits that form part of a larger system, such as, but not limited to, integrated circuits (ICs), systems on chips (SoCs), and the like. The various components and / or functional blocks disclosed herein may be implemented as modules that may include software, firmware, and / or hardware that provide the functionality described herein in conjunction with the various components and / or functional blocks.

[0061] The subject matter disclosed herein relates to systems and methods for efficiently allocating or partitioning computing resources between kernels executing in parallel on a single GPU device. The system disclosed herein may schedule parallel and / or simultaneous execution of multiple GPU operations on a single GPU device through core allocation and / or core reallocation. The system's application programming interface (API) allows a user to select GPU operations to be scheduled in parallel and to allocate computing resources for the parallel operations.

[0062] The systems and methods disclosed herein may be used to allocate computing resources to optimize the performance of a multipath neural network. In one embodiment, parallel execution may be enabled for different layers in different paths of a multipath neural network. Automatic optimization of multipath neural network applications may be provided based on causal resource distribution techniques. In one embodiment, causal resource distribution techniques may provide automatic performance optimization of a multipath neural network. An optimal resource allocation may be identified based on a relationship between computing resources and execution times of layers in the neural network affected by the allocation level of computing resources.

[0063] In addition, the causal resource distribution can be based on the relationship between the critical path through the multipath neural network and the computational resources and the execution time associated with the computational resources. The cost model template can be used for performance analysis, and one or more heuristics can be used to reduce the complexity of the analysis to determine the optimal resource allocation. In addition, the execution schedule of the multipath neural network application can be optimized using the causal resource distribution technique.

[0064] The resource allocation techniques disclosed herein can allocate computing resources between layers of a multipath neural network to improve the performance results of the overall neural network. Network performance can be analyzed based on modifications to resource utilization of the layers of the multipath neural network, and if the performance analysis indicates performance improvements based on a particular computing resource allocation, the resource utilization of individual layers can be updated or reconfigured.

[0065] The subject matter disclosed herein provides a programming interface that can partition computing resources among kernels. Unlike current GPU platforms, which have limited programmer support for parallel execution and rely on the number of thread blocks executed to determine the number of GPU cores for a kernel, the subject matter disclosed herein provides a programming interface in which a programmer can define GPU operations for parallel execution and resource allocations for the parallel operations.

[0066] Both static resource allocation and dynamic resource allocation can be supported by the systems and methods disclosed herein. For static allocation, resources can be assigned to individual layers before executing the application. For dynamic allocation, resources can be assigned during runtime to provide flexibility based on network changes (such as changes in topology, input, batch size, etc.).

[0067] Figure 1 A functional block diagram depicting a multipath neural network analyzer 100 that optimizes the allocation or partitioning of computing resources to different layers in different paths of a multipath neural network in accordance with the subject matter disclosed herein. In an example embodiment, the analyzer 100 may be implemented as a data processing system executing suitable software. The analyzer 100 may receive a multipath neural network 101 as input for execution using computing resources available to the analyzer 100. The multipath neural network 101 may be a full precision trained neural network. Alternatively, the multipath neural network 101 may be a trained neural network. The analyzer 100 may analyze different layers in different paths of the multipath neural network 101 using causal resource distribution and determine the optimal allocation or partitioning of computing resources to enable the multipath neural network 101 to execute or run in a minimum amount of time.

[0068] In one embodiment, the analyzer 100 may generate an optimized computational resource allocation for one or more layers of one or more paths of the multi-path neural network prior to executing the multi-path neural network. The optimized computational resource allocation is then used when the neural network is run. In another embodiment, the analyzer 100 may optimize the parameters defining the computational resources for one or more layers of one or more paths of the multi-path neural network associated with the neural network so that when the neural network is instantiated on a computing device, an optimized configuration of the neural network is executed. In yet another embodiment, the analyzer 100 may dynamically update the computational resource allocation for one or more layers of one or more paths of the multi-path neural network during execution of the multi-path neural network to provide flexibility based on network changes (e.g., changes in topology, input, batch size, etc.).

[0069] Initially in Figure 1 At step 102 in the example, the analyzer 100 assigns the corresponding streams to different paths of the neural network 101 having one or more sequentially dependent layers. At step 103, the analyzer 100 generates a baseline resource allocation state. In addition, a window or group size can be generated to reduce the complexity of performing causal resource distribution for different computational resource allocations, such as in combination with Figure 5 At step 104, the analyzer 100 may perform a performance analysis of different computing resource allocations to determine an optimal computing resource allocation. A device resource constraint 105 that places constraints and / or constraints on available computing resources may be received from a user input. The resource constraint 105 may be received from the user prior to receiving the multipath neural network at the neural network 101. The performance analysis may also use the window or packet size generated at step 103. The number of different computing resource allocations analyzed may be reduced at step 106 by pruning or discarding computing resource allocations that may not provide an optimal performance solution. A cost model 107 may be used during the performance analysis.

[0070] Scheduler 108 may simulate the throughput performance of portions of the different layers and paths analyzed and / or the overall analyzed neural network 101. At step 109, as better computational resource allocations are determined for the different layers and / or paths, and better scheduling allocations are determined for neural network 101, analyzer 100 updates the baseline resource allocations for the layers at step 103 with the better allocations, and the better scheduling allocations will be used when neural network 101 runs.

[0071] In operation, the analyzer 100 receives device resource constraints 105 via an application programming interface (API) that allows a user to selectively set constraints on computing resources available to execute a multi-path neural network. In one embodiment, a user may select a minimum number and / or maximum number of cores of a processing unit or computing unit for one or more kernels that execute a path of a multi-path neural network on a processing unit. Table 1 sets forth an example of a modified kernel launch function that may be used in an Open Computing Language (OpenCL) programming framework that may provide resource constraints, such as a minimum number and a maximum number of cores available for a kernel in accordance with the subject matter disclosed herein. The clEnqueueNDRangeKernel kernel launch command may be modified to include, for example, two additional variable parameters: a "min_cores" variable parameter and a "max_cores" variable parameter. Specifically, a "const cl_uint min_cores" variable parameter and a "const cl_uint max_cores" variable parameter are added to the clEnqueueNDRangeKernel kernel launch command of the OpenCL framework. Two additional parameters that set variable parameters are present at the end of the variable parameter list of the clEnqueueNDRangeKernel kernel launch command.

[0072] The "min_cores" variable parameter can be used to define the minimum number of cores to use for the kernel's processing unit when running with other kernels, and the "max_cores" variable parameter can be used to define the maximum number of cores to use for the kernel's processing unit when running alone. The user can define a number of cores to be used for a kernel regardless of the number of thread blocks of the kernel.

[0073] Table 1. Modified kernel startup functions

[0074] cl_int clEnqueueNDRangeKernel(cl_command_queue command_queue,

[0075] cl_kernel kernel,

[0076] cl_uint work_dim,

[0077] const size_t*global_work_offset,

[0078] const size_t*global_work_size,

[0079] const size_t*local_work_size,

[0080] cl_uint num_events_in_wait_list,

[0081] const cl_event*event_wait_list,

[0082] cl_event*event,

[0083] const cl_uint min_cores,

[0084] const cl_uint max_cores)

[0085] Figure 2A Depicts a processing unit, such as a GPU, having four cores 1 to 4, on which multiple GPU operations have been scheduled in parallel, for example using a modified kernel launch function, in accordance with the subject matter disclosed herein. Initially, the modified kernel launch function is used to set the min-cores / max-cores variable parameters of kernel 201 to 2 / 3. For this example, kernel 201 may have 12 thread blocks executing on GPU cores 1 to 4, the thread blocks being distributed as follows: Figure 2A As depicted in FIG. 1 , the distribution is on cores 1 to 3.

[0086] The second core 202 may be configured to set the minimum core / maximum core variable parameter to 2 / 4. Figure 2B , when kernel 202 is launched on the GPU device, the GPU scheduler (not shown) performs thread block migration so that the thread blocks of kernel 201 previously allocated on core 3 are migrated to core 1 and core 2 based on the minimum / maximum variable parameters for both the previously executed kernel 201 and the newly executed kernel 202. In this case, both kernels have a minimum core variable parameter 2, which will set the minimum value of the two GPU cores to run each of kernel 201 and kernel 202. Different cores 1 to 4 can run different numbers of thread blocks. In addition, different thread blocks can be allocated. Figure 2A and Figure 2B The specific core depicted in .

[0087] Based on the device constraints 105 input by the user, the analyzer 100 may use causal resource distribution to determine the optimal computational resource allocation or partitioning to execute the multi-path neural network 101. In one embodiment, causal resource distribution may be performed on a critical path through the multi-path neural network 101 to identify how changes in computational resource allocation affect, for example, the execution time through the critical path. The causal resource distribution may also be used to identify how changes in computational resource allocation in different paths of the multi-path neural network may affect the overall execution time of the neural network.

[0088] Figure 3 An example multipath neural network 300 is depicted having two paths 301 and 302 between an input 303 and a decision block 304. Path 301 may include five sequentially dependent layers, and path 302 may include four sequentially dependent layers. Figure 3 , path 302 is a critical path through neural network 300 that limits the overall throughput of neural network 300. That is, computations and processing along path 302 take longer to complete than computations and processing along path 301. The causal resource distribution may initially determine a baseline execution time through critical path 302 based on a baseline computational resource allocation. A baseline execution time for each of the other paths through neural network 300 may also be determined based on the baseline computational resource allocation. The computational resource allocation may be changed according to different layers in the critical path to determine how changes in the computational resource allocation affect the execution time of the critical path.

[0089] Causal resource distribution involves a mapping between resource allocation and execution time, since the execution time of a layer of a neural network depends on the resource allocation. For example, a 20% speedup of a particular function (i.e., layer) in critical path 302 can potentially improve (i.e., reduce) the overall runtime of neural network application 300 by 5%.

[0090] The causal resource distribution may be extended to also analyze other paths of the multipath neural network to determine how changing the computational resources allocated to different layers of the various paths of the multipath neural network may change the execution time associated with the critical path. Allocating fewer device cores to a particular layer of the neural network may increase the execution time of that layer, but may advantageously free up computational resources for use in advantageously reducing the execution time of another layer in another path.

[0091] In one embodiment, multiple simulations may be run to determine the performance of a multi-path neural network based on different computational resource allocations to different layers and paths of the neural network. An optimal resource allocation may be determined based on the estimated execution time of the neural network for different computational resource allocations.

[0092] Figure 4ADepicting the relative baseline execution times and sequences of a portion of two example paths through a multipath neural network. A first data stream 401 is executed through a series of layers (functions) representing a first path through a multipath neural network (i.e., a Maxpool layer and a fully connected (FC) layer). A second data stream 402 is executed through a series of layers representing a second path through the neural network (i.e., a Convolutional layer and a Rectified Linear Unit (ReLu) layer). Although neither path is identified as a critical path, Figure 4A The upper path depicts that it takes longer to complete execution (dashed line) than the lower path.

[0093] Figure 4A Each of the layers or functions in contains an indication of the memory compute resources that have been allocated to the layer's function. For example, the max pooling layer has been allocated 1GB of memory. The FC layer has been allocated 0.5GB. The convolutional layer has been allocated 5GB, and the ReLu layer has been allocated 1GB. Consider this example where a maximum of 5GB of memory can be allocated to any layer in any path. Therefore, Figure 4A The memory allocation depicted in does not cause the layers in either path to execute in parallel with the convolutional layers, because all of the available 5GB of memory has been allocated to the convolutional layers.

[0094] Figure 4B 4.5GB has been allocated for the convolutional layers. Figure 4A ) is reduced by 10%, and the FC layer in the upper path (stream 401) can be executed simultaneously with the convolutional layer. The overall execution time of the multi-path neural network (i.e., two paths) can also be reduced (dashed line) because the FC layer can now be executed in parallel with the convolutional layer because 0.5GB of memory is available to execute the FC layer.

[0095] Figure 4C 4.0 GB of memory has been allocated to the convolutional layers. Figure 4A ) is reduced by 20%, the max pooling layer in the upper path (stream 401) can be executed in parallel with the convolutional layer in the lower path. In addition, when the max pooling layer is completed, the FC layer in the upper path can be executed in parallel with the convolutional layer. When the convolutional layer is completed, the ReLu layer in the lower path can be executed in parallel with the FC layer. The overall execution time (dashed line) of the multi-path neural network (i.e., two paths) can be further reduced by reducing the memory allocation to the convolutional layer. Although in Figure 4B and Figure 4C , the convolutional layer may take slightly longer to execute based on the reduced memory allocation, but the overall execution time for both paths may be significantly reduced. The example reduction in memory allocation for the convolutional layer may be statically applied to the neural network so that each subsequent instantiation of the multipath neural network runs with an overall reduced execution time. Alternatively, the example reduction in memory allocation may be dynamically determined and applied to the current instantiation of the neural network to account for network changes, such as changes in topology, input, batch size, etc.

[0096] A cost model can be used for causal resource distribution of commonly used layers in a neural network, the model relating the execution time of each layer to the computational resource utilization. Computational resources that may be considered during causal resource distribution may include device computational units, memory space, and memory bandwidth. For a CPU's computational units, the number of cores may be considered. For a GPU's computational units, the number of streaming multiprocessors (SMs) may be considered. For a Field Programmable Gate Array (FPGA), the area of ​​the FPGA may be considered. The unit of resource allocation may typically be a discrete value (e.g., number of cores, memory / workspace for different implementations of a layer).

[0097] A definition that can be used for causal resource distribution performance analysis of multipath neural networks can be as follows. For the input of the multipath neural network, k execution streams are input into the k paths of the multipath neural network respectively. For each corresponding s stream, there are n s Depends on layer l in order. For each layer l in the path, there are m l different implementations. Each implementation has execution time t ml,k and the resource set {w r |r∈resource}. The goal of the definition is to find the parallel schedule and implementation of all layers that minimizes the overall execution time of the multipath neural network, as in equation (1).

[0098] against resource,

[0099] Where K is the total number of layers, w r,l is the utilization of resource r multiplied by layer l, M r is the maximum available resource layer r for the execution of the multipath neural network, and if layer l is executed at time t, then α l,t is 1 if executed at that time, and is 0 if not executed at that time.

[0100] Several heuristics can be used to reduce the complexity of resource allocation. For example, the search space can be pruned. For example, the implementation for layer l can be pruned if there is another implementation j for this layer, where t i >t j and Resources, w r,i >w r,j . You can also prune implementations that have relatively long execution times. For example, if t k >β×minimum{t i |i∈layerLimplementation}, then implementation k for layer L can be pruned because a long-running implementation is unlikely to be part of the optimal solution even if such an implementation may provide relatively small resource utilization.

[0101] Another technique to reduce the complexity of resource allocation may be to use window-like resource allocation of layers. That is, windows or groupings of layers may be formed based on the average or nominal execution time of layers in each path and depth of the network. The window size for each stream or layer may include tunable parameters. The window may be controlled to slide along a multipath neural network path, and performance analysis of the layers inside the window may be performed to find the optimal resource allocation.

[0102] Figure 5 An example of window class resource allocation analysis for layer 500 according to the subject matter disclosed herein is depicted. Figure 5 As depicted in FIG. 5 , a first window 501 may be defined and used to perform a causal resource distribution analysis between layers of a first path 503 and a second path 504 of a multi-path neural network. A second window 502 may be similarly defined to perform another causal resource distribution analysis between layers of the first path 503 and the second path 504. The execution time of each stream may be matched as closely as possible within the same window to fully achieve parallel execution.

[0103] Figure 6 A block diagram depicts an exemplary architecture 600 for a data processing system. In one embodiment, architecture 600 may be used to implement Figure 1 In another embodiment, the architecture 600 may be used to implement the analyzer 100 and run the optimized multi-path neural network provided by the analyzer 100.

[0104] The architecture 600 includes at least one processor, such as a central processing unit (CPU) 601, which is coupled to a storage element 602 via a system bus 603 or other suitable circuit. The architecture 600 can store program code in the storage element 602. The processor 601 can execute program code accessed from the storage element 602 via the system bus 603. Therefore, the processor 601 can act as a dedicated processor. The storage element 602 can include one or more physical memory devices, such as but not limited to a local memory 604 and one or more mass storage devices 605. The local memory 604 can be a random access memory (RAM) or other non-permanent memory device commonly used during the actual execution of the program code. The mass storage device 605 is implemented as a hard disk drive (HDD), a solid-state drive (SSD) and / or other permanent data storage device. Architecture 600 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from mass storage device 605 during execution.

[0105] The architecture 600 may also include input / output (I / O) devices such as a keyboard 606, a display device 607, and / or a pointing device 608 that may be optionally coupled to the architecture 600. In some embodiments, one or more of the I / O devices may be combined in a touch screen that serves as a display device 607. Such a display device 607 may also include a keyboard 606 and a pointing device 608. One or more network adapters 609 may also be coupled to the architecture 600 to enable the architecture 600 to be coupled to other systems, computer systems, remote printers, and / or remote storage devices through an intermediate private or public network. Modems, cable modems, Ethernet cards, and wireless transceivers are non-limiting examples of different types of network adapters that may be used with the architecture 600. Depending on the specific device implemented using the architecture 600, the specific type of one or more network adapters may vary. The I / O devices may be coupled to the architecture 600 directly or through an intermediate I / O controller.

[0106] like Figure 66, storage element 602 may store operating system 610 and application 611. In one aspect, operating system 610 and application 611 implemented in the form of executable program code may be executed by architecture 600. Thus, operating system 610 and application 611 may be considered an integral part of architecture 600. Operating system 610, application 611, and any data items used, generated, and / or manipulated by architecture 600 may be functional data structures that confer functionality when used as part of a system implemented using architecture 600.

[0107] In one arrangement, application 611 may include one or more modules that, when executed by a system using architecture 600 or an architecture similar to architecture 600, may perform the functions described herein. Figures 1 to 5 Various operations and functions described herein. One or more modules may include software, firmware, and / or hardware that combine various components and / or functional blocks to provide the functionality described herein.

[0108] In another arrangement, the architecture 600 may be coupled to the platform 612 via a communication link 613. In one example, the architecture 600 may be coupled to the platform 612 via a network adapter 609. In another example, the architecture 600 may include one or more other I / O devices, such as a Universal Serial Bus (USB) interface or other communication port, which may be used to couple the architecture 600 to the platform 612. The platform 612 may be a circuit board and have a neural network accelerator 614 coupled thereto. In one arrangement, the neural network accelerator 614 may be implemented as an integrated circuit (IC) or multiple ICs. For example, the neural network accelerator 614 may be implemented as one or more programmable ICs, such as a field programmable gate array, one or more application-specific ICs (ASICs), etc.

[0109] The architecture 600 may be configured to perform the operations described herein on a multi-path neural network to allocate computing resources to different layers in different paths of the multi-path neural network, such as Figure 1 That is, architecture 600 may receive multipath neural network 101 via an I / O device, operate on one or more layers of the multipath neural network, and output an optimized computing resource configuration for the multipath neural network according to the subject matter disclosed herein. Architecture 600 may further provide a definition of the optimized multipath neural network 101 to neural network accelerator 614 for execution therein.

[0110] Figure 6This is provided for illustrative purposes only and is therefore not intended as a limitation of the inventive arrangements described herein. In some cases, a particular system implemented using architecture 600 may include fewer or more components than those depicted. Additionally, the particular operating system and / or application included as part of architecture 600 may vary.

[0111] As those skilled in the art will recognize, the innovative concepts described herein may be modified and varied through a wide range of applications. Accordingly, the scope of the claimed subject matter should not be limited to any of the specific exemplary teachings discussed above, but instead should be defined by the following claims.

Claims

1. A computing device having a processing unit for running a multipath neural network, wherein the steps performed by the multipath neural network include: two or more paths through the multipath neural network, a first path comprising one or more layers, the first path comprising a critical path through the multipath neural network, the critical path limiting a throughput of the multipath neural network, a first layer of the critical path being allocated a first computational resource allocation capable of executing the multipath neural network, the first computational resource allocation reducing an execution time of the multipath neural network to less than a baseline execution time for a second computational resource allocation for the multipath neural network, the first computational resource allocation for the first layer being different from the second computational resource allocation for the first layer, The first computing resource allocation and the second computing resource allocation partition the computing resources within the graphics processing unit.

2. The computing device of claim 1, wherein the first computing resource allocation for the first layer is allocated prior to executing the multipath neural network.

3. The computing device of claim 1, wherein the first computing resource allocation for the first layer is allocated during execution of the multipath neural network.

4. The computing device of claim 1, wherein computing resources capable of executing the multi-path neural network include a plurality of computing unit cores, a memory size, and a memory bandwidth.

5. The computing device of claim 1, wherein the critical path includes a layer that provides a convolution function.

6. A method for allocating resources to layers of a multipath neural network, the method comprising: determining a baseline execution time for a multipath neural network based on a first computational resource allocation that allocates computational resources capable of executing the multipath neural network to the multipath neural network, the multipath neural network comprising two or more paths through the multipath neural network; iteratively determining execution times of the two or more paths of the multipath neural network through one or more second computational resource allocations for the two or more paths, each second computational resource allocation being different from the first computational resource allocation; as well as identifying a second allocation of computational resources for a first layer in a first path, the second allocation of computational resources for the first layer providing an overall execution time for the multi-path neural network that is less than an overall baseline execution time for the multi-path neural network, The first computing resource allocation and the second computing resource allocation partition the computing resources within the graphics processing unit.

7. The method of claim 6, wherein the overall baseline execution time for the multi-path neural network corresponds to an execution time of a critical path through the multi-path neural network that limits a throughput of the multi-path neural network.

8. The method of claim 6 further comprising allocating the second computing resource allocation for the first layer in the identified first path to provide the overall execution time for the multi-path neural network that is less than the overall baseline execution time for the multi-path neural network.

9. The method of claim 8, further comprising allocating the second computing resource allocation for the first layer in the identified first path prior to executing the multi-path neural network.

10. The method of claim 8, further comprising allocating the second computational resource allocation for the first layer in the identified first path during execution of the multi-path neural network.

11. The method of claim 6, wherein identifying the second computing resource allocation further comprises receiving a selection of the second computing resource allocation from a user.

12. The method of claim 6, wherein computing resources capable of executing the multi-path neural network include a number of computing unit cores, memory size, and memory bandwidth.

13. The method of claim 6, wherein at least one of the two or more paths includes a layer providing a convolution function.

14. A multipath neural network analyzer, comprising: an interface that receives a multipath neural network, the multipath neural network comprising two or more paths through the multipath neural network and a first path of the two or more paths comprising one or more layers; and a processing device that generates a multipath neural network including one or more layers of a critical path through the multipath neural network, the one or more layers of the critical path being allocated a first computational resource allocation capable of executing the multipath neural network, the critical path limiting a throughput of the multipath neural network, the first computational resource allocation reducing an execution time of the multipath neural network to less than a baseline execution time of a second computational resource allocation for the multipath neural network, the first computational resource allocation for a first layer of the critical path being different from the second computational resource allocation for the first layer of the critical path, The first computing resource allocation and the second computing resource allocation partition the computing resources within the graphics processing unit.

15. The multi-path neural network analyzer of claim 14, wherein computing resources capable of executing the multi-path neural network include a plurality of computing unit cores, memory size, and memory bandwidth.

16. The multipath neural network analyzer of claim 14, wherein the processing means is further configured to: determining a baseline execution time for two or more paths through the multi-path neural network to execute the multi-path neural network based on the second computational resource allocation, iteratively determining execution times of the two or more paths of the multipath neural network through at least one third computational resource allocation for the two or more paths, each third computational resource allocation being different from the second computational resource allocation; and The third computational resource allocation that provides the execution time for the multi-path neural network that is less than the baseline execution time for the multi-path neural network is identified.

17. The multi-path neural network analyzer of claim 16, wherein the baseline execution time for the multi-path neural network corresponds to an execution time of the critical path through the multi-path neural network for the second computational resource allocation for the critical path.

18. The multi-path neural network analyzer of claim 16, wherein the processing device is further configured to allocate the third computing resource allocation for the critical path as the first computing resource allocation for the critical path.

19. The multi-path neural network analyzer of claim 18, wherein the processing device is configured to allocate the first computing resource allocation for the critical path prior to executing the multi-path neural network.

20. The multi-path neural network analyzer of claim 18, wherein the processing device is configured to allocate the first computational resource allocation for the critical path during execution of the multi-path neural network.

Images