Neural network computing system, neural network computing method and computer system

Abstract

A neural network computing system, a neural network computing method, and a computer system are provided. The neural network computing system includes a processor and a deep learning framework under the control of the processor. The framework obtains model information of a neural network model by reading at least one neural network model file; uses the model information to create a neural network graph of the neural network model; adjusts the neural network graph so that the neural network model corresponds to the operation of a first hardware computing device and the operation of a second hardware computing device; divides the neural network model into a plurality of sub-models including a first sub-model and a second sub-model; pipelines the first hardware computing device and the second hardware computing device by assigning the first sub-model and the second sub-model to the first hardware computing device and the second hardware computing device, respectively; and detects a reduced hardware delay measurement from a plurality of hardware delay measurements obtained by changing at least one of the hardware delays of the first sub-model and the second sub-model.

Description

[0001] This application claims priority from Korean Patent Application No. 10-2019-0103543, filed on August 23, 2019, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates to a neural network computing method and a system including the neural network computing method. Background Art

[0003] An artificial neural network (ANN) is a computational model implemented as software or hardware that uses a large number of artificial neurons connected by connecting lines to mimic the computing power of biological systems. In ANNs, artificial neurons that simplify the functions of biological neurons are used. Artificial neurons are interconnected by connecting lines with strong connection strength to perform human cognitive behavior or learning processes. Recently, research has been conducted on deep learning based on ANNs, and various methods have been studied to improve the processing performance of ANNs related to deep learning.

[0004] To achieve deep learning inference, hardware accelerators can be used. Due to computational constraints, dedicated hardware can use heterogeneous accelerators as heterogeneous systems. Summary of the Invention

[0005] Exemplary embodiments of the present disclosure provide a neural network (NN) computing system that increases processing speed by eliminating stalls using pipelining between heterogeneous hardware computing devices during parallel processing.

[0006] Exemplary embodiments of the present disclosure also provide a NN computing method that increases processing speed by eliminating stalls using pipelining between heterogeneous hardware computing devices during parallel processing.

[0007] Exemplary embodiments of the present disclosure also provide a computer system that increases processing speed by eliminating stalls during parallel processing using pipelining between heterogeneous hardware computing devices.

[0008] According to an exemplary embodiment, a neural network computing system includes a processor and a deep learning framework under the control of the processor. The deep learning framework is configured to obtain model information of a neural network model by reading at least one neural network model file; create a neural network graph of the neural network model using the model information; and adjust the neural network graph so that the neural network model corresponds to the operation of a first hardware computing device and the operation of a second hardware computing device that is different from the operation of the first hardware computing device. The deep learning framework is also configured to: divide the neural network model into a plurality of sub-models including a first sub-model and a second sub-model; pipeline the first hardware computing device and the second hardware computing device by assigning the first sub-model and the second sub-model to the first hardware computing device and the second hardware computing device, respectively; and detect a reduced hardware delay measurement from a plurality of hardware delay measurements obtained by changing at least one of the hardware delays of the first sub-model and the second sub-model.

[0009] According to an exemplary embodiment, a neural network computing method includes: obtaining model information of a neural network model by reading at least one neural network model file; creating a neural network graph of the neural network model using the model information; dividing the neural network model into a plurality of sub-models including a first sub-model and a second sub-model; and pipeline-ing a first hardware computing device and a second hardware computing device by assigning the first sub-model and the second sub-model to the first hardware computing device and the second hardware computing device, respectively. The second hardware computing device performs different operations from the first hardware computing device. The method also includes compiling the first sub-model and the second sub-model into the first hardware computing device and the second hardware computing device, respectively.

[0010] According to an exemplary embodiment, a computer system includes: a processor that controls the overall operation of the computer system; a memory that stores data for controlling the computer system; a deep learning framework controlled by the processor; and multiple hardware computing devices controlled by the deep learning framework. The deep learning framework is configured to: obtain model information of a neural network model by reading at least one neural network model file; create a neural network graph of the neural network model using the model information; and adjust the neural network graph so that the neural network model corresponds to the operation of a first hardware computing device and the operation of a second hardware computing device that is different from the operation of the first hardware computing device. The deep learning framework is also configured to: divide the neural network model into multiple sub-models including a first sub-model and a second sub-model; pipeline the first hardware computing device and the second hardware computing device by assigning the first sub-model and the second sub-model to the first hardware computing device and the second hardware computing device, respectively; and detect a reduced hardware delay measurement from a plurality of hardware delay measurements obtained by changing at least one of the hardware delays of the first sub-model and the second sub-model. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The above and other features of the present disclosure will become more apparent by describing in detail exemplary embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0012] Figure 1 is a block diagram of a computer system according to an exemplary embodiment of the present disclosure.

[0013] Figure 2 is a block diagram of a neural network (NN) computing system according to an exemplary embodiment of the present disclosure.

[0014] Figure 3 According to an exemplary embodiment of the present disclosure Figure 2 Block diagram of the runtime compiler.

[0015] Figure 4 is a block diagram illustrating the operation of a NN computing system according to an exemplary embodiment of the present disclosure.

[0016] Figure 5 is a diagram showing an exemplary embodiment according to the present disclosure Figure 4 Schematic diagram of the NN graph.

[0017] Figure 6 is a diagram showing an exemplary embodiment according to the present disclosure Figure 4 Schematic diagram of the NN subgraph.

[0018] Figure 7 It shows that according to Figure 6 A pipelined timing diagram of an exemplary embodiment of the present invention.

[0019] Figure 8 A NN calculation method according to an exemplary embodiment of the present disclosure is shown.

[0020] Figure 9 A NN calculation method according to an exemplary embodiment of the present disclosure is shown.

[0021] Figure 10 is a block diagram illustrating a NN calculation method according to an exemplary embodiment of the present disclosure.

[0022] Figure 11 is a block diagram illustrating a NN calculation method according to an exemplary embodiment of the present disclosure.

[0023] Figure 12 is a block diagram illustrating a NN calculation method according to an exemplary embodiment of the present disclosure.

[0024] Figure 13 It shows that according to Figure 8 A timing diagram illustrating the benefits of an exemplary embodiment of a NN computation method.

[0025] Figure 14 It shows that according to Figure 9 A timing diagram illustrating the benefits of an exemplary embodiment of a NN computation method. DETAILED DESCRIPTION

[0026] Exemplary embodiments of the present disclosure will be described more fully hereinafter with reference to the accompanying drawings.Throughout the drawings, like reference numerals may refer to like elements.

[0027] It will be understood that the terms "first," "second," "third," etc. are used herein to distinguish one element from another, and the elements are not limited by these terms. Thus, the "first" element in an exemplary embodiment may be described as the "second" element in another exemplary embodiment.

[0028] It should be understood that descriptions of features or aspects within each exemplary embodiment should typically be considered as available for other similar features or aspects in other exemplary embodiments, unless the context clearly dictates otherwise.

[0029] As used herein, the singular forms are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0030] Figure 1 is a block diagram of a computer system 1000 according to an exemplary embodiment of the present disclosure.

[0031] The computer system 1000 may analyze input data in real time based on a neural network (NN) to extract effective information, and may determine a situation or control elements of an electronic device mounted on the computer system 1000 based on the extracted information.

[0032] Computer system 1000 may be, for example, an application processor (AP) that can be used in a mobile device. Alternatively, computer system 1000 may be, for example, a robotic device (such as a drone or an advanced driver assistance system (ADAS)), a smart television (TV), a smartphone, a medical device, a mobile device, a display device, a measurement device, or an Internet of Things (IoT) device. However, computer system 1000 is not limited thereto. Hereinafter, computer system 1000 will be described as, for example, an AP.

[0033] Reference Figure 1 , the computer system 1000 may include a processor 100, a deep learning framework 200, a hardware computing device 300, a random access memory (RAM) 400, and a memory 500. At least some of these elements of the computer system 1000 may be mounted on a single semiconductor chip.

[0034] The computer system 1000 can perform neural network (NN) computing functions and can therefore be defined as including a neural network system (NNS). The NNS can include at least some of the elements of the computer system 1000 that can be used in conjunction with NN operations. Figure 1 , NNS may include a processor 100, a deep learning framework 200, and a hardware computing device 300. However, the present disclosure is not limited thereto. For example, in addition to Figure 1 In addition to those shown in FIG, various elements associated with NN operations may be included in the NNS.

[0035] The processor 100 controls the overall operation of the computer system 1000. The processor 100 may include a single processor core or multiple processor cores. The processor 100 may process or execute programs and / or data stored in the memory 500. The processor 100 may control the deep learning framework 200 and the hardware computing device 300 by executing the programs stored in the memory 500.

[0036] The RAM 400 may temporarily store programs, data, or instructions. For example, the programs and / or data stored in the memory 500 may be temporarily stored in the RAM 400 according to the control or startup code of the processor 100. The RAM 400 may be implemented as a memory (such as, for example, a dynamic RAM (DRAM) or a static RAM (SRAM)).

[0037] The memory 500 may store control instruction codes, control data, or user data for controlling the computer system 1000. The memory 500 may include at least one of a volatile memory and a nonvolatile memory. For example, the memory 500 may be implemented as a DRAM, an SRAM, or an embedded DRAM.

[0038] The deep learning framework 200 can perform NN-based tasks based on various types of NNs. The operations required by the NN can be performed by the hardware computing device 300.

[0039] Examples of NN include various types of NNs (such as convolutional neural networks (CNNs) (such as GoogLeNet, AlexNet, or VGG networks), region-CNNs (R-CNNs), region proposal networks (RPNs), recurrent neural networks (RNNs), stacked deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNNs), deconvolution networks, deep belief networks (DBNs), restricted Boltzmann machines (RBMs), fully convolutional networks, long short-term memory (LSTM) networks, and classification networks). However, the present disclosure is not limited thereto.

[0040] A NN that performs a single task may include multiple sub-NNs, which may be implemented as heterogeneous sub-models and may be operated by heterogeneous hardware computing devices 300 .

[0041] The computer system 1000 can execute various types of applications, and the applications can send requests for homogeneous or heterogeneous hardware computing devices 300 to perform operations to the deep learning framework 200. The deep learning framework 200 can allow heterogeneous hardware computing devices 300 to operate in a non-blocking mode, so that the heterogeneous hardware computing devices 300 can simultaneously perform their operations in parallel (i.e., the heterogeneous hardware computing devices 300 can be pipelined). Even in non-blocking mode, the deep learning framework 200 can change the hardware latency of the hardware computing devices 300 to improve hardware utilization and reduce overall hardware latency.

[0042] Figure 2 is a block diagram of a NN computing system according to an exemplary embodiment of the present disclosure.

[0043] Reference Figure 2 , the deep learning framework 200 may include a model parser 210 , a model builder 220 , a model optimizer 230 , a task manager 240 , a model retainer 250 , and a runtime compiler 260 .

[0044] The deep learning framework 200, including each of the model parser 210, the model builder 220, the model optimizer 230, the task manager 240, the model holder 250, and the runtime compiler 260, can be implemented as software, hardware, firmware, or a combination thereof. For example, when these components are implemented as hardware, the components can be implemented by an application specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processor device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a processor including a general-purpose processor, a controller, a microcontroller, a microprocessor, an electronic device, other electronic units designed to perform the functions described in the present disclosure, or a combination thereof.

[0045] The deep learning framework 200 can control the hardware computing device 300 . Figure 2 The hardware computing device 300 is shown to include a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a field programmable gate array (FPGA), a neural processing unit (NPU), and an electronic control unit (ECU). However, the present disclosure is not limited to this. In addition, the hardware computing device 300 may also include a hardware accelerator capable of performing hardware operations. In one embodiment, each hardware computing device 300 may include one or more of a CPU, a GPU, a DSP, an FPGA, an NPU, and an ECU.

[0046] The model parser 210 may read an input NN model file to obtain model information of the input NN model, and may parse various information from the input NN model.

[0047] For example, the model parser 210 may parse various information. The various information may include, for example, layer topology (such as depth and branches), information about compression methods, information about operation types in each layer, data attribute information (such as format, security, and size), memory layout information of operands (such as input, kernel / filter, and output), and information about data compression methods. Kernels / filters may correspond to weights, and memory layout information may include padding, stride, etc.

[0048] The model builder 220 may create an NN graph of the input NN model using the model information acquired by the model parser 210. The NN model may include, for example, an input layer, a hidden layer, and an output layer, and each of these layers may include one or more neurons. The model builder 220 may create an NN graph using the layers of the NN model and the neurons of each layer of the NN model based on the information parsed by the model parser 210.

[0049] The model optimizer 230 can adjust the NN model for which the NN graph has been created by adjusting the NN graph. Because the type of operation required for each hidden layer of each of the multiple sub-models included in the NN model may be different, the type of operation required for each sub-model may also be different. Therefore, the sub-models can be operated by heterogeneous hardware computing devices 300 that perform different operations. The model optimizer 230 can replace, merge, or divide and adjust the sub-models so that the sub-models can correspond to the hardware computing device 300 and the NN graph can be optimized for the hardware computing device 300. For example, the model optimizer 230 can adjust the NN graph so that the NN model corresponds to one or more of the operations of a first hardware computing device 300, the operations of a second hardware computing device 300 different from the operations of the first hardware computing device 300, the operations of a third hardware computing device 300 different from the operations of the first hardware computing device 300 and the operations of the second hardware computing device 300, and the like. As a result, the hardware latency of the hardware computing device 300 can be changed. Therefore, the total hardware latency of the entire NN model can be measured, and a minimum total hardware latency measurement (e.g., a minimum total hardware latency measurement value) can be determined and achieved.

[0050] Although exemplary embodiments are described herein as determining a minimum total hardware delay measurement, the present disclosure is not limited thereto. For example, in exemplary embodiments, a reduced total hardware delay measurement (e.g., a reduced total hardware delay measurement value) that is at least slightly greater than the minimum total hardware delay measurement may be determined. Thus, when reference is made herein to the minimum total hardware delay measurement, that measurement may be substituted for the reduced total hardware delay measurement according to exemplary embodiments.

[0051] The task manager 240 may divide the NN model into a plurality of sub-models and may pipeline the hardware computing device 300 by assigning the sub-models to the hardware computing device 300 .

[0052] Additionally, the task manager 240 may pipeline the hardware computing device 300 by measuring the total hardware latency and determining a minimum total hardware latency measurement.

[0053] The task manager 240 can analyze the hardware capabilities and preferences / policies / runtime context of the host or processor (or all of the considerations of the task manager 240) and can pipeline the hardware computing device 300 by measuring the total hardware latency while adjusting the hardware latency of the hardware computing device 300 and determining the minimum total hardware latency measurement. For example, the hardware latency of the hardware computing device 300 can be changed and the effect of this on the total hardware latency can be observed, thereby allowing the minimum hardware latency measurement to be detected from among multiple hardware latency measurements. Once the minimum total hardware latency measurement is determined, the hardware latency of the hardware computing device 300 can be adjusted to a value that results in the determined minimum total hardware latency measurement. Thus, exemplary embodiments can utilize NN to reduce the total latency of a computing system and improve the operation of the computing system.

[0054] Adjusting the hardware delay of the hardware computing device 300 (e.g., by adjusting the hardware delay of the corresponding sub-model) may include, for example, assigning the sub-model assigned to the hardware computing device 300 with the longest hardware delay to another hardware computing device 300, merging, dividing, or replacing and modifying the operation of the hardware computing device 300, changing the hardware capabilities of the hardware computing device 300, and changing the performance of the hardware computing device 300 (such as the output, frequency, and mode of the hardware computing device 300). In one embodiment, adjusting the hardware delay of the corresponding sub-model may include replacing, merging, or dividing the operation of the hardware computing device 300 according to the assignment between the hardware computing device 300 and the corresponding sub-model.

[0055] The task manager 240 not only adjusts the hardware latency of the hardware computing devices 300, but also adjusts and measures the total hardware latency while adjusting the relationship between the heterogeneous hardware computing devices 300, and pipelines the hardware computing devices 300 by determining the minimum total hardware latency measurement. In addition, the task manager 240 can pipeline the hardware computing devices 300 by determining the minimum total hardware latency using a specific method specified in the NN model file. For example, the task manager 240 can pipeline the hardware computing devices 300 based on the parameters defined in each NN model file.

[0056] Adjusting the relationship between heterogeneous hardware computing devices 300 may include, for example, changing available hardware computing devices 300 according to dynamic hardware scheduling, changing operation paths between hardware computing devices 300, and adding and / or modifying pre-processing or post-processing by changing the operation paths.

[0057] The addition and / or modification of pre-processing and post-processing may include performing quantization before or after operation of the DSP if a DSP is included in the operation path, and adding data layout and adding input rearrangement and / or weight rearrangement for each hardware computing device 300 before operation of the GPU if a GPU is included in the operation path.

[0058] The model holder 250 may temporarily store model information of a sub-model that has been compiled into the hardware computing device 300 by the runtime compiler 260 or has been pre-compiled.

[0059] Figure 3 According to an exemplary embodiment of the present disclosure Figure 2 A block diagram of the runtime compiler 260 is shown.

[0060] Reference Figure 2 and Figure 3 , the runtime compiler 260 is included in the deep learning framework 200. In addition, a compiler 261 (NPU_RC), a compiler 262 (GPU_RC), a compiler 263 (CPU_RC), and a compiler 264 (DSP_RC) dedicated to the hardware computing device 300 may be set. Figure 3 Only a compiler for NPU, a compiler for GPU, a compiler for CPU, and a compiler for DSP are shown, but the present disclosure is not limited thereto, and compilers for other hardware computing devices 300 may also be provided.

[0061] The runtime compiler 260 may perform compilation during runtime and may compile the sub-model assigned to the hardware computing device 300 into the hardware computing device 300 .

[0062] Figure 4 is a block diagram illustrating the operation of a NN computing system according to an exemplary embodiment of the present disclosure.

[0063] Reference Figure 4 , the NN model file may be input to the model parser 210. The input NN model file may be in the form of, for example, tflite, onnx, and prototxt. However, the present disclosure is not limited thereto, and the input NN model file may also include NN model files in other formats besides those set forth herein.

[0064] The model parser 210 can read the input NN model file and obtain and parse the model information of the NN model. The model parser 210 can send the obtained model information to the model builder 220, and the model builder 220 can create a NN graph based on the obtained model information.

[0065] A NN model may include multiple sub-models, each with a hidden layer.

[0066] The model builder 220 may send the NN model to the adaptive path manager 270. The adaptive path manager 270 may include Figure 2 The model optimizer 230 and the task manager 240 are described below.

[0067] Thus, the NN model can be divided into sub-models, and the sub-models can be assigned to the hardware computing device 300 so that the hardware computing device 300 can be pipelined. Then, the total hardware delay can be measured while adjusting the hardware delay of the hardware computing device 300, and the minimum total hardware delay measurement can be found. Alternatively, the pipelining of the hardware computing device 300 can be performed by determining the minimum total hardware delay measurement in a specific method specified in each of the input NN model files.

[0068] The sub-model may be assigned to the hardware computing device 300 to correspond to the minimum total hardware delay measurement, and the runtime compiler 260 may compile the sub-model into the hardware computing device 300 .

[0069] Figure 5 is a diagram showing an exemplary embodiment according to the present disclosure Figure 4 Schematic diagram of the NN graph.

[0070] Reference Figure 4 and Figure 5 , the model builder 220 may send the NN graph to the adaptive path manager 270 .

[0071] NN may include an input layer, a hidden layer, and an output layer. NN may perform operations based on input data (eg, I1 and I2) and may generate output data (eg, O1 and O2) based on the results of the operations.

[0072] The NN may be a deep neural network (DNN) comprising two or more hidden layers or n-layer NNs. For example, Figure 5 As shown in , the NN may be a DNN including an input layer 10 , a first hidden layer 12 , a second hidden layer 14 , and an output layer 16 .

[0073] In the case where the NN is a DNN, since the NN includes multiple layers from which effective information is extracted, the NN can process complex data sets. Figure 5, the NN is shown as including four layers. However, the present disclosure is not limited thereto. For example, the number of layers included in the NN may vary.

[0074] Each layer of a NN may include multiple neurons. Neurons may correspond to, for example, processing elements (PEs), units, or artificial nodes. For example, Figure 5 As shown in , the input layer 10 may include two neurons (or nodes), and each of the first hidden layer 12 and the second hidden layer 14 may include three neurons (or nodes). The first hidden layer 12 may be operated by an NPU, and the second hidden layer 14 may be operated by a GPU. However, the present disclosure is not limited thereto. The number of neurons (or nodes) included in each layer of the NN may vary, the layers of the NN may perform operations different from those described herein, and the layers of the NN may be operated by hardware computing devices 300 different from those described herein.

[0075] The neurons included in each layer of the NN can be connected to each other and thus can exchange data with each other. A single neuron can receive data from other neurons to perform an operation and can output the result of the operation to other neurons.

[0076] The input and output of each neuron (or node) may be referred to as input activation and output activation, respectively. For example, activation may be a parameter corresponding not only to the output of a neuron but also to the input of neurons included in subsequent layers.

[0077] Each neuron can be based on an activation function (e.g., σ), a weight (e.g., and ) and bias (e.g., and ) and the activations received from neurons included in the previous layer (e.g., and ), to determine its activation (e.g., and ).

[0078] Weights and biases are parameters used to calculate the output activation in each neuron. Weights are the values ​​assigned to the connections between neurons, and biases are the values ​​associated with each neuron.

[0079] For each neuron to determine its activation (ie, for determining the output of each layer), a layer of a NN may include at least one operation.

[0080] A NN having a multi-layer structure may include multiple operations and may require a large amount of computation to process input data to generate output data.

[0081] Figure 6is a diagram showing an exemplary embodiment according to the present disclosure Figure 4 Schematic diagram of the NN subgraph.

[0082] Reference Figure 4 and Figure 6 , the model builder 220 may send the NN graph to the adaptive path manager 270 .

[0083] The NN graph may include a plurality of hidden layers (a first hidden layer 22, a second hidden layer 24, a third hidden layer 26, and a fourth hidden layer 28), an input layer "Input", and an output layer "Output".

[0084] The “Conv 1×1” operation may be performed by the NPU in the first hidden layer 22. The “Concatenate” operation may be performed by the GPU in the second hidden layer 24 that receives the output activations of the first hidden layer 22. The “Conv 1×1” operation and the “Conv 3×3” operation may be performed by the NPU in the third hidden layer 26 that receives the output activations of the second hidden layer 24. The “Concatenate” operation may be performed by the GPU in the fourth hidden layer 28 that receives the output activations of the third hidden layer 26, and the GPU may send the output activations of the fourth hidden layer 28 to the output layer “Output”.

[0085] The hardware computing device 300 may be assigned to each of the first hidden layer 22, the second hidden layer 24, the third hidden layer 26, and the fourth hidden layer 28. Because the first hidden layer 22, the second hidden layer 24, the third hidden layer 26, and the fourth hidden layer 28 are included in and occupy part of the NN graph, the first hidden layer 22, the second hidden layer 24, the third hidden layer 26, and the fourth hidden layer 28 may be referred to as a subgroup of the NN or a submodel of the NN.

[0086] Figure 6 The first hidden layer 22, the second hidden layer 24, the third hidden layer 26, and the fourth hidden layer 28 of the NN can be NN subgraphs and can be submodels of the NN. Therefore, when the NN is used with a heterogeneous hardware accelerator, the NN subgraph of the NN can also be used.

[0087] Figure 7 It shows that according to Figure 6 A pipelined timing diagram of an exemplary embodiment of the present invention.

[0088] Reference Figure 6 and Figure 7 , inference can be made from the input layer “Input” to the output layer “Output” through the first hidden layer 22 , the second hidden layer 24 , the third hidden layer 26 and the fourth hidden layer 28 .

[0089] exist Figure 7In the example of , two inferences are performed. In the first inference, the operation OP in the first hidden layer 22 22 1 The operation OP in the second hidden layer 24 can be performed by the NPU. 24 1 The operation OP in the third hidden layer 26 can be performed by the GPU. 26 1 The operation OP in the fourth hidden layer 28 may be performed by the NPU. 28 1 Can be executed by GPU.

[0090] In the second inference, the operation OP in the first hidden layer 22 22 2 The operation OP in the second hidden layer 24 can be performed by the NPU. 24 2 The operation OP in the third hidden layer 26 can be performed by the GPU. 26 2 The operation OP in the fourth hidden layer 28 may be performed by the NPU. 28 2 Can be executed by GPU.

[0091] In blocking mode, operate OP 24 1 Can be processed by NPU operation OP 22 1 Then start. When operating OP 24 1 The NPU does not perform operations while the GPU is executing them. 24 1 Then start operating OP 26 1 In one exemplary embodiment, until operation OP 24 1 and Operation OP 26 1 Both of them are completed before the GPU performs the operation.

[0092] In blocking mode, the operation of one hardware computing device 300 may start only after the operation of another hardware computing device 300. In the second inference, as in the first inference, the operation OP of the NPU 22 2 Can only be operated on GPU OP 28 1 Then start.

[0093] Similarly, operation OP 24 2 Can only be operated on NPU OP 222 Then start. When operating OP 24 2 The NPU does not perform operations while the GPU is executing them. 24 2 Then start operating OP 26 2 In one exemplary embodiment, until operation OP 24 2 and Operation OP 26 2 Both of them are completed before the GPU performs the operation.

[0094] In non-blocking mode, the first inference starts in the NPU and the operation OP in the first inference 22 1 Afterwards, the operation OP in the second inference 22 2 and the operation OP in the first inference 24 1 It can be started in NPU and GPU respectively.

[0095] Therefore, the OP 22 2 Available in OP 22 1 Then prepare to execute in NPU, operation OP 28 2 Operations available in NPU 26 1 And then the operation OP in NPU 26 2 Then start.

[0096] Operations OP in GPU 24 1 And then the operation OP in NPU 22 2 After that, operate OP 24 2 You can start. After that, operate OP 28 2 Operations available in NPU 26 2 Then start.

[0097] In this way, hardware utilization in non-blocking mode can be improved and, as a result, overall hardware latency can be reduced.

[0098] Figure 8 A NN calculation method according to an exemplary embodiment of the present disclosure is shown.

[0099] Figure 8The portion "i" is a diagram showing the Figure 6 Operational block diagram of the NN model. Figure 2 、 Figure 6 and Figure 8 The task manager 240 can dispatch a portion of the sub-model of the hardware computing device 300 with the longest hardware delay (e.g., the longest hardware delay relative to other hardware computing devices 300) to another hardware computing device 300 (e.g., by assigning the operation OP of the NPU to the sub-model of the hardware computing device 300 with the longest hardware delay (e.g., the longest hardware delay relative to other hardware computing devices 300)). 22 Operations OP dispatched to the GPU 24 ), to change the hardware delay of the hardware computing device 300. Therefore, the hardware delay of the NPU and GPU can be changed by changing the operation of the NPU (for example, the operation of the OP 22 and Operation OP 26 ) (for example, in Figure 8 In, operate OP 22 ) is dispatched to the GPU to change.

[0100] Reference Figure 8 Part "ii", Operation OP 24 , Operation OP 26 and Operation OP 28 It can be operated sequentially after the input layer "Input" and can then be transmitted sequentially to the output layer "Output".

[0101] Figure 9 A NN calculation method according to an exemplary embodiment of the present disclosure is shown.

[0102] Figure 9 The portion "i" is a diagram showing the Figure 6 Operational block diagram of the NN model. Figure 2 、 Figure 6 and Figure 9 , the model optimizer 230 or the task manager 240 may change the hardware delay of the hardware computing device 300 by merging, dividing, or replacing operations of the hardware computing device 300 .

[0103] Figure 9 Part "ii" is to convert the NPU operation OP 22 and GPU operation OP 24 Merge into a single operation (i.e., operation OP 30 ) and obtain the operation block diagram.

[0104] Reference Figure 9 Part "ii", Operation OP 30 , Operation OP 26 and Operation OP 28It can be operated sequentially after the input layer "Input" and can then be transmitted sequentially to the output layer "Output".

[0105] Figure 10 is a block diagram illustrating a NN calculation method according to an exemplary embodiment of the present disclosure.

[0106] Reference Figure 2 and Figure 10 , the hardware latency of the hardware computing device 300 can be changed by changing the relationship between the heterogeneous hardware computing devices 300 via the task manager 240, and the minimum total hardware latency measure can be found by adding and / or modifying pre-processing or post-processing according to the change of the operation path.

[0107] according to Figure 10 In an exemplary embodiment, a GPU may be added to the operation path. In the case where a GPU is added as the hardware computing device 300, data layout may be processed first, and operations of the GPU may be performed.

[0108] Data layout is a method of converting data into a specific format (such as the format of an image file) before performing calculations on the data or storing the data. Examples of the specific format may include NCHW, NHWC, CHWN, nChw8c, and nChw16c.

[0109] If the GPU operation is operation OP 24 , the data layout can be obtained by operating OP 22 Receive output activation is performed. As a result, the hardware latency of the GPU can be changed.

[0110] Figure 11 is a block diagram illustrating a NN calculation method according to an exemplary embodiment of the present disclosure.

[0111] Reference Figure 2 and Figure 11 , a DSP may be added to the operation path. In the case where a DSP is added as the hardware computing device 300, quantization may be performed, and then the operation of the DSP may be performed. Thereafter, dequantization may be performed.

[0112] For example, in the case where the dedicated NPU is operated in 32-bit units, 8-bit quantization may be performed before the input of the operation of the DSP, and 32-bit inverse quantization may be performed after the operation of the DSP.

[0113] In operation OP 24 In the case of DSP operation, quantization can be performed in the operation OP 22 The output is then performed, and inverse quantization can be performed in the operation OP 26 is executed before the input.

[0114] Figure 12 is a block diagram illustrating a NN calculation method according to an exemplary embodiment of the present disclosure.

[0115] Reference Figure 2 and Figure 12 , any hardware computing device C may be installed in the operation path, and input rearrangement and / or weight rearrangement may be performed before the operation of the hardware computing device C.

[0116] For example, when optimizing the operation of hardware computing device C for matrix multiplication and operating OP 22 When outputting in Fmap format, operate OP 22 It can be converted into a "matrix" before the input of the operation of the hardware computing device C. Even in the case of receiving the same output value, input rearrangement and / or weight rearrangement for preparing data in advance in the hardware computing device may be added.

[0117] Reference Figure 12 , can be operated in OP 22 Add input rearrangement and / or weight rearrangement after the output.

[0118] Figure 13 It shows that according to Figure 8 A timing diagram illustrating the benefits of an exemplary embodiment of a NN computation method.

[0119] Reference Figure 8 and Figure 13 , NPU operation OP 22 1 and Operation OP 22 2 can be dispatched to the GPU. Therefore, the operation OP 22 1 and Operation OP 22 2 But it seems that they are related to the operation OP 24 1 and Operation OP 24 2 Perform the same operation as merging.

[0120] Reference Figure 13 , operation OP 24 1 You can start in the GPU, then operate OP 24 2 Can be started in GPU. Operation OP 24 2 After that, operate OP 28 1 and Operation OP 28 2Can be operated in OP 26 1 and Operation OP 26 2 Then it starts in the GPU.

[0121] Reference Figure 13 Parts "i" and "ii" of the embodiment can reduce the total hardware delay in the NN by changing the hardware delay of each hardware computing device 300 through the dispatch of operations. As a result, stalls can be eliminated. For example, the total hardware delay can be reduced by improving hardware utilization.

[0122] Figure 14 It shows that according to Figure 9 A timing diagram illustrating the benefits of an exemplary embodiment of a NN computation method.

[0123] Reference Figure 9 and Figure 14 , NPU operation OP 22 1 and Operation OP 22 2 Compatible with GPU operations 24 1 and Operation OP 24 2 Merge to create operation OP 30 1 and Operation OP 30 2 .

[0124] As a result, stalls can be eliminated and overall hardware latency can be reduced.

[0125] As is conventional in the field of the present disclosure, exemplary embodiments are described and illustrated in the accompanying drawings in terms of functional blocks, units, and / or modules. It will be understood by those skilled in the art that these blocks, units, and / or modules are physically implemented by electronic (or optical) circuits (such as logic circuits, discrete components, microprocessors, hard-wired circuits, memory elements, wiring connections, etc.) that can be formed using semiconductor-based manufacturing techniques or other manufacturing techniques. Where blocks, units, and / or modules are implemented by microprocessors or the like, they can be programmed using software (e.g., microcode) to perform the various functions discussed herein and can optionally be driven by firmware and / or software. Alternatively, each block, unit, and / or module can be implemented by dedicated hardware, or as a combination of dedicated hardware for performing some functions and a processor (e.g., one or more programmed microprocessors and associated circuits) for performing other functions.

[0126] As will be appreciated by those skilled in the art, aspects of the present disclosure may be implemented as a system, method, or computer program product. Thus, aspects of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment (including firmware, resident software, microcode, etc.), or a combination of software and hardware aspects, which may generally be referred to herein as a "circuit," "module," or "system." Furthermore, aspects of the present disclosure may take the form of a computer program product implemented in one or more computer-readable media having a computer-readable program code implemented thereon.

[0127] Herein, the term "circuit" may refer to either an analog circuit or a digital circuit. In the case of a digital circuit, the digital circuit may be hardwired to perform the corresponding task of the circuit (such as a digital processor that executes instructions to perform the corresponding task of the circuit). Examples of such processors include application-specific integrated circuits (ASICs) and field-programmable gate arrays (FPGAs).

[0128] While the present disclosure has been particularly shown and described with reference to exemplary embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present disclosure as defined by the claims.

Claims

1. A neural network computing system comprising: processor; and A deep learning framework under control of a processor, wherein the deep learning framework is configured to: Obtaining model information of the neural network model by reading at least one neural network model file; Using the model information to create a neural network graph of the neural network model; adjusting the neural network graph so that the neural network model corresponds to operations of the first hardware computing device and operations of a second hardware computing device that is different from the operations of the first hardware computing device; Dividing the neural network model into a plurality of sub-models including a first sub-model and a second sub-model; Pipelining the first hardware computing device and the second hardware computing device by assigning the first sub-model and the second sub-model to the first hardware computing device and the second hardware computing device, respectively, so that the first hardware computing device and the second hardware computing device simultaneously perform operations of the first hardware computing device and operations of the second hardware computing device in parallel; detecting a reduced total hardware delay measurement value from among a plurality of total hardware delay measurement values ​​obtained by varying at least one of the hardware delays of the first sub-model and the second sub-model; and Once a reduced total hardware delay measurement is detected, at least one of the hardware delays of the first sub-model and the second sub-model is adjusted to a value that results in the detected reduced total hardware delay measurement.

2. The neural network computing system according to claim 1, wherein: The step of changing at least one of the hardware delays of the first sub-model and the second sub-model includes replacing, merging, or dividing the first sub-model and the second sub-model.

3. The neural network computing system according to claim 1, wherein: Changing at least one of the hardware delays of the first sub-model and the second sub-model includes dispatching a portion of the operation of the first hardware computing device having the longest hardware delay to the second hardware computing device.

4. The neural network computing system according to claim 1, wherein: Changing at least one of the hardware delays of the first sub-model and the second sub-model includes replacing, merging, or dividing operations of the first hardware computing device and operations of the second hardware computing device.

5. The neural network computing system according to claim 1, wherein: Changing at least one of the hardware delays of the first sub-model and the second sub-model includes changing an output, a frequency, or a mode of the first hardware computing device or the second hardware computing device.

6. The neural network computing system according to claim 1, wherein: Changing at least one of the hardware delays of the first sub-model and the second sub-model includes changing a hardware capability of the first hardware computing device or the second hardware computing device.

7. A neural network calculation method, comprising: Obtaining model information of the neural network model by reading at least one neural network model file; Using the model information to create a neural network graph of the neural network model; Dividing the neural network model into a plurality of sub-models including a first sub-model and a second sub-model; pipeline the first hardware computing device and the second hardware computing device by assigning the first sub-model and the second sub-model to the first hardware computing device and the second hardware computing device, respectively, so that the first hardware computing device and the second hardware computing device simultaneously perform an operation of the first hardware computing device and an operation of the second hardware computing device in parallel, wherein the operation of the first hardware computing device is different from the operation of the second hardware computing device; Compiling the first sub-model and the second sub-model into the first hardware computing device and the second hardware computing device respectively; detecting a reduced total hardware delay measurement value from among a plurality of total hardware delay measurement values ​​obtained by varying at least one of the hardware delays of the first sub-model and the second sub-model; and Once a reduced total hardware delay measurement is detected, at least one of the hardware delays of the first sub-model and the second sub-model is adjusted to a value that results in the detected reduced total hardware delay measurement.

8. The neural network calculation method according to claim 7, wherein: Changing at least one of the hardware delays of the first sub-model and the second sub-model includes dispatching a portion of the operation of the first hardware computing device having the longest hardware delay to the second hardware computing device.

9. The neural network calculation method according to claim 7, wherein: The step of changing at least one of the hardware delays of the first sub-model and the second sub-model includes replacing, merging, or dividing the first sub-model and the second sub-model.

10. The neural network calculation method according to claim 7, wherein: The first hardware computing device and the second hardware computing device are pipelined based on parameters defined in each of the at least one neural network model file.

11. The neural network computing method according to claim 7, further comprising: measuring a total hardware latency by varying a first hardware computing device and a second hardware computing device according to a dynamic hardware schedule; as well as Determine the reduced overall hardware latency measurement.

12. The neural network computing method according to claim 7, further comprising: Measuring total hardware latency by adding and / or modifying pre-processing or post-processing based on changes in the operation path; as well as Determine the reduced overall hardware latency measurement.

13. The neural network computing method according to claim 12, further comprising: When a digital signal processor is included in the operation path, quantization is performed before the operation of the digital signal processor and inverse quantization is performed after the operation of the digital signal processor.

14. The neural network computing method according to claim 12, further comprising: When a graphics processor is included in the operation path, data layout operations are added before the operations of the graphics processor.

15. The neural network computing method according to claim 12, further comprising: When the first hardware computing device or the second hardware computing device is included in the operation path, an input rearrangement operation and / or a weight rearrangement operation is added before the operation of the first hardware computing device.

16. A computer system comprising: a processor to control the operation of the computer system; a memory for storing data for controlling the computer system; Deep learning framework, controlled by the processor; and a plurality of hardware computing devices controlled by a deep learning framework and comprising a first hardware computing device and a second hardware computing device, Among them, the deep learning framework is configured as: Obtaining model information of the neural network model by reading at least one neural network model file; Using the model information to create a neural network graph of the neural network model; adjusting the neural network graph so that the neural network model corresponds to operations of the first hardware computing device and operations of a second hardware computing device that is different from the operations of the first hardware computing device; Dividing the neural network model into a plurality of sub-models including a first sub-model and a second sub-model; Pipelining the first hardware computing device and the second hardware computing device by assigning the first sub-model and the second sub-model to the first hardware computing device and the second hardware computing device, respectively, so that the first hardware computing device and the second hardware computing device simultaneously perform operations of the first hardware computing device and operations of the second hardware computing device in parallel; detecting a reduced total hardware delay measurement value from among a plurality of total hardware delay measurement values ​​obtained by varying at least one of the hardware delays of the first sub-model and the second sub-model; and Once a reduced total hardware delay measurement is detected, at least one of the hardware delays of the first sub-model and the second sub-model is adjusted to a value that results in the detected reduced total hardware delay measurement.

17. The computer system according to claim 16, wherein: Changing at least one of the hardware delays of the first sub-model and the second sub-model includes dispatching a portion of the operation of the first hardware computing device having the longest hardware delay to the second hardware computing device.

18. The computer system according to claim 16, wherein: The step of changing at least one of the hardware delays of the first sub-model and the second sub-model includes replacing, merging, or dividing the first sub-model and the second sub-model.

19. The computer system according to claim 16, wherein: Changing at least one of the hardware delays of the first sub-model and the second sub-model includes changing an output, a frequency, or a mode of the first hardware computing device or the second hardware computing device.

Images