Neural network computing system

Abstract

A neural network computing system includes a processor including heterogeneous computing devices, a memory, a memory controller to control the memory, and a system bus to communicate between the processor and the memory controller. The processor generates a frequency level combination table representing frequency level combinations of hardware devices including a heterogeneous computing device, a memory controller, and a system bus, each frequency level combination maximizes the reduction of the operation time relative to the increment of the energy consumption when the performance level of the neural network model is increased by one level; determining a target performance level; a frequency level combination is selected among the frequency level combinations according to the target performance level, and the hardware device is controlled based on the selected frequency level combination while the neural network model is operated.

Description

[0001] This application claims priority from Korean Patent Application No. 10-2024-0032461 filed on March 7, 2024, in the Korean Intellectual Property Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0002] The present disclosure relates to a neural network computing system and a neural network model operation method. Background Art

[0003] Recently, research on neural networks that mimic the learning ability of the human brain to process information has been actively conducted. Computation based on neural networks can be used to accurately identify and determine objects or specific information from various user data such as voice, images, and videos.

[0004] Processors may include heterogeneous computing devices. Heterogeneous computing devices may include highly general-purpose central processing units (CPUs) and neural processing units (NPUs) optimized for neural network computing. To perform neural network computing, computing devices (such as CPUs) may be used together with NPUs. When performing neural network computing using various hardware devices (including heterogeneous computing devices), power consumption may increase. Summary of the Invention

[0005] On the one hand, a neural network computing system and a neural network model operation method are provided that can reduce the energy consumption of hardware devices while allowing the neural network model to be executed within a target execution time.

[0006] On the other hand, a neural network computing system and a neural network model operation method are provided that can reduce the overhead for finding an operating frequency combination of heterogeneous computing devices, where the operating frequency combination can reduce the energy consumption of the hardware device.

[0007] According to one aspect of one or more embodiments, a neural network computing system is provided, the neural network computing system comprising: a processor including a plurality of heterogeneous computing devices, each having a plurality of operating frequency levels; a memory configured to store data associated with the execution of a neural network model; a memory controller configured to control data input and output of the memory; and a system bus configured to support communication between the processor and the memory controller. The processor is configured to: generate a frequency level combination table representing a plurality of frequency level combinations of hardware devices for a plurality of performance levels of the neural network model, the hardware devices including the memory controller, the system bus, and the plurality of heterogeneous computing devices, wherein each of the plurality of frequency level combinations maximizes a reduction in execution time relative to an increase in energy consumption when the performance level of the neural network model increases by one level; determine a normalized performance value by performing feedback control based on an error between a target execution time of the neural network model and an actual execution time of the neural network model; determine a target performance level among the plurality of performance levels based on the normalized performance value; refer to the frequency level combination table to select a frequency level combination among the plurality of frequency level combinations according to the target performance level; and control the hardware device based on the selected frequency level combination while the neural network model is being executed.

[0008] According to another aspect of one or more embodiments, a neural network computing system is provided, the neural network computing system comprising: a processor including a plurality of heterogeneous computing devices, each having a plurality of operating frequency levels; and a memory configured to store data associated with the execution of a neural network model. The processor is configured to: generate a lowest frequency level combination of the plurality of heterogeneous computing devices for executing the neural network model, and set the lowest frequency level combination as a current frequency level combination of the plurality of heterogeneous computing devices; generate a higher frequency level combination in which an operating frequency level that maximizes a reduction in execution time relative to an increase in energy consumption among the plurality of operating frequency levels included in the current frequency level combination is increased by one level; repeatedly set the higher frequency level combination as the current frequency level combination and generate a higher frequency level combination until the higher frequency level combination corresponds to a highest frequency level combination of the plurality of heterogeneous computing devices; map the generated frequency level combination to a plurality of performance levels of the neural network model; determine a target performance level among the plurality of performance levels by performing feedback control based on an error between a target execution time of the neural network model and an actual execution time of the neural network model; and control the operating frequencies of the plurality of heterogeneous computing devices so that the plurality of heterogeneous computing devices have a frequency level combination corresponding to the target performance level.

[0009] According to another aspect of one or more embodiments, a neural network computing system is provided, the neural network computing system comprising: a processor including a plurality of heterogeneous computing devices each having a plurality of operating frequency levels; a memory configured to store data associated with the execution of a neural network model; a memory controller configured to control data input and output of the memory; and a system bus configured to support communication between the processor and the memory controller. The processor is configured to: trigger frequency adjustment of the neural network model in response to triggering of the neural network model; determine a target performance level of the neural network model based on an error between a target execution time of the neural network model and an actual execution time of the neural network model; determine an operating frequency of each hardware device according to the target performance level with reference to a frequency level combination table, the frequency level combination table including operating frequency combinations of a plurality of hardware devices for executing the neural network model according to the performance level of the neural network model; set a system frequency of each hardware device based on the operating frequency of each of the plurality of heterogeneous computing devices of the plurality of neural network models currently executing, the plurality of neural network models including the neural network model; and execute the neural network model according to the system frequency of each hardware device. The frequency level combination table includes a combination of the multiple operating frequency levels of the multiple hardware devices for the performance level of the neural network model, and the combination of the multiple operating frequency levels of the multiple hardware devices maximizes the reduction in running time relative to the increase in energy consumption when the performance level of the neural network model is increased by one level. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The above and other aspects will be more clearly understood from the following detailed description taken in conjunction with the accompanying drawings.

[0011] Figure 1 is a block diagram illustrating a neural network computing system according to an example embodiment.

[0012] Figure 2 is a graph illustrating a time point at which operating frequency control of a hardware device for executing a neural network model is performed according to example embodiments.

[0013] Figure 3 is a diagram illustrating a reduction in execution time of a neural network model relative to an increase in power consumption of a computing device according to a change in an operating frequency level of the computing device, according to example embodiments.

[0014] Figure 4 is a diagram illustrating a structure of a neural network model according to example embodiments.

[0015] Figure 5 is a schematic block diagram illustrating a neural network computing system according to example embodiments.

[0016] Figure 6 is a diagram specifically illustrating an operating frequency decomposer of a neural network computing system according to example embodiments.

[0017] Figure 7 is a diagram illustrating a frequency level combination table including combinations of operating frequency levels according to performance levels according to example embodiments.

[0018] Figure 8 is a flowchart illustrating a method of finding an operating frequency combination according to performance levels according to example embodiments.

[0019] Figure 9 is a diagram illustrating a method of analyzing a runtime ratio using a directed acyclic graph of a neural network model according to example embodiments.

[0020] Figure 10 is a diagram illustrating a running time of a neural network model and a running time of each computing device in a longest path according to example embodiments.

[0021] Figure 11 is a block diagram illustrating a layered structure of a neural network computing system according to an example embodiment.

[0022] Figure 12 is a block diagram specifically illustrating system software layers of a neural network computing system according to an example embodiment.

[0023] Figure 13 is a diagram illustrating a method of operating a neural network model according to example embodiments.

[0024] Figure 14 is a diagram illustrating a neural network computing system according to example embodiments.

[0025] Figure 15 is a diagram illustrating an electronic device to which a neural network computing system according to example embodiments is applied. DETAILED DESCRIPTION

[0026] Hereinafter, various example embodiments will be described with reference to the accompanying drawings.

[0027] Figure 1 is a block diagram illustrating a neural network computing system according to an example embodiment.

[0028] Reference Figure 1, showing a neural network computing system 100. The neural network computing system 100 can run a neural network model. A neural network model that models the learning method of the human brain for processing information can refer to a model that can accurately identify and determine objects or specific information from various user data (such as voice, images, and videos). However, this is merely an example, and in some example embodiments, the neural network model can be configured to perform other operations besides determining objects or specific information from various user data.

[0029] The neural network computing system 100 may be a mobile system (such as a portable communication terminal (mobile phone), a smart phone, a tablet PC (tablet personal computer), a wearable device, a healthcare device, an Internet of Things (IoT) device, etc.). However, the neural network computing system 100 is not necessarily limited to a mobile system, and in some example embodiments, may be a personal computer, a laptop computer, a server, a media player, or an automotive device (such as a navigation system).

[0030] The neural network computing system 100 may include a plurality of hardware devices such as a processor 110 , a memory controller 120 , a memory 130 , and a system bus 101 . The system bus 101 may support communication between the processor 110 , the memory controller 120 , and the memory 130 .

[0031] The processor 110 may perform neural network calculations using data stored in the memory 130. For example, the neural network calculations may include operations such as reading weights and data of each node included in the neural network model, performing convolution calculations on the data and weights, and storing or outputting calculation results.

[0032] The memory 130 may store data used by the processor 110 to perform neural network calculations. For example, the memory 130 may be loaded with one or more neural network models that can be executed by the processor 110. In some example embodiments, input data and output data of the neural network model may be stored in the memory 130. The memory 130 may include volatile memory (such as dynamic random access memory (DRAM), synchronous DRAM (SDRAM), static RAM (SRAM), and / or resistive RAM (RRAM)), and in some example embodiments, may include non-volatile memory (such as flash memory).

[0033] The memory controller 120 may control an operation of storing data received from the processor 110 in the memory 130 , and an operation of outputting data stored in the memory 130 to the processor 110 .

[0034] The processor 110 may include a plurality of heterogeneous computing devices that process and calculate data, and the plurality of heterogeneous computing devices may include a central processing unit (CPU) 111 , a graphics processing unit (GPU) 112 , a neural processing unit (NPU) 113 , a digital signal processor (DSP) 114 , and the like.

[0035] Specifically, the CPU 111 may be a highly general-purpose computing device. The GPU 112 may be a computing device optimized for parallel computing (such as graphics processing). The NPU 113 (a computing device optimized for neural network computing) may include logic blocks for executing unit calculations primarily for neural network computing (such as convolution computing). The DSP 114 may be a computing device optimized for real-time digital processing of analog signals.

[0036] When processor 110 executes a neural network model, various hardware devices may operate together. For example, to execute the neural network model, multiple heterogeneous computing devices (such as NPU 113, CPU 111, GPU 112, and DSP 114) may operate together. In some example embodiments, memory controller 120 and system bus 101 may operate to read input data of the neural network model and store output data of the neural network model.

[0037] The operating frequencies of the hardware devices including the plurality of heterogeneous computing devices, the memory controller 120, and the system bus 101 may be discretely adjusted. For example, each hardware device may have multiple operating frequency levels.

[0038] To ensure user-perceived performance of the neural network computing system 100, a target runtime may be set for each neural network model. The runtime of a neural network model may refer to the time it takes for the neural network model to receive data and complete data processing.

[0039] For example, in some example embodiments, a target runtime of 10 ms may be set for a neural network model for receiving image frames from a camera application of the neural network computing system 100 and detecting an object. In some example embodiments, a target runtime of 15 ms may be set for a neural network model for recognizing an object. In order to run the above neural network model within the target runtime, heterogeneous computing may be used (i.e., accelerating the execution of the neural network model via multiple heterogeneous computing devices (such as NPU 113, CPU 111, GPU 112, and DSP 114)).

[0040] When hardware devices simultaneously operate at their highest operating frequencies to run a neural network model, the amount of energy consumed by the neural network computing system 100 for running the neural network model may increase. To reduce the energy consumption of the neural network computing system 100, a dynamic voltage and frequency scaling (DVFS) mechanism has been proposed that can dynamically adjust the operating voltage and operating frequency. For example, for each heterogeneous computing device, the Linux kernel's governor can predict future utilization based on past utilization and determine the operating frequency based on the predicted future utilization.

[0041] However, if the neural network computing system determines the operating frequency based on the utilization rate of each heterogeneous computing device without considering the target runtime of the neural network model, the neural network computing system may have difficulty maintaining the target runtime of the neural network model. In other words, the responsiveness (the property of the neural network model to complete and provide a response within the target runtime) of the neural network model may be reduced.

[0042] In some example embodiments, a system (such as a governor) may determine the operating frequency by performing feedback control based on past utilization. However, the system may operate regardless of whether the neural network model is running, and in some cases, the feedback cycle may be several times or even dozens of times longer than the running time of the neural network model. A system that receives delayed feedback and performs operating frequency control after the running of the neural network model is completed may have difficulty immediately controlling the power consumption caused by the running of the neural network model.

[0043] The neural network computing system 100 according to example embodiments may immediately control the operating frequency of a hardware device at a point in time when a neural network model is executed.

[0044] Figure 2 is a graph illustrating a time point at which operating frequency control of a hardware device for executing a neural network model is performed according to example embodiments.

[0045] Figure 2 The graph in FIG shows the operating frequency of any hardware device over time during the period of running the neural network model. Figure 2 The neural network computing system 100 according to an example embodiment may immediately set the operating frequency of the hardware device when the execution of the neural network model begins, may maintain the set operating frequency during the execution of the neural network model, and may release the set operating frequency when the execution of the neural network model ends. That is, the neural network computing system 100 may improve the responsiveness of the neural network model by performing feedforward control on the operating frequency of the hardware device according to whether the neural network model is being executed.

[0046] The neural network computing system 100 may determine an operating frequency level to be set for the operation of the neural network model by performing feedback control based on previous operation results of the neural network model.

[0047] According to example embodiments, the neural network computing system 100 may determine a target performance level for running a neural network model based on an error between a target running time and an actual running time of the neural network model, and may determine a frequency level combination of hardware devices that can minimize energy consumption for running the neural network model while maintaining the target running time of the neural network model.

[0048] The combination of operating frequencies of the hardware devices that can minimize the amount of energy consumed for running a neural network model while providing the same performance level can vary depending on the power consumption characteristics of the hardware devices and the components of the neural network model. For example, the time each hardware device runs a neural network model can vary depending on the structure of the neural network model, and the amount of power consumed can vary depending on the operating frequency of the hardware device. Therefore, the combination of operating frequencies of the hardware devices that can minimize energy consumption can vary depending on the neural network model.

[0049] According to example embodiments, the neural network computing system 100 may find an operating frequency level of a hardware device that can minimize energy consumption while achieving a target performance level based on power consumption characteristics of the hardware device and a structure of a neural network model, and may control the hardware device to have the operating frequency level.

[0050] In the following, according to example embodiments, reference is made to Figure 3 An example of power consumption characteristics of a hardware device is described, and according to an example embodiment, reference is made to Figure 4 Describe the structure of the neural network model.

[0051] Figure 3 is a diagram illustrating a reduction in execution time of a neural network model relative to an increase in power consumption of a computing device according to a change in an operating frequency level of the computing device, according to example embodiments.

[0052] exist Figure 3 In an example, the NPU 113 in the heterogeneous computing device may have multiple operating frequency levels. For example, the NPU 113 may have multiple operating frequency levels having operating frequencies of 160 MHz, 267 MHz, 332 MHz, 533 MHz, 663 MHz, 800 MHz, and 935 MHz, respectively. Figure 3 It shows the amount of reduction in average execution time of the neural network model and the amount of increase in average power consumption of the NPU 113 when the neural network model is executed by increasing the operating frequency level of the NPU 113 by one level.

[0053] Reference Figure 3 As the operating frequency level of the NPU 113 increases, the average power consumption may increase relative to the reduction in average operating time when the operating frequency level increases by one level. That is, as the operating frequency level of the NPU 113 increases, increasing the operating frequency level by one level may not significantly reduce the average operating time but may lead to a decrease in power efficiency.

[0054] As reference Figure 1 As described, the processor 110 of the neural network computing system 100 may include heterogeneous computing devices (such as a CPU 111, a GPU 112, an NPU 113, and a DSP 114). Each heterogeneous computing device may have multiple operating frequency levels, and the power efficiency of the heterogeneous computing devices according to their operating frequency levels may differ from each other. For example, the power consumption at the maximum operating frequency level of the NPU 113 may be three times or more the power consumption at the maximum operating frequency level of the CPU 111.

[0055] To run a single neural network model, heterogeneous computing devices may have different levels of participation. For example, the computational loads of the heterogeneous computing devices used to run the neural network model may differ from one another, and the runtime ratios of the heterogeneous computing devices may differ from one another. The neural network computing system 100 may individually control the operating frequency level of each heterogeneous computing device to reduce the power consumption of the neural network computing system 100 while maintaining the target runtime of the neural network model.

[0056] When the neural network computing system 100 controls the operating frequency levels of heterogeneous computing devices based on the operating time ratio of the heterogeneous computing devices in order to execute a neural network model, it may be difficult to reduce the energy consumption of the neural network computing system 100. For example, in the case where the NPU 113 is most involved in the execution of the neural network model, when the NPU 113 is set to have a relatively high operating frequency level relative to other computing devices, the neural network model may have a shortened execution time, but the NPU 113 may have reduced power efficiency. As a result, the energy consumption of the neural network computing system 100 may increase.

[0057] According to an example embodiment, the neural network computing system 100 may set the operating frequency level of each heterogeneous computing device based on the operating time ratio of the heterogeneous computing devices used to run the neural network model and the power consumption according to the operating frequency level of the heterogeneous computing devices to minimize the energy consumption while maintaining the target operating time of the neural network model.

[0058] Hereinafter, before describing in detail the neural network computing system 100 according to an example embodiment, reference will be made to Figure 4 Describe the structure of the neural network model.

[0059] Figure 4 is a diagram illustrating a structure of a neural network model according to example embodiments.

[0060] Reference Figure 4 , the neural network model NNM may include a plurality of nodes NS, N1 to N12, and NE and a plurality of edges E1 to E16. The neural network model NNM may be represented as a directed acyclic graph. A directed acyclic graph may refer to a graph in which individual elements are oriented in a specific direction and are not cyclic relative to each other.

[0061] Each of the plurality of nodes NS, N1 to N12, and NE may model a neuron (the basic unit of the nervous system). Each of the plurality of nodes NS, N1 to N12, and NE may include source code representing a computation to be executed. The plurality of nodes NS, N1 to N12, and NE may be executed by one of the heterogeneous computing devices included in the processor. The plurality of nodes NS, N1 to N12, and NE may also include attribute values ​​representing a target computing device on which the source code will be executed.

[0062] The plurality of nodes NS, N1 to N12, and NE may include a start node NS, first nodes N1 to twelfth nodes N12, and an end node NE. The start node NS may be a node that externally receives input data for running the neural network model NNM. The end node NE may be a node that outputs result data to the outside after the execution of the neural network model NNM is completed. The first node N1 to twelfth node N12 may be a node for generating result data based on the input data.

[0063] The input / output relationships of multiple nodes NS, N1 to N12, and NE can be represented as multiple edges E1 to E16. For example, the second edge E2 can represent a relationship in which data output from the first node N1 is input to the second node N2. For example, when the CPU 111 executes the first node N1 and the NPU 113 executes the second node N2, the CPU 111 can complete the calculation of the first node N1 and store the output data in the memory 130 via the memory controller 120. The NPU 113 can obtain the data stored in the memory 130 via the memory controller 120 and use the obtained data to execute the second node N2.

[0064] According to an example embodiment, the above reference Figure 1 The described neural network computing system 100 can perform feedback control on the target runtime and actual runtime of a neural network model to determine a target performance level for running the neural network model.

[0065] In some example embodiments, the neural network computing system 100 may analyze the operating time ratios of computing devices that run the plurality of nodes NS, N1 to N12, and NE, and may determine a combination of operating frequency levels capable of achieving a target performance level while minimizing energy consumption based on the operating time ratios of the computing devices and the power consumption according to the operating frequency levels of the computing devices. The neural network computing system 100 may control the operating frequencies of hardware devices including the heterogeneous computing devices based on the combination of the operating frequency levels of the heterogeneous computing devices.

[0066] In the following, reference will be made to Figures 5 to 14 A neural network computing system according to example embodiments is described in detail.

[0067] Figure 5 is a schematic block diagram illustrating a neural network computing system according to example embodiments.

[0068] Reference Figure 5 , the neural network computing system 200 may include a feedback controller 210, an operating frequency decomposer 220, a power profiler 230, a neural network model runner 240, a runtime analyzer 250, and an error calculator 260. In an example embodiment, Figure 5 The neural network computing system 200 in may correspond to Figure 1 The neural network computing system 100 in FIG. The feedback controller 210, the operating frequency decomposer 220, the power analyzer 230, the neural network model runner 240, the runtime analyzer 250, and the error calculator 260 can be implemented in software and can be implemented by hardware devices (such as, for example, Figure 1 The processor 110, memory controller 120, and memory 130 described above operate.

[0069] The neural network model runner 240 may control the operating frequency of the hardware devices included in the neural network computing system 200 based on the control signal output from the operating frequency analyzer 220, and may control the hardware devices to run the neural network model while operating according to the operating frequency. In some example embodiments, the neural network model runner 240 may feed back the running time of each of the nodes and edges of the neural network model to the running time analyzer 250, and may feed back the actual running time of the neural network model to the error calculator 260.

[0070] The runtime analyzer 250 may analyze the runtime of each hardware device and the total runtime of the neural network model based on the runtime of each node and edge of the neural network model. For example, the runtime analyzer 250 may analyze the runtime of each heterogeneous computing device based on the computing device that runs each node of the neural network model and the measured runtime of each node. In some example embodiments, the runtime analyzer 250 may analyze the total runtime of the neural network model based on the sum of the runtimes of the nodes and edges of the neural network model. The runtime of each heterogeneous computing device analyzed may be proportional to the computational load of each heterogeneous computing device, and the total runtime analyzed may be proportional to the total computational load of the neural network model.

[0071] The error calculator 260 may obtain the actual runtime from the neural network model runner 240 and may calculate the error between the target runtime and the actual runtime of the neural network model. In one example embodiment, the target runtime may be externally input.

[0072] The power analyzer 230 may store the power consumption amount according to the operating frequency level of each hardware device and provide the stored power consumption amount to the operating frequency decomposer 220. Even when the hardware device has a preset operating frequency level, the hardware device may have different power consumption amounts in the utilization period (a period in which calculations are performed) and the idle period (idle state). The power analyzer 230 may store both the power consumption amount in the utilization period and the power consumption amount in the idle period according to the operating frequency level.

[0073] The feedback controller 210 may determine a normalized target performance value by performing feedback control based on the error of the runtime obtained from the error calculator 260. The normalized performance value may refer to the performance obtained by normalizing the performance of the neural network computing system 200 based on the operating frequency range of each hardware device, and the performance of the neural network computing system 200 may vary according to the operating frequency range. In one example embodiment, the normalized performance may have a value from "0" to "1". For example, the performance of the neural network computing system 200 when each hardware device operates at the lowest operating frequency level may correspond to a normalized performance value of "0", and the performance of the neural network computing system 200 when each hardware device operates at the highest operating frequency level may correspond to a normalized performance value of "1".

[0074] When feedback controller 210 determines that the actual runtime is shorter than the target runtime based on the runtime error, feedback controller 210 may adjust the normalized target performance value downward. When feedback controller 210 determines that the actual runtime is longer than the target runtime, feedback controller 210 may adjust the normalized target performance value upward. When feedback controller 210 determines that the actual runtime is the same as the target runtime, feedback controller 210 may maintain the normalized target performance value. Thus, the actual runtime can be controlled to be closer to the target runtime. In some example embodiments, feedback controller 210 may significantly adjust the normalized target performance value as the runtime error increases.

[0075] The operating frequency decomposer 220 may determine the operating frequency level of each hardware device based on the normalized target performance value obtained from the feedback controller 210. For example, a neural network model may have multiple performance levels, each having a normalized performance value. The operating frequency decomposer 220 may determine a target performance level close to the normalized target performance value among the multiple performance levels, and may select a combination of frequency levels of the hardware devices corresponding to the target performance level.

[0076] For example, the operating frequency decomposer 220 may determine a frequency level combination that minimizes energy consumption while achieving a target performance level based on the power consumption according to the operating frequency of each hardware device and the operating time ratio of the heterogeneous computing devices analyzed by the operating time analyzer 250.

[0077] The operating frequency decomposer 220 may provide a control signal including a combination of frequency levels to the neural network model operator 240 .

[0078] According to an example embodiment, the neural network computing system 200 may perform feedback control on the operating frequency of a hardware device running the neural network model based on a target running time and an actual running time of the neural network model, thereby minimizing the amount of energy consumption caused by the running of the neural network model while ensuring user responsiveness.

[0079] In the following, reference will be made to Figure 6 A method of determining an operating frequency by an operating frequency resolver according to example embodiments is described in more detail.

[0080] Figure 6 is a diagram specifically illustrating an operating frequency decomposer of a neural network computing system according to example embodiments.

[0081] The operating frequency decomposer 220 may include an operating frequency selector 221, an energy estimator 222, and a performance estimator 223. The operating frequency selector 221 may obtain a target performance level of the neural network model and may refer to the frequency level combination table to find the target performance level of the neural network model. iIn some example embodiments, the operating frequency selector 221 may output the operating frequency level of the hardware device based on the frequency level combination. 、 、 、 、 and .

[0082] The operating frequency selector 221 can generate a frequency level combination table based on the power profile of the computing device and the operating time ratio of the computing device according to the neural network model. The frequency level combination table represents the frequency level combination of the hardware devices that can minimize the energy consumption for running the neural network model for each performance level.

[0083] Specifically, the operating frequency selector 221 can update the combination of operating frequency levels to generate a frequency level combination table, which combination of operating frequency levels can maximize the reduction in the running time of the neural network model relative to the increase in energy consumption of the neural network model when the performance level of the neural network model is increased by one level.

[0084] The operating frequency selector 221 may acquire the increase amount of energy consumption from the energy estimator 222 and the decrease amount of operating time from the performance estimator 223 in order to generate a frequency level combination table.

[0085] The energy estimator 222 may estimate the amount of increase in energy consumption when the operating frequency level of the computing device is increased by one level based on the power distribution from the power analyzer 230 and the operating time ratio of each heterogeneous computing device from the operating time analyzer 250. In some example embodiments, the performance estimator 223 may estimate the amount of decrease in operating time when the operating frequency level of the computing device is increased by one level based on the operating time ratio of each heterogeneous computing device from the operating time analyzer 250.

[0086] When the operating frequency level of the computing device is increased by one level, the operating frequency selector 221 may determine whether the amount of reduction in operating time relative to the amount of increase in energy consumption is maximizeable, and may increase the operating frequency level of the computing device by one level, thereby determining a frequency level combination corresponding to the performance level of the neural network model.

[0087] In the following, reference is made to Figure 7 and Figure 8 A method of generating a frequency level combination table according to example embodiments is described in detail.

[0088] Figure 7is a diagram illustrating a frequency level combination table including combinations of operating frequency levels according to performance levels according to example embodiments, and Figure 8 is a flowchart illustrating a method of finding an operating frequency combination according to performance levels according to example embodiments.

[0089] Reference Figure 7 , a frequency level combination table is shown, which indicates the operating frequency levels of hardware devices including heterogeneous computing devices according to performance levels PL1 to PL46 of the neural network model. Specifically, the operating frequency levels of the heterogeneous computing devices NPU, DSP, GPU and CPU, memory controller MEM, and system bus BUS are shown. According to example embodiments, the NPU, DSP, GPU, CPU, memory controller, and system bus may correspond to NPU 113, DSP 114, GPU 112, CPU 111, memory controller 120, and system bus 101.

[0090] Reference Figure 8 , refer to Figure 6 The described operation frequency selector 221 may perform operations S11 to S16 to generate a reference frequency. Figure 7 Table of frequency level combinations described.

[0091] Reference Figure 7 and Figure 8 In operation S11, the operating frequency selector 221 may generate a lowest frequency level combination of the computing device and may set the lowest frequency level combination as a current frequency level combination of the computing device.

[0092] The lowest frequency level combination of computing devices may refer to a combination in which the operating frequency level of each computing device is the minimum level. In the lowest frequency level combination, each of the operating frequency levels of the computing devices NPU, DSP, GPU, and CPU may be the first level L1.

[0093] exist Figure 7 In the example, each of the operating frequency levels of the memory controller MEM and the system bus BUS may be set to the first level L1, and a method of setting the operating frequency levels of the memory controller MEM and the system bus BUS will be described below.

[0094] In operation S12 , the operating frequency selector 221 may generate a higher frequency level combination in which an operating frequency level “maximizing a reduction in operating time relative to an increase in power consumption” in the current frequency level combination is increased by one level.

[0095] For example, the operating frequency selector 221 may generate a higher frequency level combination by increasing one operating frequency level among the operating frequency levels of the heterogeneous computing devices NPU, DSP, GPU, and CPU by one level.

[0096] In order to determine the operating frequency level that maximizes the reduction in operating time relative to the increase in energy consumption, the operating frequency selector 221 may estimate, for each of the heterogeneous computing devices NPU, DSP, GPU, and CPU, the reduction in operating time and the increase in energy consumption when the operating frequency level of one computing device is increased by one level.

[0097] For example, the amount of reduction in the operating time when the operating frequency level of the computing device p increases by one level may be estimated based on the following [Equation 1].

[0098] [Equation 1]

[0099] In the above [Equation 1], Represents the reduction in running time, t p Indicates by reference Figure 5 and Figure 6 The runtime analyzer 250 described herein analyzes the runtime of the computing device p for the neural network model, represents the operating frequency of computing device p in the current frequency level combination, and represents the operating frequency of the computing device p after the operating frequency level is increased by one level.

[0100] The runtime of computing device p analyzed by runtime analyzer 250 may be proportional to the amount of computation performed by computing device p in executing the neural network model. For example, runtime analyzer 250 may analyze the runtime of computing device p based on the result of at least one execution of the neural network model. The analyzed runtime of computing device p may be used as a value representing the amount of computation performed by computing device p in the neural network model and may be constant regardless of the operating frequency of computing device p.

[0101] When the operating frequency level of the computing device p increases by one level, the amount of reduction in the running time of the neural network model can be determined as the difference between "the running time when the computing device p processes a certain amount of calculation when operating at the current operating frequency" and "the running time when the computing device p processes the same amount of calculation when operating at an operating frequency with a level increased by one level".

[0102] The increase amount of power consumption when the operating frequency level of the computing device p increases by one level may be estimated based on the following [Equation 2].

[0103] [Equation 2]

[0104]

[0105] In the above [Equation 2], represents the increase in energy consumption, represents the energy consumption of the computing device p for the operating frequency f, represents the power consumed in the utilization period of the computing device p for the operating frequency f, represents the power consumed in the idle period of the computing device p for the operating frequency f, represents the running time of computing device p for neural network model analysis, represents the total running time of the analyzed neural network model, represents the operating frequency of computing device p in the current frequency level combination, and represents the operating frequency of the computing device p after the operating frequency level is increased by one level.

[0106] As described above, when the computing device p operates at the operating frequency f, the power consumed during the utilization period of the computing device p may be different from the power consumed during the idle period of the computing device p. Therefore, in the neural network model, the energy consumption of the computing device p for the operating frequency f is can be determined as the running time of the computing device p The energy consumption during the period and the idle time of the computing device p The total energy consumption during the period.

[0107] In some example embodiments, when the operating frequency level of the computing device p increases by one level, the increase in energy consumption for running the neural network model may be determined as the difference between: "the amount of energy consumption when the computing device p operates at an operating frequency having a level increased by one level" and "the amount of energy consumption when the computing device p operates at the current operating frequency" (as expressed in Equation 2).

[0108] The operating frequency selector 221 may estimate the reduction amount of the operating time relative to the increase amount of energy consumption (ie, energy efficiency) by dividing the reduction amount of the operating time by the increase amount of the energy consumption.

[0109] The operating frequency selector 221 may estimate the energy efficiency when the operating frequency level of the NPU is increased by one level in the current frequency level combination, the energy efficiency when the operating frequency level of the DSP is increased by one level in the current frequency level combination, the energy efficiency when the operating frequency level of the GPU is increased by one level in the current frequency level combination, and the energy efficiency when the operating frequency level of the CPU is increased by one level in the current frequency level combination. In some example embodiments, the operating frequency selector 221 may determine the frequency level combination with the highest energy efficiency among the estimated energy efficiencies as the higher frequency level combination. For example, the operating frequency selector 221 may generate the higher frequency level combination by increasing the operating frequency level of the NPU from the first level L1 to the second level L2 in the current frequency level combination, in which all performance levels of the NPU, DSP, GPU, and CPU are the first performance level PL1.

[0110] In operation S13, the operating frequency selector 221 may determine whether the higher frequency level combination corresponds to the highest frequency level combination. Figure 7 In the example, the highest frequency level combination can be a combination in which the operating frequency levels of the heterogeneous computing devices NPU, DSP, GPU and CPU are respectively the eighth level L8, the eighth level L8, the twelfth level L12 and the twenty-first level L21.

[0111] When the higher frequency level combination does not correspond to the highest frequency level combination (No in operation S13 ), the operating frequency selector 221 may set the higher frequency level combination as the current frequency level combination in operation S14 and may repeatedly perform operations S12 and S13 .

[0112] For example, the operating frequency selector 221 may set the frequency level combination (L2, L1, L1, L1) as the current frequency level combination, and may determine a higher frequency level combination corresponding to the third performance level PL3 as (L3, L1, L1, L1) based on energy efficiency. The operating frequency selector 221 may repeatedly perform operations S12 and S13 until the higher frequency level combination becomes the highest frequency level combination (L8, L8, L12, L21).

[0113] When the higher frequency level combination corresponds to the highest frequency level combination (YES in operation S13 ), the operating frequency selector 221 may map all the generated frequency level combinations to a plurality of performance levels in operation S15 .

[0114] When the operation of increasing one operating frequency level in the lowest frequency level combination is repeatedly performed to generate frequency level combinations until the highest frequency level combination is reached, when the performance levels of the NPU, DSP, GPU, and CPU are 8, 8, 12, and 21, respectively, 1+(8-1)+(8-1)+(12-1)+(21-1)=46 combinations can be generated, such as Figure 7 As shown in , the 46 combinations can be mapped to the first performance level PL1 to the forty-sixth performance level PL46 in ascending order of the sum of the operating frequency levels.

[0115] The performance levels may correspond to normalized performance values ​​of the neural network model. For example, the operating frequency selector 221 may map the normalized performance value of the first performance level PL1 corresponding to the lowest frequency level combination to "0", and may map the normalized performance value of the forty-sixth performance level PL46 corresponding to the highest frequency level combination to "1". In some example embodiments, the normalized performance values ​​of the remaining performance levels may be mapped to gradually increasing values ​​between "0" and "1". In one embodiment, the remaining frequency level combinations corresponding to the remaining performance levels may be mapped to normalized performance values ​​with equal intervals between "0" and "1" in ascending order of the plurality of performance levels.

[0116] In one example embodiment, the operating frequency selector 221 may map the performance levels to normalized performance values ​​set to have equal intervals therebetween according to the number of performance levels. The performance differences caused by the respective frequency level combinations may not be strictly equal, but may be approximately equal.

[0117] In operation S16 , the operating frequency selector 221 may determine a frequency level combination of the memory controller MEM and the system bus BUS based on the frequency level combination of the heterogeneous computing device.

[0118] When the operating frequencies of hardware devices that have dependencies on each other differ significantly due to the hardware structure, a performance bottleneck may occur. For example, when the operating frequencies of the memory controller MEM and the system bus BUS are significantly lower than the operating frequencies of the computing device, the memory controller MEM and the system bus BUS may not be able to quickly obtain values ​​generated by the computing device, and may have difficulty quickly providing values ​​used by the computing device.

[0119] According to an example embodiment, the operating frequency selector 221 may associate the operating frequencies of the memory controller and the system bus with the operating frequencies of the heterogeneous computing devices. In one example embodiment, the operating frequency selector 221 may determine the operating frequency levels of the memory controller MEM and the system bus BUS to be proportional to the "relatively highest level among the operating frequency levels of the heterogeneous computing devices in the combination of frequency levels of the heterogeneous computing devices." In one example embodiment, the processor 110 (e.g., the operating frequency selector 221) may normalize the multiple operating frequency levels of the multiple heterogeneous computing devices and determine the operating frequency levels of the memory controller and the system bus to be proportional to the highest level among the normalized multiple operating frequency levels.

[0120] For example, the number of operating frequency levels of the heterogeneous computing devices NPU, DSP, GPU, and CPU may be 8, 8, 12, and 21, respectively, and may be different from each other. In order to compare the relative sizes of the operating frequency levels of the heterogeneous computing devices NPU, DSP, GPU, and CPU, the operating frequency levels may be normalized. In some example embodiments, the number of operating frequency levels of the memory controller MEM and the system bus BUS may also be different from each other. The operating frequency levels of the memory controller MEM and the system bus BUS may be determined to be a minimum level such that the normalized operating frequency level of the memory controller MEM and the system bus BUS is not lower than the highest frequency level among the normalized operating frequency levels.

[0121] According to an example embodiment, the operating frequency selector 221 may generate a frequency level combination having a low amount of energy consumption by repeatedly performing an operation of increasing one operating frequency level in the current frequency level combination. Figure 7 In the example shown in FIG1 , when the number of operating frequency levels of the heterogeneous computing devices NPU, DSP, GPU, and CPU is 8, 8, 12, and 21, respectively, the operating frequency selector 221 can generate a frequency level combination with low energy consumption by performing (8-1) + (8-1) + (12-1) + (21-1) = 45 searches. In other words, the operating frequency selector 221 can efficiently generate frequency level combinations without performing a search for all frequency level combinations of the heterogeneous computing devices (e.g., 8×8×12×21=16,128 combinations). Therefore, the overhead of the operating frequency selector 221 in generating a frequency level combination for each neural network model can be reduced.

[0122] In the following, reference will be made to Figure 9 and Figure 10 A method of analyzing the operating time ratio of each computing device is described in detail.

[0123] Figure 9is a diagram illustrating a method of analyzing a runtime ratio using a directed acyclic graph of a neural network model according to example embodiments.

[0124] Figure 9 Show reference Figure 3 The runtime of each node and the runtime of each edge in the described neural network model NNM. For example, the runtime of the first node N1 may have a value of "4", and the runtime of the first edge E1 may have a value of "3". In an example embodiment, the runtime values ​​may be relative values ​​with arbitrary units.

[0125] Among the multiple nodes NS, N1 to N12, and NE, nodes with dependencies between them can be processed in a predetermined order. Dependency between nodes may refer to a relationship in which the output value of one node is used as the input value of another node. For example, the fifth node N5 may use the output value from the fourth node N4 as an input value. In some example embodiments, the sixth node N6 may use the output value of the fifth node N5 as each input value. That is, the fifth node N5 may be dependent on the fourth node N4, and the sixth node N6 may be dependent on the fifth node N5. It may be necessary to process the fourth to sixth nodes N4, N5, and N6 in the order of the fourth node N4, the fifth node N5, and the sixth node N6.

[0126] Independent nodes do not need to be processed in a predetermined order and can be processed in parallel. For example, the sixth node N6 and the tenth node N10 may not have a relationship in which the output value of the sixth node N6 is used as the input value of the tenth node N10. Therefore, the sixth node N6 and the tenth node N10 can be processed in any order and can be processed in parallel.

[0127] Due to the dependency and independence of nodes, the directed acyclic graph of the neural network model NNM may have various paths from the start node NS to the end node ES.

[0128] The running time of the neural network model may depend on the running time of the longest path among various paths that takes the longest running time. For example, in one example embodiment, the running time of the neural network model may be the same as the running time of the longest path. According to an example embodiment, the running time ratio of each computing device may be determined based on the analysis of the longest path of the directed acyclic graph. Figure 5 The described runtime analyzer 250 may determine the longest path with the longest runtime among various directed acyclic paths of the neural network model based on the runtime of each node and the runtime of each edge obtained from the neural network model runner 240. Figure 9 In the example, the longest path of the neural network model NNM is indicated by a thick line. For example, Figure 9The longest path of the neural network model NNM in can be NS-N3-N4-N5-N6-N7-N8-N9-NE.

[0129] Figure 10 is a diagram showing the running time of a neural network model and the running time of each computing device in the longest path.

[0130] Figure 10 The execution time of each computing device in the first period of executing a neural network model according to an example embodiment is shown.

[0131] The running time of each computing device in the first time period can be determined based on an analysis of the longest path of the neural network model. For example, in one example embodiment, the computing device that runs each of the nodes and edges included in the longest path can be predetermined. Therefore, the running time of the computing devices can be analyzed based on the running time of each of the nodes and edges and the computing devices that run each of the nodes and edges. In some example embodiments, the total running time of the neural network model can also be analyzed. In one embodiment, the running time ratio of a computing device can be the ratio of the time that the computing device runs the neural network model to the total running time of the neural network model.

[0132] A neural network model may have a predetermined connection structure between nodes and edges. Therefore, the longest path of the neural network model may be the same in each period of time during which the neural network model is run. According to example embodiments, the reduction in run time and the increase in energy consumption when the frequency of a computing device is increased by one level may be estimated based on the run time of each heterogeneous computing device analyzed based on the run results of the neural network model in the first period and the total run time of the neural network model.

[0133] Figure 11 is a block diagram illustrating a layered structure of a neural network computing system according to an example embodiment.

[0134] The neural network operating system 300 may have a layered structure including a hardware layer 310 , a system software layer 320 , and an application layer 330 .

[0135] The hardware layer 310 (the lowest layer of the neural network operating system 300) may include hardware devices (such as a processor 311, a memory controller 312, and a system bus 313). The processor 311 may include multiple heterogeneous computing devices (such as a CPU, a GPU, an NPU, and a DSP). In an example embodiment, the processor 311 may include a plurality of heterogeneous computing devices (such as a CPU, a GPU, an NPU, and a DSP). Figure 11 The hardware devices in the hardware layer 310 may correspond to the reference Figure 1 The hardware device of the neural network computing system 100 described in the embodiment of the present invention is shown in FIG. Figure 11A memory that inputs and outputs data under the control of the memory controller 312 is omitted in the example shown in FIG.

[0136] The system software layer 320 may manage the hardware devices of the hardware layer 310 and may provide an abstracted platform. For example, the system software layer 320 may run a kernel (such as Linux).

[0137] The system software layer 320 may include a frequency scaler 321 and a neural network model runner 322. According to example embodiments, the frequency scaler 321 may determine an operating frequency of a hardware device that runs a neural network model by performing feedback control.

[0138] In some example embodiments, the neural network model operator 322 may operate the neural network model using a hardware device operating at the operating frequency determined by the frequency regulator 321. In some example embodiments, the neural network model operator 322 may output the actual operating time of the neural network model as a result of operating the neural network model. The actual operating time may be fed back to the frequency regulator 321 for closed-loop control of the frequency regulator 321.

[0139] The system software layer 320 may be executed by the processor 311. For example, the system software layer 320 may be executed by a CPU. However, the computing device that may execute the system software layer 320 is not limited to a CPU.

[0140] The application layer 330 may be executed on the system software layer 320 and may include a plurality of neural network models 331 to 33N and other applications 340. For example, the other applications 340 may include a camera application. In some example embodiments, the plurality of neural network models 331 to 33N may include a model for detecting an object included in an image frame acquired by the camera application, a model for recognizing a detected object, a model for detecting a target area in an image frame, a model for recognizing a detected target area, a model for classifying a recognized target area according to meanings such as a person, a car, and a tree, etc. However, the types of the neural network models 331 to 33N and other applications 340 are not limited thereto.

[0141] According to an example embodiment, the system software layer 320 may receive feedback regarding an error between a target execution time and an actual execution time of each of the neural network models 331 to 33N in order to execute the neural network models 331 to 33N, and may control an operating frequency of a hardware device based on the error in the execution time.

[0142] When a neural network model is running, other applications can be running simultaneously, and multiple neural network models can be running simultaneously. For example, when the neural network operating system 300 is a mobile system, a neural network model for detecting objects can be running simultaneously with a camera application. When multiple applications including neural network models are running simultaneously, resource contention may occur in the hardware device.

[0143] According to an exemplary embodiment, the system software layer 320 may adjust the operating frequency of the hardware device based on the target runtime and actual runtime of each of the multiple neural network models running simultaneously among the neural network models 331 to 33N. According to an exemplary embodiment, the neural network operating system 300 may achieve the target runtime of each of the multiple neural network models while reducing the amount of power consumption of each hardware device.

[0144] Figure 12 is a block diagram specifically illustrating system software layers of a neural network computing system according to an example embodiment.

[0145] Reference Figure 12 The system software layer of the neural network computing system 400 may include multiple frequency determiners 410, a power analyzer 420, a system frequency determiner 430, and a neural network model runner 440. The multiple frequency determiners 410 may respectively correspond to the neural network models running in the system software layer. Figure 12 Shows any neural network model among the running neural network models frequency determiner 410i. Figure 12 The plurality of frequency determiners 410, power analyzers 420 and system frequency determiners 430 in the system may correspond to Figure 11 The frequency regulator 321, and Figure 12 The neural network model runner 440 in may correspond to Figure 11 Neural network model runner 322 in.

[0146] Reference Figure 12 Each of the plurality of frequency determiners 410 may be based on a target runtime of a corresponding neural network model. The actual running time The error between To output the operating frequency of each hardware device used to run the neural network model 、 、 、 、 and .

[0147] The power analyzer 420 may store a power profile indicating the amount of power consumption during a utilization period and the amount of power consumption during an idle period according to the operating frequency level of each heterogeneous computing device. The power analyzer 420 may provide the power profile to each of the plurality of frequency determiners 410. The power profile may be used to determine the operating frequency of each hardware device. 、 、 、 、 and .

[0148] The system frequency determiner 430 may obtain the operating frequency of each hardware device for running each neural network model from each of the plurality of frequency determiners 410. In some example embodiments, the system frequency determiner 430 may output a frequency for simultaneously running a plurality of neural network models (including the neural network model ) of the system operating frequency of each hardware device 、 、 、 、 and .

[0149] According to an example embodiment, the system frequency determiner 430 may determine, as the system operating frequency of each hardware device, an operating frequency having a highest value among the operating frequencies of each hardware device acquired from the plurality of frequency determiners 410. For example, the system frequency determiner 430 may determine, as the system operating frequency of the NPU, an operating frequency having a highest value among the operating frequencies of the NPU acquired from each of the plurality of frequency determiners 410.

[0150] The neural network model runner 440 may control the operating frequency of the hardware device based on the system operating frequency obtained from the system frequency determiner 430 and may use the hardware device to run multiple neural network models. The neural network model runner 440 may output the running time E of the multiple neural network models.

[0151] Neural network model The frequency determiner 410 i may include a feedback controller 411 , an operating frequency resolver 412 , a run-time analyzer 413 , and an error calculator 414 . Figure 12 The feedback controller 411, the operating frequency decomposer 412, the run time analyzer 413 and the error calculator 414 in the reference Figure 5 The feedback controller 210, the operating frequency resolver 220, the runtime analyzer 250, and the error calculator 260 are described.

[0152] The feedback controller 411 can be based on the neural network model calculated by the error calculator 414. The error in running time Perform feedback control to output the neural network model The normalized target performance value of In one example embodiment, the feedback controller 411 may perform proportional-integral-derivative (PID) control to reduce the normalized target performance value sudden changes and improve stability.

[0153] Neural network models can be predetermined For example, a device driver controlling a hardware device may provide an application programming interface (API) capable of setting a target runtime for each neural network model. The target runtime for each neural network model may be input via the API.

[0154] In some example embodiments, feedback controller 411 may determine the initial value of the normalized target performance value of the neural network model to be "1." That is, when the neural network model is first executed, feedback controller 411 may control each hardware device to operate at the highest operating frequency to ensure responsiveness. In some example embodiments, feedback controller 411 may adjust the normalized target performance value lower based on the actual execution time being shorter than the target execution time, and may control the actual execution time to converge to the target execution time.

[0155] The operating frequency decomposer 412 may be based on the normalized target performance value output by the feedback controller 411. , to output for running the neural network model The operating frequency of each hardware device 、 、 and The operating frequency resolver 412 can refer to Figure 6 and Figure 7 The frequency level combination table described above determines the operating frequency of each hardware device. For example, the operating frequency decomposer 412 may be configured to have a normalized target performance value not less than The frequency level combination corresponding to the lowest level is determined among the performance levels of the normalized performance value of the output signal, and the operating frequency of each device corresponding to the frequency level combination can be output. 、 、 、 and .

[0156] The operating frequency resolver 412 may be based on running a neural network model At least one result is generated for the neural network model For example, the operation frequency decomposer 412 can analyze the frequency level combination table for running the neural network model based on the running time of each node and the running time of each edge from the running time analyzer 413. In some example embodiments, the operating frequency decomposer 412 may generate a power distribution that can reduce the power consumption for running the neural network model for each performance level based on the operating time ratio and the power distribution obtained from the power analyzer 420. frequency level combination that determines the amount of energy consumed.

[0157] Figure 13 is a diagram illustrating a method of operating a neural network model according to example embodiments.

[0158] The neural network model operation method may include operations S21 to S28, and operations S21 to S28 may be performed by referring to Figure 12 The system software layer of the described neural network operating system 400 is executed.

[0159] In operation S21, a neural network model may be triggered. For example, when a camera application is run by a user's selection, a neural network model for detecting an object may be triggered.

[0160] In operation S22, frequency scaling of the triggered neural network model may be triggered. For example, when the neural network model When triggered, refer to Figure 12 Described neural network model The frequency determiner 410i of the neural network model can be triggered together with the frequency determiner 410i. The frequency determiner can be triggered independently for each neural network model. For example, the frequency determiner 410i can be further triggered while the frequency determiner of another neural network model is running.

[0161] In operation S23, the target performance of the neural network model may be determined based on the previous runtime error. For example, the feedback controller 411 may determine the runtime error based on the target runtime of the neural network model and the previous runtime, and determine the target performance level of the neural network model based on the runtime error.

[0162] In operation S24, the operating frequency of each hardware device according to the target performance level may be determined with reference to the frequency level combination table. For example, the operating frequency decomposer 412 may generate a frequency level combination that maximizes the reduction in operating time relative to the increase in energy consumption each time the performance level increases by one level, and may determine the operating frequency level of each hardware device according to the target performance level.

[0163] In operation S25, the system operating frequency of each hardware device may be determined based on the operating frequency of the neural network model (including the triggered neural network model) running for each hardware device. For example, the system frequency determiner 430 may set the operating frequency with the maximum value among the operating frequencies of the running neural network model for each hardware device as the system frequency.

[0164] In operation S26 , the triggered neural network model may be run by driving the hardware device according to the above-set system operating frequency.

[0165] In operation S27, when the triggered neural network model is executed, the set operating frequency may be released. Releasing the set operating frequency may include initializing the operating frequency of the hardware device or restoring the operating frequency to the operating frequency before executing the neural network model.

[0166] In operation S28, the actual running time of the neural network model may be fed back. The normalized performance value and operating frequency of the neural network model may be adjusted based on the actual running time of the neural network model. The adjusted operating frequency may be used the next time the same neural network model is triggered.

[0167] According to an example embodiment, the neural network operating system may perform feedback control based on a target runtime of a neural network model and an actual runtime of the neural network model, thereby optimizing the operating frequency of each hardware device. Each hardware device may operate at the optimized operating frequency to process one or more neural network models, thereby reducing energy consumption used to execute the neural network model while maintaining the target runtime of the neural network model.

[0168] The runtime of a neural network model can vary due to various factors. Wide variations in the operating frequency of each hardware device can make it difficult for the actual runtime to converge to the target runtime. According to example embodiments, in order to mitigate variations in the operating frequency of each hardware device, in addition to PID control performed by feedback controller 411, a smoothing technique may also be applied.

[0169] Figure 14 is a diagram illustrating a neural network computing system according to example embodiments.

[0170] Figure 14 The neural network operating system 400a may include the reference Figure 12 The components of the neural network operating system 400 described in the previous section are similar, so for the sake of brevity, their repeated description will be omitted. Figure 12 Different descriptions of the Neural Network Operating System 400 in Figure 14 Neural network operating system 400a.

[0171] and Figure 12 The multiple frequency determiners 410 are different, Figure 14 Each of the plurality of frequency determiners 410 may further include a smoothing processor 415. The smoothing processor 415 may be configured to generate a smoothing signal by the neural network model output by the feedback controller 411. The normalized target performance value of Perform a smoothing operation to output the smoothed target performance The smoothed target performance may be input to the operating frequency resolver 412 .

[0172] The smoothing processor 415 may determine the smoothed target performance value based on an exponential smoothing technique for the normalized target performance value determined when the previous neural network model was run. Specifically, the smoothed target performance value may be determined based on a weighted average of the previously determined normalized target performance values. Among the normalized target performance values, older values ​​may be given smaller weights, and more recent values ​​may be given larger weights. The weights may be determined based on parameters having values ​​between "0" and "1."

[0173] In some example embodiments, the plurality of frequency determiners 410 may allow the operating frequency of each hardware device to be gradually changed, thereby controlling the actual runtime of the neural network model to converge to the target runtime of the neural network model.

[0174] Figure 15 is a diagram illustrating an electronic device to which a neural network computing system according to example embodiments is applied.

[0175] Reference Figure 15 , the mobile system 1000 may include a camera 1100, a display 1200, an audio processor 1300, a modem 1400, DRAM 1500a and DRAM 1500b, flash memory devices 1600a and flash memory devices 1600b, input / output devices 1700a and input / output devices 1700b, and an application processor (hereinafter referred to as "AP") 1800.

[0176] The mobile system 1000 may be implemented as a laptop computer, a portable terminal, a smart phone, a tablet PC, a wearable device, a healthcare device, an Internet of Things (IoT) device, etc. In some example embodiments, the mobile system 1000 may be implemented as a server or a personal computer.

[0177] Camera 1100 can capture still images or videos under user control. Mobile system 1000 can use still images / videos captured by camera 1100 to obtain specific information, or convert the still images / videos into other forms of data (such as text), and can store the converted data. In some example embodiments, mobile system 1000 can identify character strings included in still images / videos captured by camera 1100 and provide text or audio translations corresponding to the character strings. As described, the application areas of camera 1100 in mobile system 1000 have become increasingly diverse. In one example embodiment, camera 1100 can transmit data (such as still images / videos) to AP 1800 via a D-Phy or C-Phy interface in accordance with the MIPI standard.

[0178] The display 1200 may be implemented in various forms, such as a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, an active-matrix organic light-emitting diode (AM-OLED), a plasma display panel (PDP), a field emission display (FED), electronic paper, etc. In one example embodiment, the display 1200 may provide a touch screen function and thus also serve as an input device for the mobile system 1000. In some example embodiments, the display 1200 may be integrated with a fingerprint sensor and thus may provide security functions for the mobile system 1000. In one example embodiment, the AP 1800 may transmit image data to be displayed on the display 1200 to the display 1200 via a D-Phy or C-Phy interface according to the MIPI standard.

[0179] The audio processor 1300 may process audio data stored in the flash memory devices 1600a and 1600b, or audio data included in content externally received via the modem 1400 or the input / output devices 1700a and 1700b. For example, the audio processor 1300 may perform various processes (such as encoding / decoding, amplification, noise filtering, etc.) on the audio data.

[0180] Modem 1400 can modulate and transmit signals to transmit and receive wired / wireless data, and can restore the original signal by demodulating the externally received signal. Input / output device 1700a and input / output device 1700b, which provide digital input and output devices, may include a port that can be connected to an external recording medium, an input device (such as a touch screen or mechanical button keys), and an output device that can output vibration in a tactile manner. In some examples, input / output device 1700a and input / output device 1700b can be connected to the external recording medium via a port (such as a USB, lightning cable, SD card, micro SD card, DVD, network adapter, etc.).

[0181] AP 1800 may control the overall operation of mobile system 1000. Specifically, AP 1800 may control display 1200 so that a portion of the content stored in flash memory device 1600a and flash memory device 1600b is displayed on the screen. In some example embodiments, when user input is received via input / output device 1700a and input / output device 1700b, AP 1800 may perform a control operation corresponding to the user input.

[0182] The AP 1800 may be provided as a system-on-chip (hereinafter referred to as an "SoC") that runs applications, an operating system (OS), and the like. In some example embodiments, the AP 1800 may be included in a single semiconductor package along with other devices included in the mobile system 1000, such as the DRAM 1500a, the flash memory 1620, and / or the memory controller 1610. For example, at least one device different from the AP 1800 may be provided in the form of a package, such as a package-on-package (PoP), a ball grid array (BGA), a chip-scale package (CSP), a system-in-package (SIP), a multi-chip package (MCP), a wafer-level fabrication package (WFP), a wafer-level process stack package (WSP), and the like. The kernel of the OS running on the AP 1800 may include an input / output (I / O) scheduler and a device driver for controlling the flash memory devices 1600a and 1600b. The device driver may control access performance of the flash memory devices 1600 a and 1600 b with reference to the number of synchronization queues managed by the input / output scheduler, or may control a CPU mode, a dynamic voltage and frequency scaling (DVFS) level, etc. in the SoC.

[0183] In one example embodiment, the AP 1800 may include a processor block that executes calculations or runs applications and / or an operating system, and various other peripheral components connected to the processor block via a system bus. The peripheral components may include a memory controller, internal memory, a power management block, an error detection block, a monitoring block, and the like. The processor block may include one or more cores. When the processor block includes multiple cores, each core may include a cache memory, and a common cache shared by the cores may be included in the processor block.

[0184] In one example embodiment, the AP 1800 may include an accelerator block 1820 (a dedicated circuit for AI data computation). In some example embodiments, an accelerator chip may be provided separately from the AP 1800, and the DRAM 1500b may be additionally connected to the accelerator block 1820 or the accelerator chip. The accelerator block 1820 (a functional block dedicated to performing specific functions of the AP 1800) may include a graphics processing unit (GPU) (i.e., a functional block dedicated to graphics data processing), a neural processing unit (NPU) (i.e., a block dedicated to performing AI computation and inference), and a data processing unit (DPU) (i.e., a block dedicated to data transmission).

[0185] In some example embodiments, the mobile system 1000 may include a plurality of DRAMs 1500a and 1500b. In one example embodiment, the AP 1800 may include a controller 1810 for controlling the DRAMs 1500a and 1500b, and the DRAM 1500a may be directly connected to the AP 1800.

[0186] The AP 1800 may control the DRAM by setting a command and mode register set (MRS) that complies with the Joint Electron Device Engineering Council (JEDEC) standard, or may perform control by setting specifications and functions used by the mobile system 1000, such as low voltage, high speed, reliability, and a DRAM interface protocol for CRC / ECC. For example, the AP 1800 may communicate with the DRAM 1500a via an interface that complies with the JEDEC standard, such as LPDDR4, LPDDR5, etc. In some example embodiments, the AP 1800 may perform control by setting a new DRAM interface protocol, allowing the accelerator block 1820 or accelerator chip provided separately from the AP 1800 to control the DRAM 1500b for an accelerator having a higher bandwidth than the bandwidth of the DRAM 1500a.

[0187] Figure 15Only DRAM 1500a and DRAM 1500b are shown, but the components of mobile system 1000 are not necessarily limited thereto. Depending on the bandwidth, response speed, and voltage conditions of AP 1800 or accelerator block 1820, memories other than DRAM 1500a and DRAM 1500b may also be included in mobile system 1000. For example, controller 1810 and / or accelerator block 1820 may control various memories (such as phase change random access memory (PRAM), SRAM, magnetic random access memory (MRAM), resistive random access memory (RRAM), ferroelectric random access memory (FRAM), hybrid RAM, etc.). DRAM 1500a and DRAM 1500b may have a relatively lower latency than that of input / output devices 1700a and 1700b or flash memory devices 1600a and 1600b, and may have a higher bandwidth than that of input / output devices 1700a and 1700b or flash memory devices 1600a and 1600b. DRAM 1500a and DRAM 1500b may be initialized when power is turned on in mobile system 1000. When the OS and application data are loaded, DRAM 1500a and DRAM 1500b may be used as a temporary storage location for the OS and application data, or as a runtime space for various software codes.

[0188] DRAM 1500a and DRAM 1500b can store data for four arithmetic operations (such as addition, subtraction, multiplication, and division), vector operation data, address operation data, or FFT operation data. In one example embodiment, DRAM 1500a and DRAM 1500b can be provided as a process-in-memory (PIM) with computing functions. For example, functions for inference can be performed in DRAM 1500a and DRAM 1500b. Here, inference can be performed in a deep learning algorithm using an artificial neural network. The deep learning algorithm may include a training operation to learn a model using various data, and an inference operation to distinguish data using the learned model. For example, the function used for inference may include a hyperbolic tangent function, a sigmoid function, a rectified linear unit (ReLU) function, and the like.

[0189] In one example embodiment, an image captured by a user using the camera 1100 may be signal processed and stored in the DRAM 1500 b , and the accelerator block 1820 or accelerator chip may perform AI data calculations to identify the data using the data stored in the DRAM 1500 b and a function for inference.

[0190] In one example embodiment, the mobile system 1000 may include a plurality of storage devices or a plurality of flash memory devices 1600a and 1600b having a larger capacity than the DRAM 1500a and the DRAM 1500b. The flash memory devices 1600a and 1600b may include a controller 1610 and a flash memory 1620. The controller 1610 may receive a control command and data from the AP 1800 and may write data to the flash memory 1620 in response to a control command, or may read data stored in the flash memory 1620 and transmit the data to the AP 1800.

[0191] In some example embodiments, the accelerator block 1820 or accelerator chip may use the flash memory devices 1600a and 1600b to perform training operations and AI data calculations. In one example embodiment, a block capable of executing predetermined calculations may be implemented in the flash memory devices 1600a and 1600b, and the block may alternatively use data stored in the flash memory 1620 to perform at least a portion of the training operations and inferred AI data calculations performed by the AP 1800 and / or the accelerator block 1820.

[0192] In one example embodiment, the AP 1800 may include an interface 1830 so that the flash memory device 1600a and the flash memory device 1600b can be directly connected to the AP 1800. For example, the AP 1800 may be implemented as an SoC, the flash memory device 1600a may be implemented as a chip separate from the AP 1800, and the AP 1800 and the flash memory device 1600a may be installed in a single package. However, example embodiments are not limited thereto, and in some example embodiments, a plurality of flash memory devices 1600a and 1600b may be electrically connected to the mobile system 1000 via a connection.

[0193] Flash memory devices 1600a and 1600b may store data (such as still images captured by camera 1100 or videos captured by camera 1100), or may store data received via a communication network and / or ports included in input / output devices 1700a and 1700b. For example, flash memory devices 1600a and 1600b may store augmented reality / virtual reality content, high-definition (HD) content, or ultra-high-definition (UHD) content.

[0194] According to an example embodiment, the AP 1800 may drive the camera 1100 and execute a camera application that allows a user to use the camera 1100. The AP 1800 may execute a neural network model that detects an object in an image frame generated by the camera application, a neural network model that determines an object, etc., concurrently with the execution of the camera application.

[0195] The neural network model may be executed by heterogeneous computing devices included in the AP 1800. Each heterogeneous computing device may have multiple operating frequency levels. The AP 1800 may control each of the one or more neural network models executed using the heterogeneous computing devices to have one performance level among multiple performance levels.

[0196] Specifically, the AP 1800 may generate a frequency level combination table representing frequency level combinations of hardware devices including heterogeneous computing devices, wherein the frequency level combination maximizes the reduction in runtime relative to the increase in energy consumption when the performance level of the neural network model increases by one level. For example, the AP 1800 may estimate the increase in energy consumption relative to the reduction in runtime based on the power distribution of each heterogeneous computing device and the runtime ratio of each heterogeneous computing device analyzed based on the results of at least one execution of the neural network model.

[0197] AP 1800 can determine a target performance level among performance levels by performing feedback control based on an error between a target operating time and an actual operating time of a neural network model, can select a frequency level combination of a hardware device with reference to a frequency combination table, and can control the hardware device to have an operating frequency level included in the frequency level combination while the neural network model is being executed.

[0198] According to example embodiments, AP 1800 may improve energy efficiency in running a neural network model while maintaining a target runtime of the neural network model. In some example embodiments, AP 1800 may use heuristic techniques to find frequency level combinations without searching all possible combinations, thereby reducing system overhead.

[0199] According to example embodiments, a neural network computing system may determine a target performance level of a neural network model by performing feedback control based on an error between a target runtime and an actual runtime of the neural network model, and may select an operating frequency combination of heterogeneous computing devices that maximizes energy efficiency to achieve the target performance level. As a result, the neural network computing system may improve energy efficiency in running the neural network model while maintaining the target runtime of the neural network model.

[0200] The neural network computing system can use heuristic techniques to find a combination of operating frequencies of heterogeneous computing devices that can maximize energy efficiency to achieve a target performance level in the neural network model, thereby determining the operating frequency of each hardware device without incurring excessive overhead for the system.

[0201] While example embodiments have been shown and described above with reference to the accompanying drawings, it will be apparent to those skilled in the art that modifications and variations may be made without departing from the scope of the inventive concept as defined in the appended claims.

Claims

1. A neural network computing system comprising: a processor comprising a plurality of heterogeneous computing devices each having a plurality of operating frequency levels; a memory configured to store data associated with the execution of the neural network model; a memory controller configured to control data input and output of the memory; as well as A system bus configured to support communications between the processor and the memory controller, The processor is configured as follows: generating a frequency level combination table, the frequency level combination table representing a plurality of frequency level combinations of hardware devices for a plurality of performance levels of the neural network model, the hardware devices including a memory controller, a system bus, and the plurality of heterogeneous computing devices, and each of the plurality of frequency level combinations maximizes a reduction in runtime relative to an increase in energy consumption when the performance level of the neural network model increases by one level; determining a normalized performance value by performing feedback control based on an error between a target runtime of the neural network model and an actual runtime of the neural network model; determining a target performance level among the plurality of performance levels based on the normalized performance value; referring to the frequency level combination table, selecting a frequency level combination among the plurality of frequency level combinations according to a target performance level; and While the neural network model is being executed, the hardware device is controlled based on the selected frequency level combination.

2. The neural network computing system of claim 1, wherein: The processor is configured to: repeatedly generating, starting from when the current frequency level combination of the plurality of heterogeneous computing devices is the lowest frequency level combination until the higher frequency level combination is the highest frequency level combination, a higher frequency level combination in which the operating frequency level of a computing device among the plurality of heterogeneous computing devices that maximizes a reduction in operating time relative to an increase in energy consumption, among a plurality of operating frequency levels included in the current frequency level combination, is increased by one level; mapping the generated frequency level combinations to the plurality of performance levels of the neural network model; and A frequency level combination table is generated by determining a frequency level combination of a memory controller and a system bus based on the generated frequency level combination.

3. The neural network computing system of claim 2, wherein: The processor is configured to determine a reduction in the running time of the higher frequency level combination in the neural network model relative to the current frequency level combination based on the analyzed running time of the one computing device and the operating frequencies before and after increasing the operating frequency level of the one computing device by one level.

4. The neural network computing system of claim 3, wherein: The processor is configured to determine the reduction in the runtime based on the following Equation 1: Equation 1 Wherein, ΔT represents the reduction of the running time, p represents the one computing device, and t p represents the running time of the one computing device for the neural network model analysis, represents an operating frequency before the operating frequency level of the one computing device is increased by one level, and Indicates the operating frequency after the operating frequency level of the one computing device is increased by one level.

5. The neural network computing system of claim 2, wherein: The processor is configured to determine an increase in energy consumption of a higher frequency level combination in a neural network model relative to a current frequency level combination based on the analyzed operating time of each of the multiple heterogeneous computing devices, the operating frequencies before and after the operating frequency level of the one computing device is increased by one level, and the power consumption according to the operating frequency in each of the utilization period and the idle period of the one computing device.

6. The neural network computing system of claim 5, wherein: The processor is configured to determine the amount of increase in energy consumption based on the following Equation 2: Equation 2 Wherein, p represents the one computing device, represents the increase in energy consumption, represents the energy consumption of the one computing device for the operating frequency f, represents the power consumed in the utilization period of said one computing device for the operating frequency f, represents the power consumed in an idle period of the one computing device for an operating frequency f, represents the running time of the one computing device for the neural network model analysis, represents the total running time of the analyzed neural network model, represents an operating frequency before the operating frequency level of the one computing device is increased by one level, and Indicates the operating frequency after the operating frequency level of the one computing device is increased by one level.

7. The neural network computing system of claim 2, wherein: The processor is configured to: Mapping the highest frequency level combination to a normalized performance value of "1"; Mapping the lowest frequency level combination to a normalized performance value of "0"; and The remaining frequency level combinations are mapped to normalized performance values ​​between "0" and "1" with equal intervals therebetween in ascending order of the plurality of performance levels.

8. The neural network computing system of claim 2, wherein: The processor is configured to normalize the multiple operating frequency levels of the multiple heterogeneous computing devices and determine the operating frequency level of the memory controller and the operating frequency level of the system bus to be proportional to the highest level among the normalized multiple operating frequency levels.

9. The neural network computing system of claim 1, wherein: The processor is configured to: determining, for each of a plurality of neural network models executed with the neural network model, a plurality of frequency level combinations for each hardware device; determining a frequency having a maximum level among a plurality of operating frequency levels for each of the plurality of neural network models as a system frequency for each hardware device; and Control each hardware device to operate at the system frequency.

10. The neural network computing system of claim 9, wherein: The processor is further configured to run a camera application that generates image frames, and The multiple neural network models include at least two models from the following models: a model for detecting objects in an image frame, a model for recognizing objects, a model for detecting target areas in an image frame, a model for recognizing detected target areas, and a model for classifying recognized target areas.

11. The neural network computing system of claim 1 , wherein: The processor is configured to: feedback the actual running time of the neural network model when the running of the neural network model is completed.

12. The neural network computing system of claim 1, wherein: The processor is configured to: determining a smoothed target performance level by performing smoothing on the target performance level of the neural network model and a plurality of previously determined target performance levels of the neural network model; and The smoothed target performance level is used as the target performance level of the neural network model to select a frequency level combination of the plurality of heterogeneous computing devices.

13. The neural network computing system of claim 1, wherein: Heterogeneous computing devices include: central processing units, neural processing units, graphics processing units, and digital signal processors.

14. The neural network computing system according to any one of claims 1 to 13, wherein: The processor is configured to receive a target runtime for the neural network model via an application program interface.

15. A neural network computing system comprising: a processor comprising a plurality of heterogeneous computing devices each having a plurality of operating frequency levels; as well as a memory configured to store data associated with the execution of the neural network model, The processor is configured as follows: generating a lowest frequency level combination of the plurality of heterogeneous computing devices for running a neural network model, and setting the lowest frequency level combination as a current frequency level combination of the plurality of heterogeneous computing devices; generating a higher frequency level combination in which an operating frequency level that maximizes a reduction in operating time relative to an increase in energy consumption, among the plurality of operating frequency levels included in the current frequency level combination, is increased by one level; repeatedly setting the higher frequency level combination as the current frequency level combination and generating higher frequency level combinations until the higher frequency level combination corresponds to a highest frequency level combination of the plurality of heterogeneous computing devices; mapping the generated frequency level combinations to multiple performance levels of the neural network model; determining a target performance level among the plurality of performance levels by performing feedback control based on an error between a target execution time of the neural network model and an actual execution time of the neural network model; and The operating frequencies of the plurality of heterogeneous computing devices are controlled so that the plurality of heterogeneous computing devices have a frequency level combination corresponding to a target performance level.

16. The neural network computing system of claim 15, wherein: The processor generates a combination of higher frequency levels by: analyzing an execution time ratio of each of the plurality of heterogeneous computing devices based on previous execution results of the neural network model; Performing the following operations on the plurality of heterogeneous computing devices: increasing the operating frequency level of one computing device among the plurality of heterogeneous computing devices by one level, and calculating, based on the operating time ratio of the one computing device, a reduction in operating time relative to an increase in energy consumption before and after the operating frequency level of the one computing device is increased by one level; and A frequency level combination in which the operating frequency level of the one computing device that maximizes the reduction in the operating time relative to the increase in energy consumption is increased by one level is determined as a higher frequency level combination.

17. The neural network computing system of claim 16, wherein: The processor is also configured to: A runtime ratio of each of the plurality of heterogeneous computing devices is determined based on a runtime of each of the plurality of heterogeneous computing devices in a longest path among a plurality of paths of a directed acyclic graph included in a neural network model.

18. A neural network computing system comprising: a processor comprising a plurality of heterogeneous computing devices each having a plurality of operating frequency levels; a memory configured to store data associated with the execution of the neural network model; a memory controller configured to control data input and output of the memory; as well as A system bus configured to support communications between the processor and the memory controller, The processor is configured as follows: In response to triggering of the neural network model, triggering frequency adjustment of the neural network model; determining a target performance level for the neural network model based on an error between a target runtime of the neural network model and an actual runtime of the neural network model; determining an operating frequency of each hardware device according to a target performance level with reference to a frequency level combination table, the frequency level combination table including operating frequency combinations for a plurality of hardware devices for running the neural network model according to the performance level of the neural network model; setting a system frequency of each hardware device based on an operating frequency of each of the plurality of heterogeneous computing devices currently running a plurality of neural network models, the plurality of neural network models including the neural network model; and Running the neural network model according to the system frequency of each hardware device, In which, the frequency level combination table includes a combination of the multiple operating frequency levels of the multiple hardware devices for the performance level of the neural network model, and the combination of the multiple operating frequency levels of the multiple hardware devices maximizes the reduction in running time relative to the increase in energy consumption when the performance level of the neural network model increases by one level.

19. The neural network computing system of claim 18, wherein: The processor is also configured to: When the execution of the neural network model is completed, the system frequency set for each of the plurality of heterogeneous computing devices is released.

20. The neural network computing system of claim 18, wherein: The processor is also configured to: After the operation of the neural network model is completed, the actual operation time of the neural network model is fed back.

Images