Electronic device and method for controlling the same

Abstract

An electronic device and a control method thereof are disclosed. The electronic device includes: a memory for storing a plurality of accelerators and a plurality of neural networks; and a processor configured to select a first neural network from the plurality of neural networks and select a first accelerator from the plurality of accelerators to implement the first neural network, implement the first neural network on the first accelerator to obtain information associated with the implementation, obtain a first reward value for the first accelerator and the first neural network based on the information associated with the implementation, select a second neural network from the plurality of neural networks to be implemented on the first accelerator, implement the second neural network on the first accelerator to obtain information associated with the implementation, obtain second reward values ​​for the first accelerator and the second neural network based on the information associated with the implementation, and select a neural network and an accelerator having a maximum reward value from the plurality of neural networks and the plurality of accelerators based on the first reward value and the second reward value.

Description

Technical Field

[0001] The present disclosure relates to an electronic device and a control method thereof, for example, to an electronic device and a control method thereof for determining an accelerator and a neural network pair capable of outputting optimal accuracy and efficiency metrics. Background Art

[0002] FPGA accelerators are particularly useful for low-volume DNN inference tasks, in custom hardware (HW) configurations, and when tailored for specific DNN characteristics (such as sparsity or customized precision). One advantage of FPGAs is that the HW design cycle is relatively short compared to custom application-specific integrated circuits (ASICs). However, this advantage comes with an interesting side effect: FPGA accelerator HW is often designed after the algorithm (e.g., DNN) is determined and locked.

[0003] Even if an accelerator is software-programmable, its hardware is often over-optimized for a specific DNN to maximize its efficiency. As a result, different DNNs are often inefficient for the same hardware. To address this "over-optimization" problem, FPGA designs are often configurable at the hardware level. In this case, when a new DNN is discovered, the accelerator parameters can be adjusted to the new DNN to maximize hardware efficiency. Even with hardware configurability, FPGA accelerators have the disadvantage of always having to keep up with new DNNs.

[0004] Methods for designing DNNs can be automatic and are known as Neural Architecture Search (NAS). NAS has successfully discovered DNN models that achieve state-of-the-art accuracy in image classification, super-resolution, speech recognition, and machine translation.

[0005] A further development, called FNAS, is described in "Accuracy vs. Efficiency: Achieving Both Through FPGA-Implementation Aware Neural Architecture Search," published by Jiang et al. in the arXiv e-print (January 2019). FNAS is a HW-aware NAS that has been used to try to discover DNNs that minimize latency on a given FPGA accelerator. FNAS can be used to discover convolutional neural networks (CNNs) suitable for a specific FPGA accelerator. Other HW-aware NAS add latency to the reward function so that the discovered models optimize accuracy and inference latency when running on, for example, a mobile device.

[0006] Also note that for CPUs and GPUs, algorithms are optimized to fit existing HW, and for a successful ASIC it is necessary to build in a lot of flexibility and programmability to achieve some future control accuracy. Summary of the Invention

[0007] [Technical solution]

[0008] Embodiments of the present disclosure provide an electronic device and a control method thereof for determining an accelerator and a neural network pair capable of outputting optimal accuracy and efficiency metrics.

[0009] According to an example embodiment, a method for controlling an electronic device includes a memory storing a plurality of accelerators and a plurality of neural networks, the method comprising: selecting a first neural network from among the plurality of neural networks, and selecting a first accelerator from among the plurality of accelerators that is configured to implement the first neural network, implementing the first neural network on the first accelerator to obtain information associated with an implementation result, obtaining a first reward value for the first accelerator and the first neural network based on the information associated with the implementation, selecting a second neural network from among the plurality of neural networks to be implemented on the first accelerator, implementing the second neural network on the first accelerator to obtain information associated with the implementation result, obtaining second reward values ​​for the first accelerator and the second neural network based on the information associated with the implementation, and selecting a neural network and an accelerator having a maximum reward value from among the plurality of neural networks and the plurality of accelerators based on the first reward value and the second reward value.

[0010] Selecting the first accelerator may include: identifying whether hardware performance of the first accelerator and the first neural network obtained by inputting the first accelerator and the first neural network into the first prediction model meets a predetermined criterion; and implementing the first neural network on the first accelerator based on identifying that the obtained hardware performance meets the first hardware criterion to obtain information related to the implementation.

[0011] The identifying may include selecting a second accelerator for implementing the first neural network from among accelerators other than the first accelerator based on identifying that the obtained hardware performance does not satisfy the first hardware criterion.

[0012] Information associated with an implementation may include accuracy and efficiency metrics of the implementation.

[0013] Obtaining the first reward value may include normalizing the obtained accuracy and efficiency metrics; and obtaining the first reward value by performing a weighted sum operation on the normalized metrics.

[0014] Selecting a first neural network from among a plurality of neural networks and selecting a first accelerator from among a plurality of accelerators for implementing the first neural network may include: obtaining a first probability value corresponding to a first configurable parameter included in each of the plurality of neural networks; and selecting the first neural network based on the first probability value from among the plurality of neural networks.

[0015] Selecting the first accelerator may include obtaining a second probability value corresponding to a second configurable parameter included in each of the plurality of accelerators; and selecting the first accelerator for implementing the first neural network from among the plurality of accelerators based on the second probability value.

[0016] Selecting a first neural network from a plurality of neural networks and selecting a first accelerator from a plurality of accelerators for implementing the first neural network may include: based on selecting the first neural network, and before selecting the first accelerator for implementing the first neural network, predicting hardware performance of the selected first neural network using a second prediction model.

[0017] Predicting may include identifying whether the predicted hardware performance of the first neural network satisfies a second hardware criterion, and selecting a first accelerator for implementing the first neural network based on identifying that the predicted hardware performance of the first neural network satisfies the second hardware criterion.

[0018] The identifying may include selecting another neural network from among a plurality of neural networks other than the first neural network based on identifying that hardware performance of the selected first neural network does not satisfy a second hardware criterion.

[0019] According to an example embodiment, an electronic device includes: a memory for storing a plurality of accelerators and a plurality of neural networks; and a processor configured to: select a first neural network among the plurality of neural networks, and select a first accelerator among the plurality of accelerators that is configured to implement the first neural network, implement the first neural network on the first accelerator to obtain information associated with an implementation result, obtain a first reward value for the first accelerator and the first neural network based on the information associated with the implementation, select a second neural network among the plurality of neural networks to be implemented on the first accelerator, implement the second neural network on the first accelerator to obtain information associated with the implementation result, obtain second reward values ​​for the first accelerator and the second neural network based on the information associated with the implementation, and select a neural network and an accelerator having a maximum reward value among the plurality of neural networks and the plurality of accelerators based on the first reward value and the second reward value.

[0020] The processor may be configured to: identify whether hardware performance of the first accelerator and the first neural network obtained by inputting the first accelerator and the first neural network into the first prediction model satisfies a predetermined criterion, and based on identifying that the obtained hardware performance satisfies the first hardware criterion, implement the first neural network on the first accelerator to obtain information associated with the implementation.

[0021] The processor may be further configured to select a second accelerator, other than the first accelerator, for implementing the first neural network based on identifying that the obtained hardware performance does not satisfy the first hardware criterion.

[0022] Information associated with an implementation may include accuracy and efficiency metrics of the implementation.

[0023] The processor may be further configured to normalize the obtained accuracy and efficiency metrics and obtain the first reward value by performing a weighted sum operation on the normalized metrics.

[0024] The processor may also be configured to obtain a first probability value corresponding to a first configurable parameter included in each of the plurality of neural networks; and select a first neural network from the plurality of neural networks based on the first probability value.

[0025] The processor may also be configured to obtain a second probability value corresponding to a second configurable parameter included in each of the plurality of accelerators; and select a first accelerator from among the plurality of accelerators for implementing the first neural network based on the second probability value.

[0026] The processor may also be configured to predict hardware performance of the selected first neural network using a second prediction model before selecting a first accelerator for implementing the first neural network based on the selection of the first neural network.

[0027] The processor may also be configured to identify whether the predicted hardware performance of the first neural network satisfies a second hardware criterion, and select a first accelerator for implementing the first neural network based on identifying that the predicted hardware performance of the first neural network satisfies the second hardware criterion.

[0028] The processor may also be configured to select another neural network from among the plurality of neural networks other than the first neural network based on identifying that hardware performance of the selected first neural network does not satisfy a second hardware criterion. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The above and other aspects, features and advantages of certain embodiments of the present disclosure will become more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which:

[0030] Figure 1 is a block diagram illustrating an exemplary configuration and operation of an electronic device according to an embodiment;

[0031] Figure 2 is a flowchart illustrating an exemplary process for determining whether to implement a first neural network on a first accelerator using a first prediction model by an electronic device according to an embodiment;

[0032] Figure 3 is a flowchart illustrating an exemplary process for determining whether an electronic device selects an accelerator for implementing a first neural network through a second prediction model according to an embodiment;

[0033] Figure 4A 、 Figure 4B and Figure 4C including flowcharts and block diagrams illustrating exemplary configurations and exemplary operations of electronic devices according to embodiments;

[0034] Figure 5 is a diagram showing that according to an embodiment, Figure 4A Figure 2 shows an exemplary well-defined CNN search space used in the method.

[0035] Figure 6 is a block diagram illustrating exemplary components of an FPGA accelerator according to an embodiment;

[0036] Figure 7A is a graph illustrating area and resource usage for two types of accelerator architectures according to an embodiment;

[0037] Figure 7B is a diagram showing the Figure 7A A graph showing the relationship between latency per image and parallelism for the indicated type of accelerator architecture;

[0038] Figure 8 is a graph showing latency numbers versus size and pixel_par according to an embodiment;

[0039] Figure 9 is a graph illustrating exemplary Pareto optimal points for accuracy, latency, and area according to an embodiment;

[0040] Figure 10 is a diagram showing a method according to an embodiment of the present invention. Figure 9 The graph of the accuracy and waiting time of the Pareto optimal point is shown;

[0041] Figure 11A 、 11B , 11C and 11D are diagrams showing that the 2 , less than 70mm 2 , less than 150mm 2 and less than 220mm 2An exemplary accuracy-latency Pareto frontier graph of a single convolution engine and a dual convolution engine under the area constraint of ;

[0042] Figure 12A is a graph showing accuracy versus latency with constraints imposed according to an embodiment;

[0043] Figure 12B and Figure 12C is a diagram showing the Figure 12A Figure 1 shows an exemplary arrangement of the CNNs selected in

[15] .

[0044] Figure 12D Comparison of the Examples Figure 12C Figure 2 shows the execution schedule of a CNN running on its co-designed accelerator and different accelerators.

[0045] Figure 13 is a graph showing accuracy and latency for an overall case relative to the Pareto optimum of the parameter ratio_conv_engines according to an embodiment;

[0046] Figure 14 is a block diagram illustrating an exemplary alternative architecture that may be used to implement a staged search according to an embodiment;

[0047] Figure 15A is a graph showing precision versus latency according to an embodiment, with the top search result for an unconstrained search highlighted;

[0048] Figure 15B is a graph illustrating precision versus latency according to an embodiment and highlighting the top search result for a search with one constraint;

[0049] Figure 15C is a graph illustrating precision versus latency according to an embodiment and highlighting the top search result for a search with two constraints;

[0050] Figure 16A 、 16B and 16C are diagrams illustrating the Figure 15A 、 15B and 15C , graphs of exemplary reward values ​​for each of the individual, combined, and staged search strategies in the unconstrained and constrained searches;

[0051] Figure 17 is a graph showing the highest accuracy and perf / area for each point searched using a combined search according to an embodiment;

[0052] Figure 18A and 18Bis a diagram showing an exemplary arrangement of a CNN selected from FIG. 15 according to an embodiment;

[0053] Figure 19 and Figure 20 This is to illustrate the Figure 4A A block diagram of an exemplary alternative architecture for using the method together with or performing independent searches;

[0054] Figure 21 It is shown that according to the embodiment, Figure 20 A flowchart of an exemplary method implemented on the architecture of

[0055] Figure 22 It is shown that according to the embodiment, Figure 20 Flowchart of an exemplary alternative method implemented on the architecture of FIG. DETAILED DESCRIPTION

[0056] Hereinafter, the present disclosure will be described in more detail with reference to the accompanying drawings.

[0057] Figure 1 1 is a block diagram illustrating an exemplary configuration and operation of the electronic device 100 according to an exemplary embodiment of the present disclosure. Figure 1 As shown, the electronic device 100 may include a memory 110 and a processor (eg, including a processing circuit) 120. However, Figure 1 The illustrated configuration is an example for implementing the embodiments of the present disclosure, and appropriate hardware and software configurations that are apparent to those skilled in the art may be further included in the electronic device 100 .

[0058] The memory 110 may store instructions or data related to at least one other component of the electronic device 100. An instruction may refer to, for example, an action statement that can be executed by the processor 120 in a programming language and may be the smallest unit for executing or operating a program. The memory 110 is accessible by the processor 120, and the processor 120 may perform operations such as reading, writing, modifying, and updating data.

[0059] The memory 110 may store a plurality of accelerators (e.g., including various processing circuits and / or executable program units) 10-1, 10-2, ... 10-N and a plurality of neural networks (e.g., including various processing circuits and / or executable program units) 20-1, 20-2, ... 20-N. The memory 110 may store an accelerator sub-search space including the plurality of accelerators 10-1, 10-2, ... 10-N and a neural sub-search space including the plurality of neural networks 20-1, 20-2, ... 20-N. The total search space may be defined by the following equation 1.

[0060] [Equation 1]

[0061] S=S NN ×S FGPA

[0062] Where SNN is the sub-search space of the neural network, and SFPGA is the sub-search space of the FPGA. If the accelerator is implemented as another type of accelerator instead of an FPGA, the memory 110 may store sub-search spaces for searching and selecting the implemented type of accelerator. The processor 120 may access each search space stored in the memory 110 to search for and select a neural network or accelerator. Related embodiments are described below.

[0063] A neural network (or artificial neural network) may refer to a model that can process data input using an artificial intelligence (AI) algorithm, for example. A neural network may include multiple layers, and a layer may refer to each step of the neural network. The multiple layers included in the neural network have multiple weight values, and the operation of the layer may be performed by the operation result of the previous layer and the operation of the multiple weights. A neural network may include a combination of several layers, and the layer may be represented by multiple weights. A neural network may include various processing circuits and / or executable program elements.

[0064] Examples of neural networks may include, but are not limited to, convolutional neural networks (CNNs), deep neural networks (DNNs), recurrent neural networks (RNNs), restricted Boltzmann machines (RBMs), deep confidence networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), deep Q networks, and the like. CNNs may include different blocks selected from conv1x1, conv3x3, and pool3x3. As another example, a neural network may include a GZIP compression-type neural network, which is an algorithm that includes two main computational blocks that perform LZ77 compression and Huffman encoding. The LZ77 computation block includes parameters such as the compression window size and the maximum compression length. The Huffman computation block may have parameters such as the Huffman tree size and the tree update frequency. These parameters affect the final result of the GZIP string compression algorithm, and there may generally be a trade-off between the compression ratio and the compression rate.

[0065] Each of the plurality of neural networks may include a first configurable parameter. Hardware or software characteristics of each of the plurality of neural networks may be determined by the number (or weight) corresponding to the configurable parameters included in each neural network. The first configurable parameter may include at least one of an operating mode or a layer connection scheme of each neural network. The operating mode may include the type and number of operations performed between layers included in the neural network. The layer connection scheme may include the number of layers included in each operating network, the number of stacks or units included in a layer, the connection relationship between layers, etc.

[0066] For example, an accelerator may refer to a hardware device that can increase the amount or processing speed of data processed by a neural network based on artificial intelligence (AI) algorithms. In one example, the accelerator may be implemented as a platform for implementing a neural network, such as, but not limited to, a field programmable gate array (FPGA) accelerator or an application-specific integrated circuit (ASIC).

[0067] Each of the plurality of accelerators may include a second configurable parameter. Hardware or software characteristics of each of the plurality of accelerators may be determined according to a value corresponding to the second configurable parameter included by each of the plurality of accelerators. The second configurable parameter included by each of the plurality of accelerators may include, for example, but not limited to, at least one of a parallelism parameter (e.g., parallel output functions or parallel output pixels), a buffer depth (e.g., a buffer depth for input, output, and weight buffers), a pooling engine parameter, a memory interface width parameter, a convolution engine ratio parameter, and the like.

[0068] The memory 110 may store an evaluation model 30. The evaluation model 30 may refer to, for example, an AI model that may output a reward value for the accelerator and neural network selected by the processor 120 and may be controlled by the processor 120. For example, the evaluation model 30 may perform normalization on information related to the implementation (e.g., accuracy metrics and efficiency metrics) obtained by implementing the selected neural network on the selected accelerator.

[0069] The evaluation model 30 may perform a weighted sum operation on the normalized accuracy metric and efficiency metric to output a reward value. The process of normalizing each metric and performing the weighted sum operation by the evaluation model 30 will be described in more detail below. The greater the reward value for the accelerator and neural network pair output by the evaluation model 30, the more accurate and efficient the implementation and operation of the accelerator and neural network pair can be.

[0070] The evaluation model 30 may limit the value that the evaluation model 30 can output by a threshold value corresponding to each of the accuracy metric and the efficiency metric. For example, the algorithm applied by the evaluation model 30 to output a reward value for the accuracy metric and the efficiency metric may be implemented as shown in Equation 2.

[0071] [Equation 2]

[0072]

[0073] In Equation 2, m may refer to an accuracy metric or an efficiency metric, w may refer to a weight vector of m, and th may refer to a threshold vector of m. The evaluation model 30 may output a reward value using Equation 3 below.

[0074] [Equation 3]

[0075]

[0076] In Equation 3, ar is the area of ​​the accelerator, lat (e.g., latency) is the latency, acc is the accuracy value, and w1, w2, and w3 are weighted sets for each of area, latency, and accuracy. If optimization is performed on the search space s, the evaluation model output E(s) = m satisfies a given constraint (e.g., latency less than a specific value).

[0077] The accuracy metric may refer to, for example, a value indicating the accuracy with which the neural network has been implemented on the accelerator. The efficiency metric may refer to, for example, a value indicating the degree to which the neural network can be executed on the accelerator with an optimized implementation. The efficiency metric may include, for example, but not limited to, at least one of a latency metric, a power metric, an area metric of the accelerator when the neural network is implemented on the accelerator, and the like.

[0078] The memory 110 may include a first prediction model 40 and a second prediction model 50. The first prediction model 40 may refer to, for example, an AI model capable of outputting an estimated value corresponding to the hardware performance of the input accelerator and the neural network. The hardware performance corresponding to the first accelerator and the first neural network may include the latency or power required when implementing the first neural network on the first accelerator.

[0079] When the first neural network is implemented on the first accelerator, the first prediction model 40 may output an estimated value of the latency or power that may be required. The first hardware criterion may be a predetermined value when the first prediction model 40 is designed, but may be updated by the processor 120. Embodiments associated with the first prediction model 40 will be described in more detail below.

[0080] The second prediction model 50 may refer to, for example, an AI model capable of outputting an estimated value of the hardware performance corresponding to the neural network. For example, when the first neural network is input, the second prediction model 50 may output an estimated value of the hardware performance corresponding to the first neural network. The estimated value of the hardware performance corresponding to the first neural network may include, for example, but not limited to, at least one of the waiting time predicted to be required when the first neural network is implemented on a specific accelerator, the memory footprint of the first neural network, etc. The memory footprint of the first neural network may refer to, for example, the size of the space occupied by the first neural network on the memory 110 or the first accelerator. An exemplary embodiment associated with the second prediction model 50 will be described in more detail below.

[0081] The first prediction model 40 and the second prediction model 50 may be controlled by the processor 120. The processor 120 may learn each model. For example, the processor 120 may input the first accelerator and the first neural network into the first prediction model to obtain an estimated value of the hardware performance of the first accelerator and the first neural network. The processor 120 may train the first prediction model 40 to output an optimal estimated value that minimizes and / or reduces the difference between the hardware performance value that can be obtained when the first neural network is implemented on the first accelerator and the obtained estimated value.

[0082] For example, the processor 120 may input the first neural network into the second prediction model 50 to obtain an estimated value of the hardware performance of the first neural network. The processor 120 may train the second prediction model 50 to output an optimal estimated value that may minimize and / or reduce the difference between a hardware performance value that may be obtained by the first neural network when the actual first neural network is implemented in a specific accelerator and the obtained estimated value.

[0083] The memory 110 may include a policy function model 60. The policy function model 60 may refer to, for example, an AI model that may output probability values ​​corresponding to configurable parameters included in each of the neural network and the accelerator, and may be controlled by the processor 120. In an exemplary embodiment, when a plurality of neural networks are input, the policy function model 60 may apply a policy function to the first configurable parameters included in each neural network to output a probability value corresponding to each first configurable parameter. The policy function may refer to, for example, a function that can give a high probability value to a parameter that can output a high reward value for the configurable parameter and may include a plurality of parameters. The plurality of parameters included in the policy function may be updated by the control of the processor 120.

[0084] For example, the probability value corresponding to the first configurable parameter may indicate a probability value indicating whether the neural network including the first configurable parameter is a neural network capable of outputting a higher reward value than other neural networks. For example, the first configurable parameter may be an operation method, the first neural network may execute the first operation method, and the second neural network may execute the second operation method. When the first neural network and the second neural network are input, the policy function model 60 may apply the policy function to the operation methods included in each neural network to output a probability value corresponding to each operation method. If the probability corresponding to the first operation method is 40% and the probability corresponding to the second operation method is 60%, the processor 120 may select a case where the probability of selecting the first neural network including the first operation method among the multiple neural networks is 40% and the probability of selecting the second neural network including the second operation method is 60%.

[0085] The policy function can be applied to possible parameters to output a probability value corresponding to each second configurable parameter. For example, the probability value corresponding to the second configurable parameter may refer to a probability value that the accelerator can output a higher reward value (including the second configurable parameter) than other accelerators. For example, if the second configurable parameter included in the accelerator is a convolution engine rate parameter, the first accelerator includes the convolution engine rate parameter, and the second neural network includes the convolution engine rate parameter, when the first accelerator and the second accelerator are input, the policy function model 60 can apply the policy function to the accelerator including each of the first and second convolution engine rate parameters to output a probability value corresponding to each convolution engine rate parameter. If the probability of selecting the first convolution engine rate parameter is 40% and the probability of selecting the second convolution engine rate parameter is 60%, the processor 120 may select a case where the probability of selecting the first accelerator including the first convolution engine rate parameter of the plurality of accelerators is 40%, and the probability of selecting the second accelerator including the second convolution engine rate parameter is 60%.

[0086] The evaluation model 30, the first prediction model 40, the second prediction model 50, and the policy function model 60 may have been stored in a non-volatile memory and then loaded into a volatile memory under the control of the processor 120. The volatile memory may be included in the processor 120 as an element of the processor 120, such as Figure 1 As shown, this is merely an example, and the volatile memory may be implemented as an element separate from the processor 120 .

[0087] Non-volatile memory may refer to, for example, a memory capable of retaining stored information even when power is interrupted. For example, non-volatile memory may include, but is not limited to, at least one of flash memory, programmable read-only memory (PROM), magnetoresistive random access memory (MRAM), resistive random access memory (RRAM), and the like. Volatile memory may refer to, for example, a memory requiring continuous power to maintain stored information. For example, volatile memory may include, but is not limited to, at least one of dynamic random access memory (DRAM), static random access memory (SRAM), and the like.

[0088] The processor 120 may be electrically connected to the memory 110 and control the overall operation of the electronic device 100. For example, the processor 120 may select one of a plurality of neural networks stored in the neural network sub-search space by executing at least one instruction stored in the memory 110. The processor 120 may access the neural network sub-search space stored in the memory 110. The processor 120 may input the plurality of neural networks included in the neural network sub-search space into the policy metric function model 60 to obtain probability values ​​corresponding to the first configurable parameter included in each of the plurality of neural networks. For example, if the first configurable parameter includes a layer connection scheme, the processor 120 may input the plurality of neural networks into the policy function model 60 to obtain probability values ​​corresponding to the layer connection scheme of each of the plurality of neural networks. If the probability values ​​corresponding to the layer connection scheme of each of the first and second neural networks are 60% and 40%, respectively, the processor 120 may select the first and second neural networks from the plurality of neural networks with probabilities of 60% and 40%, respectively.

[0089] The processor 120 may select an accelerator to implement the neural network selected from the plurality of accelerators. The processor 120 may access the sub-search space of the accelerator stored in the memory 110. The processor 120 may input the plurality of accelerators stored in the accelerator sub-search space into the policy function model 60 to obtain a probability value corresponding to the second configurable parameter included in each of the plurality of accelerators. For example, if the second configurable parameter is a parallelism parameter, the processor 120 may input the plurality of accelerators into the policy function model 60 to obtain a probability value corresponding to the parallelism parameter included in each of the plurality of accelerators. If the probability values ​​corresponding to the parallelism parameter included in each of the first accelerator and the second accelerator are 60% and 40% respectively, the processor 120 may select the first accelerator and the second accelerator from the plurality of accelerators with probabilities of 60% and 40% respectively as the accelerators for implementing the first neural network.

[0090] In an exemplary embodiment, when selecting a first neural network from among a plurality of neural networks, the processor 120 may obtain an estimated value of hardware performance corresponding to the first neural network via the second prediction model 50 before selecting an accelerator to implement the first neural network from among the plurality of accelerators. If the estimated value of hardware performance corresponding to the first neural network does not meet the second hardware criteria, the processor 120 may again select one of the plurality of neural networks other than the first neural network. The processor 120 may input the first neural network into the second prediction model 50 to obtain an estimated value of hardware performance corresponding to the first neural network. The estimated value of hardware performance corresponding to the first neural network may include at least one of the latency predicted to occur when the first neural network is implemented in a specific accelerator or memory footprint of the first neural network.

[0091] Processor 120 may determine whether the estimated value of the hardware performance corresponding to the neural network satisfies a second hardware criterion. If the estimated value of the hardware performance corresponding to the first neural network is determined to satisfy the second hardware criterion, processor 120 may select an accelerator from among the plurality of accelerators for implementing the first neural network. If the estimated value of the hardware performance corresponding to the first neural network is determined not to satisfy the second hardware criterion, processor 120 may select a neural network from among the plurality of neural networks other than the first neural network. If the performance of the hardware corresponding to the first neural network does not satisfy the second hardware criterion, this may mean that a high reward value cannot be obtained through the first neural network. If the hardware performance of the first neural network is determined not to satisfy the second hardware criterion, processor 120 may minimize and / or reduce unnecessary operations by excluding the first neural network. However, this is merely an exemplary embodiment, and processor 120 may select the first accelerator to implement the first neural network from among the plurality of accelerators immediately after selecting the first neural network from the plurality of neural networks.

[0092] In another embodiment, if a first neural network among a plurality of neural networks is selected, and a first accelerator among a plurality of accelerators is selected in which the first neural network is to be implemented, the processor 120 may input the first accelerator and the first neural network into the first prediction model 40 to obtain an estimated value of hardware performance corresponding to the first accelerator and the first neural network. The hardware performance corresponding to the first accelerator and the first neural network may include latency or power required when the first neural network is implemented on the first accelerator.

[0093] Processor 120 may determine whether the obtained hardware performance estimate satisfies a first hardware criterion. If the obtained hardware performance estimate is determined to satisfy the first hardware criterion, processor 120 may implement the first neural network on the first accelerator and obtain information related to the implementation. If it is determined that the obtained hardware performance does not satisfy the first hardware criterion, processor 120 may select another accelerator to implement the first neural network for a plurality of accelerators other than the first accelerator. The fact that the hardware performance of the first neural network and the first accelerator does not satisfy the first hardware criterion may mean, for example, that the reward value is not high enough to be obtained by obtaining information related to the implementation of the first neural network on the first accelerator. Therefore, if it is determined that the hardware performance of the first neural network and the first accelerator does not satisfy the first hardware criterion, processor 120 may minimize and / or reduce unnecessary operations by immediately excluding the first neural network and the first accelerator. However, this is merely an exemplary embodiment, and if the first accelerator and the first neural network are selected, processor 120 may directly implement the selected accelerator and neural network without inputting them into first prediction model 40 to obtain information related to the implementation.

[0094] The first hardware criterion and the second hardware criterion may be predetermined values ​​obtained through experiments or statistics, but may be updated by processor 120. For example, if the threshold latency of the first hardware criterion is set to 100 ms, but the average of the estimated values ​​of the latency corresponding to multiple neural networks is identified as 50 ms, processor 120 may reduce (e.g., to 60 ms) the threshold latency. Processor 120 may update the first hardware criterion or the second hardware criterion based on the estimated values ​​of the hardware performance of multiple neural networks or multiple accelerators.

[0095] The processor 120 may implement the selected neural network on the selected accelerator to obtain implementation-related information, including implementation and accuracy and efficiency metrics. The processor 120 may input the implementation-related information into the evaluation model 30 to obtain a reward value corresponding to the selected accelerator and neural network. As described above, the evaluation model 30 may normalize the accuracy metric and the efficiency metric and perform a weighted sum operation on the normalized indices to output a reward value.

[0096] If a first reward value is obtained by implementing the first neural network on the first accelerator, the processor 120 may select a second neural network to be implemented on the first accelerator from among the plurality of neural networks. The processor 120 may select the second neural network from among the plurality of neural networks by searching for neural networks that may obtain a higher reward value than when the first neural network is implemented on the first accelerator. The processor 120 may select the second neural network from among the plurality of neural networks other than the first neural network in the same manner as the first neural network was selected from among the plurality of neural networks.

[0097] The processor 120 may obtain implementation-related information by implementing the second neural network selected on the first accelerator. Before implementing the second neural network on the first accelerator, the processor 120 may input the first accelerator and the second neural network into the first prediction model 30 to identify whether the hardware performance corresponding to the first accelerator and the second neural network meets the first hardware criteria. If the hardware performance corresponding to the first accelerator and the second neural network is identified as meeting the first hardware criteria, the processor 120 may implement the second neural network on the first accelerator to obtain implementation-related information. However, this is merely an exemplary embodiment, and the processor 120 may directly obtain implementation-related information without inputting the first accelerator and the second neural network into the first prediction model 30.

[0098] Processor 120 may implement the first accelerator and the second neural network to obtain a second reward value based on the obtained accuracy metric and efficiency metric. Processor 120 may select the neural network and accelerator with the largest reward value from among the plurality of accelerators based on the first reward value and the second reward value. For example, a second reward value greater than the first reward value may indicate that implementing the first neural network on the first accelerator is more efficient and accurate than implementing the second neural network. Processor 120 may identify that the first accelerator and the second neural network are more optimized and / or improved compared to the first accelerator and the first neural network.

[0099] The processor 120 may select an accelerator to implement the second neural network in a plurality of accelerators other than the first accelerator. When the second accelerator is selected as the accelerator for implementing the second neural network, the processor 120 may implement the second neural network on the second accelerator to obtain information related to the implementation, and obtain a third reward value based on the obtained information related to the implementation. The processor 120 may compare the second reward value with the third reward value to select a pair of accelerators and neural networks that can output a higher reward value. The processor 120 may select a pair of neural networks and accelerators that can output the maximum reward value among the stored accelerators and neural networks by repeating the above operation. The pair of neural networks and accelerators that can output the maximum reward value can perform specific tasks more accurately and efficiently than other pairs, such as, but not limited to, image classification, speech recognition, etc.

[0100] The processor 120 may include various processing circuits, such as, but not limited to, a central processing unit (CPU), a dedicated processor, a microcontroller unit (MCU), a microprocessing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an advanced reduced instruction set computing (RISC) machine (ARM) processor for processing digital signals, etc., or may be defined as one or more of the corresponding terms. The processor 120 may be implemented in, for example, but not limited to, a system on chip (SoC) type or a large-scale integration (LSI) type in which processing algorithms are implemented, or in a field programmable gate array (FPGA). The processor 120 may perform various functions by executing computer-executable instructions stored in the memory 110. The processor 120 may include at least one of a graphics processing unit (GPU), a neural processing unit (NPU), and a visual processing unit (VPU) for performing AI functions, which may include only an AI processor.

[0101] AI-related functions are operated through processors and memories. The one or more processors may include, for example, but not limited to, general-purpose processors such as central processing units (CPUs), application processors (APs), digital signal processors (DSPs), dedicated processors, and graphics-only processors such as graphics processing units (GPUs), visual processing units (VPUs), and AI-only processors such as neural network processors (NPUs), but the processors are not limited thereto. The one or more processors may control the processing of input data according to predetermined operating rules or AI models stored in the memory. If the one or more processors are AI-only processors, the AI-only processors may be designed to have a hardware structure dedicated to the processing of a specific AI model.

[0102] The predetermined operating rules or AI models can be produced by learning. For example, producing by learning can mean that a predetermined operating rule or AI model set for performing a desired feature (or purpose) is produced by training a basic AI model using various training data using a learning algorithm. Learning can be implemented by a separate server and / or system, but is not limited thereto, and can be implemented in an electronic device. Examples of learning algorithms include, but are not limited to, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning.

[0103] The AI ​​model may be composed of multiple neural network layers. Each of the multiple neural network layers may include multiple weighted values, and a neural network operation may be performed by operating between the results of the previous layer and multiple parameters. The parameters included in the multiple neural network layers may be optimized and / or improved by learning the results of the AI ​​model. For example, multiple weight values ​​may be updated so that the loss value or cost value obtained by the AI ​​model during the learning process can be reduced or minimized.

[0104] Figure 2 is a flowchart illustrating an exemplary process for determining whether to implement a first neural network on a first accelerator by the electronic device 100 through a first prediction model according to an embodiment.

[0105] In step S210, the electronic device 100 may select a first neural network from a plurality of neural networks, and select a first accelerator for implementing the first neural network from a plurality of accelerators. Figure 1 The process of selecting the first neural network and the first accelerator by the electronic device 100 has been described and will not be further described here.

[0106] In step S220, the electronic device 100 may obtain estimated values ​​corresponding to the hardware performance of the first neural network and the first accelerator using the first prediction model. When the first neural network and the first accelerator are input, the first prediction model may output estimated values ​​corresponding to the hardware performance of the first neural network and the first accelerator. For example, when the first neural network is implemented on the first accelerator, the first prediction model may output the estimated latency and power required.

[0107] In step S230, the electronic device 100 may identify whether the obtained estimated value of the hardware performance satisfies the first hardware criterion. For example, if the estimated waiting time required when implementing the first neural network on the first accelerator exceeds the first hardware criterion, the electronic device 100 may identify that the estimated value of the hardware performance corresponding to the first neural network and the first accelerator does not satisfy the first hardware criterion. As another example, if the estimated power consumed when implementing the first neural network on the first accelerator does not exceed the first hardware criterion, the electronic device 100 may identify that the estimated value of the hardware performance corresponding to the first neural network and the first accelerator satisfies the first hardware criterion.

[0108] If the estimated value of the hardware performance corresponding to the first neural network and the first accelerator does not satisfy the first hardware criterion ("No" in S230), then in step S240, the electronic device 100 may select a second accelerator for implementing the first neural network from among the accelerators other than the first accelerator. The fact that the estimated value of the hardware performance corresponding to the first neural network and the first accelerator does not satisfy the first hardware criterion may mean that a high reward value cannot be obtained through the first neural network and the first accelerator. The electronic device 100 may minimize and / or reduce unnecessary operations by selecting a pair of neural networks and accelerators other than the first neural network and the first accelerator pair.

[0109] If the estimated values ​​of the hardware performance corresponding to the first neural network and the first accelerator satisfy the first hardware criterion ("Yes" in S230), then in step S250, the electronic device 100 may implement the first neural network on the first accelerator. Since the estimated values ​​of the hardware performance corresponding to the first neural network and the first accelerator satisfy the first hardware reference, the electronic device 100 may obtain information related to implementation by actually implementing the first neural network on the first accelerator.

[0110] Figure 3 is a flowchart illustrating an exemplary process for determining whether to select, by the electronic device 100, an accelerator for implementing the first neural network through the second prediction model.

[0111] In step S310, the electronic device 100 may select a first neural network from a plurality of neural networks. The process of selecting the first neural network from a plurality of neural networks by the electronic device 100 has been described above, and therefore, it is not necessary to repeat the description here.

[0112] In step S320, the electronic device 100 may obtain an estimated value of the hardware performance corresponding to the first neural network using the second prediction model. When the first neural network is input, the second prediction model may output an estimated value of the hardware performance corresponding to the first neural network. For example, when the first neural network is implemented on a specific accelerator, the second prediction model may estimate the latency or coverage area of ​​the first neural network that is estimated to be required.

[0113] In step S330, the electronic device 100 may identify whether the estimated value of the hardware performance corresponding to the obtained first neural network satisfies the second hardware reference. For example, if the estimated waiting time required when implementing the first neural network on a specific accelerator exceeds the second hardware reference, the electronic device 100 may identify that the estimated value of the hardware performance corresponding to the first neural network does not satisfy the second hardware criterion. As another example, if the capacity of the first neural network satisfies the second hardware criterion, the electronic device 100 may identify that the estimated value of the hardware performance corresponding to the first neural network satisfies the second hardware criterion.

[0114] If it is determined that the estimated value of the hardware performance corresponding to the first neural network does not meet the second hardware criterion ("No" in S330), then in step S340, the electronic device 100 may select one of the multiple neural networks other than the first neural network. The fact that the estimated value of the hardware performance corresponding to the first neural network does not meet the second hardware criterion may mean that it does not obtain a high reward value via the first neural network. Therefore, the electronic device 100 can minimize and / or reduce unnecessary operations by selecting another neural network other than the first neural network from the multiple neural networks.

[0115] If it is identified that the estimated values ​​of the hardware performance corresponding to the first neural network and the first accelerator satisfy the second hardware criterion ("Yes" in S330), then in step S350, the electronic device 100 may select an accelerator for implementing the first neural network from among the plurality of accelerators. Figure 1 The process of selecting an accelerator by the electronic device 100 to implement the first neural network has been described, and thus a detailed description thereof will not be repeated here.

[0116] Figure 4A 、 4B 4C are flow charts illustrating exemplary methods for designing accelerators and parameterizable algorithms by the electronic device 100 . Figure 4A and 4B An example in which the parameterizable algorithm is implemented as a convolutional neural network (CNN) is shown, but this is merely an example. For example, the parameterizable algorithm may be implemented as another type of neural network.

[0117] like Figure 4AAs shown, the electronic device 100 selects a first convolutional neural network (CNN) architecture from the CNN search space stored in the memory 110 (S400). At the same time or within a threshold time range, the electronic device 100 may select a first accelerator architecture from the accelerator sub-search space in step S402. In step S404, the electronic device 100 may implement the selected first CNN on the selected first accelerator architecture. In step S406, the electronic device 100 may obtain information related to or associated with the implementation by implementing the first CNN on the selected first accelerator, including accuracy metrics and efficiency metrics. The efficiency metric may include, for example, but not limited to, the latency, power, area, etc. of the accelerator, which is required to implement the neural network on the accelerator. In step S408, the electronic device 100 may obtain a reward value based on the information related to the obtained implementation. Then, in step S410, the electronic device 100 may use the obtained reward value to select or update a pair of optimized CNN and accelerator (e.g., FPGA). The electronic device 100 may repeat the above process until the best CNN and FPGA pair is selected.

[0118] Figure 4B is shown for implementing Figure 4A The processor 120 of the electronic device 100 can select the first CNN and the first FPGA from the CNN sub-search space and the FPGA sub-search space (or FPGA design space) and implementation-related input information by implanting the first CNN into the evaluation model 30. The evaluation model 30 can output the reward obtained based on the implementation-related information.

[0119] This method can be described as a reinforcement learning system to jointly optimize and / or improve the structure of the CNN and the underlying FPGA accelerator. As mentioned above, the related art NAS can adjust the CNN to a specific FPGA accelerator, or adjust the FPGA accelerator for a newly discovered CNN. However, the NAS according to the present disclosure can collaboratively design the CNN and its corresponding FPGA accelerator.

[0120] Figure 4C 1 is a diagram illustrating an exemplary configuration of the processor 120. Figure 4C As shown, the processor 120 includes a plurality of single long short-term memory (LSTM) units, followed by corresponding dedicated fully connected (FC) layers, with each output having one unit and one FC layer. The result output from the FC layer connected to a single LSTM unit can be input to the next LSTM unit. In this case, the result output from the FC layer can be a parameter for configuring CNN or accelerator hardware. In an exemplary embodiment, as Figure 4CAs shown, the processor 120 can first obtain the parameters for configuring the CNN via multiple single LSTM units and the FC layer coupled thereto, and then can obtain the hardware parameters of the FPGA accelerator. The first and second configurable parameters of each of the CNN and the FPGA accelerator are processed as outputs and have their own units and FC layers. Once all configurable parameters are obtained, the processor 120 can send the CNN and accelerator to the evaluation model 30 for evaluating the CNN and accelerator.

[0121] Figure 4C The processor 120 shown is an extension of the traditional RL-based NAS and can be referred to as an RL agent. Therefore, the processor is based on an LSTM unit. However, the processor 120 can implement a completely different algorithm, such as a genetic algorithm, and thus can have a different structure. The processor 120 is responsible for taking a finite sequence of actions that are converted into a model structure. Each action can be referred to as a Figure 4C Each decision is selected from a finite set of options and together with other decisions selected by the processor 120 in the same iteration form a model structure sequence s. The set of all possible search spaces s can be formally defined as:

[0122] S=O1×O2×...O n (1)

[0123] where Oi is the set of available options for the i-th decision. In each iteration t, the processor 120 generates a sequence of structures st.

[0124] The sequence st is passed to the evaluation model, which evaluates the proposed structure and creates a reward rt generated by a reward function R(st) based on the evaluated metric. The reward is then used to update the processor so that (as t→∞) it chooses the sequence st that maximizes the reward function.

[0125] There are different approaches to solving the problem of updating the processor. For example, in deep RL, a DNN can be used as a trainable component and backpropagation is used to update it. For example, in the above Figure 4A In the REINFORCE method outlined in [1], the processor 120 DNN (a single LSTM cell as described above) implements a policy function π that generates a sequence of probability distributions, one for each decision, samples the probability distributions to select elements from their respective sets of zeros, and thus the decision sequence s. The network is then updated by computing the gradient of the product of the observed reward r and the total probability of choosing the sequence s. This is described with reference to Equation 4 below.

[0126] [Equation 4]

[0127]

[0128] where D = {D1, D2, ..., Dn} is the set of probability distributions for each decision. Since s is a 0 generated from a sequence of independent sampled decisions s1, s2, ..., sn, the total probability p(s|D) can be easily calculated as:

[0129]

[0130] RL-based algorithms are convenient because they do not impose any restrictions on what are the elements of s (what are the available options) or how to compute the reward signal from s. Therefore, without loss of generality, we can abstract some details and in practice simply identify each available option by its index. The sequence of indices selected by the processor 120 is then converted into a model and subsequently evaluated to construct the reward signal independently of the algorithm described in this section. Different strategies can be used without compromising the basic approach. Following this property, the search space can be described using a shortened notation via Equation 5:

[0131] [Equation 5]

[0132] S=(k1,k2,...,k n )k i ∈N + (4)

[0133] It should be understood that the search space S as defined in Equation 1 has |0i| = ki, where k is the number of options available for each parameter.

[0134] An overview of the general algorithm is illustrated by the following non-limiting example of the algorithm:

[0135]

[0136] The REINFORCE algorithm or a similar algorithm can be used to combine evaluation metrics and generate a reward function to search. The algorithm can include a policy function that takes weights / parameters and a distribution Dt can be obtained from the policy function. A sequence st from this distribution can then be sampled. When searching the combinatorial space, the sequence contains FPGA parameters and CNN parameters. The sequence is then evaluated by running the evaluation model 30 of the selected CNN on the selected FPGA, or the performance is simulated as described in more detail below. Metrics mt are measured by the evaluation model 30, such as latency, accuracy, area, power. These metrics are used as inputs to the reward function R(mt). The reward function is used together with the probability of selecting the sequence to update the parameters / weights of the policy function. This allows the policy function to learn to select the sequence that maximizes the reward.

[0137] Figure 4A The presented approach extends traditional NAS by including multiple decisions related to the design selection of an FPGA accelerator. Therefore, the search space is defined as the Cartesian product of the neural network sub-search space (SNN) and the FPGA sub-search space (SFPGA), and is defined as Equation 1. Where SNN is the search space and SFPGA is the extension related to the FPGA accelerator design.

[0138] The above search space is not fundamentally different from the definition provided in Equation 5 and does not imply any changes to the search algorithm. However, since the search domains of the two parts are different, it may be helpful to clearly distinguish them and use this distinction to illustrate their synergy. Each sub-search space is discussed in more detail below.

[0139] Figure 5 is a diagram showing that according to an embodiment, Figure 4A Figure 2 shows an example of a well-defined CNN search space used in the method of [ 0 ]. It should be understood that this is just one example of a well-defined search space that can be used. The search space is described in detail in "NAS Bench 101: Towards Reproducible Neural Architecture Search" by Ying et al., published on ArXiv e-prints (February 2019), which is incorporated herein by reference in its entirety and may be referred to as NASBench. Figure 5 An example structure of a CNN within the search space is shown. As shown, the CNN comprises three stacks 302, 304, and 306, each containing three units 312, 314, and 316. Each stack uses the same unit design, but operates on the data in different dimensions due to the downsampling modules interleaved with the stack. For example, the input data for each stack is x2 smaller in both the X and Y dimensions than the previous data, but contains x2 more features, which is standard practice for classification models. This framework is fixed, with the only variable in each model being the innermost design of the individual units.

[0140] The search space for cell design can be restricted to a maximum of 7 operations (the first and last operations are fixed) and 9 connections. These operations are selected from the following available options: 3×3 or 1×1 convolution, and 3×3 max pooling, all with stride 1, and the connections need to be "forward" (e.g., the adjacency matrix of the underlying computational graph needs to be upper triangular). In addition, when more than one connection enters an operation, concatenation and element-wise addition operations are automatically inserted. As in Equation (1), the search space is defined as a list of options (e.g., configurable parameters), in this case, the CNN search space contains 5 operations, each with 3 options, and 21 connections, which can be true or false (2 options), and the 21 connections are non-zero values ​​in the adjacency matrix between the 7 operations.

[0141] SCNN=(3,3,…3,2,…2)(6)5 times 21 times

[0142] The search space does not directly capture the requirement of having at most 9 connections and therefore contains invalid points, such as points in the search space for which it may not be possible to create a valid model. Additionally, a point may be invalid if the output node of a unit is disconnected from its input.

[0143] Figure 6 4 is a diagram illustrating an exemplary FPGA accelerator 400 and its connected system-on-chip 402 and external memory 404. The FPGA accelerator 400 includes one or more convolution engines 410, a pooling engine 412, an input buffer 414, a weight buffer 416, and an output buffer 418. A library for accelerating DNNs on a system-on-chip FPGA, such as the library shown in FIG4 , is described in “Chaidnn v2-HLS based DeepNeuralNetwork Accelerator Library for Xilinx Ultrascale+MPSoCs” from Xilinx, Inc., 2019, which is incorporated herein by reference in its entirety and is hereinafter referred to as the ChaiDNN library.

[0144] The search space of the FPGA accelerator is defined by configurable parameters of each key component of the FPGA accelerator. As described in more detail below, the configurable parameters that define the search space include parallelism parameters (e.g., parallel output features or parallel output pixels), buffer depth (e.g., for input, output, and weight buffers), memory interface width, pooling engine usage, and convolution engine ratio.

[0145] Configurable parameters for the convolution engine include the parallelism parameters "filter_par" and "pixel_par", which determine the number of output feature maps and the number of output pixels to be generated in parallel, respectively. The parameter convolution engine ratio "ratio_conv_engines" is also configurable and has been newly introduced in this method. This ratio can determine the number of DSPs allocated to each convolution engine. When set to 1, this can mean, for example, that there is a single general-purpose convolution engine that runs any type of convolution, and a value of 1 can be considered the default setting used in the ChaiDNN library. When set to any number lower than 1, there are dual convolution engines - for example, one of them is dedicated to and tuned to 3×3 filters, and the other is used for 1×1 filters.

[0146] The configurable parameter used by the pooling engine is "pool_enable". If this parameter is true, additional FPGA resources are used to create a separate pooling engine. Otherwise, the pooling function in the convolution engine is used.

[0147] exist Figure 6 In the illustrated implementation, there are three buffers: input buffer 414, weight buffer 416, and output buffer 418. Each buffer has a configurable depth and resides in the FPGA's internal block memory. In the current CHaiDNN implementation, the buffers need to have enough space to accommodate the input feature maps, output feature maps, and weights for each layer. A larger buffer size of 5 allows for larger images and filters without fetching data from slower external memory. As described below, feature and filter slicing can improve the accelerator's flexibility.

[0148] The FPGA communicates with the CPU and external DDR4 memory 404 via an AXI bus. As in the CHaiDNN library, configurable parameters allow configuration of the memory interface width to achieve a trade-off between resources and performance.

[0149] The FPGA accelerator search space for parameters (filter_par, pixel_par, input, output, weight buffer depth, mem_interface_width, pool_en, and ratio_conv_engines) is defined below.

[0150] S_{FPGA}=(2,5,4,3,3,2,2,6)(7)

[0151] Considering the details of the evaluation model in more detail, it should be noted that the area and latency of the accelerator are determined by the parameters in the accelerator design space. Therefore, compiling all configurations in the design space to measure area and latency online during NAS is not practical, as each compilation takes several hours and running CNN models simultaneously requires thousands of FPGAs. Therefore, a fast evaluation model can be used to find efficiency metrics.

[0152] For each accelerator architecture, Figure 4A Step S406 can be completed in stages: first, using an area model. Equations can be used to estimate FPGA resource utilization in terms of CLBs, DSPs, and BRAMs to model the CLB, DSP, and BRAM usage of each subcomponent. An example subcomponent is the row buffer within the convolution engine, which varies in size based on the configurable parameters "filter_par" and "pixel_par." The equation uses these two variables as input and gives the number of BRAMs.

[0153] When the configurable parameter "ratio_conv_engines" is set to less than 1, it is possible to have two specialized convolution engines. In this case, the CLB and DSP usage of the convolution engine is reduced by 25% compared to the general-purpose convolution engine. This is a reasonable estimate of the potential area savings that can be achieved due to specialization, and much larger savings have been demonstrated in the literature. In addition, when a separate pooling engine is used and the configurable parameter "pool_enable" is set to 1, a fixed number of CLBs and DSPs are consumed.

[0154] The BRAMs buffer data for the convolution and pooling engines. The size of the input, output, and weight buffers is configurable by depth. This data is double buffered, thus consuming twice the amount of BRAM. A fixed number of BRAMs are also dedicated to pooling (if enabled), bias, scale, mean, variance, and beta. Assuming each BRAM is 36K bits, calculate the number of BRAMs. Then, based on the FPGA resource usage, the next step is to estimate the number of BRAMs in mm. 2The area is quantified to a single number - silicon area - by taking the FPGA size in units. The area of ​​each resource is scaled relative to the CLB. Since this data is not available for the device used, data for a similar device is used from "Design Tradeoffs for Hard and SoftFPGA-based Network on Chips" published by Abdelfattah et al., International Conference on Field Programmable Technology 95-103 (2012), which is incorporated herein by reference in its entirety. Different block characteristics are also used to take into account the smaller process nodes (20nm vs. 40nm) (8 LUTs per CLB instead of 10, and 36K bits per BRAM instead of 9K bits). The table below shows the estimated block area of ​​a device that can be used in this method.

[0155] resource Correlation area (CLB) <![CDATA[Tile area (mm 2 )]]> CLB 1 0.0044 BRAM 6 0.026 DSP 10 0.044 total 64,922 286

[0156] Figure 7A is a graph illustrating the area of ​​various example accelerator architectures. For configurable parameters "filter_par" = 8 and "filter_par" = 16, the lines plot the estimated resource usage by area. Area measurements have also been calculated and displayed on the graph. The graph shows that the predictions of the area model are valid for actual measurements. Note that the model has predicted the area of ​​accelerator architectures that are not currently supported by CHaiDNN, but, for example, the size of the smallest architecture with configurable parameters "filter_par" = 8 and "pixel_par" = 4 is 96.43 mm 2 , and the maximum frame size with configurable parameters "filter_par" = 16, "pixel_par" = 64 is 218.6220mm 2 .

[0157] Once the FPGA resource usage in terms of CLBs, DSPs, and BRAMs has been estimated, as Figure 4A As part of step S406 of , for example using a waiting time model, the waiting time may be estimated. It should be understood that in this example, the waiting time is estimated before the waiting time, but the estimation may be performed in any order.

[0158] For example, a latency model can include two components: 1) an operation latency lookup table and 2) a scheduler. 85 operations were obtained from the NASBench search space, including 3×3 and 1×1 convolutions, max pooling, and element-wise addition operations of various dimensions. Each operation was run on FPGA accelerators with different configurations, and the latency was described using the performance evaluation API provided by CHaiDNN. The latency numbers were then stored in a lookup table. The scheduler appropriately assigned operations to parallel computing units and used the latency of the operations in the lookup table to calculate the total latency of the CNN model.

[0159] The latency of the convolution operation depends on the parallelism factors "filter_par" and "pixel_par". Since CHaiDNN does not support the architectures "filter_par=8", "pixel_par=4" and "filter_par=16", "pixel_par=64", their latency is interpolated using measurements from other architectures. In the case of dual convolution engines, one is dedicated to 3×3 filters and the other is dedicated to 1×1 filters. The performance of the corresponding convolution scales proportionally with the number of available engines. For example, when the parameter ratio_conv_engines=0.75, the latency of 3×3 convolution increases by 1 / 0.75, and the latency of 1×1 convolution increases by 1 / 0.25.

[0160] In the original CHaiDNN accelerator, the data buffer must be sized to fit the entire input, output, and filter tensors to obtain the highest possible throughput. However, if the image resolution increases and the CNN becomes deeper, this partitioning scheme is not feasible and limits the feasibility of the accelerator. Figure 7A To the approach described in

[15] , a scheme can be added where slices of the input tensor are fetched from external memory into input buffers and processed independently by the accelerator. Furthermore, when the output and weight buffers are full, the output layer and filter weights overflow to external memory, so performance is limited by the memory bandwidth, which depends on the configurable parameter "mem_interface_width".

[0161] Therefore, when building the latency model, some assumptions have been made due to limitations of the current implementation of CHaiDNN. First, the performance evaluation API does not support max pooling running on a standalone engine, so the latency is modeled as 2 times faster than the latency running on the convolution engine. Second, the memory interface width cannot be configured independently. It is related to the DIET_CHAI_Z configuration, which includes a set of parameters, and the memory interface width depends on the AXI bus having a reduced width when DIET_CHAI_Z is enabled. Without bringing all parameters to the accelerator design space, the model assumes that the latency increases by 4% when the parameter "mem_interface_width" is reduced from 512 bits to 256 bits. Finally, the method used in the model does not take into account the operation fusion used by the accelerator's runtime to optimize latency.

[0162] Figure 7B is a graph illustrating the validation results of the latency model. First, the latency is estimated by the model for different accelerator architectures, and the results are presented in Figure 7B We then ran the model on an FPGA accelerator and measured Figure 7B The end-to-end latency is shown in Figure 2. This figure shows that, although assumptions can be made, the latency model is able to describe the trend of latency with respect to the level of parallelism. It should be noted that for Figure 7A and 7B , HW pooling is enabled, the memory interface width is 512 bits, the buffer size is [8192, 2048, 2048], the batch size is 2 and the clock frequency is 200MHz.

[0163] Figure 8 is a graph illustrating the latency of all convolution operations extracted from the lookup table versus the parameters GFLOPS(size) and pixel_par. As shown in the figure, the latency increases with the data size and decreases with more parallelism in the convolution engine.

[0164] like Figure 4A As shown, a reward based on these metrics (e.g., latency, size, and accuracy) is generated (step S408), and this is used to update the selection of CNN and FPGA (S410). As an illustration of the complexity of this implementation, Figure 9is a graph illustrating an exemplary Pareto optimal point, such as described in "Multiobjective Optimization, Interactive and Evolution Approach" by Branke et al., published by Springer in 2008, the entire contents of which are incorporated herein by reference. The CNN accuracy in NASBench was pre-computed and stored in a database, and the above-mentioned FPGA accelerator model was quickly run on a desktop computer. This allowed the entire co-design search space to be enumerated with 370 million data points. The Pareto optimal point within the three billion points was then located by iteratively filtering the dominant points from the search space. A dominant point is a point that is worse than at least one other point on all three metrics (area, latency, accuracy). The remaining (non-dominated) points are optimal in at least one of our evaluation metrics (area, latency, or accuracy). For our search space, there are only 3096 Pareto optimal model-accelerator pairs, and these are in Figure 9 Shown in.

[0165] like Figure 9 As shown, there is a three-way tradeoff between area, latency, and accuracy—to improve one, the others must be reduced. As shown in the scatter plot, the search space consists of approximately concentric accuracy-latency tradeoff curves, each at a different accelerator area. By modifying the CNN, we roughly move along the concentric accuracy-latency curves. By changing the accelerator hardware, we move along the horizontal line (thus affecting both latency and area).

[0166] Figure 10 is a graph illustrating the performance of the co-designed CNN and FPGA compared to models and accelerators found using other methods (e.g., GoogLeNet, Resnet, and SeczeNet). According to one embodiment, ChaiDNN is manually optimized to run GoogLeNet and Resnet, and Figure 10 As shown in , the latency of GoogLeNet is very close to the ParetoFront (e.g., the above method). However, for re-building the network, it is further away from the Pareto Front. Even though it has improved in accuracy compared to GoogLeNet, it is three times further away from the Pareto Front in terms of latency, as shown in Figure 10 This demonstrates the effectiveness of collaboratively designing the model and accelerator compared to sequentially designing the model and accelerator one after the other.

[0167] Figure 11A 、 11B, 11C and 11D are graphs showing exemplary accuracy-latency Pareto frontiers for single and dual convolution engines under different area constraints according to an embodiment. As described above, the configurable parameter ratio_conv_engines determines whether there is a single engine or dual engines, and the ratio of DSPs allocated to each dual engine. This affects the speed at which 1×1 and 3×3 convolutions run. This accelerator parameter creates an interesting tradeoff with the CNN search space. First, the CNN unit needs to be easily parallelizable to benefit from a parameter ratio_conv_engines that is less than 1. Second, different ratio_conv_engines will be more efficient based on the ratio of 3×3:1×1 operations in the CNN unit. For this parameter, we demonstrate how co-design can achieve the best results and find the right combination of CNN and accelerator to obtain the best accuracy and efficiency.

[0168] Figure 11A 、 11B Figures 11C and 11D show that dual engines with tighter area constraints are more effective, while a single general-purpose engine is generally better when the area constraint is larger. This demonstrates that dual engines are indeed a useful accelerator feature—a non-obvious conclusion given the interaction between CNN model parallelism, the dual engine's scheduling algorithm, and the ratio of DSPs allocated to each type of convolution engine. If we had studied this accelerator feature with a single CNN model or even a small number of manually designed models, it would not have been possible to obtain this conclusion—the dual engine might simply be unsuitable for these specific manually designed models. However, through co-design, we can search for the best model among hundreds of thousands of CNN models to fit a given accelerator feature.

[0169] Having determined that dual specialized engines can be useful accelerator compute cores, we consider more carefully the actual ratio of DSPs allocated to 1×1 and 3×3 convolutions. In a practical NAS search scenario, we can constrain the area for a specific FPGA device and look for the fastest model that exhibits a certain accuracy threshold. Figure 12A is a graph showing the results of these constraints when searching for Pareto optimal points according to an embodiment. The top four models found for each different ratio_conv_engines value are highlighted. The points found illustrate the independence between the CNN model and the accelerator architecture. For example, when the accelerator contains more calculations for 1×1 convolution and similar calculations for 3×3, there are more 1×1 conv1 operations in the CNN unit.

[0170] Figure 12B and 12C: is a diagram showing exemplary CNN units corresponding to ratio_conv_engines equal to 0.33 and 0.67, respectively, according to an embodiment. As shown in the figure, when ratio_conv_engines=0.67, the best model has three 1×1 convolutions and four 3×3s, while for ratio_conv_engines=0.33, the count moves to five 1×1s and two 3×3s.

[0171] Figure 12D Comparison of the Examples Figure 12C Figure 2 shows the execution schedule of the CNN in

[15] running on its co-designed accelerator or a “different” accelerator, e.g., Figure 12C Both designs are subject to the same area constraint. As shown in the figure, the latency on the co-designed accelerator is much lower (48ms vs. 72ms), and the utilization of the convolution engines is much higher, while on the "different" accelerator, it is clear that the 1×1 engine is underutilized and the 3×3 engine becomes the bottleneck.

[0172] Figure 13 is a diagram showing the overall situation of the Pareto optimal co-designed CNN model accelerator pair with respect to the parameter ratio_conv_engines according to one embodiment. As shown in the figure, when more DSPs are allocated for 1×1 convolution (ratio = 0.25), the Pareto optimal design has low precision. In contrast, when more computations are allocated to 3×3 convolution (ratio = 0.67), we obtain points with higher precision. In fact, this may be due to the higher precision caused by increasing the use of 3×3 convolution. In addition, a single convolution engine appears to be superior for low latency design. Moreover, we find similar points when ratio = 0.5 or 0.33. We can continue to make useful observations in this way to help guide the manual design of accelerators. However, as mentioned above, the goal is to use NAS for automatic search.

[0173] A machine learning task (e.g., image classification) can be represented as a DNN search space, and a hardware accelerator can be represented by its parameters (forming the FPGA search space). Figure 4A As shown, a reward based on a metric (e.g., latency, size, and accuracy) is generated (step S208), and this is used to update the selection of CNN and FPGA (S410). These steps can be performed using multi-objective optimization (MOO) of latency, accuracy, and area, as well as different search algorithms for navigating the co-design search space as described below.

[0174] As mentioned above, there are fundamental tradeoffs between the three metrics, and therefore, there is no easy solution to the optimization problem. Therefore, additional steps must be taken to be able to define "better" and "worse" co-designs. Ultimately, we want to obtain a function that will take the metrics into account and return a scalar value that is interpreted as the quality of the co-design in question. We will use this function as the reward function R from the REINFORCE algorithm shown above.

[0175] Consider two standard approaches to solving the MOO problem. The first approach is to combine the three metrics using a weighted sum into one objective function, as described in "Multiobjective Optimization, Interactive and Evolution Approach" by Branke et al., Springer 2008, which is incorporated herein by reference in its entirety. The second approach is to consider only the set of points for all metrics except one metric that is below / above a certain threshold, and then optimize for the remaining metrics (a constrained approach). We then also consider hybrid approaches in computing the reward function, where either fewer metrics are constrained, and / or constrained metrics are also considered. The general MOO reward function used in this application can be formally defined as Equation 6:

[0176] [Equation 6]

[0177]

[0178] where m is the vector of metrics we want to optimize, w is the vector of their weights, and is the vector of thresholds used to constrain the domain of the function.

[0179] For cases where at least two metrics are added together, we normalize their values ​​to make them more comparable to each other, since different metrics use different units and have values ​​from different ranges. A similar effect can be achieved by adjusting their weights relative to their absolute values, but we find that normalized values ​​are easier to reason about. That is, even after normalization, it may still be non-obvious how different metrics contribute to the objective function for a given set of weights.

[0180] A minor technical issue that must be addressed is that RL algorithms work by maximizing a reward function, but different metrics require different types of optimization (maximizing accuracy, minimizing area and latency). We handle this by including negative area and latency as input to the reward function. We also take care to produce positive values ​​for all metrics by handling negative values ​​during normalization whenever performing a weighted sum.

[0181] We explored three different normalization strategies, which are described in more detail in Marlez et al., "Function-Transformation Methods for multi-objective optimization," Engineering Optimization 37, 6 (2005), 551-570, the disclosure of which is hereby incorporated by reference in its entirety. The first strategy is maximum normalization, one of the most common methods, and normalizes values ​​relative to their maximum possible value. For negative values, we consider their absolute value and treat them similarly. In this case, our normalization function can be formally defined as Equation 7.

[0182] [Equation 7]

[0183]

[0184] Another common normalization method is min-max normalization, which considers both the minimum and maximum values ​​of the metric. This range is then linearly mapped to the [0,1] range. The specific function can be defined as Equation 8

[0185] [Equation 8]

[0186]

[0187] The third normalization method is standard deviation normalization, which uses the standard deviation of each value to normalize each value. This equation can be defined as Equation 9

[0188] [Equation 9]

[0189]

[0190] By combining the general weighted sum equation (Equation 6) with a selected normalization function (one of Equations 7 to 9, such as Equation 8), the MOO problem can be defined as Equation 10.

[0191] [Equation 10]

[0192]

[0193] where ar is area, lat is latency, and acc is accuracy), w1, w2, w3 are sets of weights for each of area, latency, and accuracy, and optimization is performed on the search space s S so that the evaluation model output satisfies the given constraints (e.g., latency is below a certain value).

[0194] If the search point does not satisfy the specified constraints, a penalty function Rv is used as feedback to the processor to prevent the processor from searching for similar points that fall below our requirements. Since the standard reward function is positive, we want to prevent the processor from choosing invalid points. A simple solution is to make the penalty function negative. We use the same function as the standard reward function R, but with two changes: 1) instead of (ar, lat, acc), we use (ar-thride, lat-latth, acc-accth), and 2) we use the opposite function to make Rv negative, thus notifying the processor that this is a bad choice.

[0195] It is also possible to consider different weights for the MOO problem to explore how their choice affects the results of the search. For example, the weights can be set equal for each metric, such as 1 / 3, or the weights can be set to prioritize one metric, such as by setting w1 to 0.5, w2 to 1, and w3 to 0.25 to prioritize area when solving the optimization problem. Each weight can be in the range [0,1], and the sum of the weights is equal to 1.

[0196] There are two methods for updating the selection of CNN and FPGA (S410). In the first method, the two sub-search spaces can be considered together so that the algorithm can be implemented directly on the two spaces. This method can be called a combined search. This strategy has the ability to update the CNN and accelerator at each step, and is therefore able to make faster changes to adapt to the reward function. However, the combined search space (e.g., SNNxSFPGA) is much larger, which may make it more difficult to find the optimal point (e.g., the best choice). Therefore, each experiment is run for a maximum number of steps (e.g., 10,000 steps), and the metrics are evaluated so that the reward function can be calculated.

[0197] When running the actual search, it is important to consider invalid and constrained points that can be selected by the processor and the appropriate response when such a point is identified. This behavior does not conform to the standard MOO formulation because MOO has no concept of exploration; instead, it simply provides a method for pre-selecting multi-dimensional points in a comparable manner. However, when running the search, the reward function has additional meaning because it is directly used to guide the processor towards the desired result. Therefore, simply ignoring invalid and constrained points can potentially lead to a situation where the processor's feedback is only related to one metric, which may later lead the processor to select more points that maximize it without considering the other two metrics. Therefore, it is preferable to provide a supplementary reward function for use with invalid and constrained points whenever weights equal to zero are used for some metrics within the standard reward function. Otherwise, we run the risk of this situation when learning to navigate the space and the processor simply does not take certain metrics into consideration.

[0198] As described above, the method is used to co-design FPGA and CNN, for example, by using a combinatorial search. As an alternative to a combinatorial search, the search can have a well-defined dedicated phase during which one part (e.g., the FPGA design) is fixed or frozen so that the search is focused on the other part (e.g., the CNN design), or vice versa.

[0199] Figure 14 is a block diagram illustrating an exemplary alternative architecture that may be used to implement a phased search according to an embodiment. As shown, there are two different processors (eg, each processor including processing circuitry) 1400 , 1420 and an evaluation model 1422 . Figure 14 The estimation model 1422 is described as being loaded into a separate volatile memory rather than into processors 1400 and 1420, but this is merely an example and the estimation model 1422 may be loaded into each processor. The first processor 1400 learns to optimize the CNN structure, while the second processor 1420 selects the best combination of options for FPGA design.

[0200] When running such a search, the number of steps for each CNN stage can be greater than the number of steps for each FPGA stage, for example, 1000 steps for each CNN stage compared to 200 steps. The two stages are interleaved and repeated multiple times until we hit the total number of steps (e.g., 10,000 steps). This staged solution is used to find the global optimal solution. This decomposition-conquer technique considers the two search spaces separately, which makes it easier to find better local optimal points (for each search space). However, the interaction between the stages is limited, which may make it more difficult to optimally adapt the CNN and accelerator to each other, for example to perform a specific task.

[0201] Figure 15A 、 15B 15C are graphs showing the top search results compared to the top 100 Pareto optimal points according to an embodiment. Each graph shows the results of the combined and staged searches described above. As a benchmark, these proposed searches are compared to a separate search strategy in which the CNN search space is first searched for a CNN and then the accelerator design space is searched, such as the sequential search approach of the prior art. As described above, there are two separate stages, rather than multiple interleaved stages. The search for the CNN by the first processor 1400 is performed in 8,333 steps, while the search for the FGPA by the second processor 1420 is performed in 1,334 steps. Figures 15A to 15C Each of the top search results shown in [1] maximizes the reward function for one of the three experimental variables. Each experiment is repeated ten times, so there are at most ten points for each strategy. A good search algorithm is expected to produce results near the highest Pareto optimal point.

[0202] Figure 15A The results of an “unconstrained” experiment are shown, where no constraints are imposed on the reward function of Equation 10 above. The weights are arbitrarily chosen to be w(area, lat, acc) = (0.1, 0.8, 0.1). Figure 15A As shown, this experiment can be used to simply search for many good points to understand the co-design space. Figure 15B The results of an experiment in which a single constraint, latency less than 100ms, was imposed are shown. The weights were chosen to be w(area, lat, acc) = (0.1, 0, 0.9). This experiment simulates a situation in which the end user knows the task and real-time requirements but is unsure which FPGA device to choose and what accuracy to achieve at each device size can help in this decision. Figure 15C The results of an experiment are shown, where two constraints are imposed, namely, the accuracy is greater than 0.92 and the area is less than 100 mm 2 The weights are chosen to be w(area, lat, acc) = (0, 1, 0) to optimize latency. The experiment is conducted as a goal by imposing two constraints. This experiment may be useful when there is a maximum FPGA area budget for the application and a minimum allowable accuracy.

[0203] Figure 16A 、 16B 16C are graphs showing exemplary reward values ​​for each of the individual, combined, and phased search strategies in three experimental scenarios. Figure 16A shows the results of an "unconstrained" experiment where there are no constraints, Figure 16B shows the results of an experiment in which a single constraint is imposed, and Figure 16C The results of an experiment in which two constraints are imposed are shown. Only the reward function R is shown in the figure instead of the penalty function Rv.

[0204] Figure 15A 、 15B , 15C, 16A, 16B, and 16C show that the search alone cannot consistently find good points within the constraints. This is because it searches for the most accurate CNN model without any context of the HW target platform. Figure 15B shows two "lucky" individual points that outperform other searches, while Figure 16B shows a higher reward. However, these curves do not show that the remaining eight points all have much higher latencies than this constraint. This is true for all Figure 15A 、 15B and 15C are both correct. Figure 15A 、 15B In

[15] and [15C], only a few individual points fit on the displayed axes; the rest are generally highly accurate but inefficient. This demonstrates the stochastic nature of CNNs designed without a hardware environment. They may or may not fall within the efficiency constraints based on chance, further motivating the need for joint co-design methods.

[0205] Figure 15A 、 15B , 15C, 16A, 16B and 16C show that the staged and combined search strategies improve the individual searches because they take into account the HW accelerator and, more importantly, they simultaneously consider all the variables of the HW accelerator and all the variables of the CNN. Figure 16A 、 16B and 16C shows that in Figure 16A In the unconstrained experiments shown, the combined search strategy is generally better, while for Figure 16B and 16C In the constrained experiments shown in

[15] , the phased search strategy achieves higher rewards. Figure 15C , where the staged search approaches the ideal point. However, Figure 15C It also shows a disadvantage of the phased search, namely that it is more prone to missing specified constraints, possibly because there are limited opportunities to switch from the CNN search phase to the FPGA search phase within the 10,000-step limit of the experiment. Increasing the number of search steps can mean, for example, that the phased search can find points within the constraints, but increases the running time of the experiment.

[0206] More generally, staged searches converge more slowly than combinatorial searches. Figure 16A 、 16B and 16C is highlighted, Figure 16A 、 16BFigures 16C and 16C show that the phased search went through several exploration phases before finding its optimal result. Thus, phased search and combined search appear to have relative advantages over each other. Combined search appears to work better when the search is unconstrained and generally converges to a solution faster. Phased search finds better results when constraints are present, but generally requires more search steps to do so.

[0207] As mentioned above Figure 5 As explained, the CNN search space used in the above analysis can be called NASBench. In this search space, CNNs have been trained to perform ImageNet classification. To verify the above results, we use a co-design approach to discover CNN model-accelerator pairs that optimize different tasks, such as Cifar-100 image classification. Note that Cifar-100 image classification is almost as difficult as ImageNet classification, which is reflected by its highest accuracy number, which is typically similar to ImageNet19. However, Cifar-100 has a much smaller training set (60K versus 251M), so training a CNN to perform Cifar-100 image classification is about two orders of magnitude faster than ImageNet classification. This makes it more feasible for the infrastructure that can be used for the experiments described in this application.

[0208] All discovered CNNs must be trained from scratch to perform such a task. However, the same search space SCNN described above can still be used. For example, the training is described in "NAS-Bench-101: To Reproducible Neural Architecture Search" by Ying et al., published in the arXiv e-prints in February 2019, which is incorporated by reference in its entirety. There are 108 training epochs using standard data augmentation (padding, random crops, and flipping).

[0209] The initial learning rate is 0.1, with cosine decay and weight decay of 10−4. Training each new CNN takes about 1-GPU hour. To be able to train many models, we co-design NAS in parallel on six machines, each with eight Nvidia-1080 GPUs, allowing 48 models to be trained in parallel.

[0210] The collaborative design search is run with the two constraints combined into one. Specifically, latency and area are combined into a metric called perf / area, and this metric is constrained to a threshold. Accuracy is then maximized under this constraint. The perf / area threshold is gradually increased according to (2, 8, 16, 30, 40), and a total of approximately 2,300 valid points are searched, starting with 300 points at the first threshold and increasing to 1,000 points at the final threshold. This appears to make it easier for the processor to learn the structure of a high-precision CNN. The combined search strategy described above is used because it converges to a solution faster.

[0211] Figure 17 is a graph showing the highest accuracy and perf / area for each point found using combined search. The top 10 points among the model-accelerator points visited at each threshold are plotted. The graph also shows the Figure 5 ResNet and GoogLeNet cells within the CNN framework shown, paired with their best accelerators in terms of performance / area. When the high-accuracy CNN cell is implemented on the best possible corresponding accelerator in our FPGA search space, we compare it to these two well-known high-accuracy CNN cells and find it difficult to surpass the baseline. However, as shown in the figure, we find that many points exceed the accuracy and efficiency of the ResNet and GoogLeNet baselines.

[0212] The best two points are marked as Cod-1 and Cod-2. Their performance is shown in Table 2 below:

[0213] [Table 2]

[0214]

[0215] Cod-1 improves ResNet's accuracy by 1.8% and its perf / area by 41%. These are substantial gains in both accuracy and efficiency. Cod-2 shows more modest improvements over GoogLeNet, but still surpasses it in efficiency and accuracy, while running 4.2% faster in absolute latency.

[0216] Figure 18A and 18B ] are diagrams showing the model structures of Cod-1 and Cod-2, respectively, and Table 3 below lists the HW parameters.

[0217] [Table 3]

[0218]

[0219] like Figure 18AAs shown in the rightmost branch of the unit in , Cod-1 achieves beyond ResNet accuracy, but uses important ResNet features: skip connections and element-by-element addition. In terms of hardware, both Cod-1 and Cod-2 use the largest convolution engine and avoid using a dedicated pooling engine. However, other HW parameters are customized for each CNN. For example, the input buffer size and memory interface width of Cod-1 are smaller than those of Cod-2. This may be due to the fact that the Cod-1 CNN uses more and smaller convolutions compared to Cod-2.

[0220] There are likely better points than Cod-1 and Cod-2, as the search space has a total of approximately three billion points. Only about 1,000 points were explored before Cod-1 was found, and about 2,000 points before Cod-2 was found. This highlights the speed of convergence achieved by the processor when using combinatorial search. It is also effective in finding good designs, especially when properly tuned with a representative reward function and search strategy as described above.

[0221] Figure 19 is a block diagram illustrating an exemplary replacement system according to an embodiment that can be used to search a CNN search space as a proxy for Figure 4A Independent improvements to the arrangement of Figure 4A In this arrangement, a processor (e.g., including processing circuitry) 1300 proposes a model architecture for a CNN that is fed to a cutoff model 1312. The cutoff model 1312 uses hardware metrics, such as thresholds on latency and memory footprint, as cutoff points to provide fast feedback to the processor 1300. If the proposed model does not meet the hardware criteria, the processor receives feedback to prevent it from proposing similar poorly performing models. This will allow the processor 1300 to focus on proposing models that meet the hardware constraints. If the proposed model does meet the hardware criteria, the model is sent to an evaluation model 1322 for more detailed evaluation, e.g., generating a reward function, as described above.

[0222] Cutoff model 1312 can be dynamic so that the hardware metric can change as the search progresses to improve the models located by the search. For example, if the initial latency threshold is 100ms, but many models have a latency equal to 50ms, the latency threshold can be updated on the fly (e.g., in real time) to, for example, 60ms. In this way, more models will be eliminated from the search and the overall search process will be accelerated.

[0223] As schematically shown, the cutoff model can use multiple hardware devices H / W 1, H / W 2, ... H / WN simultaneously to search for a model suitable for all devices.

[0224] Figure 20 is a diagram showing a method according to an embodiment of the present invention. Figure 19 14. A more complex version of the system is shown in FIG, wherein the cutoff model 1412 includes a hardware runtime estimator 1430 and a verifier 1432. The hardware runtime estimator 1430 is used to predict the hardware performance, such as latency, of the proposed model for a processor on a target hardware platform. This is not a trivial task because the relationship between the total number of FLOPS required to run the proposed model architecture or the size of its parameters and the latency on a specific hardware platform has a nonlinear relationship due to variations in on-chip / off-chip memory utilization, memory footprint, degree of parallelism, area usage, clock speed, or any other relevant task or hardware metric.

[0225] Hardware runtime estimator 1430 includes a statistical model module 1440, a discriminator 1442, a theoretical hardware model module 1444, and a deployment module 1446. Statistical model module 1440 is used to predict (e.g., estimate) hardware metrics and send them to discriminator 1442. Initially, the statistical model is based on a theoretical model that is calculated in theoretical hardware model module 1444 to give a baseline prediction that is sent to statistical model module 1440. These models may suffer from poor prediction quality, especially the initial model. Therefore, discriminator 1442 monitors the confidence level of the results from the statistical model.

[0226] When the confidence in the estimated hardware metrics is low (e.g., below a confidence threshold), the proposed architecture can be sent to the deployment module 1446 for deployment on the target hardware, such as hardware devices H / W 1, H / W 2, ..., H / WN. The latency (or other hardware metrics) is measured and the measurements are sent to the statistical model module 1440 to update the statistical model. The measurements are also sent to the discriminator 1442 to update the monitoring process within the discriminator. The actual measured values, rather than the estimated values, are then sent along with the model to the validator 1432. When the confidence in the estimated hardware metrics is good (e.g., above a threshold), the model is sent directly to the validator 1432.

[0227] Once the validator 1432 has received a model with estimated hardware values ​​or measured hardware values, the validator 1432 checks whether the proposed architecture meets all hardware metrics. In other words, the validator 1432 can compare the hardware values ​​with the defined thresholds to determine whether the hardware constraints are met. If the proposed model does meet the hardware criteria, the model is sent to the evaluation model 1422 for a more detailed evaluation, for example, to generate a reward function, as described above. Therefore, it is clear that in this configuration, the processor 1400 sends all proposed model architectures of the CNN to the hardware runtime estimator 1430. Specifically, as shown in the figure, the proposed model architecture is sent to the statistical model module 1440 and the discriminator 1442.

[0228] Figure 20 The method described in Figure 4A The implementation and evaluation steps (steps S404 and S406) in [ 404 ] are modeled. This can result in faster runtimes because the hardware does not have to be pooled for each iteration. Note also that the overall search process can be configured by providing an overall GPU time budget. Thus, at the end of the computational budget, we obtain the optimal model that meets all requirements.

[0229] Figure 21 is a flowchart illustrating an exemplary method for continuously updating a statistical model used in a statistical model module. The method can be performed using one or more modules thereof in a runtime estimator. As shown in the first step, a proposed CNN model is received (step S1500), for example, from a processor as described above. Before running the statistical model, the processor identifies how many proposed models have been previously transmitted to the statistical model. For example, the processor can identify whether the proposed neural network model (e.g., CNN model) has been transmitted by repeating N times in step S1502. Here, N times can refer to a threshold number, can be a predetermined number, and can be a number obtained through experiments, statistics, etc. If the process runs for fewer than a threshold number of iterations of the statistical model, e.g., N ("No" in S1502), the statistical model is applied to the received model to predict hardware parameters, such as the latency that occurs when the selected model runs on an FPGA (step S1504). The process then loops back to the beginning to repeat for the next received model.

[0230] If the statistical model has been iterated more than N times ("yes" in S1502), the proposed model is run on the actual hardware, for example using Figure 19The deployment module and one of the multiple hardware modules shown are used to provide actual measurements of the hardware parameters (step S1506). The statistical model is also applied to predict the hardware parameters (step S1508). These steps are shown as being in sequence, but it should be understood that they can be performed simultaneously or in other orders. If there is a difference between the predicted parameters and the measured parameters, the measured parameters can be used to update the statistical model (step S1510). The process then loops back to the beginning to repeat for the next received model.

[0231] This approach allows for scaling and improved runtimes compared to an approach that always uses actual hardware to determine performance. For example, multiple threads or processes can use a statistical model to search for new CNN models while using a single actual hardware device to infrequently update the statistical model. Using regular measurements, the statistical model is likely to be more accurate and up-to-date. The statistical model can only execute the training data on which it was created. When a search for a new CNN model is performed, the search can move to a different search space that includes data that the original model was not trained on. Therefore, updating the statistical model with measurements helps ensure that the statistical model continues to predict representative hardware metrics, which in turn are used to guide the search. Any error between the predicted and measured hardware metrics can also be used to adjust the number of iterations between implementing the CNN model on the hardware. For example, as the error increases, the number of iterations between polling the hardware can be reduced, and vice versa.

[0232] Figure 22 is a diagram showing a method according to an embodiment of the present invention. Figure 20 How to use the discriminator with Figure 21 The method shown in the flowchart is an example of an approach similar to that shown in the figure, which helps the discriminator learn how to distinguish between reliable predictions and invalid predictions. The proposed technique can improve the understanding of the hardware within the selection process by generating better statistical models without significantly affecting the runtime of the selection process.

[0233] As shown in steps S1600 and S1602, the discriminator receives the proposed model, for example, from the processor, and the predicted hardware metrics, for example, from the statistical model. These steps are shown in a specific order, but it should be understood that the information can be received simultaneously or in a different order. The discriminator determines whether the predicted hardware metrics are credible (step S1604), and in this method, when the discriminator determines that the predicted metrics are credible ("yes" in S1604), there is an optional additional step, namely, the discriminator determines whether the predicted metrics need to be verified (step S1606). The verification decision can be made according to different strategies, such as after a fixed number of iterations, at random intervals, or through the output of the evaluation system. If verification is not required ("no" in S1606), the predicted HW parameters are output (step S1608), for example, to the verifier to determine whether to pass the model to the evaluation model as described above.

[0234] When the discriminator determines that the predicted metric cannot be trusted ("No" in S1604), the proposed model is run on actual hardware to obtain measurements of the hardware metric of interest (e.g., latency) (step S1610). Figure 21 As shown, when there is a difference between the predicted parameters and the measured parameters, the measured parameters can be used to update the statistical model (step S1612). The measured HW parameters are output (step S1614), for example, to the verifier to determine whether the model is passed to the evaluation model as described above. Similarly, when the discriminator determines that the predicted metric needs to be verified ("yes" in S1606), the step of running the proposed model on the hardware (step S1610), the step of updating the statistical model as needed (step S1612) and the step of outputting the measured parameters (step S1614) are performed. In all cases, once the measured or predicted parameters are output, the process loops back to the beginning to repeat for the next received model.

[0235] In the description above, the terms hardware metrics and hardware parameters are used interchangeably. It may be difficult to estimate or measure certain metrics, such as latency, and therefore proxy metrics (such as flops and model size) can be used as estimates of the required metrics. The statistical model described above can be trained using hardware measurements previously captured for a specific type of CNN. A theoretical model can be used to build a statistical model that approximates hardware metrics (such as latency) from model characteristics (such as the number of parameters, flops, connectivity between layers, operation types, etc.). The theoretical model may have different equations for each layer type (such as convolution, max pooling, ReLU, etc.), with varying accuracy / fidelity for each layer. Theoretical models can be used instead of statistical models.

[0236] In the above description, reference has been made to the co-design or design of a CNN and an FPGA processor. However, it should be understood that the method is not only applicable to CNNs, but can be easily extended to any neural network using the above techniques. The method can also be more broadly applied to any parameterizable algorithm that can be beneficially implemented in hardware, such as compression algorithms and cryptographic algorithms. It should be understood that in order for the method to work, it is necessary to have a well-defined algorithm search space, for example, a parameterizable algorithm must be defined by means of at least one configurable parameter. For example, in the above method, the search space is defined by using the model described in conjunction with Figure 4. However, it should be understood that this model is merely illustrative and other models of parameterizable algorithms can be used by setting the parameters of the neural network to be modeled. Similarly, it should be understood that the method can be applied to other types of hardware, not just FPGA processors.

[0237] The processor, evaluation model, and other modules may include any suitable processing unit capable of accepting data as input, processing the input data according to stored computer executable instructions, and generating output data. The processor, evaluation model, and other modules may include any type of suitable processing unit, including but not limited to a central processing unit, a microprocessor, a reduced instruction set computer (RISC) microprocessor, a complex instruction set computer (CISC) microprocessor, a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a system on a chip (SoC), a digital signal processor (DSP), etc. In addition, any functionality described as being supported by the processor, evaluation model, and other modules may be implemented, at least in part, in hardware and / or firmware by any number of devices.

[0238] Some aspects of the present disclosure are described above with reference to the block diagram and the flow chart of the system, method, device and / or computer program product according to exemplary embodiments. It should be understood that the combination of one or more blocks in the block diagram and the flow chart, and the block in the block diagram and the flow chart can be realized by executing computer executable program instructions respectively. Equally, according to some embodiments, some blocks of the block diagram and the flow chart may not necessarily be performed in the order presented, or may not necessarily be performed at all. In addition, in certain embodiments, there may be additional components and / or operations other than those shown in the frame of the block diagram and / or flow chart.

[0239] However, it is understood that the present disclosure is not limited to the various exemplary embodiments described, but may also include various modifications, equivalent expressions and / or replacements of the embodiments of the present disclosure. Regarding the explanation of the drawings, similar reference numerals may be used for similar constituent elements.

[0240] In this specification, expressions such as “have”, “may have”, “include” or “may include” indicate the presence of corresponding features (for example, components such as numbers, functions, operations or parts), and do not exclude the presence of additional features.

[0241] In this document, expressions such as "at least one of A [and / or] B" or "one or more of A [and / or] B" include all possible combinations of the listed items. For example, "at least one of A and B" or "at least one of A or B" includes any of (1) at least one A, (2) at least one B, or (3) at least one A and at least one B.

[0242] As used herein, the terms "first," "second," etc. may refer to various components regardless of order and / or importance, and may be used to distinguish one component from another without limiting the components.

[0243] If it is described that a certain element (for example, a first element) is “operably or communicatively coupled” or “connected to” another element (for example, a second element), it should be understood that the certain element can be connected to the other element directly or through another element (for example, a third element). On the other hand, if it is described that a certain element (for example, a first element) is “directly coupled to” or “directly connected to” another element (for example, the second element), it should be understood that there is no element (for example, the third element) between the certain element and the other element.

[0244] In addition, the expression "configured to" used in the present disclosure may be used interchangeably with other expressions, such as "suitable for," "capable of," "designed to," "suitable for," "manufactured to," and "capable of," depending on the circumstances. At the same time, the term "configured to" does not necessarily refer to a device that is "specifically designed to" in terms of hardware. On the contrary, in some cases, the device "configured to" may refer to a device that is "capable of" performing an operation together with another device or component. For example, the phrase "a processor configured to perform A, B, and C" may refer to, for example, a dedicated processor (e.g., an embedded processor) for performing the corresponding operations, or a general-purpose processor (e.g., a central processing unit (CPU) or an application processor) that can perform the corresponding operations by executing one or more software programs stored in a memory device.

[0245] In this disclosure, the term user may refer to a person who uses an electronic device or uses an apparatus (eg, an artificial intelligence electronic device).

[0246] At the same time, various embodiments of the present disclosure can be implemented with software, including instructions stored on a machine-readable storage medium that is readable by a machine (e.g., a computer). The device according to the disclosed embodiment can call instructions from a storage medium and execute the called instructions, including an electronic device (e.g., electronic device 100). When the instruction is executed by a processor, the processor can directly execute the function corresponding to the instruction or use other components under the control of the processor. The instruction may include code generated by a compiler or code executable by an interpreter. The machine-readable storage medium can be provided in the form of a non-transitory storage medium. Here, a "non-transitory" storage medium may not include a signal, but is tangible, and the situation where data is semi-permanently stored in the storage medium cannot be distinguished from the situation where data is temporarily stored in the storage medium. For example, a "non-transitory storage medium" may include a buffer in which data is temporarily stored.

[0247] According to one embodiment, the method according to the above embodiment may be included in a computer program product. The computer program product can be traded as a product between a seller and a consumer. The computer program product can be distributed online in the form of a machine-readable storage medium (e.g., a compact disc read-only memory (CD-ROM)), or distributed through an application store (e.g., Play Store™), or distributed directly online. In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily generated in a manufacturer's server, an application storage server, or a machine-readable storage medium (e.g., a memory of a relay server).

[0248] According to various embodiments, each element (such as, module or program) of above-mentioned elements can comprise single entity or multiple entities.According to an embodiment, at least one element or the operation in above-mentioned corresponding element can be omitted, or at least one other element or operation can be added.Alternatively or additionally, multiple components (such as, module or program) can be combined to form a single entity.In this case, the integrated entity can perform the function of at least one function of the element of each of multiple elements in the same or similar manner as the mode performed by the corresponding element in the multiple elements before integration.Can continuously, in parallel, repeatedly or tentatively perform the operation of described module, program module or by other element according to various embodiments, or can perform or omit at least some operations according to different orders, or can add other operations thereto.

[0249] Although the present disclosure has been illustrated and described with reference to various exemplary embodiments thereof, it should be understood that the various exemplary embodiments are intended to be illustrative rather than restrictive. It will be further understood by those skilled in the art that various changes in form and details may be made without departing from the true spirit and full scope of the present disclosure, including the appended claims and their equivalents.

Claims

1. A method for controlling an electronic device, the electronic device comprising a memory storing a plurality of accelerators and a plurality of neural networks, the method comprising: Selecting a first neural network from the plurality of neural networks, and selecting a first accelerator from the plurality of accelerators for implementing the first neural network; implementing the first neural network on the first accelerator to obtain information associated with implementing the first neural network on the first accelerator; obtaining a first reward value for the first accelerator and the first neural network based on information associated with implementing the first neural network on the first accelerator; selecting a second neural network among the plurality of neural networks to be implemented on the first accelerator; implementing the second neural network on the first accelerator to obtain information associated with implementing the second neural network on the first accelerator; obtaining a second reward value for the first accelerator and the second neural network based on information associated with implementing the second neural network on the first accelerator; as well as Based on the first reward value and the second reward value, selecting a pair of neural networks and accelerators having a maximum reward value among the plurality of neural networks and the plurality of accelerators, The step of implementing the first neural network on the first accelerator to obtain information associated with implementing the first neural network on the first accelerator includes: identifying whether hardware performance of the first accelerator and the first neural network obtained by inputting the first accelerator and the first neural network into a first prediction model satisfies a predetermined criterion; and Based on identifying that the obtained hardware performance satisfies a first hardware criterion, the first neural network is implemented on the first accelerator to obtain information associated with implementing the first neural network on the first accelerator.

2. The method according to claim 1, wherein The method further comprises: Based on identifying that the obtained hardware performance does not meet the first hardware criterion, a second accelerator is selected from accelerators other than the first accelerator for implementing the first neural network. The method of claim 1 , wherein the information associated with the implementation includes accuracy and efficiency metrics of the implementation.

4. The method according to claim 1, wherein obtaining the first reward value comprises: Normalize the obtained accuracy and efficiency metrics; as well as The first reward value is obtained by performing a weighted sum operation on the normalized metrics.

5. The method according to claim 1, wherein Selecting a first neural network among the plurality of neural networks comprises: obtaining a first probability value corresponding to a first configurable parameter included in each of the plurality of neural networks; and The first neural network is selected from the plurality of neural networks based on the first probability value.

6. The method of claim 3, wherein selecting the first accelerator comprises: obtaining a second probability value corresponding to a second configurable parameter included in each of the plurality of accelerators; as well as The first accelerator for implementing the first neural network is selected from the plurality of accelerators based on the second probability value.

7. The method according to claim 1, wherein The method further comprises: Based on selecting the first neural network, before selecting the first accelerator for implementing the first neural network, hardware performance of the selected first neural network is predicted by a second prediction model.

8. The method according to claim 7, wherein: Selecting the first accelerator for implementing the first neural network includes: identifying whether the predicted hardware performance of the first neural network satisfies a second hardware criterion, and The first accelerator is selected for implementing the first neural network based on the predicted hardware performance of the first neural network satisfying the second hardware criterion.

9. The method according to claim 8, wherein The method further includes selecting another neural network from among a plurality of neural networks other than the first neural network based on identifying that hardware performance of the selected first neural network does not meet the second hardware criterion.

10. An electronic device comprising: Memory for storing multiple accelerators and multiple neural networks; as well as a processor configured to: selecting a first neural network from among the plurality of neural networks, and selecting a first accelerator from among the plurality of accelerators for implementing the first neural network; implementing the first neural network on the first accelerator to obtain information associated with implementing the first neural network on the first accelerator, obtaining a first reward value for the first accelerator and the first neural network based on information associated with implementing the first neural network on the first accelerator, selecting a second neural network among the plurality of neural networks to be implemented on the first accelerator; implementing the second neural network on the first accelerator to obtain information associated with implementing the second neural network on the first accelerator, obtaining a second reward value for the first accelerator and the second neural network based on information associated with implementing the second neural network on the first accelerator, and Based on the first reward value and the second reward value, selecting a pair of neural networks and accelerators having a maximum reward value from the plurality of neural networks and the plurality of accelerators, wherein implementing the first neural network on the first accelerator to obtain information associated with implementing the first neural network on the first accelerator includes: identifying whether hardware performance of the first accelerator and the first neural network obtained by inputting the first accelerator and the first neural network into a first prediction model satisfies a predetermined criterion, and Based on identifying that the obtained hardware performance satisfies a first hardware criterion, the first neural network is implemented on the first accelerator to obtain information associated with implementing the first neural network on the first accelerator.

11. The electronic device of claim 10, wherein the processor is further configured to select a second accelerator for implementing the first neural network among accelerators other than the first accelerator based on identifying that the obtained hardware performance does not satisfy the first hardware criterion.

12. The electronic device of claim 10, wherein the information associated with the implementation includes accuracy and efficiency metrics of the implementation. 13 . The electronic device of claim 10 , wherein the processor is further configured to normalize the obtained accuracy and efficiency metrics, and obtain the first reward value by performing a weighted sum operation on the normalized metrics.

Images