Hardware performance-based neural architecture search method and device

Abstract

A hardware performance-based neural architecture search method and device are provided. The method includes: obtaining hardware performance corresponding to a neural network block included in a pre-trained neural network; selecting at least one target block for hardware performance improvement from the neural network blocks; training weight and architecture parameters of a candidate block corresponding to the target block based on arbitrary input data and output data of the target block generated based on the input data; and updating the pre-trained neural network by replacing the target block in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters.

Description

[0001] This application claims the benefit of Korean Patent Application No. 10-2020-0114703, filed on September 8, 2020, with the Korean Intellectual Property Office, the entire disclosure of which is incorporated herein by reference for all purposes. Technical Field

[0002] The following description relates to hardware performance-based neural architecture search methods and devices. Background Technology

[0003] Neural Architecture Search (NAS) is a technique for automatically searching for the structure (i.e., architecture) of neural networks. NAS can optimize neural networks with high recognition rates in various domains (such as image classification and speech recognition) to have an architecture suitable for the hardware. NAS can design optimal architectures through learning. NAS techniques may include: methods for training RNN controllers based on reinforcement learning to predict the values ​​of elements that determine the architecture of a neural network; methods for searching for new neural network architectures using knowledge distillation schemes; and methods for searching for structures to replace unnecessary operations. Such methods may require operations used to search for and learn new architectures. NAS may require significant computational resources and time for computation. Summary of the Invention

[0004] The present invention is provided in a simplified form to describe the selection of concepts further described in the following detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help determine the scope of the claimed subject matter.

[0005] In one general aspect, a processor-implemented method for hardware performance-based neural architecture search includes: obtaining the hardware performance corresponding to neural network blocks included in a pre-trained neural network; selecting at least one target block from the neural network blocks for hardware performance improvement; training weights and architecture parameters of candidate blocks corresponding to the target blocks based on arbitrary input data and output data generated from the input data of the target blocks; and updating the pre-trained neural network by replacing the target blocks in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters.

[0006] In another general aspect, a hardware performance-based neural architecture search device includes: one or more processors configured to: acquire hardware performance corresponding to neural network blocks included in a pre-trained neural network; select at least one target block from the neural network blocks for hardware performance improvement; train weights and architecture parameters of candidate blocks corresponding to the target blocks based on arbitrary input data and output data generated from the input data of the target blocks; and update the pre-trained neural network by replacing the target blocks in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters.

[0007] In another general aspect, a processor-implemented neural architecture search method includes: acquiring the performance of neural network blocks included in a pre-trained neural network; selecting at least one target block from the neural network blocks for performance improvement; training weights and architecture parameters of candidate blocks corresponding to the target blocks based on arbitrary input data and output data generated from the input data of the target blocks; and updating the pre-trained neural network by replacing the target blocks in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters.

[0008] For each of the corresponding candidate blocks, the training steps may include: generating a loss function based on the degree of matching between the output data of the corresponding candidate block and the output data of the target block, and the performance of the corresponding candidate block, wherein the output data of the corresponding candidate block and the output data of the target block are generated based on the same input data; and training the weights and architecture parameters of the corresponding candidate block based on the loss function.

[0009] For each of the corresponding candidate blocks, the training steps may include: determining whether the corresponding candidate block is a target to be trained based on the architectural parameters of the corresponding candidate block; and training the corresponding candidate block in response to the corresponding candidate block being determined as a target to be trained.

[0010] The determination steps may include: determining that the corresponding candidate block is the target to be trained in response to the architectural parameters of the corresponding candidate block being greater than or equal to a predetermined threshold.

[0011] The steps to obtain the performance of a neural network block may include: determining the performance of the neural network block based on measurements of the actual hardware performance of the operations performed on the neural network block.

[0012] The steps to obtain the performance of a neural network block may include estimating the performance of the neural network block based on a predictive model of the hardware performance that has learned the operation of the neural network block.

[0013] The steps to obtain the performance of a neural network block may include: simulating the performance of the hardware based on the operation of the neural network block to estimate the performance of the neural network block.

[0014] The step of selecting a target block may include any one or both of the following: selecting at least one of the neural network blocks as a target block in ascending order of their performance; and comparing the performance of the neural network blocks with a threshold and selecting at least one of the neural network blocks with performance below the threshold as a target block.

[0015] The training steps may include: generating input data and generating output data as training data by feeding the input data into a target block; training at least one of the candidate blocks based on the training data; and removing the training data in response to the completion of training performed based on the training data.

[0016] The training steps may include: obtaining candidate blocks with the same input and output dimensions as the target block, as candidate blocks.

[0017] The steps of replacing the target block may include: selecting a candidate block with the largest architecture parameter from the candidate blocks; and replacing the target block with the selected candidate block.

[0018] The step of selecting a target block may also include: selecting a target block from a neural network block based on hardware constraints.

[0019] For each of the corresponding candidate blocks, the training steps may include: generating a loss function based on the degree of matching between the output data of the corresponding candidate block and the output data of the target block, the performance of the corresponding candidate block, and hardware constraints, wherein the output data of the corresponding candidate block and the output data of the target block are generated based on the same input data; and training the weights and architecture parameters of the corresponding candidate block based on the loss function.

[0020] Hardware constraints can correspond to the type of neural network operation and the size of the neural network supported by the hardware configured to execute the neural network.

[0021] The steps for selecting a target block based on hardware constraints may include: selecting a target block in response to a target block not meeting the hardware constraints.

[0022] Each of the neural network blocks may include at least one layer of a pre-trained neural network.

[0023] Each of the candidate blocks may include at least one layer that performs a predetermined operation of the trained neural network, and the candidate block has the same input and output dimensions as the target block.

[0024] Performance can be determined based on any one or any combination of the following: the execution speed of the neural network blocks in the hardware, the number of computations performed by the neural network blocks, and the size of the neural network.

[0025] A non-transitory computer-readable storage medium stores instructions that, when executed by one or more processors, configure the one or more processors to perform the method.

[0026] In another general aspect, a neural architecture search device includes: one or more processors configured to: acquire the performance of neural network blocks included in a pre-trained neural network; select at least one target block from the neural network blocks for performance improvement; train weights and architecture parameters of candidate blocks corresponding to the target blocks based on arbitrary input data and output data generated from the input data of the target blocks; and update the pre-trained neural network by replacing the target blocks in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters.

[0027] For the training, for each candidate block, the one or more processors may be configured to: generate a loss function based on the degree of matching between the output data of the corresponding candidate block and the output data of the target block, and the performance of the corresponding candidate block, wherein the output data of the corresponding candidate block and the output data of the target block are generated based on the same input data; and train the weights and architecture parameters of the corresponding candidate block based on the loss function.

[0028] For the training, for each of the corresponding candidate blocks, the one or more processors may be configured to: determine whether the corresponding candidate block is a target to be trained based on the architectural parameters of the corresponding candidate block; and train the corresponding candidate block in response to the corresponding candidate block being determined as a target to be trained.

[0029] For the training, the one or more processors may be configured to: generate input data and output data of the target block as training data; train at least one of the candidate blocks based on the training data; and remove the training data in response to the completion of training performed based on the training data.

[0030] To obtain the performance of a neural network block, the one or more processors may be configured to perform any one or any combination of the following: determining the performance of the neural network block based on measurements of the actual hardware performance of the operations of the neural network block; estimating the performance of the neural network block based on a predictive model of the performance of the hardware that has learned the operations of the neural network block; and estimating the performance of the neural network block based on simulations of the hardware performance of the operations of the neural network block.

[0031] For the training purpose, the one or more processors may be configured to: acquire candidate blocks having the same input and output dimensions as the target block, as candidate blocks.

[0032] In order to select a target block, the one or more processors may be configured to select a target block from a neural network block based on hardware constraints.

[0033] For the training, for each of the corresponding candidate blocks, the one or more processors may be configured to: generate a loss function based on the degree of matching between the output data of the corresponding candidate block and the output data of the target block, the performance of the corresponding candidate block, and hardware constraints, wherein the output data of the corresponding candidate block and the output data of the target block are generated based on the same input data; and train the weights and architecture parameters of the corresponding candidate block based on the loss function.

[0034] In another general aspect, a processor-implemented neural architecture search method includes: selecting a target block from neural network blocks included in a pre-trained neural network based on the performance of the neural network blocks; selecting candidate blocks based on the architecture parameters of the candidate blocks; training the architecture parameters of the selected candidate blocks based on arbitrary input data and output data generated by inputting the input data to the target blocks; and updating the pre-trained neural network by replacing the target block in the pre-trained neural network with one of the candidate blocks selected based on the trained architecture parameters.

[0035] The data used to train a pre-trained neural network may not include arbitrary input data and the output data of the target block.

[0036] Other features and aspects will become clear from the following detailed description, drawings, and claims. Attached Figure Description

[0037] Figure 1 An example of a neural architecture search method is shown.

[0038] Figure 2 This example demonstrates a method for obtaining the performance of a block using the performance prediction module.

[0039] Figure 3 An example of a neural architecture search method is shown.

[0040] Figure 4 An example of a method for training candidate blocks corresponding to a target block is shown.

[0041] Figure 5 Examples of methods for searching neural architectures are shown that also consider conditions other than the hardware performance corresponding to the operation.

[0042] Figure 6 An example of device configuration is shown.

[0043] Throughout the accompanying drawings and detailed embodiments, unless otherwise described or provided, the same reference numerals will be understood to denote the same elements, features, and structures. The drawings may not be to scale, and for clarity, illustration, and convenience, the relative dimensions, scale, and depiction of elements in the drawings may be exaggerated. Detailed Implementation

[0044] The following detailed embodiments are provided to aid the reader in gaining a comprehensive understanding of the methods, apparatus, and / or systems described herein. However, various changes, modifications, and equivalents of the methods, apparatus, and / or systems described herein will become apparent upon understanding this disclosure. For example, the order of operations described herein is merely illustrative and is not limited to the order set forth herein, but may be changed as will become clear upon understanding this disclosure, except for operations that must occur in a specific order. Furthermore, for clarity and brevity, descriptions of features known in the art may be omitted upon understanding this disclosure.

[0045] In the following, exemplary embodiments will be described in detail with reference to the accompanying drawings. However, it should be understood that this disclosure is not intended to be limited to the specific exemplary embodiments disclosed. Rather, the exemplary embodiments will cover all modifications, equivalents, and alternatives falling within the scope of the exemplary embodiments.

[0046] The terminology used herein is for the purpose of describing exemplary embodiments only and is not intended to limit this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. As used herein, the term “and / or” includes any one and any combination of any two or more of the associated listed items. As used herein, the terms “comprising,” “including,” and “having” indicate the presence of the stated features, integrals, steps, operations, elements, components, quantities, and / or combinations thereof, but do not preclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, quantities, and / or combinations thereof. The use of the term “may” (e.g., regarding what an example or embodiment may include or implement) with respect to examples or embodiments indicates the presence of at least one example or embodiment that includes or implements such features, while all examples are not limited thereto.

[0047] Unless otherwise defined, all terms used herein (including technical and scientific terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains and as commonly understood after understanding this disclosure. Unless expressly defined herein, terms (such as those defined in a general dictionary) shall be interpreted as having a meaning consistent with their meaning in the context of the relevant field and in this disclosure, and shall not be interpreted in an idealized or overly formalized sense.

[0048] Regarding the reference numerals assigned to elements in the accompanying drawings, it should be noted that the same elements should be represented by the same reference numerals whenever possible, even if they are shown in different drawings. Furthermore, in the description of embodiments, detailed descriptions of well-known related structures or functions will be omitted where such detailed descriptions would lead to an obscure interpretation of this disclosure.

[0049] Although the terms “first” or “second” are used herein to describe various components, assemblies, regions, layers, or parts, these components, assemblies, regions, layers, or parts should not be limited by these terms. Rather, these terms are used only to distinguish one component, assembly, region, layer, or part from another component, assembly, region, layer, or part. Thus, without departing from the teaching of the examples described herein, the first component, first assembly, first region, first layer, or first part referred to as a second component, second assembly, second region, second layer, or second part may also be referred to as a second component, second assembly, second region, second layer, or second part.

[0050] Throughout this specification, when an element (such as a layer, region, or substrate) is described as being "on" another element, "connected to," or "bonded to" another element, it may be directly "on" another element, directly "connected to," or "bonded to" said other element, or there may be one or more other elements in between. Conversely, when an element is described as being "directly on" another element, directly "connected to," or "bonded to" another element, there may be no other elements in between. Similarly, expressions such as "between" and "immediately between," and "adjacent" and "adjacent to" may be interpreted as described above.

[0051] Components having the same functionality as those included in one example embodiment are described using the same names in another example embodiment. Unless otherwise stated, descriptions made in one example embodiment are applicable to another example embodiment, and detailed descriptions that overlap are omitted.

[0052] Figure 1 An example of a neural architecture search method is shown.

[0053] Neural Architecture Search (NAS) techniques for searching for optimal neural networks, according to one or more embodiments, can reduce the amount of computational resources required while ensuring high accuracy. (See also...) Figure 1 The neural architecture search method may include: operations such as obtaining (e.g., determining) the hardware performance (PPM(block n, n = 1, 2, 3, 4)) corresponding to a neural network block (block n) included in a pre-trained neural network 110; operations such as selecting at least one target block 111, 112 from a plurality of neural network blocks for hardware performance improvement; operations such as training the weights and architecture parameters of a candidate block 120 corresponding to the selected at least one target block based on arbitrary input data and output data generated in response to the input data of the target block (e.g., output data generated by inputting input data into the target block); and operations such as replacing the target block with one of the candidate blocks based on the trained architecture parameters. The neural architecture search method may be executed in a server or device and may be executed by a processor (e.g., one or more processors) included in the server or device.

[0054] The pre-trained neural network 110 may include various neural networks (such as recurrent neural networks (RNNs) and / or convolutional neural networks (CNNs), as well as combinations of neural networks). The pre-trained neural network 110 may be a neural network that has already been trained and may include at least one layer in which the parameters are determined. Hereinafter, the neural network may be simply referred to as a "network".

[0055] A block is a portion or unit of a neural network structure and may include at least one layer in a pre-trained neural network 110. For example, a block may correspond to a portion or unit of a neural network structure comprising one or more layers that perform one or more operations used in the neural network (such as convolution, rectified linear unit (ReLU), etc.). In other words, a neural network may be divided into multiple blocks. For example, a neural network structure may be divided into multiple blocks comprising one or more layers based on the functions performed by the layers included in the neural network.

[0056] Block performance (e.g., the hardware performance corresponding to the block) can be determined based on the speed at which the block executes in hardware, the amount of computation required to execute the block, the amount of memory required, the size of the block's memory space, and / or the block's contribution to the accuracy of the entire network. Block performance can be evaluated according to the purpose of the neural network. For example, when the goal is to improve the execution speed of a neural network, block performance can be evaluated based on the execution speed of the blocks included in the neural network. Block performance can be a criterion for selecting blocks for performance improvement or for optimizing target blocks. The selected target block can be replaced by a block that performs a different operation. Block performance can be obtained by comparing it to the performance of another block. For example, referring to... Figure 1 The hardware performance (PPM) of a block can be obtained as a value to be compared with other blocks.

[0057] The hardware performance of a block can be measured based on measurements of the actual hardware performance of the operations executed within the block. In other words, the operation of obtaining the hardware performance of a block can include: measuring the block's performance based on measurements of the actual hardware performance of the operations executed within the block, which are included in a pre-trained neural network. Measurements of the actual hardware performance of the operations(s) executed within the block can be obtained from a database that records the hardware performance of each operation measured when it is actually executed in the hardware.

[0058] The hardware performance corresponding to a block can be estimated based on a predictive model that has learned the performance of the hardware executing the operations performed in the block. In other words, the operation of obtaining the hardware performance corresponding to a block may include the operation of estimating the block's performance based on a predictive model that has learned the performance of the hardware executing the operations included in the block within a pre-trained neural network. For example, the predictive model may correspond to a model that learns the performance of the hardware executing the operations performed in the neural network through deep learning. A processor that obtains the hardware performance corresponding to blocks included in the pre-trained neural network can input each block into the predictive model and obtain an estimated performance value for the hardware executing the operations in each block.

[0059] The hardware performance of a block can be estimated based on simulations of the hardware performance of the operations executed within the block. In other words, obtaining the hardware performance of a block can include estimating the block's performance based on simulations of the hardware performance of the operations executed within the block, which are included in a pre-trained neural network. For example, the number of operations executed within the block, or the performance related to the required amount of memory, can be calculated using simple equations. Furthermore, the sum of the hardware performance corresponding to each operation executed in the hardware can be estimated as the hardware performance corresponding to the network.

[0060] Based on the hardware performance corresponding to the acquired blocks, at least one block for hardware performance improvement can be selected from the blocks included in the neural network as a target block. The operation of selecting a target block may include at least one of the following: selecting at least one block as a target block in ascending order of the hardware performance corresponding to the blocks, and comparing the hardware performance corresponding to a block with a threshold and selecting at least one block corresponding to hardware performance below the threshold as a target block. For example, refer to... Figure 1 Blocks 111 and 112, which correspond to the two blocks with the lowest hardware performance or to a predetermined low percentage, can be selected as target blocks. Furthermore, blocks 111 and 112, which correspond to hardware performance less than or equal to (or below) the threshold "20", can be selected as target blocks.

[0061] The operations for obtaining the hardware performance corresponding to blocks included in the pre-trained neural network and the operation for selecting at least one target block from multiple blocks that needs hardware performance improvement can be performed in the performance prediction module.

[0062] Figure 2 This illustrates an example of how to obtain the hardware performance corresponding to a block using a performance prediction module. (See also...) Figure 2The performance prediction module 210 can obtain the hardware performance corresponding to each block by estimating or measuring the performance of the hardware performing the operations executed in the block. For example, to obtain the hardware performance, the performance prediction module 210 may include: a lookup table that records the measured performance values ​​of the hardware performing the operations in the neural network, a predictive model that learns the performance of the hardware performing the operations through deep learning, and / or a simulator that simulates the performance of the hardware performing the operations.

[0063] The performance prediction module 210 can select one or more blocks as target blocks for hardware performance improvement based on the acquired hardware performance. For example, blocks with hardware performance less than or equal to a threshold and / or blocks with hardware performance belonging to a predetermined low percentage can be selected as target blocks.

[0064] Return to reference Figure 1 In order to replace target blocks 111 and 112 with blocks that perform another operation, the neural architecture search method may also include the operation of obtaining candidate blocks 120 corresponding to each target block. The candidate blocks corresponding to the target blocks may be included in the neural network structure and may be trained to replace the target blocks.

[0065] At least one layer performing a predetermined operation may be included in each candidate block. The at least one layer may have the same input and output dimensions as the target block. In other words, a candidate block corresponding to the target block may include a set of operations with one or more operations having the same input and output dimensions as the target block. Each candidate block corresponding to the target block may include a different layer configuration. Furthermore, various operations may be performed in each candidate block.

[0066] Candidate blocks may include multiple blocks set by the user to correspond to the target block, and may also include multiple blocks automatically determined to correspond to the target block. Candidate blocks may include neural network blocks configured by: changing the connection structure of layers in the target block, changing the combination of layers, reducing the amount of operations performed in the target block, and / or adjusting the size of the target block. For example, when the target block is a CNN block, multiple CNN blocks with different filter sizes may correspond to candidate blocks corresponding to the target block.

[0067] The neural architecture search method may include the operation of training candidate blocks corresponding to target blocks to replace target blocks 111 and 112 with one of the candidate blocks 120. The operation of training candidate blocks can be performed for each target block. Training of candidate blocks corresponding to a target block can be performed for each candidate block. When multiple target blocks exist, the operation of training candidate blocks corresponding to each target block can be performed in parallel by multiple processors or sequentially by a single processor. Furthermore, the training of candidate blocks corresponding to a target block can be performed in parallel by multiple processors or sequentially by a single processor.

[0068] A processor training candidate blocks corresponding to a target block can acquire candidate blocks to be trained via user input or by accessing a memory storing preset candidate blocks corresponding to the corresponding target block. Training can be performed on all or part of the acquired candidate blocks corresponding to the target block. Some of the acquired candidate blocks can be excluded from the training target based on set criteria, and thus not be trained. For example, a candidate block can be excluded from the training target if the architecture parameters of the candidate block updated during training do not meet predetermined criteria. Furthermore, a candidate block can be excluded from the training target if the network size (e.g., the size of the candidate block's memory space) does not meet predetermined criteria. In other words, a processor training candidate blocks corresponding to a target block can determine whether to train each candidate block in the candidate blocks corresponding to the target block, and can perform training on the candidate blocks determined to be the target to be trained.

[0069] Training candidate blocks can include training the weights and architecture parameters of the candidate parameters corresponding to the target block. Candidate blocks can be trained based on arbitrary input data and the output data generated by the target block in response to the input data. Furthermore, candidate blocks can be trained based on or considering conditions such as the performance of the hardware executing the candidate block.

[0070] The architecture parameters of a candidate block can correspond to the parameters updated according to the loss function during the training process of the candidate block. As described below, the architecture parameters of a candidate block can be used as both a criterion for determining whether a candidate block should be trained and a criterion for selecting a candidate block to replace the target block during training.

[0071] The following will refer to Figure 3 and Figure 4 Describe the operation of training candidate blocks in neural architecture search methods.

[0072] When training of candidate blocks corresponding to the target block is complete, the target block can be replaced by one of the corresponding candidate blocks. The candidate block for replacing the target block can be determined based on the architectural parameter values ​​of the candidate blocks. When training of the candidate blocks is complete, the candidate block with the largest architectural parameter value can be selected from the candidate blocks as the block to replace the target block. In other words, the operation of replacing the target block can include: selecting the block with the largest architectural parameter from the candidate blocks and replacing the target block with the selected candidate block. In one example, the architectural parameters of the block can be associated with the hardware performance corresponding to the block (e.g., the performance of the hardware executing the block). For example, the largest architectural parameter can indicate the optimal hardware performance. In one example, the architectural parameters of the block can be associated with the degree of matching between the output data of the block and the output data of the target block. For example, the largest architectural parameter can indicate the maximum degree of matching between the output data of the block and the output data of the target block.

[0073] Reference Figure 1 The target block in the pre-trained neural network 110 can be replaced by a candidate block from candidate block 120 that includes the operation set 1 with the largest architecture parameter (e.g., the architecture parameter that is the largest among the architecture parameters "0.7", "0.3", "0.5", and "0.2" of the respective operation sets 1 to 4). When the pre-trained neural network operates, the input of the target block can be fed into the selected candidate block, such that the output of the selected candidate block can be processed as the output of the target block.

[0074] Compared to the target block, when it is determined that operations performed in the trained candidate blocks will not produce sufficient accuracy and / or hardware performance, the target block may be retained in the pre-trained neural network 110 without being replaced by another candidate block. The performance of operations performed in the candidate blocks can be estimated based on the architectural parameters of the candidate blocks. In some cases, architectural parameters may also be included in the target block. In this case, it can be determined whether to replace the target block by comparing the architectural parameters of the target block and the candidate blocks.

[0075] Figure 3 An example of a neural architecture search method is shown.

[0076] Reference Figure 3 The neural architecture search method may include: operation 310 of selecting a target block based on the hardware performance corresponding to the block included in the network in response to input to a pre-trained neural network; operation 320 of obtaining candidate blocks corresponding to the target block; operation 330 of generating input and output data of the target block as training data; operation 340 of updating the weights and architecture parameters of each candidate block using the training data; operation 350 of selecting candidate blocks based on the architecture parameters; and operation 360 of outputting the searched network in which the target block is replaced by the selected candidate block.

[0077] Training of candidate blocks corresponding to the target block can be performed by repeating training iterations including operation 330 (generating input and output data for the target block as training data) and operation 340 (updating the weights and architecture parameters of each candidate block using the training data). The training iterations can be repeated a predetermined number of times, and / or the training iterations can be repeated until the loss corresponding to each candidate block converges to a predetermined value (e.g., until the loss corresponding to each candidate block is less than or equal to a predetermined value).

[0078] To train candidate blocks corresponding to a target block, a knowledge distillation scheme can be used, employing arbitrary input data and the output data generated by the target block in response to the arbitrary input data (e.g., output data generated by inputting arbitrary input data into the target block). In other words, the training data for candidate blocks corresponding to the target block can include arbitrary input data and the output data generated by the target block in response to the input data. More specifically, training data for training candidate blocks corresponding to the target block can be generated using ground truth data obtained by generating arbitrary input data and inputting the generated arbitrary input data into the corresponding target block. Arbitrary input data can include, for example, real numbers between 0 and 1 arbitrarily generated by random number generation. By performing training using arbitrary input data and the output data generated by the target block in response to the input data as training data, candidate blocks can be trained to output the same (or substantially similar) results to the target block for the same input.

[0079] The processor that performs training on candidate blocks corresponding to the target block generates training data for each training iteration and removes the training data after training of candidate blocks based on the training data is complete. Training data can be generated during training iterations and removed after training is complete, thereby reducing the memory usage required for training.

[0080] A processor performing training on candidate blocks corresponding to a target block can determine whether to train each candidate block based on the architectural parameters of the respective candidate block. For example, when the architectural parameters of a first candidate block are below a predetermined criterion (e.g., a threshold), the first candidate block can be excluded from the target to be trained. The processor performing training on candidate blocks can train the candidate blocks identified as the target to be trained, with weights and architectural parameters, based on training iterations. Whether to train each candidate block can be determined based on the architectural parameters according to the loss function, thereby reducing the amount of operations required to train candidate blocks. Furthermore, neural architecture search for optimizing neural network structures can be performed by efficiently utilizing computational resources.

[0081] Figure 4 An example of a method for training candidate blocks corresponding to a target block is shown.

[0082] The operation of training candidate blocks corresponding to the target block may include: for each candidate block, generating a loss function based on the degree of matching between the output data of the corresponding candidate block and the output data generated by the target block in response to the same input data and the hardware performance corresponding to the corresponding candidate block; and training the weights and architecture parameters of the corresponding candidate block based on the loss function.

[0083] The loss function used to train each candidate block corresponding to the target block can be determined based on the degree of matching between the output data of the candidate block and the output data of the target block, as well as the hardware performance corresponding to the candidate block (e.g., the hardware performance corresponding to the operation performed in the candidate block).

[0084] Reference Figure 4 The loss function 402 used to train the candidate block 410 corresponding to the target block 420 can be determined based on the degree of matching between the output data of the candidate block 410 and the output data of the target block 420 (generated based on the same input data 401) and the hardware performance corresponding to the candidate block 410.

[0085] The degree of matching between the output data of candidate block 410 and the output data of target block 420 can be defined as the degree to which the output data of the candidate block matches the output data generated by the target block in response to the same input data 401. This degree of matching can be included in the loss function to train the candidate block to output the same result as the target block.

[0086] The hardware performance corresponding to candidate block 410 can be determined based on the execution speed, number of computations, and / or memory space size of the candidate block in the hardware. The hardware performance corresponding to candidate block 410 can be determined by measuring the hardware performance corresponding to the operations of the actual candidate block (e.g., the performance of the hardware executing the operations of the actual candidate block) or by estimating the performance using a learning model. Values ​​indicating the hardware performance corresponding to the candidate block can be included in the loss function to train the candidate block, such that the architectural parameter values ​​defined according to the loss function are updated based on the degree of matching between the output data of the target block and the output data of the candidate block, and the hardware performance corresponding to the candidate block.

[0087] The performance prediction model 430 can be used to measure or estimate the hardware performance corresponding to the candidate block 410. To obtain the hardware performance corresponding to the candidate block, the performance prediction model may include a lookup table, a prediction model, and / or a simulator. The lookup table records the measured performance of the actual hardware performing the operation in the candidate block. The performance of the operation is learned through deep learning in the prediction model, and the simulator simulates the performance of the hardware performing the operation.

[0088] For example, when the target block is the i-th block of a pre-trained neural network and the candidate block to be trained is the j-th candidate block, the loss function L corresponding to the candidate block can be defined as shown in Equation 1 below. ij .

[0089] Equation 1:

[0090]

[0091] In equation 1, A ijh represents the structure of candidate block j corresponding to target block i. ij X represents the operation performed in candidate block j corresponding to target block i. i Y represents any input data. i Let f represent the output data of target block i, f represent the cost function, λ represent the balance constant, and PPM correspond to the function of output values ​​indicating hardware performance.

[0092] The balancing constant λ can be used in the loss function to adjust the proportion of the hardware performance corresponding to the candidate block and the degree of matching of the data output from the target block and the candidate block. Furthermore, depending on the definition of hardware performance according to the embodiment, it can be inferred that performance is better as the value representing hardware performance decreases, or better as the value representing hardware performance increases. Therefore, this value can be set to a positive or negative number according to the embodiment.

[0093] As described above, the architectural parameters of a candidate block can be parameters updated according to the loss function, and can correspond to the degree of matching between the output data of the candidate block and the output data of the target block, as well as the values ​​applied to the hardware performance corresponding to the operations performed in the candidate block. The processor performing the neural architecture search can define loss-dependent architectural parameters for each candidate block, select candidate blocks to be trained from the candidate blocks based on the architectural parameters during training, select candidate blocks from the candidate blocks based on the architectural parameters after training of the candidate blocks is complete, and replace the target block with the selected candidate block.

[0094] Figure 5 Examples of methods for searching neural architectures are shown that also consider conditions other than the hardware performance corresponding to the operation.

[0095] Reference Figure 5 The target block selection operation may further include selecting at least one target block from multiple blocks by adding conditions other than hardware performance conditions. By reflecting conditions other than hardware performance conditions in the target block selection operation and the candidate block training operation, neural architecture search methods can be applied even when it is impossible to estimate the hardware performance corresponding to the operation performed in the block, or when the neural network architecture is optimized based on conditions other than hardware performance.

[0096] Here, conditions other than hardware performance can include hardware limitations. Conditions other than hardware performance conditions can include a variety of conditions. The following description will give examples of conditions other than hardware performance conditions that are hardware limitations.

[0097] Hardware constraints can be set based on the hardware executing the neural network. Hardware constraints may include, for example, the types of operations supported in the hardware executing the neural network, limitations on the number of layers in the entire network, the acceptable range of weights, the amount of memory, and / or whether operations are optimized.

[0098] When further hardware constraints are added in the target block selection operation, block 523, which does not meet the hardware constraints, can be selected as the target block based on conditions (e.g., based on blocks 521 and 522 that require performance improvement and hardware constraint 510). Although Figure 5 The example shows the selection of a block that does not meet hardware performance conditions or hardware constraints as the target block. However, in some cases, a block that does not match both hardware performance conditions and hardware constraints can be selected as the target block.

[0099] Hardware limitations can be reflected in the training operations of candidate blocks corresponding to the target block. (See reference...) Figure 4 The loss function 402 used to train the candidate block 410 corresponding to the target block can be determined based on the degree of matching between the output data of the candidate block 410 and the output data of the target block 420, the hardware performance corresponding to the candidate block 410 according to the performance prediction model 430, and the hardware constraints 440. The operation of training the candidate block corresponding to the target block may include: generating a loss function for each candidate block and training the weights and architecture parameters of the corresponding candidate block based on the loss function. The operation of generating a loss function for each candidate block is based on the degree of matching between the output data of the corresponding candidate block and the output data generated by the target block in response to the same input data, the hardware performance corresponding to the corresponding candidate block, and the hardware constraints.

[0100] For example, when the target block is the i-th block of a pre-trained neural network and the candidate block to be trained is the j-th candidate block, a loss function reflecting the hardware constraints for training the candidate block can be defined, for example, as shown in Equation 2 below.

[0101] Equation 2:

[0102]

[0103] In equation 2, A ij h represents the structure of candidate block j corresponding to target block i. ij X represents the operation performed in candidate block j corresponding to target block i. i Y represents any input data. i Let f represent the output data of target block i, f represent the cost function, λ represent the balance constant, PPM represent the hardware performance, and Con correspond to the function related to hardware constraints.

[0104] Con can output various values ​​based on hardware constraints. For example, if the types of operations supported in the hardware are specified by hardware constraints, a block that includes operations not supported in the hardware can output "0", and a block that includes operations supported in the hardware can output "1".

[0105] Neural architecture search methods can be used to optimize neural network architectures based on the hardware executing the neural network. The hardware executing the neural network can correspond to hardware including learnable processors (such as central processing units (CPUs), graphics processing units (GPUs), neural processing units / neural processors (NPUs), etc.). By setting hardware constraints based on the hardware executing the neural network, the neural network architecture can be optimized hardware-wise. For example, when using neural architecture search methods to optimize the neural network architecture in an NPU, the supported operations in the NPU and / or the NPU's memory capacity can be set as hardware constraints.

[0106] In terms of the hardware for executing a predetermined neural network, the time frame from hardware design to actual product implementation can be quite long. Therefore, once the hardware is implemented as a physical product, it may be necessary to support operations of a form not yet used in the neural network at the time of hardware design. In this case, operations not supported in the hardware can be set as hardware constraints, allowing neural architecture search methods to be applied to replace the target block.

[0107] In some cases, when a user wants to optimize a pre-trained neural network based on specific hardware, training data may not be available, or the hardware resources required for training may not be sufficient. In such situations, neural network search methods can be used to generate input and output data for the target block to be optimized as training data, or to set the hardware's memory capacity as a hardware constraint, thereby training candidate blocks corresponding to the target block.

[0108] Figure 6 An example of device configuration is shown.

[0109] Reference Figure 6 The device 601 may include a processor 602 (e.g., one or more processors) and a memory 603.

[0110] Device 601 may be a device for performing the neural architecture search method described above. Device 601 may include a computer, a terminal, an SDK, and other means including a learnable processor. Processor 602 may execute reference... Figures 1 to 5 At least one of the methods described. Processor 602 may be a learnable processor and may include, for example, an NPU, GPU, and CPU. Memory 603 may store information associated with the neural architecture search method, or store programs implementing the neural architecture search method. Memory 603 may be volatile memory or non-volatile memory.

[0111] Processor 602 can execute programs and control device 601. The code of the program executed by processor 602 can be stored in memory 603. Device 601 can be connected to external devices (e.g., personal computers or networks) via input and output devices (not shown) and perform data exchange.

[0112] In this regard Figures 1 to 6The performance prediction module, device, processor, memory, performance prediction module 210, device 601, processor 602, memory 603, and other devices, apparatuses, units, modules, and components described are implemented by or represent hardware components. Examples of hardware components that can be used to perform the operations described in this application include, where appropriate, controllers, sensors, generators, drivers, memories, comparators, arithmetic logic units, adders, subtractors, multipliers, dividers, integrators, and any other electronic components configured to perform the operations described in this application. In other examples, one or more hardware components performing the operations described in this application are implemented by computing hardware (e.g., by one or more processors or computers). The processor or computer may be implemented by one or more processing elements (such as logic gate arrays, controllers and arithmetic logic units, digital signal processors, microcomputers, programmable logic controllers, field-programmable gate arrays, programmable logic arrays, microprocessors, or any other means or combination of means configured to respond to and execute instructions in a defined manner to achieve a desired result). In one example, the processor or computer includes or is connected to one or more memories storing instructions or software executed by the processor or computer. Hardware components implemented by a processor or computer can execute instructions or software (such as an operating system (OS) and one or more software applications running on the OS) for performing the operations described in this application. The hardware components can also access, manipulate, process, create, and store data in response to the execution of instructions or software. For simplicity, the singular terms "processor" or "computer" are used in the description of the examples described in this application; however, in other examples, multiple processors or computers may be used, or a processor or computer may include multiple processing elements, or multiple types of processing elements, or both. For example, a single hardware component, or two or more hardware components, may be implemented by a single processor, or two or more processors, or a processor and a controller. One or more hardware components may be implemented by one or more processors, or a processor and a controller, and one or more other hardware components may be implemented by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may implement a single hardware component, or two or more hardware components. The hardware components can have any one or more different processing configurations, examples of which include: a single processor, a discrete processor, a parallel processor, a single instruction single data (SISD) multiprocessing, a single instruction multiple data (SIMD) multiprocessing, multiple instruction single data (MISD) multiprocessing, and multiple instruction multiple data (MIMD) multiprocessing.

[0113] Figures 1 to 6The methods for performing the operations described in this application, as shown, are executed by computing hardware (e.g., one or more processors or a computer), which is implemented to execute instructions or software as described above to perform the operations performed by the methods described in this application. For example, a single operation, or two or more operations, may be executed by a single processor, or two or more processors, or a processor and a controller. One or more operations may be executed by one or more processors, or a processor and a controller, and one or more other operations may be executed by one or more other processors, or additional processors and additional controllers. One or more processors, or a processor and a controller, may execute a single operation, or two or more operations.

[0114] Instructions or software for controlling computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above can be written as computer programs, code segments, instructions, or any combination thereof to individually or collectively instruct or configure one or more processors or computers to operate as a machine or special-purpose computer to perform operations performed by the hardware components and methods described above. In one example, the instructions or software include machine code (such as machine code generated by a compiler) that is directly executed by one or more processors or computers. In another example, the instructions or software include high-level code that is executed by one or more processors or computers using an interpreter. The instructions or software can be written in any programming language based on the block diagrams and flowcharts shown in the accompanying drawings and the corresponding description used herein, which disclose algorithms for performing operations performed by the hardware components and methods described above.

[0115] Instructions or software used to control computing hardware (e.g., one or more processors or computers) to implement hardware components and perform the methods described above, as well as any associated data, data files, and data structures, may be recorded, stored, or fixed in, or on, one or more non-transitory computer-readable storage media. Examples of non-transitory computer-readable storage media include: read-only memory (ROM), programmable random access read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disc storage devices, hard disk drives (HDDs), solid-state drives (SSDs), flash memory, card storage (such as multimedia cards or microcards (e.g., Secure Digital (SD) or Extreme Digital (XD))), magnetic tape, floppy disks, magneto-optical data storage devices, optical data storage devices, hard disks, solid-state drives, and any other device configured to store instructions or software and any associated data, data files, and data structures in a non-transitory manner, and to provide instructions or software and any associated data, data files, and data structures to one or more processors or computers, enabling one or more processors or computers to execute the instructions. In one example, the instructions or software and any associated data, data files, and data structures are distributed across a networked computer system, such that the instructions and software and any associated data, data files, and data structures are stored, accessed, and executed in a distributed manner by one or more processors or computers.

[0116] While this disclosure includes specific examples, it will be clear upon understanding this disclosure that various changes in form and detail may be made in these examples without departing from the spirit and scope of the claims and their equivalents. The examples described herein should be considered descriptive only and not for limiting purposes. The description of features or aspects in each example should be considered applicable to similar features or aspects in other examples. Suitable results may be achieved if the described techniques are performed in a different order, and / or if components in the described system, architecture, apparatus, or circuit are combined in a different manner, and / or replaced or supplemented by other components or their equivalents.

Claims

1. A processor-implemented neural architecture search method based on hardware performance, comprising: Obtain the hardware performance corresponding to the neural network blocks included in the pre-trained neural network; Select at least one target block from the neural network blocks for hardware performance improvement; Based on arbitrary input data and target blocks, output data generated from the input data is used to train the weights and architecture parameters of candidate blocks corresponding to the target blocks. and The pre-trained neural network is updated by replacing the target block in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters. For each of the corresponding candidate blocks, the training steps include: A loss function is generated based on the matching degree between the output data of the corresponding candidate block and the output data of the target block, as well as the hardware performance corresponding to the candidate block. The output data of the corresponding candidate block and the output data of the target block are generated based on the same input data. The weights and architecture parameters of the corresponding candidate blocks are trained based on the loss function.

2. The neural architecture search method according to claim 1, wherein, For each of the corresponding candidate blocks, the training steps include: The architecture parameters of the corresponding candidate block are used to determine whether the corresponding candidate block is a target to be trained; and In response to the corresponding candidate block being identified as the target to be trained, the corresponding candidate block is trained.

3. The neural architecture search method according to claim 2, wherein, The steps include: determining that the corresponding candidate block is the target to be trained in response to the architectural parameters of the corresponding candidate block being greater than or equal to a predetermined threshold.

4. The neural architecture search method according to claim 1, wherein, The steps to obtain the hardware performance corresponding to a neural network block include: The hardware performance corresponding to the neural network block is determined by measuring the actual hardware performance of the hardware that performs the operations of the neural network block.

5. The neural architecture search method according to claim 1, wherein, The steps to obtain the hardware performance corresponding to a neural network block include: The hardware performance corresponding to a neural network block is estimated based on a predictive model that has learned the performance of the hardware that performs the operations of the neural network block.

6. The neural architecture search method according to claim 1, wherein, The steps to obtain the hardware performance corresponding to a neural network block include: The hardware performance corresponding to a neural network block is estimated by simulating the performance of the hardware that performs the operations of the neural network block.

7. The neural architecture search method according to any one of claims 1 to 6, wherein, The steps for selecting a target block include any one or both of the following: At least one target block is selected from the neural network blocks in ascending order of their corresponding hardware performance; and The hardware performance corresponding to a neural network block is compared with a threshold, and at least one block in the neural network that corresponds to hardware performance below the threshold is selected as the target block.

8. The neural architecture search method according to any one of claims 1 to 6, wherein, The training steps include: The input data is generated and the output data is generated by feeding the input data into the target block, which serves as the training data; Train at least one of the candidate blocks based on the training data; and Remove training data in response to the completion of training performed based on the training data.

9. The neural architecture search method according to any one of claims 1 to 6, wherein, The training steps include: Obtain blocks with the same input and output dimensions as the target block, and use them as candidate blocks.

10. The neural architecture search method according to any one of claims 1 to 6, wherein, The steps to replace the target block include: Select the candidate block with the largest architecture parameter from the candidate blocks; and Replace the target block with the selected candidate block.

11. The neural architecture search method according to any one of claims 1 to 6, wherein, The steps for selecting a target block also include: Selecting target blocks from neural network blocks based on hardware constraints.

12. The neural architecture search method according to claim 11, wherein, The loss function is generated based on the degree of matching between the output data of the corresponding candidate block and the output data of the target block, the hardware performance of the corresponding candidate block, and the hardware constraints.

13. The neural architecture search method according to claim 11, wherein, The hardware constraints correspond to the type of neural network operation supported by the hardware configured to execute the neural network and / or the memory capacity of the hardware.

14. The neural architecture search method according to claim 11, wherein, The steps for selecting a target block based on hardware constraints include: selecting at least one block in the neural network that does not meet the hardware constraints as the target block.

15. The neural architecture search method according to any one of claims 1 to 6, wherein, Each of the neural network blocks includes at least one layer of a pre-trained neural network.

16. The neural architecture search method according to any one of claims 1 to 6, wherein, Each candidate block includes at least one layer that performs a predetermined operation of a pre-trained neural network, and the candidate block has the same input and output dimensions as the target block.

17. The neural architecture search method according to any one of claims 1 to 6, wherein, Hardware performance is determined based on any one or any combination of the following: the execution speed of the neural network block in the hardware, the amount of computation required to execute the neural network block, and the size of the memory space of the neural network block.

18. A non-transitory computer-readable storage medium storing instructions, which, when executed by one or more processors, configure the one or more processors to perform the neural architecture search method of any one of claims 1 to 17.

19. A neural architecture search device based on hardware performance, comprising: One or more processors are configured as follows: Obtain the hardware performance corresponding to the neural network blocks included in the pre-trained neural network; Select at least one target block from the neural network blocks for hardware performance improvement; Based on arbitrary input data and target blocks, output data generated from the input data is used to train the weights and architecture parameters of candidate blocks corresponding to the target blocks. and The pre-trained neural network is updated by replacing the target block in the pre-trained neural network with one of the candidate blocks based on the trained architecture parameters. For the training, the one or more processors are configured to: A loss function is generated based on the matching degree between the output data of the corresponding candidate block and the output data of the target block, as well as the hardware performance corresponding to the candidate block. The output data of the corresponding candidate block and the output data of the target block are generated based on the same input data. The weights and architecture parameters of the corresponding candidate blocks are trained based on the loss function.

20. The neural architecture search device according to claim 19, wherein, For the training, for each of the corresponding candidate blocks, the one or more processors are configured to: The architecture parameters of the corresponding candidate block are used to determine whether the corresponding candidate block is a target to be trained. and In response to the corresponding candidate block being identified as the target to be trained, the corresponding candidate block is trained.

21. The neural architecture search device according to claim 19, wherein, For the purpose of the training, the one or more processors are configured to: Generate input data and output data for the target block as training data; Train at least one of the candidate blocks based on the training data; and Remove training data in response to the completion of training performed based on the training data.

22. The neural architecture search device according to any one of claims 19 to 21, wherein, In order to obtain the hardware performance corresponding to the neural network block, the one or more processors are configured to perform any one or any combination of the following: The hardware performance corresponding to the neural network block is determined based on the performance measurements of the actual hardware that performs the operations of the neural network block. The hardware performance corresponding to the neural network block is estimated based on a predictive model that has learned the performance of the hardware that performs the operations of the neural network block. and The hardware performance corresponding to a neural network block is estimated by simulating the performance of the hardware that performs the operations of the neural network block.

23. The neural architecture search device according to any one of claims 19 to 21, wherein, For the training purpose, the one or more processors are configured to: acquire blocks having the same input and output dimensions as the target block as candidate blocks.

24. The neural architecture search device according to any one of claims 19 to 21, wherein, In order to select a target block, the one or more processors are configured to select a target block from a neural network block based on hardware constraints.

25. The neural architecture search device according to claim 24, wherein, The loss function is generated based on the degree of matching between the output data of the corresponding candidate block and the output data of the target block, the hardware performance of the corresponding candidate block, and the hardware constraints.

Images